Netflix has released FlameScope, a visualization tool that allows software engineering teams to monitor performance issues. From application startup to single threaded execution, FlameScope will provide real time insight into the time based metrics crucial to software performance.
The team at Netflix has made FlameScope open source, encouraging engineers to contribute to the project and help develop it further – we’re sure that many development teams could derive a lot of value from the tool, and we’re likely to see many customisations as its community grows.
How does FlameScope work?
Watch the video below to learn more about FlameScope.
Essentially, FlameScope allows you to build something a bit like a flame graph, but with an extra dimension. One of the challenges that Netflix identified that flame graphs sometimes have is that while they allow you to analyze steady and consistent workloads, “often there are small perturbations or variation during that minute that you want to know about, which become a needle-in-a-haystack search when shown with the full profile”. With FlameScope, you get the flame graph, but by using a subsecond-offset heat map, you’re also able to see the “small perturbations” you might have otherwise missed. As Netflix explains:
“You can select an arbitrary continuous time-slice of the captured profile, and visualize it as a flame graph.”
Why Netflix built FlameScope
FlameScope was built by the Netflix cloud engineering team. The key motivations for building it are actually pretty interesting. The team had a microservice that was suffering from strange spikes in latency, the cause a mystery. One of the members of the team found that these spikes, which occurred around every fifteen minutes appeared to correlate with “an increase in CPU utilization that lasted only a few seconds.”
CPU frame graphs, of course, didn’t help for the reasons outlined above. To tackle this, the team effectively sliced up a flame graph into smaller chunks. Slicing it down into one second snapshots was, as you might expect, a pretty arduous task, so by using subsecond heatmaps, the team was able to create flamegraphs on a really small scale. This made it much easier to visualize those variations.
The team are planning to continue to develop the FlameScope project. It will be interesting to see where they decide to take it and how the community responds.
To learn more read the post on the Netflix Tech Blog.