Creating effective dashboards using Splunk [Tutorial]

0
10500
9 min read

Splunk is easy to use for developing a powerful analytical dashboard with multiple panels. A dashboard with too many panels, however, will require scrolling down the page and can cause the viewer to miss crucial information. An effective dashboard should generally meet the following conditions:

  • Single screen view: The dashboard fits in a single window or page, with no scrolling
  • Multiple data points: Charts and visualizations should display a number of data points
  • Crucial information highlighted: The dashboard points out the most important information, using appropriate titles, labels, legends, markers, and conditional formatting as required
  • Created with the user in mind: Data is presented in a way that is meaningful to the user
  • Loads quickly: The dashboard returns results in 10 seconds or less
  • Avoid redundancy: The display does not repeat information in multiple places

In this tutorial, we learn to create different types of dashboards using Splunk. We will also discuss how to gather business requirements for your dashboards.

Types of Splunk dashboards

There are three kinds of dashboards typically created with Splunk:

  • Dynamic form-based dashboards
  • Real-time dashboards
  • Dashboards as scheduled reports

Dynamic form-based dashboards allow Splunk users to modify the dashboard data without leaving the page. This is accomplished by adding data-driven input fields (such as time, radio button, textbox, checkbox, dropdown, and so on) to the dashboard. Updating these inputs changes the data based on the selections. Dynamic form-based dashboards have existed in traditional business intelligence tools for decades now, so users who frequently use them will be familiar with changing prompt values on the fly to update the dashboard data.


Real-time dashboards are often kept on a big panel screen for constant viewing, simply because they are so useful. You see these dashboards in data centers, network operations centers (NOCs), or security operations centers (SOCs) with constant format and data changing in real time. The dashboard will also have indicators and alerts for operators to easily identify and act on a problem. Dashboards like this typically show the current state of security, network, or business systems, using indicators for web performance and traffic, revenue flow, login failures, and other important measures.

Dashboards as scheduled reports may not be exposed for viewing; however, the dashboard view will generally be saved as a PDF file and sent to email recipients at scheduled times. This format is ideal when you need to send information updates to multiple recipients at regular intervals, and don’t want to force them to log in to Splunk to capture the information themselves.

We will create the first two types of dashboards, and you will learn how to use the Splunk dashboard editor to develop advanced visualizations along the way.

Gathering business requirements

As a Splunk administrator, one of the most important responsibilities is to be responsible for the data. As a custodian of data, a Splunk admin has significant influence over how to interpret and present information to users. It is common for the administrator to create the first few dashboards. A more mature implementation, however, requires collaboration to create an output that is beneficial to a variety of user requirements and may be completed by a Splunk development resource with limited administrative rights.

Make it a habit to consistently request users input regarding the Splunk delivered dashboards and reports and what makes them useful. Sit down with day-to-day users and layout, on a drawing board, for example, the business process flows or system diagrams to understand how the underlying processes and systems you’re trying to measure really work. Look for key phrases like these, which signify what data is most important to the business:

  • If this is broken, we lose tons of revenue…
  • This is a constant point of failure…
  • We don’t know what’s going on here…
  • If only I can see the trend, it will make my work easier…
  • This is what my boss wants to see…

Splunk dashboard users may come from many areas of the business. You want to talk to all the different users, no matter where they are on the organizational chart. When you make friends with the architects, developers, business analysts, and management, you will end up building dashboards that benefit the organization, not just individuals. With an initial dashboard version, ask for users thoughts as you observe them using it in their work and ask what can be improved upon, added, or changed.

We hope that at this point, you realize the importance of dashboards and are ready to get started creating some, as we will do in the following sections.

Dynamic form-based dashboard

In this section, we will create a dynamic form-based dashboard in our Destinations app to allow users to change input values and rerun the dashboard, presenting updated data. Here is a screenshot of the final output of this dynamic form-based dashboard:

Dynamic dashboard with form input

Let’s begin by creating the dashboard itself and then generate the panels:

  1. Go the search bar in the Destinations app
  2. Run this search command:
SPL> index=main status_type="*" http_uri="*" server_ip="*" 
           | top status_type, status_description, http_uri, server_ip

Be careful when copying commands with quotation marks. It is best to type in the entire search command to avoid problems.

  1. Go to Save As | Dashboard Panel
  2. Fill in the information based on the following screenshot:
dashboard panel
  1. Click on Save
  2. Close the pop-up window that appears (indicating that the dashboard panel was created) by clicking on the X in the top-right corner of the window

Creating a Status Distribution panel

We will go to the after all the panel searches have been generated. Let’s go ahead and create the second panel:

  1. In the search window, type in the following search command:
SPL> index=main status_type="*" http_uri=* server_ip=* 
           | top status_type
  1. You will save this as a dashboard panel in the newly created dashboard. In the Dashboard option, click on the Existing button and look for the new dashboard, as seen here. Don’t forget to fill in the Panel Title as Status Distribution:
Panel Title as Status Distribution
  1. Click on Save when you are done and again close the pop-up window, signaling the addition of the panel to the dashboard.

Creating the Status Types Over Time panel

Now, we’ll move on to create the third panel:

  1. Type in the following search command and be sure to run it so that it is the active search:
SPL> index=main status_type="*" http_uri=* server_ip=* 
           | timechart count by http_status_code
  1. You will save this as a Dynamic Form-based Dashboard panel as well. Type in Status Types Over Time in the Panel Title field:
Dynamic Form-based Dashboard panel
  1. Click on Save and close the pop-up window, signaling the addition of the panel to the dashboard.

Creating the Hits vs Response Time panel

Now, on to the final panel. Run the following search command:

SPL> index=main status_type="*" http_uri=* server_ip=* 
     | timechart count, avg(http_response_time) as response_time

Save this dashboard panel as Hits vs Response Time:

Dashboard panel

Arrange the dashboard

We’ll move on to look at the dashboard we’ve created and make a few changes:

  1. Click on the View Dashboard button. If you missed out on the View Dashboard button, you can find your dashboard by clicking on Dashboards in the main navigation bar.
  2. Let’s edit the panel arrangement. Click on the Edit button.
  3. Move the Status Distribution panel to the upper-right row.
  4. Move the Hits vs Response Time panel to the lower-right row.
  5. Click on Save to save your layout changes.

Look at the following screenshot. The dashboard framework you’ve created should now look much like this.

The dashboard probably looks a little plainer than you expected it to. But don’t worry; we will improve the dashboard visuals one panel at a time:

Dynamic dashboard with four panels in tabular format

Panel options in dashboards

In this section, we will learn how to alter the look of our panels and create visualizations.

Go to the edit dashboard mode by clicking on the Edit button.

Each dashboard panel will have three setting options to work with: edit search, select visualization, and visualization format options. They are represented by three drop-down icons:

status distribution

The Edit Search window allows you to modify the search string, change the time modifier for the search, add auto-refresh and progress bar options, as well as convert the panel into a report:

Edit search

The Select Visualization dropdown allows you to change the type of visualization to use for the panel, as shown in the following screenshot:

Visualization drop down

Finally, the Visualization Options dropdown will give you the ability to fine-tune your visualization. These options will change depending on the visualization you select. For a normal statistics table, this is how it will look:

visualization

Pie chart – Status Distribution

Go ahead and change the Status Distribution visualization panel to a pie chart. You do this by selecting the Select Visualization icon and selecting the Pie icon. Once done, the panel will look like the following screenshot:

Pie chart – Status Distribution

Stacked area chart – Status Types Over Time

We will change the view of the Status Types Over Time panel to an area chart. However, by default, area charts will not be stacked. We will update this through adjusting the visualization options:

  1. Change the Status Types Over Time panel to an Area Chart using the same Select Visualization button as the prior pie chart exercise.
  2. Make the area chart stacked using the Format Visualization icon. In the Stack Mode section, click on Stacked. For Null Values, select Zero. Use the chart that follows for guidance:
Stacked area chart
  1. Click on Apply. The panel will change right away.
  2. Remove the _time label as it is already implied. You can do this in the X-Axis section by setting the Title to None. Close the Format Visualization window by clicking on the X in the upper-right corner:
Format Visualization window

Here is the new stacked area chart panel:

stacked area chart panel

Column with overlay combination chart – Hits vs Response Time

When representing two or more kinds of data with different ranges, using a combination chart—in this case combining a column and a line—can tell a bigger story than one metric and scale alone. We’ll use the Hits vs Response Time panel to explore the combination charting options:

  1. In the Hits vs Response Time panel, change the chart panel visualization to Column
  2. In the Visualization Options window, click on Chart Overlay
  3. In the Overlay selection box, select response_time
  4. Turn on View as Axis
  5. Click on X-Axis from the list of options on the left of the window and change the Title to None
  6. Click on Legend from the list of options on the left
  7. Change the Legend Position to Bottom
  8. Click on the X in the upper-right-hand corner to close the Visualization Options window

The new panel will now look similar to the following screenshot. From this and the prior screenshot, you can see there was clearly an outage in the overnight hours:

hits vs response time
  1. Click on Done to save all the changes you made and exit the Edit mode

The dashboard has now come to life. This is how it should look now:

Dynamic form-based dashboard with four panels in different formats

To summarize we saw how to create different types of dashboards.

To know more about core Splunk functionalities to transform machine data into powerful insights, check out this book Splunk 7 Essentials, Third Edition.

Read Next:

LEAVE A REPLY

Please enter your comment!
Please enter your name here