Optimizing Your Splunk Dashboard for Speed: One Search, Multiple Visualizations

By | October 17, 2023
0 Comment

As an enthusiast of efficient solutions, I often find myself seeking ways to streamline my processes. In the realm of data analysis and visualization, Splunk is a powerful tool, but it’s essential to harness its potential wisely. In this blog post, I’ll share how to create a more efficient Splunk dashboard by running one search query for multiple visualizations.

The Problem

You’ve built a Splunk dashboard with five charts and drilldowns, and they all share the same base search data. However, each chart currently executes its query, leading to a slow and inefficient process. It’s clear that running essentially the same search five times is not the way to go.


The Solution

To address this issue, you can optimize your dashboard by executing a single search query and sharing the results among multiple visualizations. Here’s how to do it step by step:

First go to the edit dashboard and click the source so you can modify the dashboard more freely using XML


1. Define Your Base Search Query

Start by crafting a base search query that captures the common data you need for all your visualizations. You need to give it an ID for reference. For instance:

<search id="base">
    <query>index=application_data MSG_TYPE_ID=9999</query>
</search>

This base query will serve as the foundation for all your visualizations.

2. Use the Base Search in Your Visualizations


Now, let’s refer to this base search in each of your visualizations. Replace the existing queries in your visualizations with references to the base search. It should look something like this:

<search base="base">
    <query> ... your visualization-specific query ... </query>
</search>

This way, you are sharing the results of the base search across the visualization, so if you have multiple you are reducing the workload on your system and speeding up your dashboard.

3. Perform Aggregation, Sorting, and More

Each individual visualization may require different aggregations, sorting, or calculations. You can still perform these operations in your visualizations’ queries, just like before:

<search base="base">
    <query>
        | stats count by CURRENT_APP 
        | eventstats sum(count) as percentage 
        | eval percentage=round(count*100/percentage,4)
        | sort - count
    </query>
</search>

With this approach, you maintain the flexibility to customize each visualization while optimizing the performance of your dashboard.


Conclusion

By running one search query for multiple visualizations in your Splunk dashboard, you can significantly enhance its efficiency. You’ll save time, reduce system load, and provide a smoother experience for users. It’s crucial to keep looking for ways to improve processes, and this optimization technique is a great step toward that goal.

So, give it a try and watch your Splunk dashboard perform better than ever before.

Here is a more complicated sample for reference

<form theme="dark" version="1.1">
  <label>Applications (Warning &amp; Errors)</label>
  <description>Count the HTTP Error and Warning responses of an application based on the status captured in MSG_TYPE_ID=9999</description>
  <fieldset submitButton="false">
    <input type="dropdown" token="application" searchWhenChanged="true">
      <label>Application</label>
      <choice value="*">All</choice>
      <default>*</default>
      <initialValue>*</initialValue>
      <fieldForLabel>CURRENT_APP</fieldForLabel>
      <fieldForValue>CURRENT_APP</fieldForValue>
      <search>
        <query>index=application_data |
stats count by CURRENT_APP | sort CURRENT_APP</query>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </search>
    </input>
    <input type="time" token="timepicker">
      <label>timepicker</label>
      <default>
        <earliest>-4h@m</earliest>
        <latest>now</latest>
      </default>
    </input>
  </fieldset>
  <search id="base">
	<!-- The base query as you have it in the search tag of your visualization -->
	<query>index=application_data MSG_TYPE_ID=9999 (PARAM1="5*" OR PARAM1="4*" OR PARAM1="3*")  CURRENT_APP=$application$ NOT /health NOT swagger NOT api-docs  </query>
          <earliest>$timepicker.earliest$</earliest>
          <latest>$timepicker.latest$</latest>
  </search>
  <row>
    <panel>
      <table>
        <title>HTTP Response All Warnings and Errors - Detailed</title>
        <search base="base">
          <query>
| stats count by PARAM1 
| rename PARAM1 as HTTPStatus
| sort - HTTPStatus</query>
        </search>
        <option name="count">100</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="refresh.display">progressbar</option>
        <format type="color" field="HTTPStatus">
          <colorPalette type="list">[#53A051,#006D9C,#DC4E41]</colorPalette>
          <scale type="threshold">400,500</scale>
        </format>
        <drilldown>
          <link target="_blank">search?q=index%3Dapplication_data%20MSG_TYPE_ID%3D9999%20CURRENT_APP%3D$application$%20PARAM1%3D$row.HTTPStatus$&amp;earliest=$timepicker.earliest$&amp;latest=$timepicker.latest$</link>
        </drilldown>
      </table>
    </panel>
    <panel>
      <table>
        <title>HTTP Response All Warnings and Errors - Grouped</title>
        <search base="base">
          <query>
| eval HTTPStatus = case(match(PARAM1, "^1.*"), "HTTP1XX", match(PARAM1, "^2.*"), "HTTP2XX", match(PARAM1, "^3.*"), "HTTP3XX", match(PARAM1, "^4.*"), "HTTP4XX", match(PARAM1, "^5.*"), "HTTP5XX", 1=1, PARAM1)
| stats count by HTTPStatus
| sort - HTTPStatus</query>
        </search>
        <option name="drilldown">none</option>
        <format type="color" field="HTTPStatus">
          <colorPalette type="map">{"HTTP1XX":#53A051,"HTTP2XX":#53A051,"HTTP3XX":#53A051,"HTTP4XX":#006D9C,"HTTP5XX":#DC4E41}</colorPalette>
        </format>
      </table>
    </panel>
    <panel>
      <table>
        <title>HTTP 5XX by application - Most errors first</title>
        <search base="base">
          <query>
| stats count by CURRENT_APP 
| eventstats sum(count) as percentage 
| eval percentage=round(count*100/percentage,4)
| sort - count</query>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">cell</option>
        <option name="percentagesRow">false</option>
        <option name="refresh.display">progressbar</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <drilldown>
          <link target="_blank">/app/sample_application/all_applications__basic_stats?form.application=$row.CURRENT_APP$&amp;form.timepicker.earliest=$form.timepicker.earliest$&amp;form.timepicker.latest=$form.timepicker.latest$</link>
        </drilldown>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <chart>
        <title>HTTP 5XX by application PIE Chart</title>
        <search base="base">
          <query>
| stats count by CURRENT_APP 
| eventstats sum(count) as percentage 
| eval percentage=round(count*100/percentage,4)
| sort - count</query>
        </search>
        <option name="charting.chart">pie</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
    <panel>
      <chart>
        <title>HTTP Response Count</title>
        <search base="base">
          <query>
| eval HTTPStatus = case(match(PARAM1, "^1.*"), "HTTP1XX", match(PARAM1, "^2.*"), "HTTP2XX", match(PARAM1, "^3.*"), "HTTP3XX", match(PARAM1, "^4.*"), "HTTP4XX", match(PARAM1, "^5.*"), "HTTP5XX", 1=1, PARAM1)
| timechart count by HTTPStatus</query>
        </search>
        <option name="charting.axisTitleX.visibility">visible</option>
        <option name="charting.axisTitleY.visibility">visible</option>
        <option name="charting.axisTitleY2.visibility">visible</option>
        <option name="charting.axisY.scale">log</option>
        <option name="charting.chart">area</option>
        <option name="charting.drilldown">all</option>
        <option name="charting.fieldColors">{
"200":#53A051,"201":#53A051,"202":#53A051,"203":#53A051,"204":#53A051,"205":#53A051,"206":#53A051,'207":#53A051,"208":#53A051,"226":#53A051,"HTTP2XX":#53A051,
"300":#FFFFFF,"301":#FFFFFF,"302":#FFFFFF,"303":#FFFFFF,"304":#FFFFFF,"305":#FFFFFF,"306":#FFFFFF,"307":#FFFFFF,"308":#FFFFFF,"HTTP3XX":#FFFFFF,
"400":#006D9C,"401":#006D9C,"402":#006D9C,"403":#006D9C,"404":#006D9C,"405":#006D9C,"406":#006D9C,"407":#006D9C,"408":#006D9C,"409":#006D9C,"410":#006D9C,"411":#006D9C,"412":#006D9C,"413":#006D9C,"414":#006D9C,"415":#006D9C,"416":#006D9C,"417":#006D9C,"418":#006D9C,"421":#006D9C,"422":#006D9C,"423":#006D9C,"424":#006D9C,"425":#006D9C,"426":#006D9C,"428":#006D9C,"429":#006D9C,"431":#006D9C,"451":#006D9C,"HTTP4XX":#006D9C,
"500":#DC4E41,"501":#DC4E41,"502":#DC4E41,"503":#DC4E41,"504":#DC4E41,"505":#DC4E41,"506":#DC4E41,"507":#DC4E41,"508":#DC4E41,"510":#DC4E41,"511":#DC4E41,"HTTP5XX":#DC4E41
}</option>
        <option name="charting.legend.placement">right</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
  </row>
</form>

Stay tuned for more tech tips and tricks on my blog. And if you have any questions or suggestions, feel free to leave a comment below!

Happy Splunking!

Recommended

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.