RSS

Troubleshooting Splunk Search Performance by Search Job Inspector

How to use Splunk Search Job Inspector to troubleshooting search performance issue, understand how search was processed and where splunk spent time.

Last Update: September 19, 2023

What is Search Job Inspector

The Search Job Inspector and the Job Details dashboard are tools that let you take a closer look at what your search is doing and see where the Splunk software is spending most of its time.

Search Job Inspector allows you to examine:

Overall stats of search (e.g., records processed and returned, processing time)
How search was processed
Where Splunk spent its time
- Use to troubleshoot search’s performance and understand impact of knowledge objects on processing (e.g., event types, tags, lookups)
- Any existing (i.e., not expired) search job can be inspected

Search Job Inspector – 3 Components

The Job Details dashboard displays basic search job facts and metrics, shows you the search strings that were run in the background to carry out the search, and gives you a concise overview of search costs and indexer usage metrics.

Search Job Inspector have 3 components:

Header
Execution costs
Search job properties

Top of Search job inspector provides basic information, including time to run and # of events scanned

Search Job Inspector – Execution Costs

Search Job Inspector – Execution Costs Provides details on cost to retrieve results, such as:

command.search.index: Time to search the index for the location to read in rawdata files
command.search.filter: Time to filter out events that do not match
command.search.rawdata: Time to read events from the raw data files

The command.search component, and everything under it, gives you the performance impact of the search command portion of your search, which is everything before the pipe character.

The command.prededup gives you the performance impact of processing the results of the search command before passing it into the dedup command.

The Input count of command.prededup matches the Output count of command.search.
The Input count of command.dedup matches the Output count of command.prededup.

In this case, the Output count of command.prededup should match the number of events returned at the completion of the search. This is the value of resultCount, under Search job properties.

Search Job Inspector – Search Job Properties

The Search job properties fields provide information about the search job.

Example:

Produces scanCount of 127,201 events
Returns resultCount of 2,144 in 3.01 seconds
To calculate performance:
Do not use resultCount/time 2,144 / 3.01 = 712 EPS
Rather, calculate scanCount/time 127,201/ 3.01 = 40,892 EPS

EPS= events per second

Troubleshooting Notes

When troubleshooting search performance, it’s important to understand the difference between the scanCount and resultCount costs. For dense searches, the scanCount and resultCount are similar (scanCount = resultCount); and for sparse searches, the scanCount is much greater than the result count (scanCount » resultCount).

scanCount: The number of events that are scanned or read off disk.
resultCount: The total number of results returned by the search.

Search performance should not so much be measured using the resultCount/time rate but scanCount/time instead. Typically, the scanCount/second event rate should hover between 10k and 20k events per second for performance to be deemed good.

Search Job Inspector Debug Messages

Configure the Search Job Inspector to display DEBUG messages when there are errors in your search. For example, DEBUG messages can warn you when there are fields missing from your results.

The Search Job Inspector displays DEBUG messages at the top of the Search Job Inspector window, after the search has completed.

By default the Search Job Inspector hides DEBUG messages.