Troubleshooting Splunk Search Performance by Search Job Inspector
What is Search Job Inspector
The Search Job Inspector and the Job Details dashboard are tools that let you take a closer look at what your search is doing and see where the Splunk software is spending most of its time.
Search Job Inspector allows you to examine:
- Overall stats of search (e.g., records processed and returned, processing time)
- How search was processed
- Where Splunk spent its time
- Use to troubleshoot search’s performance and understand impact of knowledge objects on processing (e.g., event types, tags, lookups)
- Any existing (i.e., not expired) search job can be inspected
Search Job Inspector – 3 Components
The Job Details dashboard displays basic search job facts and metrics, shows you the search strings that were run in the background to carry out the search, and gives you a concise overview of search costs and indexer usage metrics.
Search Job Inspector have 3 components:
- Execution costs
- Search job properties
Search Job Inspector – Header
Top of Search job inspector provides basic information, including time to run and # of events scanned
Search Job Inspector – Execution Costs
Search Job Inspector – Execution Costs Provides details on cost to retrieve results, such as:
- Time to search the index for the location to read in rawdata files
- Time to filter out events that do not match
- Time to read events from the raw data files
command.search component, and everything under it, gives you the performance impact of the
search command portion of your search, which is everything before the pipe character.
command.prededup gives you the performance impact of processing the results of the
search command before passing it into the
- The Input count of
command.prededupmatches the Output count of
- The Input count of
command.dedupmatches the Output count of
In this case, the Output count of
command.prededup should match the number of events returned at the completion of the search. This is the value of
resultCount, under Search job properties.
Search Job Inspector – Search Job Properties
The Search job properties fields provide information about the search job.
- Produces scanCount of 127,201 events
- Returns resultCount of 2,144 in 3.01 seconds
- To calculate performance:
- Do not use resultCount/time 2,144 / 3.01 = 712 EPS
- Rather, calculate scanCount/time 127,201/ 3.01 = 40,892 EPS
EPS= events per second
When troubleshooting search performance, it’s important to understand the difference between the
resultCount costs. For dense searches, the
resultCount are similar (
resultCount); and for sparse searches, the
scanCount is much greater than the result count (
- The number of events that are scanned or read off disk.
- The total number of results returned by the search.
Search performance should not so much be measured using the resultCount/time rate but scanCount/time instead. Typically, the scanCount/second event rate should hover between 10k and 20k events per second for performance to be deemed good.
Search Job Inspector Debug Messages
Configure the Search Job Inspector to display DEBUG messages when there are errors in your search. For example, DEBUG messages can warn you when there are fields missing from your results.
The Search Job Inspector displays DEBUG messages at the top of the Search Job Inspector window, after the search has completed.
By default the Search Job Inspector hides DEBUG messages.
Encrypted your DNS to protect your privacy and firewall to block phishing, malicious domains, block ads in all browsers and apps