Troubleshooting Splunk Search Performance by Search Job Inspector
What is Search Job Inspector
The Search Job Inspector and the Job Details dashboard are tools that let you take a closer look at what your search is doing and see where the Splunk software is spending most of its time.
Search Job Inspector allows you to examine:
- Overall stats of search (e.g., records processed and returned, processing time)
- How search was processed
- Where Splunk spent its time
- Use to troubleshoot search’s performance and understand impact of knowledge objects on processing (e.g., event types, tags, lookups)
- Any existing (i.e., not expired) search job can be inspected
Search Job Inspector – 3 Components
The Job Details dashboard displays basic search job facts and metrics, shows you the search strings that were run in the background to carry out the search, and gives you a concise overview of search costs and indexer usage metrics.
Search Job Inspector have 3 components:
- Header
- Execution costs
- Search job properties
Search Job Inspector – Header
Top of Search job inspector provides basic information, including time to run and # of events scanned
Search Job Inspector – Execution Costs
Search Job Inspector – Execution Costs Provides details on cost to retrieve results, such as:
- command.search.index
- Time to search the index for the location to read in rawdata files
- command.search.filter
- Time to filter out events that do not match
- command.search.rawdata
- Time to read events from the raw data files
The command.search
component, and everything under it, gives you the performance impact of the search
command portion of your search, which is everything before the pipe character.
The command.prededup
gives you the performance impact of processing the results of the search
command before passing it into the dedup
command.
- The Input count of
command.prededup
matches the Output count ofcommand.search
. - The Input count of
command.dedup
matches the Output count ofcommand.prededup
.
In this case, the Output count of command.prededup
should match the number of events returned at the completion of the search. This is the value of resultCount
, under Search job properties.
Search Job Inspector – Search Job Properties
The Search job properties fields provide information about the search job.
Example:
- Produces scanCount of 127,201 events
- Returns resultCount of 2,144 in 3.01 seconds
- To calculate performance:
- Do not use resultCount/time 2,144 / 3.01 = 712 EPS
- Rather, calculate scanCount/time 127,201/ 3.01 = 40,892 EPS
EPS= events per second
Troubleshooting Notes
When troubleshooting search performance, it’s important to understand the difference between the scanCount
and resultCount
costs. For dense searches, the scanCount
and resultCount
are similar (scanCount
= resultCount
); and for sparse searches, the scanCount
is much greater than the result count (scanCount
» resultCount
).
scanCount
- The number of events that are scanned or read off disk.
resultCount
- The total number of results returned by the search.
Search performance should not so much be measured using the resultCount/time rate but scanCount/time instead. Typically, the scanCount/second event rate should hover between 10k and 20k events per second for performance to be deemed good.
Search Job Inspector Debug Messages
Configure the Search Job Inspector to display DEBUG messages when there are errors in your search. For example, DEBUG messages can warn you when there are fields missing from your results.
The Search Job Inspector displays DEBUG messages at the top of the Search Job Inspector window, after the search has completed.
By default the Search Job Inspector hides DEBUG messages.
Related pages:
- Troubleshooting Splunk Search Performance by Search Job Inspector
- Splunk != vs. NOT Difference Detail Explained with Examples
- Splunk Search Best Practices for Better Performance Response Time
- Install Splunk and Forwarder on Linux
References
OmniLock - Block / Hide App on iOS
Block distractive apps from appearing on the Home Screen and App Library, enhance your focus and reduce screen time.
DNS Firewall for iOS and Mac OS
Encrypted your DNS to protect your privacy and firewall to block phishing, malicious domains, block ads in all browsers and apps