RSS

Troubleshooting Splunk Search Performance by Search Job Inspector

How to use Splunk Search Job Inspector to troubleshooting search performance issue, understand how search was processed and where splunk spent time.

Splunk timechart Command – Overview

timechart command performs statistical aggregations against time

  • Plots and trends data over time
  • _time is always the x-axis
  • You can optionally split data using the by clause for one other field
    • Each distinct value of the split by field is a separate series in the chart
  • Timecharts are best represented as line or area charts

Scenario How many usage violations have occurred during the last 7 days?

index=network sourcetype=cisco_wsa_squid usage=Violation
| timechart count

Note

Functions and arguments used with stats and chart can also be used with timechart.

What is Search Job Inspector

The Search Job Inspector and the Job Details dashboard are tools that let you take a closer look at what your search is doing and see where the Splunk software is spending most of its time.

Search Job Inspector allows you to examine:

  • Overall stats of search (e.g., records processed and returned, processing time)
  • How search was processed
  • Where Splunk spent its time
    • Use to troubleshoot search’s performance and understand impact of knowledge objects on processing (e.g., event types, tags, lookups)
    • Any existing (i.e., not expired) search job can be inspected

Search Job Inspector – 3 Components

The Job Details dashboard displays basic search job facts and metrics, shows you the search strings that were run in the background to carry out the search, and gives you a concise overview of search costs and indexer usage metrics.

Search Job Inspector have 3 components:

  • Header
  • Execution costs
  • Search job properties

Splunk Search Job Inspector – 3 Components Overview

Search Job Inspector – Header

Top of Search job inspector provides basic information, including time to run and # of events scanned

Splunk Search Job Inspector – Header

Search Job Inspector – Execution Costs

Search Job Inspector – Execution Costs Provides details on cost to retrieve results, such as:

command.search.index
Time to search the index for the location to read in rawdata files
command.search.filter
Time to filter out events that do not match
command.search.rawdata
Time to read events from the raw data files

Splunk Search Job Inspector – Execution Cost

The command.search component, and everything under it, gives you the performance impact of the search command portion of your search, which is everything before the pipe character.

The command.prededup gives you the performance impact of processing the results of the search command before passing it into the dedup command.

  • The Input count of command.prededup matches the Output count of command.search.
  • The Input count of command.dedup matches the Output count of command.prededup.

In this case, the Output count of command.prededup should match the number of events returned at the completion of the search. This is the value of resultCount, under Search job properties.

Search Job Inspector – Search Job Properties

The Search job properties fields provide information about the search job.

Example:

  • Produces scanCount of 127,201 events
  • Returns resultCount of 2,144 in 3.01 seconds
  • To calculate performance:
  • Do not use resultCount/time 2,144 / 3.01 = 712 EPS
  • Rather, calculate scanCount/time 127,201/ 3.01 = 40,892 EPS

EPS= events per second

Troubleshooting Notes

When troubleshooting search performance, it’s important to understand the difference between the scanCount and resultCount costs. For dense searches, the scanCount and resultCount are similar (scanCount = resultCount); and for sparse searches, the scanCount is much greater than the result count (scanCount » resultCount).

scanCount
The number of events that are scanned or read off disk.
resultCount
The total number of results returned by the search.

Search performance should not so much be measured using the resultCount/time rate but scanCount/time instead. Typically, the scanCount/second event rate should hover between 10k and 20k events per second for performance to be deemed good.

Search Job Inspector Debug Messages

Configure the Search Job Inspector to display DEBUG messages when there are errors in your search. For example, DEBUG messages can warn you when there are fields missing from your results.

The Search Job Inspector displays DEBUG messages at the top of the Search Job Inspector window, after the search has completed.

By default the Search Job Inspector hides DEBUG messages.

Related pages:

References

OmniLock - Block / Hide App on iOS

Block distractive apps from appearing on the Home Screen and App Library, enhance your focus and reduce screen time.

DNS Firewall for iOS and Mac OS

Encrypted your DNS to protect your privacy and firewall to block phishing, malicious domains, block ads in all browsers and apps

Ad