RSS

Splunk Search Best Practices for Better Performance Response Time

Tips on splunk search best practices for better performance response time. Guidelines on create splunk search.
Last Update:

Splunk search best practices, a quick guideline on splunk search, write better search to improve your search quality and boost query time. Notes from Splunk Fundamentals.

Time is the most efficient filter

keep the time range short (e.g. last 60 minutes) to reduce query time.

Specifying a narrow time range is a great way to filter the data in your dataset and to reduce query time and avoid producing more results.

An example of using a time range in a search that goes back 5 minutes, snapping to the beginning of the minute. The end of the time range is the beginning of the current minute.

| from main where earliest=-5m@m and latest=@m

Limit the number of events retrieved

By default, a Splunk search retrieves all events. However in some situations you might want to retrieve a sample set of events, instead of retrieving the entire event set. Limiting the number of events retrieved is useful in several situations:

  • You are creating a search and want to determine if you are retrieving the correct events
  • You need only a subset or sample set of events for your search

You can specify a limit to the number of events retrieved in a couple of ways:

Use the head command

The head command retrieves only the most recent N events for a historical search, or the first N captured events for a realtime search. For example:

index=foo status=404 | head 1000 ...

Use event sampling

Event sampling uses a ratio that you specify to select events. For example, if the sample ratio value is 100, each event has a 1 in 100 chance of being included in the result set. To learn more about event sampling and sampling ratios, see Event sampling.

By default, event sampling is not active. You must specify a sampling ratio before you run your search. In Splunk Web, click the Sampling drop-down and choose a sampling ratio.

Specify one or more index values at the beginning of your search string

Use OR to search multiple indexes:

(index=foo OR  index=bar) "fun"

use a wildcard (*) in index values

index=foo* "fun"

Include as many search terms as possible

If you want to find events with "error" and "sshd", and 90% of the events include "error" but only 5% "sshd", include both values in the search:

"error" AND "sshd"

Make your search terms as specific as possible

Searching for "access denied" is always better than searching for "denied"

Inclusion is generally better than exclusion (NOT)

Searching for "access denied" is faster than searching for NOT "access granted"

More resources are used tracking NOT expressions than if you specify what you are looking for. Where ever possible, avoid using NOT expressions. For example, instead of using a string of NOT or != expressions such as:

(NOT host=d NOT host=e)

or

(host!=d AND host!=e)

Use the specific terms you are searching for:

(host=a OR host=b OR host=c).

Filter as early as possible

For example, remove duplicate events, then sort.

The sooner filters and required fields are added to a search, the faster the search will run. It is always best to filter in the foundation of the search if possible.

slower:

index=foo
| stats count by host
| search host="bar"

faster:

index=foo host="bar"
| stats count by host

Avoid using wildcards at the beginning or middle of a string

  • Wildcards at beginning of string scan all events within timeframe
  • Wildcards in middle of string may return inconsistent results
  • So use fail* (not *fail or *fail* or f*il)

When possible, use OR instead of wildcards

  • For example, use (user=admin OR user=administrator) instead of user=admin*

Troubleshooting a slow search: Use the Search Job Inspector

When you wrote a search take a long time, what can you do?

The Search Job Inspector is a tool you can use both to troubleshoot the performance of a search and to determine which phase of the search takes the greatest amounts of time. It dissects the behavior of your searches to help you understand the execution costs of knowledge objects such as event types, tags, lookups, search commands, and other components within the search.

Related pages:

References

OmniLock - Block / Hide App on iOS

Block distractive apps from appearing on the Home Screen and App Library, enhance your focus and reduce screen time.

DNS Firewall for iOS and Mac OS

Encrypted your DNS to protect your privacy and firewall to block phishing, malicious domains, block ads in all browsers and apps

Ad