Splunk Search Best Practices for Better Performance Response Time
Splunk search best practices, a quick guidelines on splunk search, write better search to improve your search quality and boost query time. Notes from Splunk Fundamentals.
Time is the most efficient filter
keep the time range short (e.g. last 60 minutes) to reduce query time.
Specifying a narrow time range is a great way to filter the data in your dataset and to reduce query time and avoid producing more results.
An example of using a time range in a search that goes back 5 minutes, snapping to the beginning of the minute. The end of the time range is the beginning of the current minute.
| from main where earliest=-5m@m and latest=@m
Limit the number of events retrieved
By default, a Splunk search retrieves all events. However in some situations you might want to retrieve a sample set of events, instead of retrieving the entire event set. Limiting the number of events retrieved is useful in several situations:
- You are creating a search and want to determine if you are retrieving the correct events
- You need only a subset or sample set of events for your search
You can specify a limit to the number of events retrieved in a couple of ways:
Use the head command
The head command retrieves only the most recent N events for a historical search, or the first N captured events for a realtime search. For example:
index=foo status=404 | head 1000 ...
Use event sampling
Event sampling uses a ratio that you specify to select events. For example, if the sample ratio value is 100, each event has a 1 in 100 chance of being included in the result set. To learn more about event sampling and sampling ratios, see Event sampling.
By default, event sampling is not active. You must specify a sampling ratio before you run your search. In Splunk Web, click the Sampling drop-down and choose a sampling ratio.
Specify one or more index values at the beginning of your search string
specify multiple index values in a search
OR to search multiple indexes:
(index=foo OR index=bar) "fun"
use a wildcard (*) in index values
HintFor best performance, specify the index values at the beginning of the search string.
Include as many search terms as possible
If you want to find events with
"sshd", and 90% of the events
"error" but only 5%
"sshd", include both values in the search:
"error" AND "sshd"
Make your search terms as specific as possible
"access denied" is always better than searching for
Inclusion is generally better than exclusion (NOT)
"access denied" is faster than searching for
NOT "access granted"
More resources are used tracking NOT expressions than if you specify what you are looking for. Where ever possible, avoid using NOT expressions. For example, instead of using a string of NOT or != expressions such as:
(NOT host=d NOT host=e)
(host!=d AND host!=e)
Use the specific terms you are searching for:
(host=a OR host=b OR host=c).
Filter as early as possible
For example, remove duplicate events, then sort.
The sooner filters and required fields are added to a search, the faster the search will run. It is always best to filter in the foundation of the search if possible.
index=foo | stats count by host | search host="bar"
index=foo host="bar" | stats count by host
Avoid using wildcards at the beginning or middle of a string
- Wildcards at beginning of string scan all events within timeframe
- Wildcards in middle of string may return inconsistent results
- So use
When possible, use OR instead of wildcards
- For example, use
(user=admin OR user=administrator)instead of
HintRemember, field names are case sensitive and field values are case insensitive.
Troubleshooting a slow search: Use the Search Job Inspector
When you wrote a search take a long time, what can you do?
The Search Job Inspector is a tool you can use both to troubleshoot the performance of a search and to determine which phase of the search takes the greatest amounts of time. It dissects the behavior of your searches to help you understand the execution costs of knowledge objects such as event types, tags, lookups, search commands, and other components within the search.
- Searching specific time ranges
- Write better searches
- Quick tips for optimization
- Splunk > Clara-fication: Search Best Practices
- Search Job Inspector
Encrypted your DNS to protect your privacy and firewall to block phishing, malicious domains, block ads in all browsers and apps