RSS

Install Splunk and Forwarder on Linux

Step by step install Splunk, Splunk Forwarder, Splunk app free trial version on Linux.

Install Splunk

Download from https://www.splunk.com/en_us/download/splunk-enterprise

Download Splunk 8 via Command Line (wget):

# .deb For Debian and Ubuntu
$ wget -O splunk-8.0.3-a6754d8441bf-linux-2.6-amd64.deb 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.3&product=splunk&filename=splunk-8.0.3-a6754d8441bf-linux-2.6-amd64.deb&wget=true'

# .rpm For Redhat and CentOS
$ wget -O splunk-8.0.3-a6754d8441bf-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.3&product=splunk&filename=splunk-8.0.3-a6754d8441bf-linux-2.6-x86_64.rpm&wget=true'

# .tgz For all Linux platforms
$ wget -O splunk-8.0.3-a6754d8441bf-Linux-x86_64.tgz 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.3&product=splunk&filename=splunk-8.0.3-a6754d8441bf-Linux-x86_64.tgz&wget=true'

Install Splunk on Debian Buster using .deb:

$ sudo dpkg -i splunk-8.0.3-linux-2.6-amd64.deb
Selecting previously unselected package splunk.
(Reading database ... 89422 files and directories currently installed.)
Preparing to unpack splunk-8.0.3-linux-2.6-amd64.deb ...
Unpacking splunk (8.0.3) ...
Setting up splunk (8.0.3) ...
complete

The Splunk install location is /opt/splunk:

$ ls -l /opt/splunk
total 2936
-r--r--r--  1 splunk splunk     842 Mar 31 02:36 README-splunk.txt
drwxr-xr-x  4 splunk splunk    4096 Apr  9 15:02 bin
-r--r--r--  1 splunk splunk      57 Mar 31 02:33 copyright.txt
drwxr-xr-x 15 splunk splunk    4096 Apr  9 15:02 etc
-rw-r--r--  1 splunk splunk     426 Apr  9 15:02 ftr
drwxr-xr-x  4 splunk splunk    4096 Apr  9 15:02 include
drwxr-xr-x  8 splunk splunk    4096 Apr  9 15:02 lib
-r--r--r--  1 splunk splunk   85709 Mar 31 02:33 license-eula.txt
drwxr-xr-x  3 splunk splunk    4096 Apr  9 15:02 openssl
drwxr-xr-x  4 splunk splunk    4096 Apr  9 15:02 share
-r--r--r--  1 splunk splunk 2881101 Mar 31 02:56 splunk-8.0.3-linux-2.6-x86_64-manifest

Start, Stop and Restart Splunk

Optional add splunk to PATH:

export PATH="$PATH:/opt/splunk/bin"

Start, stop and restart splunk:

$ splunk start
$ splunk stop
$ splunk restart

Note: Should use splunk user to execute all splunk related command and config change.

Example:

$ sudo -u splunk splunk start --accept-license

This appears to be your first time running this version of Splunk.

Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.

Please enter an administrator username: admin
Password must contain at least:
   * 8 total printable ASCII character(s).
Please enter a new password:
Please confirm new password:
Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.
Generating RSA private key, 2048 bit long modulus
.....................................................................................................+++++
..................................................................................................................................+++++
e is 65537 (0x10001)
writing RSA key

Generating RSA private key, 2048 bit long modulus
..........+++++
....................................................+++++
e is 65537 (0x10001)
writing RSA key

Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.

Splunk> CSI: Logfiles.

Checking prerequisites...
	Checking http port [8000]: open
	Checking mgmt port [8089]: open
	Checking appserver port [127.0.0.1:8065]: open
	Checking kvstore port [8191]: open
	Checking configuration... Done.
		Creating: /opt/splunk/var/lib/splunk
		Creating: /opt/splunk/var/run/splunk
		Creating: /opt/splunk/var/run/splunk/appserver/i18n
		Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
		Creating: /opt/splunk/var/run/splunk/upload
		Creating: /opt/splunk/var/run/splunk/search_telemetry
		Creating: /opt/splunk/var/spool/splunk
		Creating: /opt/splunk/var/spool/dirmoncache
		Creating: /opt/splunk/var/lib/splunk/authDb
		Creating: /opt/splunk/var/lib/splunk/hashDb
New certs have been generated in '/opt/splunk/etc/auth'.
	Checking critical directories...	Done
	Checking indexes...
		Validated: _audit _internal _introspection _metrics _telemetry _thefishbucket history main summary
	Done
	Checking filesystem compatibility...  Done
	Checking conf files for problems...
	Done
	Checking default conf files for edits...
	Validating installed files against hashes from '/opt/splunk/splunk-8.0.3-a6754d8441bf-linux-2.6-x86_64-manifest'
	All installed files intact.
	Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Generating a RSA private key
..............................................................................................+++++
..........+++++
writing new private key to 'privKeySecure.pem'
-----
Signature ok
subject=/CN=buster/O=SplunkUser
Getting CA Private Key
writing RSA key
Done


Waiting for web server at http://127.0.0.1:8000 to be available........ Done


If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://127.0.0.1:8000

sudo /opt/splunk/bin/splunk enable boot-start -user 

$ sudo /opt/splunk/bin/splunk enable boot-start -user splunk
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.

Splunk ports

  • 8000: For clients to the Splunk Web (webserver)
  • 8089: For Splunk Management port (inter Splunk communication)
  • 9997: For forwarders to the Splunk indexer. (forwarding and receiving data) This need manually enable, see blow Splunk Forwarder.

Config Splunk

Disable telemetry to splunk

If you donot want to send your splunk usage to Splunk Inc. edit /opt/splunk/etc/apps/splunk_instrumentation/local/telemetry.conf to disable telemetry.

$ sudo -u splunk cat /opt/splunk/etc/apps/splunk_instrumentation/local/telemetry.conf
[general]
telemetrySalt = ffbebefe-512d-4aa7-b832-634c9b854f6b
deploymentID = 7bfc7c72-c062-520a-8a3c-78975cdf5d38
reportStartDate = 2020-04-09
sendAnonymizedUsage = false
sendAnonymizedWebAnalytics = false
sendLicenseUsage = false
optInVersionAcknowledged = 4
sendSupportUsage = false
showOptInModal = false

see telemetry.conf reference for more configuration.

Add local file monitor

sudo -u splunk vim /opt/splunk/etc/apps/search/local/inputs.conf

[monitor:///var/log/nginx/splunk/access.log]
disabled = false
index = splunk_web
sourcetype = access_combined

[monitor:///var/log/nginx/splunk/error.log]
disabled = false
index = splunk_web
sourcetype = nginx_error_log

[monitor:///var/log/nginx/buster/access.log]
disabled = false
index = buster_web
sourcetype = access_combined

[monitor:///var/log/auth.log]
disabled = false
index = buster_os
sourcetype = linux_secure

[monitor:///var/log/syslog]
disabled = false
index = buster_os
sourcetype = syslog

Splunk Forwarder

Enable forwarder receiver on Splunk server

Before use splunk forwarder, you need enable receiver on splunk server:

Settings -> Forwarding and receiving -> Receive data -> Add new

Enter port you want (9997 by default). For example, 9997 will receive data on TCP port 9997.

Or use command line:

$ sudo -u splunk /opt/splunk/bin/splunk enable listen 9997

Install Forwarder

Download forwarder at https://www.splunk.com/en_us/download/universal-forwarder.html

Download the current latest version 8.0.3 via Command Line (wget):

# .rpm format (For Redhat, CentOS)
$ wget -O splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.3&product=universalforwarder&filename=splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-x86_64.rpm&wget=true'

# .deb format (For Debian, Ubuntu)
$ wget -O splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-amd64.deb 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.3&product=universalforwarder&filename=splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-amd64.deb&wget=true'

# .tgz format (For all linux platform)
$ wget -O splunkforwarder-8.0.3-a6754d8441bf-Linux-x86_64.tgz 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.3&product=universalforwarder&filename=splunkforwarder-8.0.3-a6754d8441bf-Linux-x86_64.tgz&wget=true'

Install on CentOS:

# sudo rpm -ivh splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-x86_64.rpm
warning: splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID b3cd4420: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:splunkforwarder-8.0.3-a6754d8441b################################# [100%]
complete

Splunk forwarder install location: /opt/splunkforwarder/

Start Splunk Forwarder

# cd /opt/splunkforwarder/bin
# sudo -u splunk ./splunk start --accept-license

This appears to be your first time running this version of Splunk.

Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.

Please enter an administrator username: fadmin
Password must contain at least:
   * 8 total printable ASCII character(s).
Please enter a new password:
Please confirm new password:

Splunk> Finding your faults, just like mom.

Checking prerequisites...
	Checking mgmt port [8089]: open
		Creating: /opt/splunkforwarder/var/lib/splunk
		Creating: /opt/splunkforwarder/var/run/splunk
		Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
		Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
		Creating: /opt/splunkforwarder/var/run/splunk/upload
		Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry
		Creating: /opt/splunkforwarder/var/spool/splunk
		Creating: /opt/splunkforwarder/var/spool/dirmoncache
		Creating: /opt/splunkforwarder/var/lib/splunk/authDb
		Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
New certs have been generated in '/opt/splunkforwarder/etc/auth'.
	Checking conf files for problems...
	Done
	Checking default conf files for edits...
	Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-8.0.3-a6754d8441bf-linux-2.6-x86_64-manifest'
	All installed files intact.
	Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
 [  OK  ]

Config Forwarder

The first step is use splunk add forward-server to add a forwarder server.

$ ./splunk add forward-server <splunk server>:<receiver port>

Example:

# ./splunk add forward-server 1.2.3.4:9997
Splunk username: fadmin
Password:
Added forwarding to: 1.2.3.4:9997.

Remove forwarder server:

# ./splunk remove forward-server 1.2.3.4:9997
Stopped forwarding to: 1.2.3.4:9997

Install Splunk forwarder as system service so it can startup at boot time:

# ./splunk enable boot-start
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.

Test Forwarder connection:

# /opt/splunkforwarder/bin/splunk list forward-server
Splunk username: fadmin
Password:
Active forwards:
	None
Configured but inactive forwards:
	1.2.3.4:9997

Add data to monitor

./splunk add monitor /var/log/auth.log -sourcetype linux_secure
./splunk add monitor /var/log/syslog -sourcetype syslog

Note: This is not permanent, need use inputs.conf to make it permanent.

Forwarder Config File - inputs.conf

You can configure data inputs on a forwarder by editing the inputs.conf configuration file.

In nearly all cases, edit inputs.conf in the $SPLUNK_HOME/etc/system/local directory. If you have an app installed and want to make changes to its input configuration, edit $SPLUNK_HOME/etc/apps/<appname>/local/inputs.conf. For example, if you have the Splunk Add-on for Unix and Linux installed, you would make edits in $SPLUNK_HOME/etc/apps/Splunk_TA_nix/local/inputs.conf.

Do not make changes to the inputs.conf in $SPLUNK_HOME/etc/system/default. When you upgrade, the installation overwrites that file, which removes any changes you made.

Whenever you make a change to a configuration file, you must restart the forwarder for the change to take effect.

Edit inputs.conf

  1. Using your operating system file management tools or a shell or command prompt, navigate to $SPLUNK_HOME/etc/system/local.
  2. Open inputs.conf for editing (e.g. /opt/splunkforwarder/etc/system/local/inputs.conf). You might need to create this file if it does not exist.
  3. Add your data inputs.
  4. Once you have added your inputs, save the file and close it.
  5. Restart the forwarder.

Example inputs.conf:

# /opt/splunkforwarder/etc/system/local/inputs.conf
#
# Linux login log.
# For Debian, the log path is /var/log/auth.log
# For CentOS, the log path is /var/log/secure
[monitor:///var/log/secure]
index=foo_os_log
sourcetype=linux_secure

[monitor:///var/log/syslog]
index=foo_os_log
sourcetype=syslog

Add splunk user to root (CentOS/Redhat) or adm (Debian/Ubuntu) group to have read access to /var/log/secure:

# usermod -a -G root splunk

Install Splunk app

Get Splunk app at https://splunkbase.splunk.com/

To mannually install apps and add-ons directly into Splunk Enterprise

  1. Put the downloaded file in the $SPLUNK_HOME/etc/apps directory.
  2. Untar and ungzip your app or add-on, using a tool like tar -xvf (on *nix) or WinZip (on Windows).
  3. Restart Splunk.

Troubleshooting

Enable debug log

Change $SPLUNKFORWARDER/etc/log.cfg to enable DEBUG logging, change category.TailingProcessor and category.WatchedFile to DEBUG:

[splunkd]

rootCategory=WARN,A1
# TailingProcessor is meant to be used at level INFO -- without it, analyzing a
# normal diag becomes much harder.  Do NOT remove the TailingProcessor logger.
category.TailingProcessor=DEBUG
category.WatchedFile=DEBUG

Log file: /opt/splunkforwarder/var/log/splunk/splunkd.log

Forwarder can not connect to Splunk server error:

WARN  TcpOutputProc - Cooked connection to ip=1.2.3.4:9997 timed out

Sample success log of forwarder connected to Splunk server error:

INFO  TcpOutputProc - Connected to idx=1.2.3.4:9997, pset=0, reuse=0.
INFO  TcpOutputProc - Found currently active indexer. Connected to idx=1.2.3.4:9997, reuse=1.

Related pages:

References

OmniLock - Block / Hide App on iOS

Block distractive apps from appearing on the Home Screen and App Library, enhance your focus and reduce screen time.

DNS Firewall for iOS and Mac OS

Encrypted your DNS to protect your privacy and firewall to block phishing, malicious domains, block ads in all browsers and apps

Ad