1. Introduction

Splunk software is used as a monitoring, searching, tracking structures and unstructured data for analyzing, Splunk is a machine learning mechanism captures, indexes, and get real-time data and convert into graphics like reports, alerts and dashboard, depending upon our needs we can generate it.

Splunk analyse high volume of datas, using splunk API we can integrate lot of application in to the splunk, Lot of plugins are available in the splunkbase place. Few most common API are useful integrating Splunk with VMWare, Splunk with OS logs to monitor availability. Splunk with Service now.

In this chapter we are going to see few basic scenarios how to install and integrate splunk in a linux server

2. Splunk Feature

• Realtime visibility – Splunk’s unique investigative approach allows you to ingest any data — in the cloud or on-premises —for complete visibility
• Why should we use splunk — We can use most of the use cases across business, IT, Security, and DevOps functions.
• Built for Enterprise Scale — Analyse at an unprecedented scale. Powerful search capabilities provide cohesive analytical experiences on massive data sets of any scale and across any number of data sources.
• Take action on data in motion — Stream processing provides more control over the explosion of enterprise data. Get better visibility in any environment
• Interactive Dashboards and Visualization — Create and share dashboard instantly, helps us to create our own dashboards and requirements

3. How to install splunk

Splunk is easily installable and Splunk forwarder agent to be installed on the nodes to push the data, Splunk supports Windows, Linux and OSX

Splunk Enterprise 8.0.6 Downloads

Operating System	32-bit	64-bit	Arm
Linux	Yes	Yes	Yes
Windows	Yes	Yes	No
Mac	NA	Yes	NA

Splunk enterprise downloads — https://www.splunk.com/en_us/download/splunk-enterprise.html#

Splunk Universal Forwarder 8.0.5 Downloads

Universal Forwarders provide reliable, secure data collection from remote sources and forward that data into Splunk software for indexing. It can scale to thousands of remote systems, collecting data for processing

Operating Systems	Architecture
Linux	64-bit2.6+, 3.x+, or 4.x+ kernel Linux distributions ppcle2.6+, 3.x+, or 4.x+ kernel Linux distributions s390x2.6+ kernel Linux distributions ARMv62.6+, 3.x+, or 4.x+ kernel Linux distributions
Mac	OSX 10.13/10.14/10.15
Solaris	64-bitSolaris 11 SPARCSolaris 10, 11
Windows	64-bit Windows 10, Windows Server 2016, 2019 32-bitWindows 10
BSD	64-bit FreeBSD 11
AIX PPC	AIX 7.1, 7.2

Splunk forwarder downloads — https://www.splunk.com/en_us/download/universal-forwarder.html

4. Howto install the splunk

Register and download the splunk enterprise software from the site, in this chapter we are going to install the splunk enterprise in CentOS 8 server running on oracle virtual box

find the downloaded files stored in the centos server

[root@control splunk]# ls -lrt
total 528132
-rw-r--r-- 1 root root 513781002 Sep 7 18:55 splunk-8.0.6-152fb4b2bb96-linux-2.6-x86_64.rpm
-rw-r--r-- 1 root root 27021496 Sep 7 18:55 splunkforwarder-8.0.5-a1a6394cc5ae-linux-2.6-x86_64.rpm

Install the software

[root@control splunk]# rpm -ivh splunk-8.0.6-152fb4b2bb96-linux-2.6-x86_64.rpm
warning: splunk-8.0.6-152fb4b2bb96-linux-2.6-x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID b3cd4420: NOKEY
Verifying… ################################# [100%]
Preparing… ################################# [100%]
Updating / installing…
1:splunk-8.0.6-152fb4b2bb96 ################################# [100%]
complete
[root@control splunk]#

Start the service

[root@control splunk]# /opt/splunk/bin/splunk start --accept-license

Create credentails

This appears to be your first time running this version of Splunk.
Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.
Please enter an administrator username:

Create passwords

Please enter an administrator username: admin
Password must contain at least:
8 total printable ASCII character(s).
Please enter a new password:
Please confirm new password:
Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.
Generating RSA private key, 2048 bit long modulus
……..+++++
…………….+++++
e is 65537 (0x10001)
writing RSA key

Finally we will see the splunk service is started and url will be displayed

writing RSA key
Done
[ OK ]
Waiting for web server at http://127.0.0.1:8000 to be available…. Done
If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com
The Splunk web interface is at http://control.computercarriage.com:8000
[root@control splunk]#

Environment Variables

Once the service is started, export the env variables for the bash shell

[root@control bin]# export SPLUNK_HOME=/opt/splunk/bin
[root@control bin]# echo $SPLUNK_HOME
/opt/splunk/bin

And also add the path in the .bash_profile for permanent

Check the service

Now the splunk is installed and running, check by executing the below command

[root@control bin]# splunk status
Warning: overriding $SPLUNK_HOME setting in environment ("/opt/splunk/bin") with "/opt/splunk". If this is not correct, edit /opt/splunk/etc/splunk-launch.conf
splunkd is running (PID: 2841).
splunk helpers are running (PIDs: 2845 2864 2926 2986 3249).
[root@control bin]#

Enable the port 9997 for the splunk forwarder

By enabling the port 9997 for the splunk, the splunk accepts the data from the splunk installed agents/forwarders

5. Install the forwarders in the client

We have already downloaded the latest version of the agent

[root@server1 ~]# ls -l splunkforwarder-8.0.5-a1a6394cc5ae-linux-2.6-x86_64.rpm
-rw-r--r-- 1 root root 27021496 Sep 7 09:34 splunkforwarder-8.0.5-a1a6394cc5ae-linux-2.6-x86_64.rpm

Install the forwarders

[root@server1 ~]# rpm -ivh splunkforwarder-8.0.5-a1a6394cc5ae-linux-2.6-x86_64.rpm
warning: splunkforwarder-8.0.5-a1a6394cc5ae-linux-2.6-x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID b3cd4420: NOKEY
Verifying… ################################# [100%]
Preparing… ################################# [100%]
Updating / installing…
1:splunkforwarder-8.0.5-a1a6394cc5a################################# [100%]
complete
[root@server1 ~]#

Enable the service

Make sure enable the forwarders service at boot

[root@server1 ~]# /opt/splunkforwarder/bin/splunk enable boot-start
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.
[root@server1 ~]#

Start the service

Start the service and accept the license

root@server1 ~]# /opt/splunkforwarder/bin/splunk start --accept-license

Add the splunk enterprise server

Enable by adding the splunk enterprise server

[root@server1 ~]# /opt/splunkforwarder/bin/splunk add forward-server 192.168.43.185:9997
Splunk username: admin
Password:
Added forwarding to: 192.168.43.185:9997.
[root@server1 ~]#

Enable to monitor logs for the server

Add the system logs to the splunk server by the following command

[root@server1 ~]# /opt/splunkforwarder/bin/splunk add monitor /var/log/messages
Added monitor of '/var/log/messages'.

Verify the forward-server and monitoring

[root@server1 ~]# /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
192.168.225.185:9997
Configured but inactive forwards:
None

Note: splunk forward-server show be in active state

Add the log file path in the splunk enterprise server configuration

[root@control local]# pwd
/opt/splunk/etc/system/local
[root@control local]# cat inputs.conf
[default]
host = control.balapobi.com
[monitor:///var/log/messages]
disabled = 0

6. Screen shots of splunk enterprise

Open the web browser enter the splunk enterprise server url with the credentials provided at the time of installation

click the forward option

Click search in the left panel, execute host=server1 and press enter

Now we will see the logs from the client server will be displayed in the dashboard, it will take some time

Note: You need to open ports for spunk like 8000, 9997

Let us know your comments, in the coming posts we will go in details how to install splunk in windows and mac

Also go through Terraform infrastructure provisioning https://computercarriage.com/2020/09/07/terraform-infrastructure-provisioning/