1. Introduction
Splunk software is used as a monitoring, searching, tracking structures and unstructured data for analyzing, Splunk is a machine learning mechanism captures, indexes, and get real-time data and convert into graphics like reports, alerts and dashboard, depending upon our needs we can generate it.
Splunk analyse high volume of datas, using splunk API we can integrate lot of application in to the splunk, Lot of plugins are available in the splunkbase place. Few most common API are useful integrating Splunk with VMWare, Splunk with OS logs to monitor availability. Splunk with Service now.
In this chapter we are going to see few basic scenarios how to install and integrate splunk in a linux server
2. Splunk Feature
- Realtime visibility – Splunk’s unique investigative approach allows you to ingest any data — in the cloud or on-premises —for complete visibility
- Why should we use splunk — We can use most of the use cases across business, IT, Security, and DevOps functions.
- Built for Enterprise Scale — Analyse at an unprecedented scale. Powerful search capabilities provide cohesive analytical experiences on massive data sets of any scale and across any number of data sources.
- Take action on data in motion — Stream processing provides more control over the explosion of enterprise data. Get better visibility in any environment
- Interactive Dashboards and Visualization — Create and share dashboard instantly, helps us to create our own dashboards and requirements
3. How to install splunk
Splunk is easily installable and Splunk forwarder agent to be installed on the nodes to push the data, Splunk supports Windows, Linux and OSX
Splunk Enterprise 8.0.6 Downloads
Operating System | 32-bit | 64-bit | Arm |
Linux | Yes | Yes | Yes |
Windows | Yes | Yes | No |
Mac | NA | Yes | NA |
Splunk enterprise downloads — https://www.splunk.com/en_us/download/splunk-enterprise.html#
Splunk Universal Forwarder 8.0.5 Downloads
Universal Forwarders provide reliable, secure data collection from remote sources and forward that data into Splunk software for indexing. It can scale to thousands of remote systems, collecting data for processing
Operating Systems | Architecture |
Linux | 64-bit2.6+, 3.x+, or 4.x+ kernel Linux distributions ppcle2.6+, 3.x+, or 4.x+ kernel Linux distributions s390x2.6+ kernel Linux distributions ARMv62.6+, 3.x+, or 4.x+ kernel Linux distributions |
Mac | OSX 10.13/10.14/10.15 |
Solaris | 64-bitSolaris 11 SPARCSolaris 10, 11 |
Windows | 64-bit Windows 10, Windows Server 2016, 2019 32-bitWindows 10 |
BSD | 64-bit FreeBSD 11 |
AIX PPC | AIX 7.1, 7.2 |
Splunk forwarder downloads — https://www.splunk.com/en_us/download/universal-forwarder.html
4. Howto install the splunk
Register and download the splunk enterprise software from the site, in this chapter we are going to install the splunk enterprise in CentOS 8 server running on oracle virtual box
find the downloaded files stored in the centos server
[root@control splunk]# ls -lrt total 528132 -rw-r--r-- 1 root root 513781002 Sep 7 18:55 splunk-8.0.6-152fb4b2bb96-linux-2.6-x86_64.rpm -rw-r--r-- 1 root root 27021496 Sep 7 18:55 splunkforwarder-8.0.5-a1a6394cc5ae-linux-2.6-x86_64.rpm
Install the software
[root@control splunk]# rpm -ivh splunk-8.0.6-152fb4b2bb96-linux-2.6-x86_64.rpm warning: splunk-8.0.6-152fb4b2bb96-linux-2.6-x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID b3cd4420: NOKEY Verifying… ################################# [100%] Preparing… ################################# [100%] Updating / installing… 1:splunk-8.0.6-152fb4b2bb96 ################################# [100%] complete [root@control splunk]#
Start the service
[root@control splunk]# /opt/splunk/bin/splunk start --accept-license
Create credentails
This appears to be your first time running this version of Splunk. Splunk software must create an administrator account during startup. Otherwise, you cannot log in. Create credentials for the administrator account. Characters do not appear on the screen when you type in credentials. Please enter an administrator username:
Create passwords
Please enter an administrator username: admin Password must contain at least: 8 total printable ASCII character(s). Please enter a new password: Please confirm new password: Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'. Generating RSA private key, 2048 bit long modulus ……..+++++ …………….+++++ e is 65537 (0x10001) writing RSA key
Finally we will see the splunk service is started and url will be displayed
writing RSA key Done [ OK ] Waiting for web server at http://127.0.0.1:8000 to be available…. Done If you get stuck, we're here to help. Look for answers here: http://docs.splunk.com The Splunk web interface is at http://control.computercarriage.com:8000 [root@control splunk]#
Environment Variables
Once the service is started, export the env variables for the bash shell
[root@control bin]# export SPLUNK_HOME=/opt/splunk/bin [root@control bin]# echo $SPLUNK_HOME /opt/splunk/bin
And also add the path in the .bash_profile for permanent
Check the service
Now the splunk is installed and running, check by executing the below command
[root@control bin]# splunk status Warning: overriding $SPLUNK_HOME setting in environment ("/opt/splunk/bin") with "/opt/splunk". If this is not correct, edit /opt/splunk/etc/splunk-launch.conf splunkd is running (PID: 2841). splunk helpers are running (PIDs: 2845 2864 2926 2986 3249). [root@control bin]#
Enable the port 9997 for the splunk forwarder
By enabling the port 9997 for the splunk, the splunk accepts the data from the splunk installed agents/forwarders
5. Install the forwarders in the client
We have already downloaded the latest version of the agent
[root@server1 ~]# ls -l splunkforwarder-8.0.5-a1a6394cc5ae-linux-2.6-x86_64.rpm -rw-r--r-- 1 root root 27021496 Sep 7 09:34 splunkforwarder-8.0.5-a1a6394cc5ae-linux-2.6-x86_64.rpm
Install the forwarders
[root@server1 ~]# rpm -ivh splunkforwarder-8.0.5-a1a6394cc5ae-linux-2.6-x86_64.rpm warning: splunkforwarder-8.0.5-a1a6394cc5ae-linux-2.6-x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID b3cd4420: NOKEY Verifying… ################################# [100%] Preparing… ################################# [100%] Updating / installing… 1:splunkforwarder-8.0.5-a1a6394cc5a################################# [100%] complete [root@server1 ~]#
Enable the service
Make sure enable the forwarders service at boot
[root@server1 ~]# /opt/splunkforwarder/bin/splunk enable boot-start Init script installed at /etc/init.d/splunk. Init script is configured to run at boot. [root@server1 ~]#
Start the service
Start the service and accept the license
root@server1 ~]# /opt/splunkforwarder/bin/splunk start --accept-license
Add the splunk enterprise server
Enable by adding the splunk enterprise server
[root@server1 ~]# /opt/splunkforwarder/bin/splunk add forward-server 192.168.43.185:9997 Splunk username: admin Password: Added forwarding to: 192.168.43.185:9997. [root@server1 ~]#
Enable to monitor logs for the server
Add the system logs to the splunk server by the following command
[root@server1 ~]# /opt/splunkforwarder/bin/splunk add monitor /var/log/messages Added monitor of '/var/log/messages'.
Verify the forward-server and monitoring
[root@server1 ~]# /opt/splunkforwarder/bin/splunk list forward-server Active forwards: 192.168.225.185:9997 Configured but inactive forwards: None
Note: splunk forward-server show be in active state
Add the log file path in the splunk enterprise server configuration
[root@control local]# pwd /opt/splunk/etc/system/local [root@control local]# cat inputs.conf [default] host = control.balapobi.com [monitor:///var/log/messages] disabled = 0
6. Screen shots of splunk enterprise
Open the web browser enter the splunk enterprise server url with the credentials provided at the time of installation

click the forward option

Click search in the left panel, execute host=server1 and press enter

Now we will see the logs from the client server will be displayed in the dashboard, it will take some time

Note: You need to open ports for spunk like 8000, 9997
Let us know your comments, in the coming posts we will go in details how to install splunk in windows and mac
Also go through Terraform infrastructure provisioning https://computercarriage.com/2020/09/07/terraform-infrastructure-provisioning/