Are you familiar with a pain of trying to install or update a large number of Splunk universal forwarders using only Splunk toolkit? It seems impossible. That is where the work of “configuration management” tools makes a true difference in the everyday life of Splunk admins. There is a vast variety of DevOps tools, such as Ansible, Chef, Puppet or Saltstack. Often it depends on your own preferences, your environment, types of tasks or any other reason, which one to choose. 

We prefer Ansible, an automation tool that is easy to use, highly scalable and agentless. In this case there is no software or agent to be installed on the client that communicates back to the server. This tool is well suited for simplifying the complex orchestration and configuration management tasks. To get started, first you need to install Ansible. There is a clear and straightforward installation and configuration guide. You may find it under the Ansible documentation link.

Important Tip: don`t forget to specify an extra variable for python interpreter if you are using OS with different python version (ansible uses “/usr/bin/python” as python interpreter by default). So pass “-e ansible_python_interpreter=/usr/bin/python2” with command.

After configuring clients, we created a playbook, that transfers, executes and deletes script on the remote system by SSH:

—

– name: Transfer and execute a script.

 hosts: [uf-all]

 become: true

 become_user: root

 tasks:

    – name: Transfer the script

      copy: src=/home/user/scripts/uf-update.sh dest=/tmp mode=0777

     – name: Execute the script

      command: sh /tmp/uf-update.sh

     – name: Delete the script

      command: rm /tmp/uf-update.sh

Splunk UFs remote update

After Ansible and SSH clients configuration is over, we are able to install or update Splunk forwarders on remote systems. This can be done using a simple bash script listed below.

At first, this script deletes the old Splunk UF version, downloads and installs the new one. You should only specify a new admin password (it is necessary for the latest versions) and IP of your deployment server. After that, Splunk will start on behalf of a proper Linux user and will get the current configuration from the deployment server.

!/bin/sh

/opt/splunkforwarder/bin/splunk stop;

/opt/splunkforwarder/bin/splunk disable boot-start;

rm -rf /opt/splunkforwarder;

wget -O /tmp/splunkforwarder_Linux-x86_64.tgz ‘<link for current Splunk UF version>’;

tar -xzvf /tmp/splunkforwarder_Linux-x86_64.tgz -C /opt;

/opt/splunkforwarder/bin/splunk start –answer-yes –no-prompt –accept-license –seed-passwd <password>;

/opt/splunkforwarder/bin/splunk stop -auth admin:<password>;

/opt/splunkforwarder/bin/splunk set deploy-poll <your deployment server ip>:8089;

/opt/splunkforwarder/bin/splunk enable boot-start -user splunk;

rm /tmp/splunkforwarder_Linux-x86_64.tgz;

chown -R splunk:splunk /opt/splunkforwarder;

runuser -l splunk -c ‘/opt/splunkforwarder/bin/splunk start’;

Summary

In this article we have learned how to update Splunk universal forwarders remotely with the help of Ansible. It is a great automation tool that does a vast variety of work: helps with the configuration management, application deployment, task automation. It also does IT orchestration, where you have to run tasks in a sequence, and creates a chain of events which must happen on several different servers or devices, using only SSH connection. To sum up, UnderDefense team recommends Ansible as a good choice for system administrators.