Overview

If a Log Receiver environment has been set up within a Manage Splunk Environment, it is recommended to forward the granular logs created into Splunk for analysis and reporting purposes.

This is made easier in Gemini Manage by the use of its integral Splunk instance which can be re-purposed as a Heavy Forwarder as shown in the diagram below.

In order to use the integral Splunk instance in this way, it first needs to be ‘Activated’ from the Home screen of the Manage web interface. Once activated, Log Receiver rules will be automatically created as Splunk Monitored Inputs. (Note: Splunk will need to be restarted to input any changes).

Enable the Indexers to receive events from the Log Receiver

In order for the Splunk Indexer Cluster to receive logs from the Log Receiver, all Production Indexers must;

  • Have their Reciever Port set
  • Contain the Indexes used in Rule settings

The Receiver Port, usually set to 9997, is probably already open to receive events from Universal Forwarders if you already have data in Splunk. If this is a new installation however, please verify this before continuing.

It is crucial that any Index specified in the creation of Log Receiver Rules has been created at all your Production Indexers before forwarding is enabled. This is usually achieved at the Cluster Master by a Base App setting in indexes.conf.

Please verify that this has been achieved and that the indexes exist, before proceeding with this document.

This process is normally achieved using Deployment Server or a similar log management tool, this is not meant to be a manual process. Please refer to your Splunk Admin if in any doubt.

Convert Log Receiver Rules into Splunk inputs

Each time a ‘New Rule’ is created at the Log Reciever dashboard, the option is given to provide a Splunk sourcetype and index to store those specific events. When this option is taken, the local Splunk instance is modified to include a 'monitor stanza' in an /etc/system/local/inputs.conf file that creates params similar to those below;

[monitor:///opt/sbox/data/syslog_UDP/.../firewall1_dest.log]
index = syslog
sourcetype = syslog

    # If the destination has Log File Splitting set to 'Host' then the 
    # following param is also added accordingly to extract an additional host 
    # field value in Splunk

host_segment = 5
Important: It is vital that the index referred to in this feature is present at all your Production Indexers before forwarding is enabled!

Testing Log Receiver Rules before enabling forwarding (optional)

This is purely optional, and you may wish to omit this step if you are familiar with both syslog and the Log Receiver feature.

You will have to create a local Splunk index referred to in any of the Log Receiver rules, for this to be successful. And it is only recommended if this is a new Log Receiver environment and you want to test the facility.

As Log Receiver Rules are saved at the dashboard, the necessary Splunk input stanza is automatically added to the local Splunk instance, although please note that a restart of Splunk will be required to activate the monitor input(s).

Logon to the Splunk web interface at your Log Receiver instance, and run a search at the index (ie. index=syslog) to verify that the Rules are working correctly.

Create a Heavy Forwarder to forward Log Receiver data

The following tasks are all to be completed from the Splunk interface acting as a Heavy Forwarder on the Log Receiver node.

In order to create a Heavy Forwarder from this Splunk instance, we need to complete three tasks;

  • Setup forwarding of all logs to the Production Indexer Cluster.
  • Delete the local Indexes used in testing the syslog rules
  • Change the license mode of this instance to that of a ‘Forwarding Licence’

To setup forwarding of the logs to the Clustered Indexers. Open the Forwarding and Receiving dashboard located in the Settings menu of Splunk (see below), and select the '+ Add New button'

Add each Indexer and its receiving port to the ‘Host’ input box one-by-one, until all Clustered Indexers have been added. If there are many Indexers and you have been granted access to the CLI, it may be easier to edit the /etc/system/local/outputs.conf file directly.

If you have conducted testing of the Log Receiver Rules on this instance, delete the Index(s) used throughout the testing process. This action will reset the ‘fishbucket’ index into resending the events received during testing, which will now be forwarded onto the Production Indexers.

In order to make this instance into a Heavy Forwarder, open the Settings / Licensing menu, select the ‘Change Licensing Group’ button and choose the ‘Forwarder License’ option.

Restart the Splunk instance to commit all these changes at the Heavy Forwarder.