If a Log Receiver environment has been set up within a Manage Splunk Environment, it is recommended to forward the granular logs created into Splunk for analysis and reporting purposes.
This is made easier in Gemini Manage by the use of its integral Splunk instance which can be re-purposed as a Heavy Forwarder as shown in the diagram below.
In order to use the integral Splunk instance in this way, it first needs to be ‘Activated’ from the Home screen of the Manage web interface. Once activated, Log Receiver rules will be automatically created as Splunk Monitored Inputs. (Note: Splunk will need to be restarted to input any changes).
Enable the Indexers to receive events from the Log Receiver
In order for the Splunk Indexer Cluster to receive logs from the Log Receiver, all Production Indexers must;
- Have their Reciever Port set
- Contain the Indexes used in Rule settings
The Receiver Port, usually set to 9997, is probably already open to receive events from Universal Forwarders if you already have data in Splunk. If this is a new installation however, please verify this before continuing.
It is crucial that any Index specified in the creation of Log Receiver Rules has been created at all your Production Indexers before forwarding is enabled. This is usually achieved at the Cluster Master by a Base App setting in indexes.conf.
Please verify that this has been achieved and that the indexes exist, before proceeding with this document.
Convert Log Receiver Rules into Splunk inputs
Each time a ‘New Rule’ is created at the Log Reciever dashboard, the option is given to provide a Splunk sourcetype and index to store those specific events. When this option is taken, the local Splunk instance is modified to include a 'monitor stanza' in an /etc/system/local/inputs.conf file that creates params similar to those below;
[monitor:///opt/sbox/data/syslog_UDP/.../firewall1_dest.log] index = syslog sourcetype = syslog # If the destination has Log File Splitting set to 'Host' then the # following param is also added accordingly to extract an additional host # field value in Splunk host_segment = 5
Testing Log Receiver Rules before enabling forwarding (optional)
This is purely optional, and you may wish to omit this step if you are familiar with both syslog and the Log Receiver feature.
As Log Receiver Rules are saved at the dashboard, the necessary Splunk input stanza is automatically added to the local Splunk instance, although please note that a restart of Splunk will be required to activate the monitor input(s).
Logon to the Splunk web interface at your Log Receiver instance, and run a search at the index (ie. index=syslog) to verify that the Rules are working correctly.
Create a Heavy Forwarder to forward Log Receiver data
The following tasks are all to be completed from the Splunk interface acting as a Heavy Forwarder on the Log Receiver node.
In order to create a Heavy Forwarder from this Splunk instance, we need to complete three tasks;
- Setup forwarding of all logs to the Production Indexer Cluster.
- Delete the local Indexes used in testing the syslog rules
- Change the license mode of this instance to that of a ‘Forwarding Licence’
To setup forwarding of the logs to the Clustered Indexers. Open the Forwarding and Receiving dashboard located in the Settings menu of Splunk (see below), and select the '+ Add New button'
Add each Indexer and its receiving port to the ‘Host’ input box one-by-one, until all Clustered Indexers have been added. If there are many Indexers and you have been granted access to the CLI, it may be easier to edit the /etc/system/local/outputs.conf file directly.
If you have conducted testing of the Log Receiver Rules on this instance, delete the Index(s) used throughout the testing process. This action will reset the ‘fishbucket’ index into resending the events received during testing, which will now be forwarded onto the Production Indexers.
In order to make this instance into a Heavy Forwarder, open the Settings / Licensing menu, select the ‘Change Licensing Group’ button and choose the ‘Forwarder License’ option.
Restart the Splunk instance to commit all these changes at the Heavy Forwarder.