Overview

After setting up a Splunk Environment using Gemini Enterprise Manage, one task that remains is the addition of a valid Splunk Enterprise License to replace the default trial license applied during installation.

Currently, the Splunk license model dictates the amount of data ingested over a 24hr period and the Enterprise License chosen by the Customer dictates this quantity. Although from Splunk 8.1 onwards the license model is beginning to change, please refer to the Splunk documentation for details.

Splunk operates a system in which one instance can be made into a License Master and the other instances become License Slaves. We can leverage this system by enabling the Cluster Master Gemini instance to act as a Splunk License Master and pointing all other instances at this instance to become License Slaves.

Create a License Master from the Cluster Master instance

Ensure that you have access to the actual Splunk Enterprise License file purchased by the Customer for the Splunk environment.

Login to the Gemini Management Center, and using a combination of the Gemini Splunk Environments and Node Name dashboards determine which Gemini instance is running as the Cluster Master in the environment.

Login to the Splunk web interface of the Cluster Master instance using the following URI:

http://<CM_gemini_instance>:8000

The default admin passwords used for the Splunk admin account on a Gemini instance are as follows, but check with the local Administrator as these may have been changed.

  • changeme (Gemini Manage 2.2 - 2.7)
  • gemini123 (Gemini Manage 2.8and above)

Navigate to the Settings menu and select the Licensing option.

Select the ‘Add License’ button to reveal the following screen;

Use the ‘Browse’ button to locate and select the Splunk Enterprise License file, and then use the ‘Install’ button to apply the license on this instance.

A message confirming the successful addition of the license should follow.

Accept this by selecting the ‘OK’ button and then review the License details to confirm they are as you expected. Multiple ‘valid’ licenses are automatically summated to give an overall total.

Restart the Splunk instance when prompted. The Cluster Master can be restarted at any time without adversely affecting the Indexer cluster.

 


Create License Slaves

All other instances except Heavy Forwarders will require a valid Enterprise License. For convenience, it is best practice to point all other instances to the License Master.

From the Gemini Management Center, use a combination of the Gemini Splunk Environments and Node Name dashboards to determine a list of all Gemini instances running as Indexers and Search Heads, etc in your environment that need to be made into License Slaves. Once identified, there are two preferred methods of modifying these instances;

  1. Using the Gemini web interface at each instance (more suited to a small number of instances)
  2. Using a setting in a deployable app distributed by a Deployment Server (a better option for a large number of instances)

Option 1: Using the Gemini web interface

From the list of instances that need to become license slaves, login as the Gemini admin user to the first of these instances using the following URI, and complete the following procedure:

https://<gemini_instance>

Navigate to Gemini's Splunk / Config Editor dashboard and follow the instructions below.

If the Splunk Icon is not yet present, first ‘Activate’ Splunk from the Home menu.

Please consult your administrator for the admin account password for Gemini instances

  • Select the /system/local directory to reveal the server.conf file.
  • Select the server.conf file to open in ‘edit’ mode.
  • Important!: This is a crucial Splunk file. Please edit with care ensuring no additional changes.
  • Feel free to make a copy of this file first using the ellipsis menu option prior to making any changes.

Copy and paste the following to add a [license] stanza that points to the License Master instance replacing the ‘xxx.xx…’ with the instance name of the License Master or IP address.

[license]
master_uri = https://<LM_instance_name>:8089
  • Select the ‘Save’ button on completion.

Complete the above server.conf file edit for all of the other instances in your environment chosen to become License Slaves.

When all the edits have been completed, it will be necessary to restart all of these instances. Potentially, most of these instances will be taking part in either an Indexer or Search Head cluster so the restarts must be controlled by rolling restarts.

Rolling Restart Procedures

Indexer Cluster

A rolling restart of the Indexers with a cluster should be achieved with the Cluster Master;

  • Open Splunk’s Indexer Clustering dashboard, located under the ‘Edit’ menu, an select the ‘Rolling Restart’ option;

  • Alternatively, from Gemini’s Splunk / Command dashboard use the following command:
    rolling-restart cluster-peers -auth admin:password

  • Note the confirmation message written back to the terminal regarding the Rolling restart.
  • Verify the progress of the rolling restart at Splunk’s Indexer Clustering dashboard on the Cluster Master.

 

Search Head Cluster

Prior to restarting the Search Head Cluster, we should upgrade first the Deployer instance.

  • Locate the Deployer instance, and perform a restart using either Splunk’s Settings / Server Controls menu or Gemini’s Splunk / Daemon dashboard

To perform a rolling restart of the Search Head Cluster there are two options available;

  • Use Splunk’s Search Head Clustering dashboard (from any of the clustered Search heads)
  • Use the rolling restart CLI command from the Gemini Splunk / Command dashboard.

The Splunk dashboard can be found on any Search Head that participates in the cluster (not the Deployer!).

  • Navigate to the Settings / Search head Clustering dashboard.
  • Select the ‘Begin Rolling Restart’ button and accept the prompt. Wait for the on-screen completion message.

The rolling restart command should ideally be performed at the ‘Captain’ of the cluster, to ensure this role is maintained following the restart, but can equally be performed by any member of the cluster. The Captain can be identified using the show shcluster-status command. This command can also be used throughout the process to check on progress.

Navigate to the Splunk / Command dashboard and use the following;

#To find the current Captain and check on the cluster health:
show shcluster-status -auth admin:<password>

#To complete the restart:
rolling-restart shcluster-members

#To check on progress and verify completion of the rolling restart:
show shcluster-status

Note: The cluster is back in business when the service_ready_ flag is set to '1'.

Other Instances such as stand-alone search heads should be restarted at your earliest convenience in order to have them acting as License Slaves.

 

Option 2: Deployment Server

It is possible to create a Splunk Deployment Server that can operate in a ‘special cluster mode’ that can deliver configuration files and parameters to both Indexer and Search Head clusters. Although non-standard this can be very effective for a large number of Gemini instances.

Please refer to the separate document, Gemini Manage - Create a Splunk Deployment Server for details on the creation process.

Once the Deployment Server has been set up specifically to work with clustered Indexers and Search Heads, the following setting should be added to the appropriate Indexer and Search Head ‘Base-apps’ in order to enable them as License Slaves.

[license]
master_uri = https://<LM_instance_name>:8089

Once the Base-apps have been deployed to the Cluster Master and Deployer respectively, the rolling restart procedures (see above) should be completed.

 

Heavy Forwarders

Any Gemini Instances that have been setup as a Heavy Forwarder will need their Splunk License setting changed from the default ‘trial’ license to a ‘Forwarder’ License. This can be achieved manually at each instance as described below, or a Deployment Server could be used to deliver the setting.

Login to the Splunk web interface of each Heavy Forwarder instance using the following URI:

http://<HF_gemini_instance>:8000

The default admin passwords used for the Splunk admin account on a Gemini instance are as follows, but check with the local Administrator as these may have been changed.

  • changeme (Gemini Manage 2.2 - 2.7)
  • gemini123 (Gemini Manage 2.8 and above)

Navigate to the Splunk Settings menu and select the Licensing option.

  • Select the ‘Change License group’ button to reveal a list of options.
  • Select the ‘Forwarder license’ radio button option.
  • Select the ‘Save’ button to confirm