When upgrading Gemini Manage from V2.6 to the new 2.7 version and above, note that there have been some internal changes around the Splunk Environments feature, especially involving the secret key and password area of Splunk. For this reason, following the upgrade, the Splunk Environments Dashboard will need to be re-created. This FAQ document describes this process.
The simplest way of recreating the Splunk Environments dashboard is by using the new JSON Manifest import facility now built-in to Manage.
This is achieved at the Management Center, and to obtain the Manifest files, you will need CLI access to the Management Center node. There can be up to three separate Manifest files depending on the complexity of your Splunk Environment; an indexer cluster manifest, a search head cluster manifest, and possibly a standalone instance manifest file.
- At your local workstation, SSH to the Management Center from a terminal screen as the 'sbox' user.
- Navigate to the /opt/sbox/tmp/upgrade-26-to-27 directory, and copy the JSON Manifest files to your local workstation.
- Login to the Management Center UI, and navigate to the Splunk Environments dashboard. Notice that all the instances are now in the 'Unassigned Nodes' panel.
- Select the '+ Add Nodes' button, and point Manage to each Manifest file in turn to re-create the Splunk Clusters. Refresh the web browser to see the Splunk Clusters format correctly.
- Use the 'Add to Cluster' button to place each complete Splunk Cluster back in its relevant Splunk Environment.
If the JSON manifest import results in a 'Validation error', this means that one or more of the Manage instances does not reflect the JSON manifest.
This is almost certainly due to one or more of the Manage nodes that has NOT been upgraded from 2.6 to 2.7.1. However, there could be other reasons of course, namely DNS or IP issues.
Suggestions following a 'Validation Error' message:
Double-check that all Manage instances listed within the JSON Manifests have been upgraded to 2.7.1 Ping both the DNS and IP address of each instance mentioned in the JSON Manifest files.
Troubleshooting the Manage Upgrade
Log files provided;
/opt/sbox/tmp/upgrade-271.log /var/log/gemini/stat/ACCESS.log /var/log/gemini/stat/INFO.log