Node

The Node tab is the starting point for the configuration of the host and server functions related to the Gemini appliance.

System Time

Accurate timekeeping is vital to ensure correct event order. Distributed environments may become out of sync and transactional searches may return inaccurate information if time is not accurately set and frequently updated.

Manage uses pool.ntp.org as a time source by default. You can:

  • Choose Add NTP Server to add additional internal or external network time sources.

or

  • Set the NTP Sync toggle to the OFF position to halt further network time updates and allow for manual editing of the system time. This may be required under special circumstances, but is not advised for general operations.
    • Choose Edit Time to set the date and time manually then choose Sync with Browser to update the date time settings with the client PC.

Timezone

Accurate timezone configuration is essential in maintaining event-order integrity, particularly in geographically distributed environments.

Name

Hostname

To prevent conflicts in distributed environments as well as declare the source path of received events, Manage requires that each device has a unique hostname. Splunk uses this hostname as a default value to populate both server.conf and inputs.conf when started for the first time.

Local Hosts

Note: DNS settings should be configured separately in each network interface in the Network tab.

While not required in normal operation, manually configuring local hosts can ensure connectivity between hosts in the absence or failure of a DNS server. In the case of high latency DNS servers manually configuring hosts may improve performance. Manually configuring local hosts is not considered best practice and should only be used in exceptional cases as multiple static configurations can be complicated to manage and easily become out of date.

To add a static host, choose Add New Record and specify the host’s IP address and hostname.

Network

Manage supports multiple network interface cards (NICs) and Gemini appliances contain either four or six NICs. Additionally Bonding and Port Redirects may also be configured here. Static route rules may be added to a specific network interface to communicate with networks which do not directly connect to the Gemini appliance.

Each NIC can be configured with an IP address either manually or via DHCP. Advanced configurations like MTU and TX Queue Length can be configured especially for network performances.

Note: If this ethernet interface is used for iSCSI connection, set the MTU to a value larger than 1,500 to enable Jumbo Frame. Consult your NAS vendor for more details.

Bonding

You can bind multiple physical NICs into one Virtual Interface to increase throughput and provide redundancy in the event of a single NIC failure.

  1. Choose Create Virtual Interface.
  2. Select the Physical Network Interfaces to be included in the Bonding group.
  3. Choose the Mode for load balancing and fault tolerance from the drop-down menu. Options include: Round Robin, Active-Backup, XOR, Broadcast, IEEE 802.3ad, Dynamic Link Aggregation, Adaptive Transmit Load Balancing, Adaptive Load Balancing.
  4. Specify the frequency of the MII link monitoring used to verify the status of the NIC. The default value is 100.
  5. Configure the Network configurations if required.
  6. Choose Add.

Port Redirect

You can redirect incoming connections on privileged ports to a port above 1024. An example of this is in the case of Splunk which, as a non-root user, will be unable to bind to and listen on ports below 1024. Add the External Port and Internal Port and choose Add.

OS Users

The OS users can access the Gemini appliance using SSH but don’t have access to the Manage web interface.

When an OS user has three failed login attempts, the account is locked and can only be unlocked from this screen. To unlock an OS user, select Yes in Allow Login and save the changes. You can also change OS user passwords here if they’ve been forgotten.

The password can be changed to default, which is a short alphanumeric string. Ensure you copy it, so you can change the password back to one of your choosing.

Note: In some cases you might need a dedicated OS user account to run scripts or applications. You may create a new OS account here, and assign it to desired groups so that you can share access permissions to other accounts. For security considerations, you should lock out unused OS user accounts.

FTP

Enabling the FTP service allows for the uploading of data to the /opt directory. This is useful when hosts are not able to have a Splunk Forwarder installed locally, allowing them to move data to a host such as a Gemini appliance that can ingest the data correctly.

Note: The FTP protocol is not natively encrypted and should only be used when security practices allow for it.

The FTP service requires the configuration of an FTP user as well as an entry in inputs.conf instructing Splunk to ingest any new data found.

FTP Service

Use the FTP Service toggle and enter the desired port (the default listening port is 2121) to enable FTP.

FTP User

The FTP protocol requires both user credentials and a directory to store received files. The default FTP user is named splunk and the default home directory is /opt/sbox. Choose Add FTP User and provide the desired username, password, and root folder to add subsequent accounts. Select an existing username to change its root folder or password.

SSH

The SSH service provides remote command line access to Manage and is natively encrypted. SSH is enabled by default for the users sbox and splunk. Toggle SSH Service to start or stop the service. Port: The listening port of SSH service. The default is 22. Session Timeout: The session timeout interval in minutes. Forward SSHD Log: When enabled a copy of the SSHD logs is forwarded to /var/log/sshd/sshd.log. Allowed Authentication Method: SSH login with password or authorized keys. Note that authorized keys is available as an option when Manage is running on AWS. Rekey Limit: Renegotiates a new key when the traffic exceeds 1GB. Fail to Ban: Bans connections from IP addresses with three failed logins within an hour.

Note: The SSH password can be changed in the OS Users tab.

SNMP

SNMP Service

The SNMP Service allows a remote Simple Network Management Protocol-enabled host to interrogate the Gemini appliance for monitoring and alerting data. To enable SNMP toggle SNMP Service.

SNMP Agent

Select Add SNMP Agent to create a new SNMP Agent entry. Multiple SNMP Agents can be configured. Select a unique name for the SNMP Agent entry and choose an Agent Version. Note: Only Alphanumeric, dot, hyphen, and underscore characters are allowed in the input fields.

SNMP Agent version 1

This version is not encrypted and authentication happens in plain text. As such, it should only be used when other, more secure, versions are not possible. SNMP v1 supports a maximum of 32 bits per counter. Input the Network and Maskbit (subnet mask) of the host network. Then enter a Community String for SNMP authentication. The default value is geminipub (public) and changing this is strongly recommended.

SNMP Agent version 2c

This version is not encrypted and authentication happens in plain text. As such, it should only be used when other, more secure versions are not possible. SNMP v2c supports a maximum of 64 bits per counter. Input the Network and Maskbit (subnet mask) of the host network. Then enter a Community String for SNMP authentication. The default value is geminipub (public) and changing this is strongly recommended.

SNMP Agent version 3

This version supports authentication, encryption, and 64-bit counters. Manage supports MD5 or SHA password hashing. Enter the authorization password and select an encryption method (DES or AES128) for SNMP communication. AES128 is considered to be the more robust of the two. Enter the encryption password and select Add.

SNMP Trap Thresholds

Enable SNMP traps for Gemini appliance’s performance data and specify the frequency and threshold. SNMP traps may be enabled for:

  • Process - ftp, splunk, ssh, syslog-ng
  • Disk
  • Link
  • CPU
  • Memory

SNMP Trap Destination

Choose Add trap Destination and provide the following information:

  • Host Address: The IP address of your SNMP host.
  • Protocol:
    • Use trapsink to send SNMPv1 traps.
    • Use trap2sink to send SNMPv2 traps.
    • Use informsink to send inform notifications.
  • Community String Enter a Community String.
  • Port.

Note: Only alphanumeric, dot, hyphen, and underscore characters are allowed in the input fields.

Failover

Manage nodes configured in a Failover Group provide for a high level of availability in the event that one of the nodes become unavailable due to a maintenance window, network outage, or similar event.

Each group has one active master peer to hold a virtual IP and several standby slave peers that are ready to take over for a failed master. Each Gemini appliance can be part of different Failover Group and each group should be provisioned using a different port number.

Creating a Failover Group

  1. Select Create New Failover Group to create a new group.
  2. Virtual NIC - IP Address: Assign an IP to this virtual group. This IP must be reachable from every member within the group.
  3. Monitor: Determine the event that will trigger a handover to another group member. For example, if Detect Splunk is selected and Splunk on the master node goes down, one slave node will step in and take over the master role.
  4. Remote Node: Add group members to join into this group automatically.

Joining a Failover Group

Choose Join Existing Group and enter the IP of Virtual NIC assigned on the master node. Once joined, the Failover group will display all members and their role within the system (mMaster or slave).

If the master node becomes unavailable or detects the monitoring events, the slave node will step in and serve the virtual IP which had been managed by the master node.

Log Receiver

The Gemini appliance can receive events from remote devices using the syslog protocol. Source filtering rules can be specified to parse, split, and direct different components into different destinations. You can create multiple rules, each with multiple filters and specify a destination to store the received log data. Each rule can only utilize a single data source.

The filter is used to parse and split log entries with defined types. Only matched logs will be stored into defined destination. There must be at least one destination for each filter and multiple destinations are allowed for each data source.

To create a new rule:

  1. Choose Create New Rule.
  2. Enter the rule name.
  3. Select the log rotation frequency and number of copies.
  4. Choose Save.
  5. Enter the Source name, select the protocol, and encryption (for TCP, if applicable).
  6. Add the private and and certificate if TLS is selected.
  7. Choose Save.

To add an optional filter to a rule:

  1. Select the three dots in the Source entry.
  2. Choose Add Filter.
  3. Enter the filter’s name.
  4. Select the type of filtering to be performed.
  5. Enter the appropriate regex filter.
  6. Choose Save.

To add a destination: You can add a destination directly for a source, or for filtered source information.

  1. Select the three dots in the Source or Filter entry.
  2. Choose Add Destination.
  3. Select the appropriate settings and custom path as required.
  4. Choose Save.

Once you’ve configured source, filter, and destination choose Save Rule.

If there are multiple data sources, create rules as needed for each data source, and then specify filters and destinations.

If you have other appliances that function as log receivers, you can replicate the settings to other appliances by selecting Cluster > Replicate Syslog Settings.

Storage

You can manage local storages and attached storages, including direct attached and network attached, to extend disk capacity for data applications such as Splunk and Cloudera. This allows the volume of an existing system to be extended. The mount point for Splunk indexes may also be defined.

Storage Devices

All the detected attached storages are listed here. You can create a new logical volume for grouping storage devices as one, merge storage devices with the existing logical volume to extend its disk capacity, or mount it to a designated mount point.

You can also create a RAID disk from multiple storage devices, Plan your storage use by considering the data growth and expansion plan. Some actions are not revertible so plan it before doing actions.

Mount disk and mount points

You can mount a new storage device to a custom mount point under /opt/mnt/. The owner of this mount point is sbox and permission is open to all. You may maintain owners and permissions of files and folders under this mount point by your own. If this storage device is entirely planned for Splunk, mount it to /opt/splunk directly.

Note: The custom path /opt/sbox/mount is deprecated. Existing mounts will continue without any impact until unmounted.

Encryption and Decryption

Manage supports disk encryptions and this has been simplified and implemented as an option while mounting disks. This is optional and it’s disabled to all disks by default. You may encrypt a disk and mount it with a new key, or mount it with an existing key.

Create New Key File: Encrypt the disk with a new key file. All the data on this disk will be erased. Use Existing Key File: If this disk was encrypted from this machine, this will allow the disk mounted again with an existing key. Upload Key File: If this disk was encrypted from somewhere else, this will allow the disk mounted again with the uploaded key file.

Once mounted, download the encryption key file and store it somewhere safe.

Note:

  • An encrypted disk can’t be used for creating RAID disk or merging into logical volume. It must be decrypted before being reallocated.
  • Encrypting logical volumes is not supported.
  • Encryption with a new key and decryption will erase all data.
  • Backing up the key file is recommended.

Creating Logical Volumes

The major advantage of a logical volume is disk space extensibility for growing data. You may extend its disk capacity by merging more storage devices into an existing logical volume any time.

Note:

  • A logical volume can be created with one or more storage devices.
  • The size of each storage device can be different.
  • There’s no way to split storage devices from an existing logical volume but remove the logical volume entirely. Plan the storage devices carefully.
  • The default logical volume rootvg-lv01 can’t be removed.
  • The use of variant storage types and speed, e.g. SSD, HDD and iSCSI connected disks in one logical volume, is not recommended. It will make the disk performance unpredictable.

Merge Disk

Merge storage device into a logical volume - You may select a target logical volume if there’re more than one logical volumes existed.

Note:

  • Once a storage device has been merged into the default logical volume rootvg-lv01, this action cannot be reversed.
  • When a device has been merged into a logical volume, it must remain attached otherwise the partition might become corrupted and data will be lost.
  • Merging a RAID disk into a logical volume is not supported.
  • Merging an encrypted disk into a logical volume is not supported.

Creating Software RAID

Depending on the RAID level you select, you can group more than one storage devices as a disk array with redundancy or acceleration. This is specifically beneficial to software instances without a hardware RAID controller, e.g., VMware, Hyper-V, and AWS. Disk drives on Gemini appliances are supported and managed by a RAID controller already.

Refer to this page to understand more about the RAID and RAID level: https://en.wikipedia.org/wiki/RAID.

Select the most appropriate RAID level for your use cases:

  • RAID 0 (Striping): Select this option if disk redundancy isn’t critical but disk performance is.
  • RAID 1 (Mirroring): Select this option if you have only two 2 disks and data integrity and availability are critical.
  • RAID 5: Select this if you have more than three disks. This option balances performance, capacity, and availability equally.

Note:

  • Merging a RAID disk into a logical volume is not supported.
  • The size of each storage device can be different when selecting RAID 5, but this can result in wasted disk space.
  • The use of variant storage types and speed, e.g., SSD, HDD and iSCSI connected disks in one RAID disk, is not recommended. It will slow down the RAID disk performance and increase latency.

Add NFS Mount

To define an NFS Mount Point:

  1. Choose Add NFS Mount Point.
  2. Enter the local mount point (located in /opt/sbox/mount).
  3. Enter the IP address of the remote server.
  4. Enter the remote path (starting with a leading /).
  5. Select the mount type. Hard mount is recommended by Splunk when the mount point is used for cold buckets.
  6. Select NFS version. This must match the version of NFS server.
  7. Choose Add.

Note: A mount point will not be detected and validated until it is mounted in the web interface. Once enabled, Manage automatically mounts the NFS mount point on boot.

Add CIFS Mount

To define a CIFS Mount Point:

  1. Choose Add CIFS Mount Point.
  2. Enter the local mount point (located in the /opt/sbox/data folder).
  3. Enter the IP address of the remote server.
  4. Enter the remote path (starting with a leading /).
  5. Enter the username.
  6. Enter the password.
  7. Add the domain.
  8. Choose Add.

Note: The mount point will not be detected and validated until you enable the configuration. When enabled, Manage automatically mounts the CIFS mount point on boot.

Add S3 Mount

To define an Amazon S3 bucket:

  1. Choose Add S3 Bucket. Enter the S3 bucket name you want to mount and the local mount point update to include that bucket name (e.g., /opt/sbox/data/s3/).
  2. Enter the IAM Access Key ID.
  3. Enter the IAM Secret Access Key.
  4. (Optional) Enable Server-Side Encryption (SSE) and select a key option.
    1. To obtain your S3 Access credentials, log in to the AWS Console, open the Users section in the IAM Service and choose the desired user. Create an Access Key in the Security credentials tab. Please note that access to S3 storage requires a connection to the public internet from the node.
  5. Choose Add.

Note: S3 is designed for data archiving and not Splunk indexing. Specifying hot/warm/cold buckets to S3 mount mounts will cause Splunk malfunctions.

Add iSCSI Target

Before proceeding, contact your NAS administrator to get the iSCSI target information and CHAP credentials. To add an iSCSI target:

  1. Choose the pencil icon to open the Initiator Settings.
  2. Specify the Login CHAP and Discovery CHAP (these must match the settings on the iSCSI target).
  3. Choose Save.
  4. In the Target Discovery field enter the iSCSI target IP address and port (e.g., 192.168.1.100:3260). Note: Thee default discovery port is 3260/tcp.
  5. The iSCSI devices that are successfully connected are listed on the screen.
  6. Select Login to connect to an iSCSI target.
  7. Once connected, a new block device will be listed in the Undefined Storage tab and can be mounted.

Please note that connected iSCSI target only means there are new block devices available. Don’t forget to mount them in Undefined Storage.

Note: Set the MTU to a value larger than 1,500 to enable Jumbo Frame for the ethernet interface used for iSCSI connection. This will improve iSCSI performance. Consult your NAS vendor for more details.

Manage Swap Space

Note: Swap Space is not available on all Gemini appliance models. Swap space is disabled by default. However it’s recommended that you enable it on an appliance with heavy loading or for applications that might consume a lot of memory. Once enabled, it will allocate disk space as swap space using the following formula:

Swap space size is equal to RAM size and if RAM size is larger than 64GB, swap space size will be set to 64GB.

Monitoring

Enabling Monitoring allows Manage to send the output of the Gemini appliance Admin and System logs to the destination of your choosing, either local storage on the Gemini appliance or a central syslog server.

  1. Choose Add Forwarding Rule.
  2. Enter the name of the file.
  3. To create a local file: Select File and provide a destination file and path (e.g., admin_file.log).
  4. To send a file to a syslog server: Select UDP or TCP.
  5. Enter the IP address of your syslog server.
  6. Enter the port of your syslog server (the default is 514)
  7. Choose Add.

Diagnostics

Manage provides network diagnostic tools in the web interface. The following commands can be executed and the resulting output shown in the window.

  • Ping
  • TCP Connect
  • NS Lookup
  • Traceroute
  • TCP Dump
  • IOSTAT

Rsync Backup

You can back up your Splunk configurations and data from the /opt/sbox folder to the remote storage regularly.

To enable rsync backup: Step 1: Choose Download SSH Public Key to obtain the SSH public key from Manage.

  • This step enables Manage to log into the remote server via SSH, using a public key.
  • The default file name should be id_rsa.pub.

Step 2: Add this public key into the authorized list in the remote server. The authorized list is usually located at ~/.ssh/authorized_keys in the remote server.

  • You can add it manually using the following command to add it into the list on the remote server: cat id_rsa.pub >> ~/.ssh/authorized_keys.

Step 3: Configure the remote server information:

  • Remote Hostname/IP.
  • Remote Port. This is the listening SSH port on the remote server. The default port is 22.
  • Destination Path. The name of the folder for the backed up data.
  • User Name. The same as the used for the SSH public key.

Step 4: Determine your Backup Scope. You can select either Splunk configuration or folders in /opt/sbox - or both. If you select /opt/sbox/ you can specify which folders to back up.

Step 5: Configure the Backup Plan, such as your policy and schedule.

  • Always create a new full copy may consume disk space rapidly. Monitor the free disk space of the remote server regularly.
  • Keep a single copy up-to-date maintains a single copy and you will not be able to restore data from older copies.

Choose Save. Then choose Backup Configuration Through Rsync to enable rsync backup and confirm whether you successfully exchanged the public key and added it into the authorized key list in the remote server.

Benchmark

You can run disk benchmarks on specific devices, monitor the disk IOPS(Input and Output Operations Per Second) in real time, and download the results. You can use this information to evaluate whether the hardware specs are able to handle running disk I/O intensive tasks, e.g., such as those on Splunk indexers. Disk benchmark attempts to drain all of the system resources including CPU and Disk I/O. The system will slow down, Manage and applications like Splunk might be not responsive during benchmarking.

Note:

  1. Avoid benchmarking on a production environment. Running disk benchmark on a production environment may cause system instability.
  2. Disk benchmark requires a large allocation (approx 15GB) disk space. If available disk space is low, the benchmark process will fail.
  3. Each benchmark job runs five times and takes about 10 minutes. During benchmarking, no additional benchmark jobs are allowed.
  4. For more accurate benchmark results it’s recommended that running applications and services are stopped.
  5. During the benchmark process there’s no way to cancel or stop. The process monitors the operating system and displays the IOPS in real time. It also records the max IOPS on screen. When the benchmark is completed, the results are result in average.
  6. The detailed benchmark methodology is available in the Gemini Support Portal.

To start a benchmark, choose Run Benchmark. Select the target device, and choose Run Benchmark. Once completed, you can download the result for deeper analysis. Choose the three dots on the far right of the screen to download or delete the report.