Gemini Appliances are designed for optimal performance and to meet or exceed Splunk recommendations. While IO performance is generally communicated as a single measure I/O operations per second (IOPS), there are numerous variables and dependencies that can affect accurate measurement.

In some cases, IOPS as measured can be impacted by specific configuration settings or operating characteristics. In these cases, our support team can help tweak and optimize these settings to achieve optimal performance under those circumstances.

In other cases, the tools and methodology for measurement can come into play and deliver different results. This guide provides details on the consistent measurement methodology used by Gemini to measure IOPS. This should provide some clarity to what could otherwise be a “your mileage may vary” situation.

Definition of Terms

  • IOPS: Input/output Operations Per Second (IOPS) is an input/output performance measurement used to characterize computer storage devices like hard disk drives (HDD), solid state drives (SSD), and storage area networks (SAN).
  • Average IOPS: Benchmarked and observed IOPS in average. This represents the I/O performance in average and is usually calculated by benchmark tools.
  • Max IOPS: This represents the peak (highest) IOPS monitored/observed during benchmarking or high I/O operations. This is a representation of the true system I/O capacity. It can be easily captured by monitoring tools and is also the metric used within the specification sheets for Gemini Appliances.

Tools

Benchmaark Tools

  • Bonnie++. A well-known and widely-used disk benchmark tool. The value in Random Seek field usually referenced to IOPS.
  • FIO. A well-known and widely-used disk benchmark tool. It's extremely powerful and be able to simulate various read/write scenarios and present the IOPS. ###Monitoring Tools
  • NMON. A disk monitoring tool contributed by IBM. The value in then Xfers field presents the current IOPS.
  • IOSTAT. A system command to display the I/O statistics. It's the most straightforward way to gather IOPS information from systm inside. The value in tps field presents the current IOPS.

You can download the above tools by your own, or request a tool pack from Gemini Support.

Testing Methodology

While test results are extremely difficult to replicate exactly, the following guidelines and steps will help ensure that your testing mirrors Gemini test methodology as closely as possible.

General Guidelines

  • Applications must not be running - Splunk and all I/O intensive processes must be stopped before testing to ensure that the only load is that imposed by the test.
  • Disable Any Disk-Caching - any OS or application level disk caching must be disabled to prevent irregular disk access patterns than can affect results. Test data size must be significantly larger than RAM to eliminate the cache effect.
  • Consistency of tests - run the test multiple times with consistent benchmark parameters, environment, and procedure to prevent variables occasionally affect results.
  • If not using local disks (SAN, virtualized) testing must be performed simultaneously on all Indexers.

Methodology

The general methodology is to use disk benchmark tools to generate disk I/O pressure to the target appliance and record and calculate the performance indicators. Meanwhile, use the monitoring tools to observe the current I/O status during benchmarking. We need Average IOPS and Max IOPS mentioned above.

Benchmark will run with the following assumptions:

  • Benchmark will run with the following assumptions:
  • Benchmark tool: FIO 2.99
  • Benchmark command and parameters:

./fio --name fio_test_rw --direct=1 --refill_buffers --ioengine=libaio --group_reporting --runtim=120 --bs=4k --size=1G --norandommap --userspace_reap --ramp_time=5 --time_based --numjobs= --iodepth=32 --rw=randrw --rwmixwrite=20

<Num> for numjobs is equal to (CPU threads - 1).

  • Iteration: the benchmark will run 5 times.
  • Monitoring tool: IOSTAT
  • Monitor command and parameters:

iostat -d -m -t 1

  • How to collect Average IOPS: The IOPS in each benchmark is read IOPS + write IOPS in the benchmark report. Average the 5 times results and it is the Average IOPS.
  • How to collect Max IOPS: Running the monitoring tools and record the highest IOPS during benchmark, and it is the Max IOPS.

Since Gemini Enterprise: Manage 2.4 has released, a new benchmark application is provided with the benchmark tool are pre-installed, the parameters are consistent, the disk I/O is transparently visible, and the procedures are automatically performed.

Steps

  1. Login to the target appliance and make sure it’s Manage 2.4 and above.
  2. Stop running applications on this machine like Splunk and Cloudera.
  3. From the top menu, go to NODE -> Benchmark.
  4. Click “Run Benchmark” and select the target device to benchmark from side panel.
  5. Read the note carefully before you click “Run Benchmark”. When clicked, the disk benchmark will start. During the benchmark process there’s no way to cancel or stop.
  6. During the benchmark process, it will monitor the operating system and display the IOPS in real time. It will also record the max IOPS on screen.
  7. When benchmark is completed, you can see the benchmark result in average. You can download the complete result in below for deeper analysis.