Xtykutl

When I run one instance of Elasticsearch, I can index at ~6,000 EPS. On the same server, I start another instance of Elasticsearch, join it to the cluster, and my index speed increases to ~10,000. In other words, a single instance of Elasticsearch does NOT utilize all of the CPU or disk IO that the server has available. Even when running two, not all resources are utilized. It seems like there is some sort of throttling somewhere and I'd like to change it. The primary use of this node will be indexing.

Single ES on server:
~6000 EPS

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

          29.45    0.00    3.87    6.26    0.00   60.43



Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

sdb               0.00   733.13    0.00  800.60     0.00     6.48    16.59     1.75    2.19    0.00    2.19   0.89  71.22

Dual ES on server:
~10,0000 EPS

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

          52.87    0.00    5.22    5.41    0.00   36.49



Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

sdb               0.00  1076.40    0.00  989.40     0.00     9.75    20.18     2.15    2.17    0.00    2.17   0.89  88.32

Maybe useful notes:

• Both ES instances are stock ES installs with the only change being to increase the JVM size.


• I have TBs of logs that look like the below:

{"timestamp":"1541290120","computername":"somenamehere","type":"server","owner":"somenamehere"}

• Disks are SSDs in software raid0. A FIO 512B write test shows IOPS=46.4k, BW=22.7MiB/s and for 4k, IOPS=46.1k, BW=180MiB/s.


• I use Logstash to process the files from a file and send to ES. The docID is created within Logstash. Stock tar.gz logstash yml config excluding config for xpack monitoring.


• I provide a mapping up front but it isn't static.


• System swap is turned off.


• Index refresh_interval is 90s.


• number_of_replicas is set to 0.


• _node stats shows total_indexing_buffer": 1062404096


• index rate is according to Kibana xpack monitoring


• Elasticsearch 6.4.2 and Logstash 6.4.2

Is there a limiter somewhere that I need to change?

first, please have a look on my yesterdays answer, where i explain how indexing works: ElasticSearch - How does sharding affect indexing performance?.

have you seen this (https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html) and this (https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-disk-usage.html) already?

i don't know the shard count for your index, but lowering the number of primary shards can improve your indexing speed.