G1GC Testing

Table of Contents

Author Lee Hinman (lee@elastic.co)
Date 2018-11-14 09:58:37

Introduction

All testing was done with JDK 10.0.1

Options for CMS: 

-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

Options for G1GC: 

-XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=75

And an additional test with 

-XX:MaxGCPauseMillis=250

31gb

CMS

elasticlogs-1bn-load

./rally --track-repository=eventdata --track=eventdata --pipeline=benchmark-only --track-params="number_of_replicas:1,shard_count:3" --target-hosts=g1gc-1:9200,g1gc-2:9200,g2gc-3:9200 --challenge=elasticlogs-1bn-load
Lap Metric Task Value Unit
All Total indexing time   4116.96 min
All Min indexing time per shard   118.7 min
All Median indexing time per shard   137.78 min
All Max indexing time per shard   172.41 min
All Total merge time   2056.68 min
All Min merge time per shard   49.3475 min
All Median merge time per shard   69.6099 min
All Max merge time per shard   75.5772 min
All Total merge throttle time   1051.26 min
All Min merge throttle time per shard   23.9423 min
All Median merge throttle time per shard   35.5916 min
All Max merge throttle time per shard   41.9042 min
All Total refresh time   85.3203 min
All Min refresh time per shard   2.45742 min
All Median refresh time per shard   2.86104 min
All Max refresh time per shard   3.03292 min
All Total flush time   65.7714 min
All Min flush time per shard   2.03267 min
All Median flush time per shard   2.16882 min
All Max flush time per shard   2.42153 min
All Total Young Gen GC   5407.64 s
All Total Old Gen GC   6.079 s
All Store size   431.862 GB
All Translog size   31.9376 GB
All Heap used for segments   342.928 MB
All Heap used for doc values   0.135151 MB
All Heap used for terms   259.084 MB
All Heap used for norms   0 MB
All Heap used for points   37.5948 MB
All Heap used for stored fields   46.114 MB
All Segment count   827  
All Min Throughput create_elasticlogs_q_write 1.44 ops/s
All Median Throughput create_elasticlogs_q_write 1.44 ops/s
All Max Throughput create_elasticlogs_q_write 1.44 ops/s
All 100th percentile latency create_elasticlogs_q_write 696.096 ms
All 100th percentile service time create_elasticlogs_q_write 696.096 ms
All error rate create_elasticlogs_q_write 0 %
All Min Throughput index-append-1000-elasticlogs_q_write 783.03 docs/s
All Median Throughput index-append-1000-elasticlogs_q_write 47188.9 docs/s
All Max Throughput index-append-1000-elasticlogs_q_write 54164.7 docs/s
All 50th percentile latency index-append-1000-elasticlogs_q_write 305.588 ms
All 90th percentile latency index-append-1000-elasticlogs_q_write 635.242 ms
All 99th percentile latency index-append-1000-elasticlogs_q_write 1530.61 ms
All 99.9th percentile latency index-append-1000-elasticlogs_q_write 2413.63 ms
All 99.99th percentile latency index-append-1000-elasticlogs_q_write 3662.98 ms
All 100th percentile latency index-append-1000-elasticlogs_q_write 8454.28 ms
All 50th percentile service time index-append-1000-elasticlogs_q_write 305.588 ms
All 90th percentile service time index-append-1000-elasticlogs_q_write 635.242 ms
All 99th percentile service time index-append-1000-elasticlogs_q_write 1530.61 ms
All 99.9th percentile service time index-append-1000-elasticlogs_q_write 2413.63 ms
All 99.99th percentile service time index-append-1000-elasticlogs_q_write 3662.98 ms
All 100th percentile service time index-append-1000-elasticlogs_q_write 8454.28 ms
All error rate index-append-1000-elasticlogs_q_write 0 %
All error rate rollover_elasticlogs_q_write_100M 0 %
All Min Throughput node_storage 5.49 ops/s
All Median Throughput node_storage 5.49 ops/s
All Max Throughput node_storage 5.49 ops/s
All 100th percentile latency node_storage 182.115 ms
All 100th percentile service time node_storage 182.115 ms
All error rate node_storage 0 %

Another

Lap Metric Task Value Unit
All Total indexing time   4750.82 min
All Min indexing time per shard   129.441 min
All Median indexing time per shard   157.176 min
All Max indexing time per shard   183.672 min
All Total merge time   2220.99 min
All Min merge time per shard   52.8165 min
All Median merge time per shard   75.8962 min
All Max merge time per shard   84.8014 min
All Total merge throttle time   1156.31 min
All Min merge throttle time per shard   25.6832 min
All Median merge throttle time per shard   39.2651 min
All Max merge throttle time per shard   47.9848 min
All Total refresh time   91.7721 min
All Min refresh time per shard   2.74202 min
All Median refresh time per shard   3.11066 min
All Max refresh time per shard   3.25633 min
All Total flush time   64.0476 min
All Min flush time per shard   1.93832 min
All Median flush time per shard   2.13275 min
All Max flush time per shard   2.34372 min
All Total Young Gen GC   5956.04 s
All Total Old Gen GC   7.651 s
All Store size   440.552 GB
All Translog size   31.9554 GB
All Heap used for segments   338.386 MB
All Heap used for doc values   0.174004 MB
All Heap used for terms   254.507 MB
All Heap used for norms   0 MB
All Heap used for points   37.5647 MB
All Heap used for stored fields   46.1394 MB
All Segment count   844  
All Min Throughput create_elasticlogs_q_write 1.58 ops/s
All Median Throughput create_elasticlogs_q_write 1.58 ops/s
All Max Throughput create_elasticlogs_q_write 1.58 ops/s
All 100th percentile latency create_elasticlogs_q_write 631.086 ms
All 100th percentile service time create_elasticlogs_q_write 631.086 ms
All error rate create_elasticlogs_q_write 0 %
All Min Throughput index-append-1000-elasticlogs_q_write 776.93 docs/s
All Median Throughput index-append-1000-elasticlogs_q_write 43988.7 docs/s
All Max Throughput index-append-1000-elasticlogs_q_write 52511.7 docs/s
All 50th percentile latency index-append-1000-elasticlogs_q_write 346.677 ms
All 90th percentile latency index-append-1000-elasticlogs_q_write 689.348 ms
All 99th percentile latency index-append-1000-elasticlogs_q_write 1538.78 ms
All 99.9th percentile latency index-append-1000-elasticlogs_q_write 2320.88 ms
All 99.99th percentile latency index-append-1000-elasticlogs_q_write 3349.96 ms
All 100th percentile latency index-append-1000-elasticlogs_q_write 4888.04 ms
All 50th percentile service time index-append-1000-elasticlogs_q_write 346.677 ms
All 90th percentile service time index-append-1000-elasticlogs_q_write 689.348 ms
All 99th percentile service time index-append-1000-elasticlogs_q_write 1538.78 ms
All 99.9th percentile service time index-append-1000-elasticlogs_q_write 2320.88 ms
All 99.99th percentile service time index-append-1000-elasticlogs_q_write 3349.96 ms
All 100th percentile service time index-append-1000-elasticlogs_q_write 4888.04 ms
All error rate index-append-1000-elasticlogs_q_write 0 %
All error rate rollover_elasticlogs_q_write_100M 0 %
All Min Throughput node_storage 1.18 ops/s
All Median Throughput node_storage 1.18 ops/s
All Max Throughput node_storage 1.18 ops/s
All 100th percentile latency node_storage 850.914 ms
All 100th percentile service time node_storage 850.914 ms
All error rate node_storage 0 %

elasticlogs-querying

./rally --track-repository=eventdata --track=eventdata --pipeline=benchmark-only --track-params="number_of_replicas:1,shard_count:3" --target-hosts=g1gc-1:9200,g1gc-2:9200,g2gc-3:9200 --challenge=elasticlogs-querying
Lap Metric Task Value Unit
All Total refresh time   3.33333e-05 min
All Min refresh time per shard   0 min
All Median refresh time per shard   0 min
All Max refresh time per shard   1.66667e-05 min
All Total flush time   0.0007 min
All Min flush time per shard   0 min
All Median flush time per shard   1.66667e-05 min
All Max flush time per shard   0.0001 min
All Total Young Gen GC   6.616 s
All Total Old Gen GC   0 s
All Store size   418.812 GB
All Translog size   6.65896e-06 GB
All Heap used for segments   338.896 MB
All Heap used for doc values   0.130119 MB
All Heap used for terms   255.223 MB
All Heap used for norms   0 MB
All Heap used for points   37.4296 MB
All Heap used for stored fields   46.1134 MB
All Segment count   772  
All Min Throughput fieldstats_elasticlogs_q-* 0.07 ops/s
All Median Throughput fieldstats_elasticlogs_q-* 0.07 ops/s
All Max Throughput fieldstats_elasticlogs_q-* 0.07 ops/s
All 50th percentile latency fieldstats_elasticlogs_q-* 13693.4 ms
All 100th percentile latency fieldstats_elasticlogs_q-* 13736.4 ms
All 50th percentile service time fieldstats_elasticlogs_q-* 13693.4 ms
All 100th percentile service time fieldstats_elasticlogs_q-* 13736.4 ms
All error rate fieldstats_elasticlogs_q-* 0 %
All Min Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_50% 0.06 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_50% 233.719 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_50% 613.704 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_50% 19288 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_50% 205.025 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_50% 585.55 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_50% 19288 ms
All error rate relative-kibana-content_issues-dashboard_50% 0 %
All Min Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_75% 0.04 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_75% 349.212 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_75% 2048.6 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_75% 24699.1 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_75% 273.106 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_75% 1958.78 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_75% 24699.1 ms
All error rate relative-kibana-content_issues-dashboard_75% 0 %
All Min Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-traffic-dashboard_50% 0.04 ops/s
All 50th percentile latency relative-kibana-traffic-dashboard_50% 388.149 ms
All 90th percentile latency relative-kibana-traffic-dashboard_50% 3703.65 ms
All 100th percentile latency relative-kibana-traffic-dashboard_50% 24897.9 ms
All 50th percentile service time relative-kibana-traffic-dashboard_50% 348.363 ms
All 90th percentile service time relative-kibana-traffic-dashboard_50% 3666.3 ms
All 100th percentile service time relative-kibana-traffic-dashboard_50% 24897.9 ms
All error rate relative-kibana-traffic-dashboard_50% 0 %
All Min Throughput relative-kibana-discover_50% 0.02 ops/s
All Median Throughput relative-kibana-discover_50% 0.02 ops/s
All Max Throughput relative-kibana-discover_50% 0.05 ops/s
All 50th percentile latency relative-kibana-discover_50% 548.368 ms
All 90th percentile latency relative-kibana-discover_50% 791.318 ms
All 100th percentile latency relative-kibana-discover_50% 21344 ms
All 50th percentile service time relative-kibana-discover_50% 488.685 ms
All 90th percentile service time relative-kibana-discover_50% 731.564 ms
All 100th percentile service time relative-kibana-discover_50% 21344 ms
All error rate relative-kibana-discover_50% 0 %

5 laps part 2:

Lap Metric Task Final Unit
All Total flush time   0.00783333 min
All Min flush time per shard   0 min
All Median flush time per shard   1.66667e-05 min
All Max flush time per shard   0.000333333 min
All Total Young Gen GC   28.21 s
All Total Old Gen GC   0.092 s
All Store size   418.483 GB
All Translog size   6.14673e-06 GB
All Heap used for segments   333.173 MB
All Heap used for doc values   0.13308 MB
All Heap used for terms   249.58 MB
All Heap used for norms   0 MB
All Heap used for points   37.3245 MB
All Heap used for stored fields   46.1352 MB
All Segment count   786  
All Min Throughput fieldstats_elasticlogs_q-* 0.08 ops/s
All Median Throughput fieldstats_elasticlogs_q-* 181.52 ops/s
All Max Throughput fieldstats_elasticlogs_q-* 297.47 ops/s
All 50th percentile latency fieldstats_elasticlogs_q-* 19.0765 ms
All 90th percentile latency fieldstats_elasticlogs_q-* 14219.2 ms
All 100th percentile latency fieldstats_elasticlogs_q-* 14283.6 ms
All 50th percentile service time fieldstats_elasticlogs_q-* 19.0765 ms
All 90th percentile service time fieldstats_elasticlogs_q-* 14219.2 ms
All 100th percentile service time fieldstats_elasticlogs_q-* 14283.6 ms
All error rate fieldstats_elasticlogs_q-* 0 %
All Min Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_50% 4.26 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_50% 204.005 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_50% 365.552 ms
All 99th percentile latency relative-kibana-content_issues-dashboard_50% 2938.91 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_50% 15148.1 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_50% 174.899 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_50% 335.416 ms
All 99th percentile service time relative-kibana-content_issues-dashboard_50% 2909.12 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_50% 15148.1 ms
All error rate relative-kibana-content_issues-dashboard_50% 0 %
All Min Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_75% 3.71 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_75% 321.488 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_75% 473.447 ms
All 99th percentile latency relative-kibana-content_issues-dashboard_75% 5124.13 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_75% 20584.1 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_75% 232.122 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_75% 401.631 ms
All 99th percentile service time relative-kibana-content_issues-dashboard_75% 5053.75 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_75% 20584.1 ms
All error rate relative-kibana-content_issues-dashboard_75% 0 %
All Min Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-traffic-dashboard_50% 2.74 ops/s
All 50th percentile latency relative-kibana-traffic-dashboard_50% 443.283 ms
All 90th percentile latency relative-kibana-traffic-dashboard_50% 834.863 ms
All 99th percentile latency relative-kibana-traffic-dashboard_50% 12605.8 ms
All 100th percentile latency relative-kibana-traffic-dashboard_50% 21268.1 ms
All 50th percentile service time relative-kibana-traffic-dashboard_50% 405.711 ms
All 90th percentile service time relative-kibana-traffic-dashboard_50% 795.978 ms
All 99th percentile service time relative-kibana-traffic-dashboard_50% 12576.9 ms
All 100th percentile service time relative-kibana-traffic-dashboard_50% 21268.1 ms
All error rate relative-kibana-traffic-dashboard_50% 0 %
All Min Throughput relative-kibana-discover_50% 0.02 ops/s
All Median Throughput relative-kibana-discover_50% 0.02 ops/s
All Max Throughput relative-kibana-discover_50% 2.06 ops/s
All 50th percentile latency relative-kibana-discover_50% 618.171 ms
All 90th percentile latency relative-kibana-discover_50% 889.818 ms
All 99th percentile latency relative-kibana-discover_50% 5542.13 ms
All 100th percentile latency relative-kibana-discover_50% 19707 ms
All 50th percentile service time relative-kibana-discover_50% 561.12 ms
All 90th percentile service time relative-kibana-discover_50% 838.234 ms
All 99th percentile service time relative-kibana-discover_50% 5491.25 ms
All 100th percentile service time relative-kibana-discover_50% 19707 ms
All error rate relative-kibana-discover_50% 0 %

pmc

./rally --track=pmc --pipeline=benchmark-only --track-params="number_of_replicas:1,shard_count:3" --target-hosts=g1gc-1:9200,g1gc-2:9200,g2gc-3:9200
Lap Metric Task Value Unit
All Total indexing time   67.9994 min
All Min indexing time per shard   10.045 min
All Median indexing time per shard   13.2481 min
All Max indexing time per shard   16.4707 min
All Total merge time   46.233 min
All Min merge time per shard   8.16337 min
All Median merge time per shard   9.33915 min
All Max merge time per shard   9.97948 min
All Total merge throttle time   28.4157 min
All Min merge throttle time per shard   5.14693 min
All Median merge throttle time per shard   5.58062 min
All Max merge throttle time per shard   6.3123 min
All Total refresh time   7.4279 min
All Min refresh time per shard   1.39738 min
All Median refresh time per shard   1.48072 min
All Max refresh time per shard   1.5554 min
All Total flush time   3.20542 min
All Min flush time per shard   0.584133 min
All Median flush time per shard   0.599233 min
All Max flush time per shard   0.7663 min
All Total Young Gen GC   72.418 s
All Total Old Gen GC   0.296 s
All Store size   40.9026 GB
All Translog size   5.25423 GB
All Heap used for segments   22.4921 MB
All Heap used for doc values   0.0475197 MB
All Heap used for terms   20.524 MB
All Heap used for norms   0.0405884 MB
All Heap used for points   0.00933361 MB
All Heap used for stored fields   1.87071 MB
All Segment count   133  
All Min Throughput index-append 1240.15 docs/s
All Median Throughput index-append 1340.36 docs/s
All Max Throughput index-append 1375.46 docs/s
All 50th percentile latency index-append 2604.22 ms
All 90th percentile latency index-append 5227.75 ms
All 99th percentile latency index-append 7027.65 ms
All 100th percentile latency index-append 7680.86 ms
All 50th percentile service time index-append 2604.22 ms
All 90th percentile service time index-append 5227.75 ms
All 99th percentile service time index-append 7027.65 ms
All 100th percentile service time index-append 7680.86 ms
All error rate index-append 0 %
All Min Throughput default 20.02 ops/s
All Median Throughput default 20.03 ops/s
All Max Throughput default 20.03 ops/s
All 50th percentile latency default 10.7641 ms
All 90th percentile latency default 12.9347 ms
All 99th percentile latency default 13.8706 ms
All 100th percentile latency default 14.2149 ms
All 50th percentile service time default 10.6144 ms
All 90th percentile service time default 12.7797 ms
All 99th percentile service time default 13.749 ms
All 100th percentile service time default 14.0598 ms
All error rate default 0 %
All Min Throughput term 20.02 ops/s
All Median Throughput term 20.03 ops/s
All Max Throughput term 20.03 ops/s
All 50th percentile latency term 9.88134 ms
All 90th percentile latency term 10.7244 ms
All 99th percentile latency term 14.9585 ms
All 100th percentile latency term 57.1547 ms
All 50th percentile service time term 9.72369 ms
All 90th percentile service time term 10.5478 ms
All 99th percentile service time term 11.9907 ms
All 100th percentile service time term 56.9583 ms
All error rate term 0 %
All Min Throughput phrase 20.02 ops/s
All Median Throughput phrase 20.03 ops/s
All Max Throughput phrase 20.03 ops/s
All 50th percentile latency phrase 9.6383 ms
All 90th percentile latency phrase 10.4424 ms
All 99th percentile latency phrase 12.7418 ms
All 100th percentile latency phrase 45.2764 ms
All 50th percentile service time phrase 9.48708 ms
All 90th percentile service time phrase 10.2973 ms
All 99th percentile service time phrase 12.6105 ms
All 100th percentile service time phrase 45.157 ms
All error rate phrase 0 %
All Min Throughput articles_monthly_agg_uncached 20.02 ops/s
All Median Throughput articles_monthly_agg_uncached 20.03 ops/s
All Max Throughput articles_monthly_agg_uncached 20.03 ops/s
All 50th percentile latency articles_monthly_agg_uncached 11.6784 ms
All 90th percentile latency articles_monthly_agg_uncached 13.2621 ms
All 99th percentile latency articles_monthly_agg_uncached 14.9504 ms
All 100th percentile latency articles_monthly_agg_uncached 15.8437 ms
All 50th percentile service time articles_monthly_agg_uncached 11.5078 ms
All 90th percentile service time articles_monthly_agg_uncached 13.0542 ms
All 99th percentile service time articles_monthly_agg_uncached 14.7822 ms
All 100th percentile service time articles_monthly_agg_uncached 15.6935 ms
All error rate articles_monthly_agg_uncached 0 %
All Min Throughput articles_monthly_agg_cached 20.03 ops/s
All Median Throughput articles_monthly_agg_cached 20.03 ops/s
All Max Throughput articles_monthly_agg_cached 20.04 ops/s
All 50th percentile latency articles_monthly_agg_cached 4.76733 ms
All 90th percentile latency articles_monthly_agg_cached 5.19338 ms
All 99th percentile latency articles_monthly_agg_cached 6.09308 ms
All 100th percentile latency articles_monthly_agg_cached 47.8448 ms
All 50th percentile service time articles_monthly_agg_cached 4.59801 ms
All 90th percentile service time articles_monthly_agg_cached 5.01184 ms
All 99th percentile service time articles_monthly_agg_cached 5.92383 ms
All 100th percentile service time articles_monthly_agg_cached 47.6912 ms
All error rate articles_monthly_agg_cached 0 %
All Min Throughput scroll 24.91 pages/s
All Median Throughput scroll 25.01 pages/s
All Max Throughput scroll 25.03 pages/s
All 50th percentile latency scroll 958.894 ms
All 90th percentile latency scroll 992.332 ms
All 99th percentile latency scroll 1144.59 ms
All 100th percentile latency scroll 1203.42 ms
All 50th percentile service time scroll 956.644 ms
All 90th percentile service time scroll 986.589 ms
All 99th percentile service time scroll 1017.56 ms
All 100th percentile service time scroll 1203.24 ms
All error rate scroll 0 %

With 5 laps

Lap Metric Task Value Unit
All Total indexing time   364.02 min
All Min indexing time per shard   10.153 min
All Median indexing time per shard   15.1734 min
All Max indexing time per shard   20.1896 min
All Total merge time   247.767 min
All Min merge time per shard   8.47562 min
All Median merge time per shard   10.0001 min
All Max merge time per shard   11.1394 min
All Total merge throttle time   154.31 min
All Min merge throttle time per shard   4.26385 min
All Median merge throttle time per shard   6.14298 min
All Max merge throttle time per shard   7.74525 min
All Total refresh time   41.2019 min
All Min refresh time per shard   1.48305 min
All Median refresh time per shard   1.65043 min
All Max refresh time per shard   1.8129 min
All Total flush time   16.8058 min
All Min flush time per shard   0.438833 min
All Median flush time per shard   0.660567 min
All Max flush time per shard   1.03523 min
All Total Young Gen GC   386.543 s
All Total Old Gen GC   1.697 s
All Store size   40.9082 GB
All Translog size   5.34391 GB
All Heap used for segments   22.5691 MB
All Heap used for doc values   0.0542717 MB
All Heap used for terms   20.6015 MB
All Heap used for norms   0.0393677 MB
All Heap used for points   0.00924587 MB
All Heap used for stored fields   1.86835 MB
All Segment count   129  
All Min Throughput index-append 890.58 docs/s
All Median Throughput index-append 1256.56 docs/s
All Max Throughput index-append 1382.66 docs/s
All 50th percentile latency index-append 2746.42 ms
All 90th percentile latency index-append 5239.32 ms
All 99th percentile latency index-append 7743.78 ms
All 99.9th percentile latency index-append 9072.23 ms
All 100th percentile latency index-append 10008.2 ms
All 50th percentile service time index-append 2746.42 ms
All 90th percentile service time index-append 5239.32 ms
All 99th percentile service time index-append 7743.78 ms
All 99.9th percentile service time index-append 9072.23 ms
All 100th percentile service time index-append 10008.2 ms
All error rate index-append 0 %
All Min Throughput default 20.02 ops/s
All Median Throughput default 20.03 ops/s
All Max Throughput default 20.03 ops/s
All 50th percentile latency default 11.4203 ms
All 90th percentile latency default 14.4558 ms
All 99th percentile latency default 16.4097 ms
All 99.9th percentile latency default 18.7965 ms
All 100th percentile latency default 19.2748 ms
All 50th percentile service time default 11.2665 ms
All 90th percentile service time default 14.3385 ms
All 99th percentile service time default 16.283 ms
All 99.9th percentile service time default 18.6289 ms
All 100th percentile service time default 19.1138 ms
All error rate default 0 %
All Min Throughput term 20 ops/s
All Median Throughput term 20.03 ops/s
All Max Throughput term 20.03 ops/s
All 50th percentile latency term 10.811 ms
All 90th percentile latency term 11.9757 ms
All 99th percentile latency term 13.6379 ms
All 99.9th percentile latency term 30.3861 ms
All 100th percentile latency term 54.0043 ms
All 50th percentile service time term 10.653 ms
All 90th percentile service time term 11.8187 ms
All 99th percentile service time term 13.4147 ms
All 99.9th percentile service time term 30.2024 ms
All 100th percentile service time term 53.8333 ms
All error rate term 0 %
All Min Throughput phrase 20.02 ops/s
All Median Throughput phrase 20.03 ops/s
All Max Throughput phrase 20.03 ops/s
All 50th percentile latency phrase 10.7866 ms
All 90th percentile latency phrase 11.8076 ms
All 99th percentile latency phrase 23.1027 ms
All 99.9th percentile latency phrase 68.0101 ms
All 100th percentile latency phrase 68.5183 ms
All 50th percentile service time phrase 10.641 ms
All 90th percentile service time phrase 11.6411 ms
All 99th percentile service time phrase 14.0891 ms
All 99.9th percentile service time phrase 67.8411 ms
All 100th percentile service time phrase 68.3497 ms
All error rate phrase 0 %
All Min Throughput articles_monthly_agg_uncached 19.98 ops/s
All Median Throughput articles_monthly_agg_uncached 20.02 ops/s
All Max Throughput articles_monthly_agg_uncached 20.03 ops/s
All 50th percentile latency articles_monthly_agg_uncached 13.576 ms
All 90th percentile latency articles_monthly_agg_uncached 15.9956 ms
All 99th percentile latency articles_monthly_agg_uncached 18.9626 ms
All 99.9th percentile latency articles_monthly_agg_uncached 43.5139 ms
All 100th percentile latency articles_monthly_agg_uncached 73.6872 ms
All 50th percentile service time articles_monthly_agg_uncached 13.4091 ms
All 90th percentile service time articles_monthly_agg_uncached 15.8167 ms
All 99th percentile service time articles_monthly_agg_uncached 18.8093 ms
All 99.9th percentile service time articles_monthly_agg_uncached 23.689 ms
All 100th percentile service time articles_monthly_agg_uncached 73.5566 ms
All error rate articles_monthly_agg_uncached 0 %
All Min Throughput articles_monthly_agg_cached 20.03 ops/s
All Median Throughput articles_monthly_agg_cached 20.03 ops/s
All Max Throughput articles_monthly_agg_cached 20.04 ops/s
All 50th percentile latency articles_monthly_agg_cached 5.3961 ms
All 90th percentile latency articles_monthly_agg_cached 6.23134 ms
All 99th percentile latency articles_monthly_agg_cached 7.49517 ms
All 99.9th percentile latency articles_monthly_agg_cached 10.8551 ms
All 100th percentile latency articles_monthly_agg_cached 28.3347 ms
All 50th percentile service time articles_monthly_agg_cached 5.22992 ms
All 90th percentile service time articles_monthly_agg_cached 6.0625 ms
All 99th percentile service time articles_monthly_agg_cached 7.33509 ms
All 99.9th percentile service time articles_monthly_agg_cached 10.6964 ms
All 100th percentile service time articles_monthly_agg_cached 28.1285 ms
All error rate articles_monthly_agg_cached 0 %
All Min Throughput scroll 24.76 pages/s
All Median Throughput scroll 25.01 pages/s
All Max Throughput scroll 25.05 pages/s
All 50th percentile latency scroll 940.9 ms
All 90th percentile latency scroll 1033.73 ms
All 99th percentile latency scroll 1352.93 ms
All 100th percentile latency scroll 1487.67 ms
All 50th percentile service time scroll 935.939 ms
All 90th percentile service time scroll 1005.77 ms
All 99th percentile service time scroll 1092.95 ms
All 100th percentile service time scroll 1179.35 ms
All error rate scroll 0 %

G1GC

elasticlogs-1bn-load

./rally --track-repository=eventdata --track=eventdata --pipeline=benchmark-only --track-params="number_of_replicas:1,shard_count:3" --target-hosts=g1gc-1:9200,g1gc-2:9200,g2gc-3:9200 --challenge=elasticlogs-1bn-load
Lap Metric Task Value Unit
All Total indexing time   3928.3 min
All Min indexing time per shard   111.075 min
All Median indexing time per shard   131.605 min
All Max indexing time per shard   145.907 min
All Total merge time   1829.04 min
All Min merge time per shard   44.5192 min
All Median merge time per shard   62.4442 min
All Max merge time per shard   67.29 min
All Total merge throttle time   909.907 min
All Min merge throttle time per shard   20.8663 min
All Median merge throttle time per shard   30.8697 min
All Max merge throttle time per shard   35.7357 min
All Total refresh time   67.436 min
All Min refresh time per shard   2.09203 min
All Median refresh time per shard   2.26085 min
All Max refresh time per shard   2.43188 min
All Total flush time   63.4883 min
All Min flush time per shard   1.89793 min
All Median flush time per shard   2.11157 min
All Max flush time per shard   2.3732 min
All Total Young Gen GC   62.962 s
All Total Old Gen GC   0 s
All Store size   436.525 GB
All Translog size   31.8007 GB
All Heap used for segments   340.745 MB
All Heap used for doc values   0.136097 MB
All Heap used for terms   257.098 MB
All Heap used for norms   0 MB
All Heap used for points   37.6013 MB
All Heap used for stored fields   45.9095 MB
All Segment count   817  
All Min Throughput create_elasticlogs_q_write 1.46 ops/s
All Median Throughput create_elasticlogs_q_write 1.46 ops/s
All Max Throughput create_elasticlogs_q_write 1.46 ops/s
All 100th percentile latency create_elasticlogs_q_write 683.1 ms
All 100th percentile service time create_elasticlogs_q_write 683.1 ms
All error rate create_elasticlogs_q_write 0 %
All Min Throughput index-append-1000-elasticlogs_q_write 792.13 docs/s
All Median Throughput index-append-1000-elasticlogs_q_write 54095.4 docs/s
All Max Throughput index-append-1000-elasticlogs_q_write 65569.4 docs/s
All 50th percentile latency index-append-1000-elasticlogs_q_write 249.792 ms
All 90th percentile latency index-append-1000-elasticlogs_q_write 547.884 ms
All 99th percentile latency index-append-1000-elasticlogs_q_write 1567.75 ms
All 99.9th percentile latency index-append-1000-elasticlogs_q_write 2388.23 ms
All 99.99th percentile latency index-append-1000-elasticlogs_q_write 3421.53 ms
All 100th percentile latency index-append-1000-elasticlogs_q_write 6255.37 ms
All 50th percentile service time index-append-1000-elasticlogs_q_write 249.792 ms
All 90th percentile service time index-append-1000-elasticlogs_q_write 547.884 ms
All 99th percentile service time index-append-1000-elasticlogs_q_write 1567.75 ms
All 99.9th percentile service time index-append-1000-elasticlogs_q_write 2388.23 ms
All 99.99th percentile service time index-append-1000-elasticlogs_q_write 3421.53 ms
All 100th percentile service time index-append-1000-elasticlogs_q_write 6255.37 ms
All error rate index-append-1000-elasticlogs_q_write 0 %
All error rate rollover_elasticlogs_q_write_100M 0 %
All Min Throughput node_storage 1.65 ops/s
All Median Throughput node_storage 1.65 ops/s
All Max Throughput node_storage 1.65 ops/s
All 100th percentile latency node_storage 606.212 ms
All 100th percentile service time node_storage 606.212 ms
All error rate node_storage 0 %

For fun I wanted to try one with MaxPauseMillis=250, but it actually lowered the throughput:

Lap Metric Task Value Unit
All Total indexing time   4190.03 min
All Min indexing time per shard   118.748 min
All Median indexing time per shard   140.601 min
All Max indexing time per shard   157.904 min
All Total merge time   1906.48 min
All Min merge time per shard   44.0058 min
All Median merge time per shard   64.419 min
All Max merge time per shard   71.6829 min
All Total merge throttle time   956.279 min
All Min merge throttle time per shard   20.7336 min
All Median merge throttle time per shard   31.9254 min
All Max merge throttle time per shard   38.1027 min
All Total refresh time   70.3355 min
All Min refresh time per shard   2.11305 min
All Median refresh time per shard   2.34045 min
All Max refresh time per shard   2.65013 min
All Total flush time   62.9792 min
All Min flush time per shard   1.95497 min
All Median flush time per shard   2.08343 min
All Max flush time per shard   2.28715 min
All Total Young Gen GC   67.948 s
All Total Old Gen GC   0 s
All Store size   443.46 GB
All Translog size   31.9911 GB
All Heap used for segments   342.882 MB
All Heap used for doc values   0.13409 MB
All Heap used for terms   259.094 MB
All Heap used for norms   0 MB
All Heap used for points   37.622 MB
All Heap used for stored fields   46.0316 MB
All Segment count   827  
All Min Throughput create_elasticlogs_q_write 1.49 ops/s
All Median Throughput create_elasticlogs_q_write 1.49 ops/s
All Max Throughput create_elasticlogs_q_write 1.49 ops/s
All 100th percentile latency create_elasticlogs_q_write 670.58 ms
All 100th percentile service time create_elasticlogs_q_write 670.58 ms
All error rate create_elasticlogs_q_write 0 %
All Min Throughput index-append-1000-elasticlogs_q_write 876.12 docs/s
All Median Throughput index-append-1000-elasticlogs_q_write 51724.2 docs/s
All Max Throughput index-append-1000-elasticlogs_q_write 64216.8 docs/s
All 50th percentile latency index-append-1000-elasticlogs_q_write 268.761 ms
All 90th percentile latency index-append-1000-elasticlogs_q_write 571.664 ms
All 99th percentile latency index-append-1000-elasticlogs_q_write 1544.43 ms
All 99.9th percentile latency index-append-1000-elasticlogs_q_write 2441.8 ms
All 99.99th percentile latency index-append-1000-elasticlogs_q_write 3426.02 ms
All 100th percentile latency index-append-1000-elasticlogs_q_write 6794.39 ms
All 50th percentile service time index-append-1000-elasticlogs_q_write 268.761 ms
All 90th percentile service time index-append-1000-elasticlogs_q_write 571.664 ms
All 99th percentile service time index-append-1000-elasticlogs_q_write 1544.43 ms
All 99.9th percentile service time index-append-1000-elasticlogs_q_write 2441.8 ms
All 99.99th percentile service time index-append-1000-elasticlogs_q_write 3426.02 ms
All 100th percentile service time index-append-1000-elasticlogs_q_write 6794.39 ms
All error rate index-append-1000-elasticlogs_q_write 0 %
All error rate rollover_elasticlogs_q_write_100M 0 %
All Min Throughput node_storage 1.55 ops/s
All Median Throughput node_storage 1.55 ops/s
All Max Throughput node_storage 1.55 ops/s
All 100th percentile latency node_storage 643.98 ms
All 100th percentile service time node_storage 643.98 ms
All error rate node_storage 0 %

elasticlogs-querying

./rally --track-repository=eventdata --track=eventdata --pipeline=benchmark-only --track-params="number_of_replicas:1,shard_count:3" --target-hosts=g1gc-1:9200,g1gc-2:9200,g2gc-3:9200 --challenge=elasticlogs-querying
Lap Metric Task Value Unit
All Total refresh time   3.33333e-05 min
All Min refresh time per shard   0 min
All Median refresh time per shard   0 min
All Max refresh time per shard   1.66667e-05 min
All Total flush time   0.00101667 min
All Min flush time per shard   0 min
All Median flush time per shard   1.66667e-05 min
All Max flush time per shard   0.000133333 min
All Total Young Gen GC   0.91 s
All Total Old Gen GC   0 s
All Store size   419.01 GB
All Translog size   6.04428e-06 GB
All Heap used for segments   338.198 MB
All Heap used for doc values   0.113178 MB
All Heap used for terms   254.705 MB
All Heap used for norms   0 MB
All Heap used for points   37.4793 MB
All Heap used for stored fields   45.9001 MB
All Segment count   781  
All Min Throughput fieldstats_elasticlogs_q-* 0.05 ops/s
All Median Throughput fieldstats_elasticlogs_q-* 0.05 ops/s
All Max Throughput fieldstats_elasticlogs_q-* 0.05 ops/s
All 50th percentile latency fieldstats_elasticlogs_q-* 20800.7 ms
All 100th percentile latency fieldstats_elasticlogs_q-* 20878 ms
All 50th percentile service time fieldstats_elasticlogs_q-* 20800.7 ms
All 100th percentile service time fieldstats_elasticlogs_q-* 20878 ms
All error rate fieldstats_elasticlogs_q-* 0 %
All Min Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_50% 0.07 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_50% 271.436 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_50% 1003.93 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_50% 18885.6 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_50% 241.516 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_50% 975.823 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_50% 18885.6 ms
All error rate relative-kibana-content_issues-dashboard_50% 0 %
All Min Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_75% 0.04 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_75% 459.979 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_75% 2023.77 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_75% 27489.9 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_75% 371.975 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_75% 1934 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_75% 27489.9 ms
All error rate relative-kibana-content_issues-dashboard_75% 0 %
All Min Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-traffic-dashboard_50% 0.04 ops/s
All 50th percentile latency relative-kibana-traffic-dashboard_50% 440.939 ms
All 90th percentile latency relative-kibana-traffic-dashboard_50% 4531.26 ms
All 100th percentile latency relative-kibana-traffic-dashboard_50% 27305.3 ms
All 50th percentile service time relative-kibana-traffic-dashboard_50% 401.014 ms
All 90th percentile service time relative-kibana-traffic-dashboard_50% 4491.54 ms
All 100th percentile service time relative-kibana-traffic-dashboard_50% 27305.3 ms
All error rate relative-kibana-traffic-dashboard_50% 0 %
All Min Throughput relative-kibana-discover_50% 0.02 ops/s
All Median Throughput relative-kibana-discover_50% 0.02 ops/s
All Max Throughput relative-kibana-discover_50% 0.04 ops/s
All 50th percentile latency relative-kibana-discover_50% 612.857 ms
All 90th percentile latency relative-kibana-discover_50% 3114.53 ms
All 100th percentile latency relative-kibana-discover_50% 26515.3 ms
All 50th percentile service time relative-kibana-discover_50% 553.179 ms
All 90th percentile service time relative-kibana-discover_50% 3060.23 ms
All 100th percentile service time relative-kibana-discover_50% 26515.3 ms
All error rate relative-kibana-discover_50% 0 %

5 laps part 2:

Lap Metric Task Value Unit
All Total flush time   0.0025 min
All Min flush time per shard   0 min
All Median flush time per shard   1.66667e-05 min
All Max flush time per shard   8.33333e-05 min
All Total Young Gen GC   2.723 s
All Total Old Gen GC   0 s
All Store size   418.483 GB
All Translog size   9.42498e-06 GB
All Heap used for segments   333.887 MB
All Heap used for doc values   0.151627 MB
All Heap used for terms   250.277 MB
All Heap used for norms   0 MB
All Heap used for points   37.322 MB
All Heap used for stored fields   46.1361 MB
All Segment count   798  
All Min Throughput fieldstats_elasticlogs_q-* 0.19 ops/s
All Median Throughput fieldstats_elasticlogs_q-* 245.61 ops/s
All Max Throughput fieldstats_elasticlogs_q-* 279.74 ops/s
All 50th percentile latency fieldstats_elasticlogs_q-* 14.5087 ms
All 90th percentile latency fieldstats_elasticlogs_q-* 5284.25 ms
All 100th percentile latency fieldstats_elasticlogs_q-* 5686.06 ms
All 50th percentile service time fieldstats_elasticlogs_q-* 14.5087 ms
All 90th percentile service time fieldstats_elasticlogs_q-* 5284.25 ms
All 100th percentile service time fieldstats_elasticlogs_q-* 5686.06 ms
All error rate fieldstats_elasticlogs_q-* 0 %
All Min Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_50% 7.96 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_50% 202.649 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_50% 287.623 ms
All 99th percentile latency relative-kibana-content_issues-dashboard_50% 1283.2 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_50% 6589.45 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_50% 172.609 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_50% 257.651 ms
All 99th percentile service time relative-kibana-content_issues-dashboard_50% 1253.22 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_50% 6589.45 ms
All error rate relative-kibana-content_issues-dashboard_50% 0 %
All Min Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_75% 7.44 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_75% 288.43 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_75% 442.273 ms
All 99th percentile latency relative-kibana-content_issues-dashboard_75% 2247.11 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_75% 6740.44 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_75% 199.955 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_75% 352.357 ms
All 99th percentile service time relative-kibana-content_issues-dashboard_75% 2163.23 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_75% 6740.44 ms
All error rate relative-kibana-content_issues-dashboard_75% 0 %
All Min Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-traffic-dashboard_50% 3.42 ops/s
All 50th percentile latency relative-kibana-traffic-dashboard_50% 356.247 ms
All 90th percentile latency relative-kibana-traffic-dashboard_50% 624.395 ms
All 99th percentile latency relative-kibana-traffic-dashboard_50% 2065.35 ms
All 100th percentile latency relative-kibana-traffic-dashboard_50% 4571.93 ms
All 50th percentile service time relative-kibana-traffic-dashboard_50% 317.506 ms
All 90th percentile service time relative-kibana-traffic-dashboard_50% 584.601 ms
All 99th percentile service time relative-kibana-traffic-dashboard_50% 2025.71 ms
All 100th percentile service time relative-kibana-traffic-dashboard_50% 4571.93 ms
All error rate relative-kibana-traffic-dashboard_50% 0 %
All Min Throughput relative-kibana-discover_50% 0.02 ops/s
All Median Throughput relative-kibana-discover_50% 0.02 ops/s
All Max Throughput relative-kibana-discover_50% 7.49 ops/s
All 50th percentile latency relative-kibana-discover_50% 579.897 ms
All 90th percentile latency relative-kibana-discover_50% 818.757 ms
All 99th percentile latency relative-kibana-discover_50% 1279.49 ms
All 100th percentile latency relative-kibana-discover_50% 4788.81 ms
All 50th percentile service time relative-kibana-discover_50% 520.546 ms
All 90th percentile service time relative-kibana-discover_50% 764.965 ms
All 99th percentile service time relative-kibana-discover_50% 1220.7 ms
All 100th percentile service time relative-kibana-discover_50% 4788.81 ms
All error rate relative-kibana-discover_50% 0 %

pmc

./rally --track=pmc --pipeline=benchmark-only --track-params="number_of_replicas:1,shard_count:3" --target-hosts=g1gc-1:9200,g1gc-2:9200,g2gc-3:9200
Lap Metric Task Value Unit
All Total indexing time   61.0517 min
All Min indexing time per shard   8.89928 min
All Median indexing time per shard   13.2891 min
All Max indexing time per shard   15.8802 min
All Total merge time   50.3876 min
All Min merge time per shard   8.72638 min
All Median merge time per shard   10.4134 min
All Max merge time per shard   11.0188 min
All Total merge throttle time   33.1992 min
All Min merge throttle time per shard   5.17798 min
All Median merge throttle time per shard   6.77245 min
All Max merge throttle time per shard   7.56282 min
All Total refresh time   8.29097 min
All Min refresh time per shard   1.57005 min
All Median refresh time per shard   1.59133 min
All Max refresh time per shard   1.89875 min
All Total flush time   3.21873 min
All Min flush time per shard   0.525083 min
All Median flush time per shard   0.59845 min
All Max flush time per shard   0.828783 min
All Total Young Gen GC   1.919 s
All Total Old Gen GC   0 s
All Store size   40.8977 GB
All Translog size   5.39226 GB
All Heap used for segments   22.7355 MB
All Heap used for doc values   0.0664101 MB
All Heap used for terms   20.7439 MB
All Heap used for norms   0.0418091 MB
All Heap used for points   0.00935555 MB
All Heap used for stored fields   1.87403 MB
All Segment count   137  
All Min Throughput index-append 1079.95 docs/s
All Median Throughput index-append 1178.94 docs/s
All Max Throughput index-append 1215.67 docs/s
All 50th percentile latency index-append 2420.04 ms
All 90th percentile latency index-append 5266.67 ms
All 99th percentile latency index-append 7594.6 ms
All 100th percentile latency index-append 9319.36 ms
All 50th percentile service time index-append 2420.04 ms
All 90th percentile service time index-append 5266.67 ms
All 99th percentile service time index-append 7594.6 ms
All 100th percentile service time index-append 9319.36 ms
All error rate index-append 0 %
All Min Throughput default 20.02 ops/s
All Median Throughput default 20.03 ops/s
All Max Throughput default 20.03 ops/s
All 50th percentile latency default 11.6597 ms
All 90th percentile latency default 13.1125 ms
All 99th percentile latency default 14.6665 ms
All 100th percentile latency default 14.9636 ms
All 50th percentile service time default 11.5267 ms
All 90th percentile service time default 13.006 ms
All 99th percentile service time default 14.5584 ms
All 100th percentile service time default 14.8322 ms
All error rate default 0 %
All Min Throughput term 20.02 ops/s
All Median Throughput term 20.03 ops/s
All Max Throughput term 20.03 ops/s
All 50th percentile latency term 11.392 ms
All 90th percentile latency term 13.1072 ms
All 99th percentile latency term 14.056 ms
All 100th percentile latency term 15.6592 ms
All 50th percentile service time term 11.2565 ms
All 90th percentile service time term 12.9624 ms
All 99th percentile service time term 13.8959 ms
All 100th percentile service time term 15.5081 ms
All error rate term 0 %
All Min Throughput phrase 20.02 ops/s
All Median Throughput phrase 20.03 ops/s
All Max Throughput phrase 20.03 ops/s
All 50th percentile latency phrase 9.85028 ms
All 90th percentile latency phrase 11.0882 ms
All 99th percentile latency phrase 12.2849 ms
All 100th percentile latency phrase 13.2761 ms
All 50th percentile service time phrase 9.71091 ms
All 90th percentile service time phrase 10.9512 ms
All 99th percentile service time phrase 12.1646 ms
All 100th percentile service time phrase 13.1643 ms
All error rate phrase 0 %
All Min Throughput articles_monthly_agg_uncached 19.88 ops/s
All Median Throughput articles_monthly_agg_uncached 20.02 ops/s
All Max Throughput articles_monthly_agg_uncached 20.03 ops/s
All 50th percentile latency articles_monthly_agg_uncached 13.3704 ms
All 90th percentile latency articles_monthly_agg_uncached 17.1365 ms
All 99th percentile latency articles_monthly_agg_uncached 156.14 ms
All 100th percentile latency articles_monthly_agg_uncached 227.944 ms
All 50th percentile service time articles_monthly_agg_uncached 13.0723 ms
All 90th percentile service time articles_monthly_agg_uncached 16.7336 ms
All 99th percentile service time articles_monthly_agg_uncached 18.2286 ms
All 100th percentile service time articles_monthly_agg_uncached 227.766 ms
All error rate articles_monthly_agg_uncached 0 %
All Min Throughput articles_monthly_agg_cached 20.03 ops/s
All Median Throughput articles_monthly_agg_cached 20.03 ops/s
All Max Throughput articles_monthly_agg_cached 20.04 ops/s
All 50th percentile latency articles_monthly_agg_cached 5.21348 ms
All 90th percentile latency articles_monthly_agg_cached 5.70236 ms
All 99th percentile latency articles_monthly_agg_cached 6.42172 ms
All 100th percentile latency articles_monthly_agg_cached 7.15797 ms
All 50th percentile service time articles_monthly_agg_cached 5.07522 ms
All 90th percentile service time articles_monthly_agg_cached 5.56663 ms
All 99th percentile service time articles_monthly_agg_cached 6.27613 ms
All 100th percentile service time articles_monthly_agg_cached 6.99592 ms
All error rate articles_monthly_agg_cached 0 %
All Min Throughput scroll 24.94 pages/s
All Median Throughput scroll 25.01 pages/s
All Max Throughput scroll 25.03 pages/s
All 50th percentile latency scroll 969.721 ms
All 90th percentile latency scroll 1133.77 ms
All 99th percentile latency scroll 1230.47 ms
All 100th percentile latency scroll 1263.8 ms
All 50th percentile service time scroll 966.593 ms
All 90th percentile service time scroll 998.132 ms
All 99th percentile service time scroll 1050.76 ms
All 100th percentile service time scroll 1248.66 ms
All error rate scroll 0 %

With 5 laps

Lap Metric Task Value Unit
All Total indexing time   332.804 min
All Min indexing time per shard   10.6766 min
All Median indexing time per shard   13.292 min
All Max indexing time per shard   15.5201 min
All Total merge time   225.971 min
All Min merge time per shard   7.54777 min
All Median merge time per shard   8.93043 min
All Max merge time per shard   9.97797 min
All Total merge throttle time   138.606 min
All Min merge throttle time per shard   4.2712 min
All Median merge throttle time per shard   5.56677 min
All Max merge throttle time per shard   6.83867 min
All Total refresh time   38.2058 min
All Min refresh time per shard   1.39037 min
All Median refresh time per shard   1.52447 min
All Max refresh time per shard   1.68265 min
All Total flush time   16.802 min
All Min flush time per shard   0.491817 min
All Median flush time per shard   0.649383 min
All Max flush time per shard   0.960117 min
All Total Young Gen GC   8.07 s
All Total Old Gen GC   0 s
All Store size   40.9249 GB
All Translog size   5.31991 GB
All Heap used for segments   22.7053 MB
All Heap used for doc values   0.0608864 MB
All Heap used for terms   20.7195 MB
All Heap used for norms   0.0424194 MB
All Heap used for points   0.00941467 MB
All Heap used for stored fields   1.8717 MB
All Segment count   139  
All Min Throughput index-append 1233.81 docs/s
All Median Throughput index-append 1344.69 docs/s
All Max Throughput index-append 1445.33 docs/s
All 50th percentile latency index-append 2540.33 ms
All 90th percentile latency index-append 5163.92 ms
All 99th percentile latency index-append 7936.91 ms
All 99.9th percentile latency index-append 9496.53 ms
All 100th percentile latency index-append 10578.6 ms
All 50th percentile service time index-append 2540.33 ms
All 90th percentile service time index-append 5163.92 ms
All 99th percentile service time index-append 7936.91 ms
All 99.9th percentile service time index-append 9496.53 ms
All 100th percentile service time index-append 10578.6 ms
All error rate index-append 0 %
All Min Throughput default 20.02 ops/s
All Median Throughput default 20.03 ops/s
All Max Throughput default 20.03 ops/s
All 50th percentile latency default 11.8426 ms
All 90th percentile latency default 13.8528 ms
All 99th percentile latency default 15.3184 ms
All 99.9th percentile latency default 16.5164 ms
All 100th percentile latency default 17.0798 ms
All 50th percentile service time default 11.6998 ms
All 90th percentile service time default 13.6945 ms
All 99th percentile service time default 15.1599 ms
All 99.9th percentile service time default 16.3505 ms
All 100th percentile service time default 16.9164 ms
All error rate default 0 %
All Min Throughput term 20.02 ops/s
All Median Throughput term 20.03 ops/s
All Max Throughput term 20.03 ops/s
All 50th percentile latency term 10.566 ms
All 90th percentile latency term 12.972 ms
All 99th percentile latency term 13.9885 ms
All 99.9th percentile latency term 16.0098 ms
All 100th percentile latency term 17.1992 ms
All 50th percentile service time term 10.4125 ms
All 90th percentile service time term 12.8095 ms
All 99th percentile service time term 13.8143 ms
All 99.9th percentile service time term 15.8536 ms
All 100th percentile service time term 17.0424 ms
All error rate term 0 %
All Min Throughput phrase 20.02 ops/s
All Median Throughput phrase 20.03 ops/s
All Max Throughput phrase 20.03 ops/s
All 50th percentile latency phrase 10.2963 ms
All 90th percentile latency phrase 11.6155 ms
All 99th percentile latency phrase 13.5286 ms
All 99.9th percentile latency phrase 15.8142 ms
All 100th percentile latency phrase 16.0346 ms
All 50th percentile service time phrase 10.1663 ms
All 90th percentile service time phrase 11.4941 ms
All 99th percentile service time phrase 13.3483 ms
All 99.9th percentile service time phrase 15.6341 ms
All 100th percentile service time phrase 15.9149 ms
All error rate phrase 0 %
All Min Throughput articles_monthly_agg_uncached 20.02 ops/s
All Median Throughput articles_monthly_agg_uncached 20.02 ops/s
All Max Throughput articles_monthly_agg_uncached 20.03 ops/s
All 50th percentile latency articles_monthly_agg_uncached 13.1431 ms
All 90th percentile latency articles_monthly_agg_uncached 15.437 ms
All 99th percentile latency articles_monthly_agg_uncached 18.8106 ms
All 99.9th percentile latency articles_monthly_agg_uncached 20.9767 ms
All 100th percentile latency articles_monthly_agg_uncached 25.819 ms
All 50th percentile service time articles_monthly_agg_uncached 12.9907 ms
All 90th percentile service time articles_monthly_agg_uncached 15.2686 ms
All 99th percentile service time articles_monthly_agg_uncached 18.6325 ms
All 99.9th percentile service time articles_monthly_agg_uncached 20.8124 ms
All 100th percentile service time articles_monthly_agg_uncached 25.6715 ms
All error rate articles_monthly_agg_uncached 0 %
All Min Throughput articles_monthly_agg_cached 20.03 ops/s
All Median Throughput articles_monthly_agg_cached 20.03 ops/s
All Max Throughput articles_monthly_agg_cached 20.04 ops/s
All 50th percentile latency articles_monthly_agg_cached 4.99289 ms
All 90th percentile latency articles_monthly_agg_cached 5.66604 ms
All 99th percentile latency articles_monthly_agg_cached 6.91026 ms
All 99.9th percentile latency articles_monthly_agg_cached 11.9497 ms
All 100th percentile latency articles_monthly_agg_cached 14.8604 ms
All 50th percentile service time articles_monthly_agg_cached 4.82022 ms
All 90th percentile service time articles_monthly_agg_cached 5.49551 ms
All 99th percentile service time articles_monthly_agg_cached 6.75064 ms
All 99.9th percentile service time articles_monthly_agg_cached 11.7942 ms
All 100th percentile service time articles_monthly_agg_cached 14.6955 ms
All error rate articles_monthly_agg_cached 0 %
All Min Throughput scroll 24.88 pages/s
All Median Throughput scroll 25.01 pages/s
All Max Throughput scroll 25.07 pages/s
All 50th percentile latency scroll 949.031 ms
All 90th percentile latency scroll 1008.76 ms
All 99th percentile latency scroll 1238.21 ms
All 100th percentile latency scroll 1517.27 ms
All 50th percentile service time scroll 945.794 ms
All 90th percentile service time scroll 989.41 ms
All 99th percentile service time scroll 1071.53 ms
All 100th percentile service time scroll 1444.52 ms
All error rate scroll 0 %

Comparison

elasticlogs-1bn-load

Metric Task CMS G1GC Unit Change %
Total indexing time   4116.96 3928.3 min -4.5825075
Min indexing time per shard   118.7 111.075 min -6.4237574
Median indexing time per shard   137.78 131.605 min -4.4817826
Max indexing time per shard   172.41 145.907 min -15.372078
Total merge time   2056.68 1829.04 min -11.068324
Min merge time per shard   49.3475 44.5192 min -9.7842849
Median merge time per shard   69.6099 62.4442 min -10.294082
Max merge time per shard   75.5772 67.29 min -10.965212
Total merge throttle time   1051.26 909.907 min -13.446055
Min merge throttle time per shard   23.9423 20.8663 min -12.847554
Median merge throttle time per shard   35.5916 30.8697 min -13.266894
Max merge throttle time per shard   41.9042 35.7357 min -14.720481
Total refresh time   85.3203 67.436 min -20.961366
Min refresh time per shard   2.45742 2.09203 min -14.868846
Median refresh time per shard   2.86104 2.26085 min -20.978036
Max refresh time per shard   3.03292 2.43188 min -19.817206
Total flush time   65.7714 63.4883 min -3.4712656
Min flush time per shard   2.03267 1.89793 min -6.6287199
Median flush time per shard   2.16882 2.11157 min -2.6396843
Max flush time per shard   2.42153 2.3732 min -1.9958456
Total Young Gen GC   5407.64 62.962 s -98.835684
Total Old Gen GC   6.079 0 s -100.
Store size   431.862 436.525 GB 1.0797431
Translog size   31.9376 31.8007 GB -0.42864836
Heap used for segments   342.928 340.745 MB -0.63657677
Heap used for doc values   0.135151 0.136097 MB 0.69995783
Heap used for terms   259.084 257.098 MB -0.76654676
Heap used for norms   0 0 MB 0
Heap used for points   37.5948 37.6013 MB 0.017289625
Heap used for stored fields   46.114 45.9095 MB -0.44346619
Segment count   827 817   -1.2091898
Min Throughput create_elasticlogs_q_write 1.44 1.46 ops/s 1.3888889
Median Throughput create_elasticlogs_q_write 1.44 1.46 ops/s 1.3888889
Max Throughput create_elasticlogs_q_write 1.44 1.46 ops/s 1.3888889
100th percentile latency create_elasticlogs_q_write 696.096 683.1 ms -1.8669839
100th percentile service time create_elasticlogs_q_write 696.096 683.1 ms -1.8669839
error rate create_elasticlogs_q_write 0 0 % 0
Min Throughput index-append-1000-elasticlogs_q_write 783.03 792.13 docs/s 1.1621522
Median Throughput index-append-1000-elasticlogs_q_write 47188.9 54095.4 docs/s 14.635857
Max Throughput index-append-1000-elasticlogs_q_write 54164.7 65569.4 docs/s 21.055595
50th percentile latency index-append-1000-elasticlogs_q_write 305.588 249.792 ms -18.258570
90th percentile latency index-append-1000-elasticlogs_q_write 635.242 547.884 ms -13.751924
99th percentile latency index-append-1000-elasticlogs_q_write 1530.61 1567.75 ms 2.4264836
99.9th percentile latency index-append-1000-elasticlogs_q_write 2413.63 2388.23 ms -1.0523568
99.99th percentile latency index-append-1000-elasticlogs_q_write 3662.98 3421.53 ms -6.5916276
100th percentile latency index-append-1000-elasticlogs_q_write 8454.28 6255.37 ms -26.009430
50th percentile service time index-append-1000-elasticlogs_q_write 305.588 249.792 ms -18.258570
90th percentile service time index-append-1000-elasticlogs_q_write 635.242 547.884 ms -13.751924
99th percentile service time index-append-1000-elasticlogs_q_write 1530.61 1567.75 ms 2.4264836
99.9th percentile service time index-append-1000-elasticlogs_q_write 2413.63 2388.23 ms -1.0523568
99.99th percentile service time index-append-1000-elasticlogs_q_write 3662.98 3421.53 ms -6.5916276
100th percentile service time index-append-1000-elasticlogs_q_write 8454.28 6255.37 ms -26.009430
error rate index-append-1000-elasticlogs_q_write 0 0 % 0
error rate rollover_elasticlogs_q_write_100M 0 0 % 0
Min Throughput node_storage 5.49 1.65 ops/s -69.945355
Median Throughput node_storage 5.49 1.65 ops/s -69.945355
Max Throughput node_storage 5.49 1.65 ops/s -69.945355
100th percentile latency node_storage 182.115 606.212 ms 232.87318
100th percentile service time node_storage 182.115 606.212 ms 232.87318
error rate node_storage 0 0 % 0

elasticlogs-querying

Metric Task CMS G1GC Unit Change %
Total refresh time   3.33333e-05 3.33333e-05 min 0.
Min refresh time per shard   0 0 min 0
Median refresh time per shard   0 0 min 0
Max refresh time per shard   1.66667e-05 1.66667e-05 min 0.
Total flush time   0.0007 0.00101667 min 45.238571
Min flush time per shard   0 0 min 0
Median flush time per shard   1.66667e-05 1.66667e-05 min 0.
Max flush time per shard   0.0001 0.000133333 min 33.333
Total Young Gen GC   6.616 0.91 s -86.245466
Total Old Gen GC   0 0 s 0
Store size   418.812 419.01 GB 0.047276582
Translog size   6.65896e-06 6.04428e-06 GB -9.2308709
Heap used for segments   338.896 338.198 MB -0.20596289
Heap used for doc values   0.130119 0.113178 MB -13.019621
Heap used for terms   255.223 254.705 MB -0.20295976
Heap used for norms   0 0 MB 0
Heap used for points   37.4296 37.4793 MB 0.13278261
Heap used for stored fields   46.1134 45.9001 MB -0.46255535
Segment count   772 781   1.1658031
Min Throughput fieldstats_elasticlogs_q-* 0.07 0.05 ops/s -28.571429
Median Throughput fieldstats_elasticlogs_q-* 0.07 0.05 ops/s -28.571429
Max Throughput fieldstats_elasticlogs_q-* 0.07 0.05 ops/s -28.571429
50th percentile latency fieldstats_elasticlogs_q-* 13693.4 20800.7 ms 51.903107
100th percentile latency fieldstats_elasticlogs_q-* 13736.4 20878 ms 51.990332
50th percentile service time fieldstats_elasticlogs_q-* 13693.4 20800.7 ms 51.903107
100th percentile service time fieldstats_elasticlogs_q-* 13736.4 20878 ms 51.990332
error rate fieldstats_elasticlogs_q-* 0 0 % 0
Min Throughput relative-kibana-content_issues-dashboard_50% 0.03 0.03 ops/s 0.
Median Throughput relative-kibana-content_issues-dashboard_50% 0.03 0.03 ops/s 0.
Max Throughput relative-kibana-content_issues-dashboard_50% 0.06 0.07 ops/s 16.666667
50th percentile latency relative-kibana-content_issues-dashboard_50% 233.719 271.436 ms 16.137755
90th percentile latency relative-kibana-content_issues-dashboard_50% 613.704 1003.93 ms 63.585377
100th percentile latency relative-kibana-content_issues-dashboard_50% 19288 18885.6 ms -2.0862713
50th percentile service time relative-kibana-content_issues-dashboard_50% 205.025 241.516 ms 17.798317
90th percentile service time relative-kibana-content_issues-dashboard_50% 585.55 975.823 ms 66.650670
100th percentile service time relative-kibana-content_issues-dashboard_50% 19288 18885.6 ms -2.0862713
error rate relative-kibana-content_issues-dashboard_50% 0 0 % 0
Min Throughput relative-kibana-content_issues-dashboard_75% 0.01 0.01 ops/s 0.
Median Throughput relative-kibana-content_issues-dashboard_75% 0.01 0.01 ops/s 0.
Max Throughput relative-kibana-content_issues-dashboard_75% 0.04 0.04 ops/s 0.
50th percentile latency relative-kibana-content_issues-dashboard_75% 349.212 459.979 ms 31.719128
90th percentile latency relative-kibana-content_issues-dashboard_75% 2048.6 2023.77 ms -1.2120473
100th percentile latency relative-kibana-content_issues-dashboard_75% 24699.1 27489.9 ms 11.299197
50th percentile service time relative-kibana-content_issues-dashboard_75% 273.106 371.975 ms 36.201695
90th percentile service time relative-kibana-content_issues-dashboard_75% 1958.78 1934 ms -1.2650732
100th percentile service time relative-kibana-content_issues-dashboard_75% 24699.1 27489.9 ms 11.299197
error rate relative-kibana-content_issues-dashboard_75% 0 0 % 0
Min Throughput relative-kibana-traffic-dashboard_50% 0.03 0.03 ops/s 0.
Median Throughput relative-kibana-traffic-dashboard_50% 0.03 0.03 ops/s 0.
Max Throughput relative-kibana-traffic-dashboard_50% 0.04 0.04 ops/s 0.
50th percentile latency relative-kibana-traffic-dashboard_50% 388.149 440.939 ms 13.600447
90th percentile latency relative-kibana-traffic-dashboard_50% 3703.65 4531.26 ms 22.345794
100th percentile latency relative-kibana-traffic-dashboard_50% 24897.9 27305.3 ms 9.6690886
50th percentile service time relative-kibana-traffic-dashboard_50% 348.363 401.014 ms 15.113832
90th percentile service time relative-kibana-traffic-dashboard_50% 3666.3 4491.54 ms 22.508796
100th percentile service time relative-kibana-traffic-dashboard_50% 24897.9 27305.3 ms 9.6690886
error rate relative-kibana-traffic-dashboard_50% 0 0 % 0
Min Throughput relative-kibana-discover_50% 0.02 0.02 ops/s 0.
Median Throughput relative-kibana-discover_50% 0.02 0.02 ops/s 0.
Max Throughput relative-kibana-discover_50% 0.05 0.04 ops/s -20.
50th percentile latency relative-kibana-discover_50% 548.368 612.857 ms 11.760168
90th percentile latency relative-kibana-discover_50% 791.318 3114.53 ms 293.58766
100th percentile latency relative-kibana-discover_50% 21344 26515.3 ms 24.228355
50th percentile service time relative-kibana-discover_50% 488.685 553.179 ms 13.197458
90th percentile service time relative-kibana-discover_50% 731.564 3060.23 ms 318.31337
100th percentile service time relative-kibana-discover_50% 21344 26515.3 ms 24.228355
error rate relative-kibana-discover_50% 0 0 % 0

The querying looked a little wonky, so I re-ran with 5 laps to try and spread the load out a bit

Metric Task CMS G1GC Unit Change %
Total flush time   0.00783333 0.0025 min -68.085093
Min flush time per shard   0 0 min 0
Median flush time per shard   1.66667e-05 1.66667e-05 min 0.
Max flush time per shard   0.000333333 8.33333e-05 min -74.999985
Total Young Gen GC   28.21 2.723 s -90.347395
Total Old Gen GC   0.092 0 s -100.
Store size   418.483 418.483 GB 0.
Translog size   6.14673e-06 9.42498e-06 GB 53.333236
Heap used for segments   333.173 333.887 MB 0.21430308
Heap used for doc values   0.13308 0.151627 MB 13.936730
Heap used for terms   249.58 250.277 MB 0.27926917
Heap used for norms   0 0 MB 0
Heap used for points   37.3245 37.322 MB -0.0066980134
Heap used for stored fields   46.1352 46.1361 MB 0.001950788
Segment count   786 798   1.5267176
Min Throughput fieldstats_elasticlogs_q-* 0.08 0.19 ops/s 137.5
Median Throughput fieldstats_elasticlogs_q-* 181.52 245.61 ops/s 35.307404
Max Throughput fieldstats_elasticlogs_q-* 297.47 279.74 ops/s -5.9602649
50th percentile latency fieldstats_elasticlogs_q-* 19.0765 14.5087 ms -23.944644
90th percentile latency fieldstats_elasticlogs_q-* 14219.2 5284.25 ms -62.837220
100th percentile latency fieldstats_elasticlogs_q-* 14283.6 5686.06 ms -60.191688
50th percentile service time fieldstats_elasticlogs_q-* 19.0765 14.5087 ms -23.944644
90th percentile service time fieldstats_elasticlogs_q-* 14219.2 5284.25 ms -62.837220
100th percentile service time fieldstats_elasticlogs_q-* 14283.6 5686.06 ms -60.191688
error rate fieldstats_elasticlogs_q-* 0 0 % 0
Min Throughput relative-kibana-content_issues-dashboard_50% 0.03 0.03 ops/s 0.
Median Throughput relative-kibana-content_issues-dashboard_50% 0.03 0.03 ops/s 0.
Max Throughput relative-kibana-content_issues-dashboard_50% 4.26 7.96 ops/s 86.854460
50th percentile latency relative-kibana-content_issues-dashboard_50% 204.005 202.649 ms -0.66468959
90th percentile latency relative-kibana-content_issues-dashboard_50% 365.552 287.623 ms -21.318171
99th percentile latency relative-kibana-content_issues-dashboard_50% 2938.91 1283.2 ms -56.337554
100th percentile latency relative-kibana-content_issues-dashboard_50% 15148.1 6589.45 ms -56.499825
50th percentile service time relative-kibana-content_issues-dashboard_50% 174.899 172.609 ms -1.3093271
90th percentile service time relative-kibana-content_issues-dashboard_50% 335.416 257.651 ms -23.184642
99th percentile service time relative-kibana-content_issues-dashboard_50% 2909.12 1253.22 ms -56.920993
100th percentile service time relative-kibana-content_issues-dashboard_50% 15148.1 6589.45 ms -56.499825
error rate relative-kibana-content_issues-dashboard_50% 0 0 % 0
Min Throughput relative-kibana-content_issues-dashboard_75% 0.01 0.01 ops/s 0.
Median Throughput relative-kibana-content_issues-dashboard_75% 0.01 0.01 ops/s 0.
Max Throughput relative-kibana-content_issues-dashboard_75% 3.71 7.44 ops/s 100.53908
50th percentile latency relative-kibana-content_issues-dashboard_75% 321.488 288.43 ms -10.282810
90th percentile latency relative-kibana-content_issues-dashboard_75% 473.447 442.273 ms -6.5844751
99th percentile latency relative-kibana-content_issues-dashboard_75% 5124.13 2247.11 ms -56.146507
100th percentile latency relative-kibana-content_issues-dashboard_75% 20584.1 6740.44 ms -67.254143
50th percentile service time relative-kibana-content_issues-dashboard_75% 232.122 199.955 ms -13.857799
90th percentile service time relative-kibana-content_issues-dashboard_75% 401.631 352.357 ms -12.268475
99th percentile service time relative-kibana-content_issues-dashboard_75% 5053.75 2163.23 ms -57.195548
100th percentile service time relative-kibana-content_issues-dashboard_75% 20584.1 6740.44 ms -67.254143
error rate relative-kibana-content_issues-dashboard_75% 0 0 % 0
Min Throughput relative-kibana-traffic-dashboard_50% 0.03 0.03 ops/s 0.
Median Throughput relative-kibana-traffic-dashboard_50% 0.03 0.03 ops/s 0.
Max Throughput relative-kibana-traffic-dashboard_50% 2.74 3.42 ops/s 24.817518
50th percentile latency relative-kibana-traffic-dashboard_50% 443.283 356.247 ms -19.634410
90th percentile latency relative-kibana-traffic-dashboard_50% 834.863 624.395 ms -25.209885
99th percentile latency relative-kibana-traffic-dashboard_50% 12605.8 2065.35 ms -83.615875
100th percentile latency relative-kibana-traffic-dashboard_50% 21268.1 4571.93 ms -78.503345
50th percentile service time relative-kibana-traffic-dashboard_50% 405.711 317.506 ms -21.740845
90th percentile service time relative-kibana-traffic-dashboard_50% 795.978 584.601 ms -26.555633
99th percentile service time relative-kibana-traffic-dashboard_50% 12576.9 2025.71 ms -83.893408
100th percentile service time relative-kibana-traffic-dashboard_50% 21268.1 4571.93 ms -78.503345
error rate relative-kibana-traffic-dashboard_50% 0 0 % 0
Min Throughput relative-kibana-discover_50% 0.02 0.02 ops/s 0.
Median Throughput relative-kibana-discover_50% 0.02 0.02 ops/s 0.
Max Throughput relative-kibana-discover_50% 2.06 7.49 ops/s 263.59223
50th percentile latency relative-kibana-discover_50% 618.171 579.897 ms -6.1914907
90th percentile latency relative-kibana-discover_50% 889.818 818.757 ms -7.9860151
99th percentile latency relative-kibana-discover_50% 5542.13 1279.49 ms -76.913389
100th percentile latency relative-kibana-discover_50% 19707 4788.81 ms -75.699954
50th percentile service time relative-kibana-discover_50% 561.12 520.546 ms -7.2308954
90th percentile service time relative-kibana-discover_50% 838.234 764.965 ms -8.7408767
99th percentile service time relative-kibana-discover_50% 5491.25 1220.7 ms -77.770089
100th percentile service time relative-kibana-discover_50% 19707 4788.81 ms -75.699954
error rate relative-kibana-discover_50% 0 0 % 0

pmc

Metric Task CMS G1GC Unit Change %
Total indexing time   67.9994 61.0517 min -10.217296
Min indexing time per shard   10.045 8.89928 min -11.405874
Median indexing time per shard   13.2481 13.2891 min 0.30947834
Max indexing time per shard   16.4707 15.8802 min -3.5851542
Total merge time   46.233 50.3876 min 8.9862220
Min merge time per shard   8.16337 8.72638 min 6.8967840
Median merge time per shard   9.33915 10.4134 min 11.502653
Max merge time per shard   9.97948 11.0188 min 10.414571
Total merge throttle time   28.4157 33.1992 min 16.834004
Min merge throttle time per shard   5.14693 5.17798 min 0.60327224
Median merge throttle time per shard   5.58062 6.77245 min 21.356588
Max merge throttle time per shard   6.3123 7.56282 min 19.810845
Total refresh time   7.4279 8.29097 min 11.619300
Min refresh time per shard   1.39738 1.57005 min 12.356696
Median refresh time per shard   1.48072 1.59133 min 7.4700146
Max refresh time per shard   1.5554 1.89875 min 22.074707
Total flush time   3.20542 3.21873 min 0.41523420
Min flush time per shard   0.584133 0.525083 min -10.108999
Median flush time per shard   0.599233 0.59845 min -0.13066704
Max flush time per shard   0.7663 0.828783 min 8.1538562
Total Young Gen GC   72.418 1.919 s -97.350106
Total Old Gen GC   0.296 0 s -100.
Store size   40.9026 40.8977 GB -0.011979679
Translog size   5.25423 5.39226 GB 2.6270262
Heap used for segments   22.4921 22.7355 MB 1.0821577
Heap used for doc values   0.0475197 0.0664101 MB 39.752776
Heap used for terms   20.524 20.7439 MB 1.0714286
Heap used for norms   0.0405884 0.0418091 MB 3.0075095
Heap used for points   0.00933361 0.00935555 MB 0.23506446
Heap used for stored fields   1.87071 1.87403 MB 0.17747272
Segment count   133 137   3.0075188
Min Throughput index-append 1240.15 1079.95 docs/s -12.917792
Median Throughput index-append 1340.36 1178.94 docs/s -12.043033
Max Throughput index-append 1375.46 1215.67 docs/s -11.617204
50th percentile latency index-append 2604.22 2420.04 ms -7.0723672
90th percentile latency index-append 5227.75 5266.67 ms 0.74448855
99th percentile latency index-append 7027.65 7594.6 ms 8.0674194
100th percentile latency index-append 7680.86 9319.36 ms 21.332247
50th percentile service time index-append 2604.22 2420.04 ms -7.0723672
90th percentile service time index-append 5227.75 5266.67 ms 0.74448855
99th percentile service time index-append 7027.65 7594.6 ms 8.0674194
100th percentile service time index-append 7680.86 9319.36 ms 21.332247
error rate index-append 0 0 % 0
Min Throughput default 20.02 20.02 ops/s 0.
Median Throughput default 20.03 20.03 ops/s 0.
Max Throughput default 20.03 20.03 ops/s 0.
50th percentile latency default 10.7641 11.6597 ms 8.3202497
90th percentile latency default 12.9347 13.1125 ms 1.3745970
99th percentile latency default 13.8706 14.6665 ms 5.7380358
100th percentile latency default 14.2149 14.9636 ms 5.2670086
50th percentile service time default 10.6144 11.5267 ms 8.5949276
90th percentile service time default 12.7797 13.006 ms 1.7707771
99th percentile service time default 13.749 14.5584 ms 5.8869736
100th percentile service time default 14.0598 14.8322 ms 5.4936770
error rate default 0 0 % 0
Min Throughput term 20.02 20.02 ops/s 0.
Median Throughput term 20.03 20.03 ops/s 0.
Max Throughput term 20.03 20.03 ops/s 0.
50th percentile latency term 9.88134 11.392 ms 15.288007
90th percentile latency term 10.7244 13.1072 ms 22.218492
99th percentile latency term 14.9585 14.056 ms -6.0333590
100th percentile latency term 57.1547 15.6592 ms -72.602078
50th percentile service time term 9.72369 11.2565 ms 15.763666
90th percentile service time term 10.5478 12.9624 ms 22.891977
99th percentile service time term 11.9907 13.8959 ms 15.888981
100th percentile service time term 56.9583 15.5081 ms -72.772888
error rate term 0 0 % 0
Min Throughput phrase 20.02 20.02 ops/s 0.
Median Throughput phrase 20.03 20.03 ops/s 0.
Max Throughput phrase 20.03 20.03 ops/s 0.
50th percentile latency phrase 9.6383 9.85028 ms 2.1993505
90th percentile latency phrase 10.4424 11.0882 ms 6.1844021
99th percentile latency phrase 12.7418 12.2849 ms -3.5858356
100th percentile latency phrase 45.2764 13.2761 ms -70.677660
50th percentile service time phrase 9.48708 9.71091 ms 2.3593139
90th percentile service time phrase 10.2973 10.9512 ms 6.3502083
99th percentile service time phrase 12.6105 12.1646 ms -3.5359423
100th percentile service time phrase 45.157 13.1643 ms -70.847709
error rate phrase 0 0 % 0
Min Throughput articles_monthly_agg_uncached 20.02 19.88 ops/s -0.69930070
Median Throughput articles_monthly_agg_uncached 20.03 20.02 ops/s -0.049925112
Max Throughput articles_monthly_agg_uncached 20.03 20.03 ops/s 0.
50th percentile latency articles_monthly_agg_uncached 11.6784 13.3704 ms 14.488286
90th percentile latency articles_monthly_agg_uncached 13.2621 17.1365 ms 29.214076
99th percentile latency articles_monthly_agg_uncached 14.9504 156.14 ms 944.38677
100th percentile latency articles_monthly_agg_uncached 15.8437 227.944 ms 1338.7043
50th percentile service time articles_monthly_agg_uncached 11.5078 13.0723 ms 13.595127
90th percentile service time articles_monthly_agg_uncached 13.0542 16.7336 ms 28.185565
99th percentile service time articles_monthly_agg_uncached 14.7822 18.2286 ms 23.314527
100th percentile service time articles_monthly_agg_uncached 15.6935 227.766 ms 1351.3397
error rate articles_monthly_agg_uncached 0 0 % 0
Min Throughput articles_monthly_agg_cached 20.03 20.03 ops/s 0.
Median Throughput articles_monthly_agg_cached 20.03 20.03 ops/s 0.
Max Throughput articles_monthly_agg_cached 20.04 20.04 ops/s 0.
50th percentile latency articles_monthly_agg_cached 4.76733 5.21348 ms 9.3584879
90th percentile latency articles_monthly_agg_cached 5.19338 5.70236 ms 9.8005538
99th percentile latency articles_monthly_agg_cached 6.09308 6.42172 ms 5.3936597
100th percentile latency articles_monthly_agg_cached 47.8448 7.15797 ms -85.039189
50th percentile service time articles_monthly_agg_cached 4.59801 5.07522 ms 10.378620
90th percentile service time articles_monthly_agg_cached 5.01184 5.56663 ms 11.069587
99th percentile service time articles_monthly_agg_cached 5.92383 6.27613 ms 5.9471659
100th percentile service time articles_monthly_agg_cached 47.6912 6.99592 ms -85.330795
error rate articles_monthly_agg_cached 0 0 % 0
Min Throughput scroll 24.91 24.94 pages/s 0.12043356
Median Throughput scroll 25.01 25.01 pages/s 0.
Max Throughput scroll 25.03 25.03 pages/s 0.
50th percentile latency scroll 958.894 969.721 ms 1.1291133
90th percentile latency scroll 992.332 1133.77 ms 14.253093
99th percentile latency scroll 1144.59 1230.47 ms 7.5031234
100th percentile latency scroll 1203.42 1263.8 ms 5.0173672
50th percentile service time scroll 956.644 966.593 ms 1.0399898
90th percentile service time scroll 986.589 998.132 ms 1.1699907
99th percentile service time scroll 1017.56 1050.76 ms 3.2627069
100th percentile service time scroll 1203.24 1248.66 ms 3.7748080
error rate scroll 0 0 % 0

Comparison of the 5 lap runs

Metric Task CMS G1GC Unit Change %
Total indexing time   364.02 332.804 min -8.5753530
Min indexing time per shard   10.153 10.6766 min 5.1570964
Median indexing time per shard   15.1734 13.292 min -12.399330
Max indexing time per shard   20.1896 15.5201 min -23.128244
Total merge time   247.767 225.971 min -8.7969746
Min merge time per shard   8.47562 7.54777 min -10.947282
Median merge time per shard   10.0001 8.93043 min -10.696593
Max merge time per shard   11.1394 9.97797 min -10.426325
Total merge throttle time   154.31 138.606 min -10.176917
Min merge throttle time per shard   4.26385 4.2712 min 0.17237942
Median merge throttle time per shard   6.14298 5.56677 min -9.3799752
Max merge throttle time per shard   7.74525 6.83867 min -11.704980
Total refresh time   41.2019 38.2058 min -7.2717520
Min refresh time per shard   1.48305 1.39037 min -6.2492836
Median refresh time per shard   1.65043 1.52447 min -7.6319505
Max refresh time per shard   1.8129 1.68265 min -7.1846213
Total flush time   16.8058 16.802 min -0.022611241
Min flush time per shard   0.438833 0.491817 min 12.073841
Median flush time per shard   0.660567 0.649383 min -1.6930909
Max flush time per shard   1.03523 0.960117 min -7.2556823
Total Young Gen GC   386.543 8.07 s -97.912263
Total Old Gen GC   1.697 0 s -100.
Store size   40.9082 40.9249 GB 0.040823111
Translog size   5.34391 5.31991 GB -0.44910936
Heap used for segments   22.5691 22.7053 MB 0.60347998
Heap used for doc values   0.0542717 0.0608864 MB 12.188120
Heap used for terms   20.6015 20.7195 MB 0.57277383
Heap used for norms   0.0393677 0.0424194 MB 7.7517864
Heap used for points   0.00924587 0.00941467 MB 1.8256800
Heap used for stored fields   1.86835 1.8717 MB 0.17930259
Segment count   129 139   7.7519380
Min Throughput index-append 890.58 1233.81 docs/s 38.540053
Median Throughput index-append 1256.56 1344.69 docs/s 7.0135927
Max Throughput index-append 1382.66 1445.33 docs/s 4.5325677
50th percentile latency index-append 2746.42 2540.33 ms -7.5039506
90th percentile latency index-append 5239.32 5163.92 ms -1.4391181
99th percentile latency index-append 7743.78 7936.91 ms 2.4940016
99.9th percentile latency index-append 9072.23 9496.53 ms 4.6769096
100th percentile latency index-append 10008.2 10578.6 ms 5.6993266
50th percentile service time index-append 2746.42 2540.33 ms -7.5039506
90th percentile service time index-append 5239.32 5163.92 ms -1.4391181
99th percentile service time index-append 7743.78 7936.91 ms 2.4940016
99.9th percentile service time index-append 9072.23 9496.53 ms 4.6769096
100th percentile service time index-append 10008.2 10578.6 ms 5.6993266
error rate index-append 0 0 % 0
Min Throughput default 20.02 20.02 ops/s 0.
Median Throughput default 20.03 20.03 ops/s 0.
Max Throughput default 20.03 20.03 ops/s 0.
50th percentile latency default 11.4203 11.8426 ms 3.6978013
90th percentile latency default 14.4558 13.8528 ms -4.1713361
99th percentile latency default 16.4097 15.3184 ms -6.6503349
99.9th percentile latency default 18.7965 16.5164 ms -12.130450
100th percentile latency default 19.2748 17.0798 ms -11.387926
50th percentile service time default 11.2665 11.6998 ms 3.8459149
90th percentile service time default 14.3385 13.6945 ms -4.4914043
99th percentile service time default 16.283 15.1599 ms -6.8973776
99.9th percentile service time default 18.6289 16.3505 ms -12.230459
100th percentile service time default 19.1138 16.9164 ms -11.496406
error rate default 0 0 % 0
Min Throughput term 20 20.02 ops/s 0.1
Median Throughput term 20.03 20.03 ops/s 0.
Max Throughput term 20.03 20.03 ops/s 0.
50th percentile latency term 10.811 10.566 ms -2.2662103
90th percentile latency term 11.9757 12.972 ms 8.3193467
99th percentile latency term 13.6379 13.9885 ms 2.5707770
99.9th percentile latency term 30.3861 16.0098 ms -47.312093
100th percentile latency term 54.0043 17.1992 ms -68.152166
50th percentile service time term 10.653 10.4125 ms -2.2575800
90th percentile service time term 11.8187 12.8095 ms 8.3833247
99th percentile service time term 13.4147 13.8143 ms 2.9788217
99.9th percentile service time term 30.2024 15.8536 ms -47.508807
100th percentile service time term 53.8333 17.0424 ms -68.342271
error rate term 0 0 % 0
Min Throughput phrase 20.02 20.02 ops/s 0.
Median Throughput phrase 20.03 20.03 ops/s 0.
Max Throughput phrase 20.03 20.03 ops/s 0.
50th percentile latency phrase 10.7866 10.2963 ms -4.5454545
90th percentile latency phrase 11.8076 11.6155 ms -1.6269183
99th percentile latency phrase 23.1027 13.5286 ms -41.441477
99.9th percentile latency phrase 68.0101 15.8142 ms -76.747277
100th percentile latency phrase 68.5183 16.0346 ms -76.598077
50th percentile service time phrase 10.641 10.1663 ms -4.4610469
90th percentile service time phrase 11.6411 11.4941 ms -1.2627673
99th percentile service time phrase 14.0891 13.3483 ms -5.2579654
99.9th percentile service time phrase 67.8411 15.6341 ms -76.954825
100th percentile service time phrase 68.3497 15.9149 ms -76.715479
error rate phrase 0 0 % 0
Min Throughput articles_monthly_agg_uncached 19.98 20.02 ops/s 0.2002002
Median Throughput articles_monthly_agg_uncached 20.02 20.02 ops/s 0.
Max Throughput articles_monthly_agg_uncached 20.03 20.03 ops/s 0.
50th percentile latency articles_monthly_agg_uncached 13.576 13.1431 ms -3.1887154
90th percentile latency articles_monthly_agg_uncached 15.9956 15.437 ms -3.4922104
99th percentile latency articles_monthly_agg_uncached 18.9626 18.8106 ms -0.80157784
99.9th percentile latency articles_monthly_agg_uncached 43.5139 20.9767 ms -51.793105
100th percentile latency articles_monthly_agg_uncached 73.6872 25.819 ms -64.961350
50th percentile service time articles_monthly_agg_uncached 13.4091 12.9907 ms -3.1202691
90th percentile service time articles_monthly_agg_uncached 15.8167 15.2686 ms -3.4653246
99th percentile service time articles_monthly_agg_uncached 18.8093 18.6325 ms -0.93996055
99.9th percentile service time articles_monthly_agg_uncached 23.689 20.8124 ms -12.143189
100th percentile service time articles_monthly_agg_uncached 73.5566 25.6715 ms -65.099665
error rate articles_monthly_agg_uncached 0 0 % 0
Min Throughput articles_monthly_agg_cached 20.03 20.03 ops/s 0.
Median Throughput articles_monthly_agg_cached 20.03 20.03 ops/s 0.
Max Throughput articles_monthly_agg_cached 20.04 20.04 ops/s 0.
50th percentile latency articles_monthly_agg_cached 5.3961 4.99289 ms -7.4722485
90th percentile latency articles_monthly_agg_cached 6.23134 5.66604 ms -9.0718850
99th percentile latency articles_monthly_agg_cached 7.49517 6.91026 ms -7.8038257
99.9th percentile latency articles_monthly_agg_cached 10.8551 11.9497 ms 10.083739
100th percentile latency articles_monthly_agg_cached 28.3347 14.8604 ms -47.554059
50th percentile service time articles_monthly_agg_cached 5.22992 4.82022 ms -7.8337718
90th percentile service time articles_monthly_agg_cached 6.0625 5.49551 ms -9.3524124
99th percentile service time articles_monthly_agg_cached 7.33509 6.75064 ms -7.9678641
99.9th percentile service time articles_monthly_agg_cached 10.6964 11.7942 ms 10.263266
100th percentile service time articles_monthly_agg_cached 28.1285 14.6955 ms -47.755835
error rate articles_monthly_agg_cached 0 0 % 0
Min Throughput scroll 24.76 24.88 pages/s 0.48465267
Median Throughput scroll 25.01 25.01 pages/s 0.
Max Throughput scroll 25.05 25.07 pages/s 0.079840319
50th percentile latency scroll 940.9 949.031 ms 0.86417260
90th percentile latency scroll 1033.73 1008.76 ms -2.4155244
99th percentile latency scroll 1352.93 1238.21 ms -8.4793744
100th percentile latency scroll 1487.67 1517.27 ms 1.9896886
50th percentile service time scroll 935.939 945.794 ms 1.0529532
90th percentile service time scroll 1005.77 989.41 ms -1.6266144
99th percentile service time scroll 1092.95 1071.53 ms -1.9598335
100th percentile service time scroll 1179.35 1444.52 ms 22.484419
error rate scroll 0 0 % 0

8gb

CMS

elasticlogs-1bn-load

Lap Metric Task Value Unit
All Total indexing time   3975.49 min
All Min indexing time per shard   104.302 min
All Median indexing time per shard   139.947 min
All Max indexing time per shard   149.099 min
All Total merge time   1878.01 min
All Min merge time per shard   45.5764 min
All Median merge time per shard   64.5762 min
All Max merge time per shard   67.3346 min
All Total merge throttle time   969.918 min
All Min merge throttle time per shard   22.0315 min
All Median merge throttle time per shard   32.524 min
All Max merge throttle time per shard   37.6408 min
All Total refresh time   70.787 min
All Min refresh time per shard   2.12968 min
All Median refresh time per shard   2.38066 min
All Max refresh time per shard   2.47522 min
All Total flush time   64.3053 min
All Min flush time per shard   1.75375 min
All Median flush time per shard   2.12187 min
All Max flush time per shard   2.47807 min
All Total Young Gen GC   1623.59 s
All Total Old Gen GC   27.516 s
All Store size   435.285 GB
All Translog size   32.0145 GB
All Heap used for segments   343.496 MB
All Heap used for doc values   0.119415 MB
All Heap used for terms   259.676 MB
All Heap used for norms   0 MB
All Heap used for points   37.6844 MB
All Heap used for stored fields   46.0162 MB
All Segment count   806  
All Min Throughput create_elasticlogs_q_write 1.57 ops/s
All Median Throughput create_elasticlogs_q_write 1.57 ops/s
All Max Throughput create_elasticlogs_q_write 1.57 ops/s
All 100th percentile latency create_elasticlogs_q_write 636.027 ms
All 100th percentile service time create_elasticlogs_q_write 636.027 ms
All error rate create_elasticlogs_q_write 0 %
All Min Throughput index-append-1000-elasticlogs_q_write 984.22 docs/s
All Median Throughput index-append-1000-elasticlogs_q_write 52716.2 docs/s
All Max Throughput index-append-1000-elasticlogs_q_write 61198.2 docs/s
All 50th percentile latency index-append-1000-elasticlogs_q_write 262.524 ms
All 90th percentile latency index-append-1000-elasticlogs_q_write 565.88 ms
All 99th percentile latency index-append-1000-elasticlogs_q_write 1528.3 ms
All 99.9th percentile latency index-append-1000-elasticlogs_q_write 2353.09 ms
All 99.99th percentile latency index-append-1000-elasticlogs_q_write 3128.78 ms
All 100th percentile latency index-append-1000-elasticlogs_q_write 7113.7 ms
All 50th percentile service time index-append-1000-elasticlogs_q_write 262.524 ms
All 90th percentile service time index-append-1000-elasticlogs_q_write 565.88 ms
All 99th percentile service time index-append-1000-elasticlogs_q_write 1528.3 ms
All 99.9th percentile service time index-append-1000-elasticlogs_q_write 2353.09 ms
All 99.99th percentile service time index-append-1000-elasticlogs_q_write 3128.78 ms
All 100th percentile service time index-append-1000-elasticlogs_q_write 7113.7 ms
All error rate index-append-1000-elasticlogs_q_write 0 %
All error rate rollover_elasticlogs_q_write_100M 0 %
All Min Throughput node_storage 7.18 ops/s
All Median Throughput node_storage 7.18 ops/s
All Max Throughput node_storage 7.18 ops/s
All 100th percentile latency node_storage 139.261 ms
All 100th percentile service time node_storage 139.261 ms
All error rate node_storage 0 %

elasticlogs-querying

Lap Metric Task Value Unit
All Total flush time   0.0007 min
All Min flush time per shard   0 min
All Median flush time per shard   1.66667e-05 min
All Max flush time per shard   0.000266667 min
All Total Young Gen GC   3.587 s
All Total Old Gen GC   0.312 s
All Store size   419.119 GB
All Translog size   6.65896e-06 GB
All Heap used for segments   339.578 MB
All Heap used for doc values   0.11544 MB
All Heap used for terms   255.983 MB
All Heap used for norms   0 MB
All Heap used for points   37.485 MB
All Heap used for stored fields   45.9951 MB
All Segment count   764  
All Min Throughput fieldstats_elasticlogs_q-* 0.11 ops/s
All Median Throughput fieldstats_elasticlogs_q-* 0.11 ops/s
All Max Throughput fieldstats_elasticlogs_q-* 0.11 ops/s
All 50th percentile latency fieldstats_elasticlogs_q-* 9412.5 ms
All 100th percentile latency fieldstats_elasticlogs_q-* 9429.81 ms
All 50th percentile service time fieldstats_elasticlogs_q-* 9412.5 ms
All 100th percentile service time fieldstats_elasticlogs_q-* 9429.81 ms
All error rate fieldstats_elasticlogs_q-* 0 %
All Min Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_50% 0.16 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_50% 211.759 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_50% 815.589 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_50% 6234.4 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_50% 181.838 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_50% 785.457 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_50% 6234.4 ms
All error rate relative-kibana-content_issues-dashboard_50% 0 %
All Min Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_75% 0.14 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_75% 303.836 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_75% 2023.73 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_75% 7344.46 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_75% 213.77 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_75% 1933.67 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_75% 7344.46 ms
All error rate relative-kibana-content_issues-dashboard_75% 0 %
All Min Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-traffic-dashboard_50% 0.23 ops/s
All 50th percentile latency relative-kibana-traffic-dashboard_50% 360.103 ms
All 90th percentile latency relative-kibana-traffic-dashboard_50% 3812.55 ms
All 100th percentile latency relative-kibana-traffic-dashboard_50% 9897.44 ms
All 50th percentile service time relative-kibana-traffic-dashboard_50% 320.432 ms
All 90th percentile service time relative-kibana-traffic-dashboard_50% 3773.92 ms
All 100th percentile service time relative-kibana-traffic-dashboard_50% 9861.63 ms
All error rate relative-kibana-traffic-dashboard_50% 0 %
All Min Throughput relative-kibana-discover_50% 0.02 ops/s
All Median Throughput relative-kibana-discover_50% 0.02 ops/s
All Max Throughput relative-kibana-discover_50% 0.22 ops/s
All 50th percentile latency relative-kibana-discover_50% 465.032 ms
All 90th percentile latency relative-kibana-discover_50% 879.546 ms
All 100th percentile latency relative-kibana-discover_50% 4616.18 ms
All 50th percentile service time relative-kibana-discover_50% 405.12 ms
All 90th percentile service time relative-kibana-discover_50% 819.774 ms
All 100th percentile service time relative-kibana-discover_50% 4616.18 ms
All error rate relative-kibana-discover_50% 0 %

pmc

Lap Metric Task Value Unit
All Total indexing time   55.1542 min
All Min indexing time per shard   8.31752 min
All Median indexing time per shard   9.65123 min
All Max indexing time per shard   14.6086 min
All Total merge time   44.4351 min
All Min merge time per shard   7.9564 min
All Median merge time per shard   9.02675 min
All Max merge time per shard   9.31437 min
All Total merge throttle time   28.8702 min
All Min merge throttle time per shard   4.51987 min
All Median merge throttle time per shard   6.11615 min
All Max merge throttle time per shard   6.54187 min
All Total refresh time   7.37102 min
All Min refresh time per shard   1.44172 min
All Median refresh time per shard   1.46093 min
All Max refresh time per shard   1.53483 min
All Total flush time   3.52218 min
All Min flush time per shard   0.5386 min
All Median flush time per shard   0.73755 min
All Max flush time per shard   0.828617 min
All Total Young Gen GC   26.109 s
All Total Old Gen GC   1.143 s
All Store size   40.9117 GB
All Translog size   5.34441 GB
All Heap used for segments   22.9091 MB
All Heap used for doc values   0.0668411 MB
All Heap used for terms   20.9177 MB
All Heap used for norms   0.0421143 MB
All Heap used for points   0.00927925 MB
All Heap used for stored fields   1.87319 MB
All Segment count   138  
All Min Throughput index-append 974.09 docs/s
All Median Throughput index-append 1119.11 docs/s
All Max Throughput index-append 1155.76 docs/s
All 50th percentile latency index-append 2305.29 ms
All 90th percentile latency index-append 4345.72 ms
All 99th percentile latency index-append 6800.68 ms
All 100th percentile latency index-append 9696.25 ms
All 50th percentile service time index-append 2305.29 ms
All 90th percentile service time index-append 4345.72 ms
All 99th percentile service time index-append 6800.68 ms
All 100th percentile service time index-append 9696.25 ms
All error rate index-append 0 %
All Min Throughput default 20.02 ops/s
All Median Throughput default 20.03 ops/s
All Max Throughput default 20.03 ops/s
All 50th percentile latency default 9.31461 ms
All 90th percentile latency default 10.6594 ms
All 99th percentile latency default 11.2845 ms
All 100th percentile latency default 11.6693 ms
All 50th percentile service time default 9.19263 ms
All 90th percentile service time default 10.5448 ms
All 99th percentile service time default 11.1222 ms
All 100th percentile service time default 11.5191 ms
All error rate default 0 %
All Min Throughput term 20.02 ops/s
All Median Throughput term 20.03 ops/s
All Max Throughput term 20.03 ops/s
All 50th percentile latency term 9.8853 ms
All 90th percentile latency term 10.6476 ms
All 99th percentile latency term 12.7413 ms
All 100th percentile latency term 29.2749 ms
All 50th percentile service time term 9.77414 ms
All 90th percentile service time term 10.5436 ms
All 99th percentile service time term 12.6598 ms
All 100th percentile service time term 29.1359 ms
All error rate term 0 %
All Min Throughput phrase 20.02 ops/s
All Median Throughput phrase 20.03 ops/s
All Max Throughput phrase 20.03 ops/s
All 50th percentile latency phrase 10.0304 ms
All 90th percentile latency phrase 10.6479 ms
All 99th percentile latency phrase 14.5024 ms
All 100th percentile latency phrase 18.0932 ms
All 50th percentile service time phrase 9.89359 ms
All 90th percentile service time phrase 10.524 ms
All 99th percentile service time phrase 14.3045 ms
All 100th percentile service time phrase 17.9279 ms
All error rate phrase 0 %
All Min Throughput articles_monthly_agg_uncached 20.02 ops/s
All Median Throughput articles_monthly_agg_uncached 20.03 ops/s
All Max Throughput articles_monthly_agg_uncached 20.03 ops/s
All 50th percentile latency articles_monthly_agg_uncached 12.4331 ms
All 90th percentile latency articles_monthly_agg_uncached 13.4298 ms
All 99th percentile latency articles_monthly_agg_uncached 16.9639 ms
All 100th percentile latency articles_monthly_agg_uncached 20.095 ms
All 50th percentile service time articles_monthly_agg_uncached 12.2865 ms
All 90th percentile service time articles_monthly_agg_uncached 13.2836 ms
All 99th percentile service time articles_monthly_agg_uncached 16.8036 ms
All 100th percentile service time articles_monthly_agg_uncached 19.9565 ms
All error rate articles_monthly_agg_uncached 0 %
All Min Throughput articles_monthly_agg_cached 20.03 ops/s
All Median Throughput articles_monthly_agg_cached 20.03 ops/s
All Max Throughput articles_monthly_agg_cached 20.04 ops/s
All 50th percentile latency articles_monthly_agg_cached 4.65779 ms
All 90th percentile latency articles_monthly_agg_cached 5.02153 ms
All 99th percentile latency articles_monthly_agg_cached 5.63183 ms
All 100th percentile latency articles_monthly_agg_cached 5.94603 ms
All 50th percentile service time articles_monthly_agg_cached 4.5034 ms
All 90th percentile service time articles_monthly_agg_cached 4.87476 ms
All 99th percentile service time articles_monthly_agg_cached 5.46725 ms
All 100th percentile service time articles_monthly_agg_cached 5.80725 ms
All error rate articles_monthly_agg_cached 0 %
All Min Throughput scroll 25 pages/s
All Median Throughput scroll 25.03 pages/s
All Max Throughput scroll 25.07 pages/s
All 50th percentile latency scroll 872.236 ms
All 90th percentile latency scroll 901.938 ms
All 99th percentile latency scroll 943.159 ms
All 100th percentile latency scroll 1011.53 ms
All 50th percentile service time scroll 871.527 ms
All 90th percentile service time scroll 901.688 ms
All 99th percentile service time scroll 942.98 ms
All 100th percentile service time scroll 1011.4 ms
All error rate scroll 0 %

G1GC

elasticlogs-1bn-load

Lap Metric Task Value Unit
All Total indexing time   3955.63 min
All Min indexing time per shard   109.698 min
All Median indexing time per shard   135.065 min
All Max indexing time per shard   152.197 min
All Total merge time   1838.6 min
All Min merge time per shard   44.3991 min
All Median merge time per shard   62.4085 min
All Max merge time per shard   70.9957 min
All Total merge throttle time   959.575 min
All Min merge throttle time per shard   21.3453 min
All Median merge throttle time per shard   32.4583 min
All Max merge throttle time per shard   38.9493 min
All Total refresh time   67.0896 min
All Min refresh time per shard   1.96207 min
All Median refresh time per shard   2.23758 min
All Max refresh time per shard   2.35863 min
All Total flush time   62.6435 min
All Min flush time per shard   1.83213 min
All Median flush time per shard   2.08168 min
All Max flush time per shard   2.35955 min
All Total Young Gen GC   184.039 s
All Total Old Gen GC   0 s
All Store size   439.278 GB
All Translog size   31.8434 GB
All Heap used for segments   342.724 MB
All Heap used for doc values   0.123569 MB
All Heap used for terms   259.046 MB
All Heap used for norms   0 MB
All Heap used for points   37.6617 MB
All Heap used for stored fields   45.8927 MB
All Segment count   803  
All Min Throughput create_elasticlogs_q_write 1.68 ops/s
All Median Throughput create_elasticlogs_q_write 1.68 ops/s
All Max Throughput create_elasticlogs_q_write 1.68 ops/s
All 100th percentile latency create_elasticlogs_q_write 593.953 ms
All 100th percentile service time create_elasticlogs_q_write 593.953 ms
All error rate create_elasticlogs_q_write 0 %
All Min Throughput index-append-1000-elasticlogs_q_write 1916.79 docs/s
All Median Throughput index-append-1000-elasticlogs_q_write 53650.2 docs/s
All Max Throughput index-append-1000-elasticlogs_q_write 67386.6 docs/s
All 50th percentile latency index-append-1000-elasticlogs_q_write 257.466 ms
All 90th percentile latency index-append-1000-elasticlogs_q_write 549.837 ms
All 99th percentile latency index-append-1000-elasticlogs_q_write 1579.82 ms
All 99.9th percentile latency index-append-1000-elasticlogs_q_write 2390.89 ms
All 99.99th percentile latency index-append-1000-elasticlogs_q_write 3128.13 ms
All 100th percentile latency index-append-1000-elasticlogs_q_write 5829.07 ms
All 50th percentile service time index-append-1000-elasticlogs_q_write 257.466 ms
All 90th percentile service time index-append-1000-elasticlogs_q_write 549.837 ms
All 99th percentile service time index-append-1000-elasticlogs_q_write 1579.82 ms
All 99.9th percentile service time index-append-1000-elasticlogs_q_write 2390.89 ms
All 99.99th percentile service time index-append-1000-elasticlogs_q_write 3128.13 ms
All 100th percentile service time index-append-1000-elasticlogs_q_write 5829.07 ms
All error rate index-append-1000-elasticlogs_q_write 0 %
All error rate rollover_elasticlogs_q_write_100M 0 %
All Min Throughput node_storage 4.25 ops/s
All Median Throughput node_storage 4.25 ops/s
All Max Throughput node_storage 4.25 ops/s
All 100th percentile latency node_storage 235.245 ms
All 100th percentile service time node_storage 235.245 ms
All error rate node_storage 0 %

elasticlogs-querying

Lap Metric Task Value Unit
All Total flush time   0.0007 min
All Min flush time per shard   0 min
All Median flush time per shard   1.66667e-05 min
All Max flush time per shard   6.66667e-05 min
All Total Young Gen GC   1.014 s
All Total Old Gen GC   0 s
All Store size   418.928 GB
All Translog size   31.8434 GB
All Heap used for segments   338.788 MB
All Heap used for doc values   0.128914 MB
All Heap used for terms   255.269 MB
All Heap used for norms   0 MB
All Heap used for points   37.505 MB
All Heap used for stored fields   45.8849 MB
All Segment count   780  
All Min Throughput fieldstats_elasticlogs_q-* 0.09 ops/s
All Median Throughput fieldstats_elasticlogs_q-* 0.18 ops/s
All Max Throughput fieldstats_elasticlogs_q-* 0.27 ops/s
All 50th percentile latency fieldstats_elasticlogs_q-* 10921.9 ms
All 100th percentile latency fieldstats_elasticlogs_q-* 11023.1 ms
All 50th percentile service time fieldstats_elasticlogs_q-* 10921.9 ms
All 100th percentile service time fieldstats_elasticlogs_q-* 11023.1 ms
All error rate fieldstats_elasticlogs_q-* 0 %
All Min Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_50% 0.1 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_50% 198.267 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_50% 1888.08 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_50% 10013.7 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_50% 168.396 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_50% 1859.92 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_50% 10013.7 ms
All error rate relative-kibana-content_issues-dashboard_50% 0 %
All Min Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_75% 0.1 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_75% 305.723 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_75% 798.621 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_75% 10007.9 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_75% 218.01 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_75% 708.718 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_75% 10007.9 ms
All error rate relative-kibana-content_issues-dashboard_75% 0 %
All Min Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-traffic-dashboard_50% 0.14 ops/s
All 50th percentile latency relative-kibana-traffic-dashboard_50% 351.977 ms
All 90th percentile latency relative-kibana-traffic-dashboard_50% 5342.88 ms
All 100th percentile latency relative-kibana-traffic-dashboard_50% 12223 ms
All 50th percentile service time relative-kibana-traffic-dashboard_50% 312.173 ms
All 90th percentile service time relative-kibana-traffic-dashboard_50% 5304.03 ms
All 100th percentile service time relative-kibana-traffic-dashboard_50% 12190 ms
All error rate relative-kibana-traffic-dashboard_50% 0 %
All Min Throughput relative-kibana-discover_50% 0.02 ops/s
All Median Throughput relative-kibana-discover_50% 0.02 ops/s
All Max Throughput relative-kibana-discover_50% 0.13 ops/s
All 50th percentile latency relative-kibana-discover_50% 440.001 ms
All 90th percentile latency relative-kibana-discover_50% 918.561 ms
All 100th percentile latency relative-kibana-discover_50% 7702.11 ms
All 50th percentile service time relative-kibana-discover_50% 380.053 ms
All 90th percentile service time relative-kibana-discover_50% 858.726 ms
All 100th percentile service time relative-kibana-discover_50% 7702.11 ms
All error rate relative-kibana-discover_50% 0 %

pmc

Lap Metric Task Value Unit
All Total indexing time   62.5757 min
All Min indexing time per shard   10.3166 min
All Median indexing time per shard   11.5227 min
All Max indexing time per shard   15.07 min
All Total merge time   44.8156 min
All Min merge time per shard   8.1999 min
All Median merge time per shard   8.90982 min
All Max merge time per shard   9.95788 min
All Total merge throttle time   27.4901 min
All Min merge throttle time per shard   5.09343 min
All Median merge throttle time per shard   5.51017 min
All Max merge throttle time per shard   6.03607 min
All Total refresh time   7.20438 min
All Min refresh time per shard   1.3669 min
All Median refresh time per shard   1.4654 min
All Max refresh time per shard   1.50385 min
All Total flush time   4.50015 min
All Min flush time per shard   0.6131 min
All Median flush time per shard   0.778383 min
All Max flush time per shard   1.3403 min
All Total Young Gen GC   5.464 s
All Total Old Gen GC   0 s
All Store size   40.9331 GB
All Translog size   5.33372 GB
All Heap used for segments   23.1308 MB
All Heap used for doc values   0.0636253 MB
All Heap used for terms   21.1481 MB
All Heap used for norms   0.0405884 MB
All Heap used for points   0.00939941 MB
All Heap used for stored fields   1.86913 MB
All Segment count   133  
All Min Throughput index-append 1356.54 docs/s
All Median Throughput index-append 1478.88 docs/s
All Max Throughput index-append 1528.32 docs/s
All 50th percentile latency index-append 2374.97 ms
All 90th percentile latency index-append 4902.01 ms
All 99th percentile latency index-append 7012.87 ms
All 100th percentile latency index-append 7392.95 ms
All 50th percentile service time index-append 2374.97 ms
All 90th percentile service time index-append 4902.01 ms
All 99th percentile service time index-append 7012.87 ms
All 100th percentile service time index-append 7392.95 ms
All error rate index-append 0 %
All Min Throughput default 20.02 ops/s
All Median Throughput default 20.03 ops/s
All Max Throughput default 20.03 ops/s
All 50th percentile latency default 10.0342 ms
All 90th percentile latency default 12.0661 ms
All 99th percentile latency default 12.5932 ms
All 100th percentile latency default 12.7473 ms
All 50th percentile service time default 9.90447 ms
All 90th percentile service time default 11.9607 ms
All 99th percentile service time default 12.4939 ms
All 100th percentile service time default 12.6518 ms
All error rate default 0 %
All Min Throughput term 20.02 ops/s
All Median Throughput term 20.03 ops/s
All Max Throughput term 20.03 ops/s
All 50th percentile latency term 8.8444 ms
All 90th percentile latency term 9.36376 ms
All 99th percentile latency term 10.421 ms
All 100th percentile latency term 20.235 ms
All 50th percentile service time term 8.69404 ms
All 90th percentile service time term 9.24857 ms
All 99th percentile service time term 10.2819 ms
All 100th percentile service time term 20.079 ms
All error rate term 0 %
All Min Throughput phrase 20.02 ops/s
All Median Throughput phrase 20.03 ops/s
All Max Throughput phrase 20.03 ops/s
All 50th percentile latency phrase 9.49498 ms
All 90th percentile latency phrase 9.99144 ms
All 99th percentile latency phrase 10.3404 ms
All 100th percentile latency phrase 10.4212 ms
All 50th percentile service time phrase 9.37132 ms
All 90th percentile service time phrase 9.87008 ms
All 99th percentile service time phrase 10.2058 ms
All 100th percentile service time phrase 10.3087 ms
All error rate phrase 0 %
All Min Throughput articles_monthly_agg_uncached 20.02 ops/s
All Median Throughput articles_monthly_agg_uncached 20.03 ops/s
All Max Throughput articles_monthly_agg_uncached 20.03 ops/s
All 50th percentile latency articles_monthly_agg_uncached 11.8155 ms
All 90th percentile latency articles_monthly_agg_uncached 13.0478 ms
All 99th percentile latency articles_monthly_agg_uncached 14.8695 ms
All 100th percentile latency articles_monthly_agg_uncached 19.2801 ms
All 50th percentile service time articles_monthly_agg_uncached 11.6596 ms
All 90th percentile service time articles_monthly_agg_uncached 12.8601 ms
All 99th percentile service time articles_monthly_agg_uncached 14.7171 ms
All 100th percentile service time articles_monthly_agg_uncached 19.1226 ms
All error rate articles_monthly_agg_uncached 0 %
All Min Throughput articles_monthly_agg_cached 20.03 ops/s
All Median Throughput articles_monthly_agg_cached 20.03 ops/s
All Max Throughput articles_monthly_agg_cached 20.04 ops/s
All 50th percentile latency articles_monthly_agg_cached 4.6134 ms
All 90th percentile latency articles_monthly_agg_cached 4.8619 ms
All 99th percentile latency articles_monthly_agg_cached 5.06305 ms
All 100th percentile latency articles_monthly_agg_cached 5.13005 ms
All 50th percentile service time articles_monthly_agg_cached 4.47007 ms
All 90th percentile service time articles_monthly_agg_cached 4.72233 ms
All 99th percentile service time articles_monthly_agg_cached 4.90648 ms
All 100th percentile service time articles_monthly_agg_cached 4.96033 ms
All error rate articles_monthly_agg_cached 0 %
All Min Throughput scroll 24.97 pages/s
All Median Throughput scroll 25.03 pages/s
All Max Throughput scroll 25.07 pages/s
All 50th percentile latency scroll 879.935 ms
All 90th percentile latency scroll 950.875 ms
All 99th percentile latency scroll 1080.29 ms
All 100th percentile latency scroll 1111.04 ms
All 50th percentile service time scroll 879.643 ms
All 90th percentile service time scroll 937.052 ms
All 99th percentile service time scroll 1048.64 ms
All 100th percentile service time scroll 1052 ms
All error rate scroll 0 %

Comparison

elasticlogs-1bn-load

Metric Task CMS G1GC Unit Change %
Total indexing time   3975.49 3955.63 min -0.49956106
Min indexing time per shard   104.302 109.698 min 5.1734387
Median indexing time per shard   139.947 135.065 min -3.4884635
Max indexing time per shard   149.099 152.197 min 2.0778141
Total merge time   1878.01 1838.6 min -2.0984979
Min merge time per shard   45.5764 44.3991 min -2.5831351
Median merge time per shard   64.5762 62.4085 min -3.3568095
Max merge time per shard   67.3346 70.9957 min 5.4371749
Total merge throttle time   969.918 959.575 min -1.0663788
Min merge throttle time per shard   22.0315 21.3453 min -3.1146313
Median merge throttle time per shard   32.524 32.4583 min -0.20200467
Max merge throttle time per shard   37.6408 38.9493 min 3.4762811
Total refresh time   70.787 67.0896 min -5.2232755
Min refresh time per shard   2.12968 1.96207 min -7.8701965
Median refresh time per shard   2.38066 2.23758 min -6.0100980
Max refresh time per shard   2.47522 2.35863 min -4.7102884
Total flush time   64.3053 62.6435 min -2.5842349
Min flush time per shard   1.75375 1.83213 min 4.4692801
Median flush time per shard   2.12187 2.08168 min -1.8940840
Max flush time per shard   2.47807 2.35955 min -4.7827543
Total Young Gen GC   1623.59 184.039 s -88.664688
Total Old Gen GC   27.516 0 s -100.
Store size   435.285 439.278 GB 0.91733003
Translog size   32.0145 31.8434 GB -0.53444533
Heap used for segments   343.496 342.724 MB -0.22474789
Heap used for doc values   0.119415 0.123569 MB 3.4786250
Heap used for terms   259.676 259.046 MB -0.24261002
Heap used for norms   0 0 MB 0
Heap used for points   37.6844 37.6617 MB -0.060237127
Heap used for stored fields   46.0162 45.8927 MB -0.26838374
Segment count   806 803   -0.37220844
Min Throughput create_elasticlogs_q_write 1.57 1.68 ops/s 7.0063694
Median Throughput create_elasticlogs_q_write 1.57 1.68 ops/s 7.0063694
Max Throughput create_elasticlogs_q_write 1.57 1.68 ops/s 7.0063694
100th percentile latency create_elasticlogs_q_write 636.027 593.953 ms -6.6151280
100th percentile service time create_elasticlogs_q_write 636.027 593.953 ms -6.6151280
error rate create_elasticlogs_q_write 0 0 % 0
Min Throughput index-append-1000-elasticlogs_q_write 984.22 1916.79 docs/s 94.752190
Median Throughput index-append-1000-elasticlogs_q_write 52716.2 53650.2 docs/s 1.7717514
Max Throughput index-append-1000-elasticlogs_q_write 61198.2 67386.6 docs/s 10.112062
50th percentile latency index-append-1000-elasticlogs_q_write 262.524 257.466 ms -1.9266810
90th percentile latency index-append-1000-elasticlogs_q_write 565.88 549.837 ms -2.8350534
99th percentile latency index-append-1000-elasticlogs_q_write 1528.3 1579.82 ms 3.3710659
99.9th percentile latency index-append-1000-elasticlogs_q_write 2353.09 2390.89 ms 1.6063984
99.99th percentile latency index-append-1000-elasticlogs_q_write 3128.78 3128.13 ms -0.020774871
100th percentile latency index-append-1000-elasticlogs_q_write 7113.7 5829.07 ms -18.058535
50th percentile service time index-append-1000-elasticlogs_q_write 262.524 257.466 ms -1.9266810
90th percentile service time index-append-1000-elasticlogs_q_write 565.88 549.837 ms -2.8350534
99th percentile service time index-append-1000-elasticlogs_q_write 1528.3 1579.82 ms 3.3710659
99.9th percentile service time index-append-1000-elasticlogs_q_write 2353.09 2390.89 ms 1.6063984
99.99th percentile service time index-append-1000-elasticlogs_q_write 3128.78 3128.13 ms -0.020774871
100th percentile service time index-append-1000-elasticlogs_q_write 7113.7 5829.07 ms -18.058535
error rate index-append-1000-elasticlogs_q_write 0 0 % 0
error rate rollover_elasticlogs_q_write_100M 0 0 % 0
Min Throughput node_storage 7.18 4.25 ops/s -40.807799
Median Throughput node_storage 7.18 4.25 ops/s -40.807799
Max Throughput node_storage 7.18 4.25 ops/s -40.807799
100th percentile latency node_storage 139.261 235.245 ms 68.923819
100th percentile service time node_storage 139.261 235.245 ms 68.923819
error rate node_storage 0 0 % 0

elasticlogs-querying

Metric Task CMS G1GC Unit Change %
Total flush time   0.0007 0.0007 min 0.
Min flush time per shard   0 0 min 0
Median flush time per shard   1.66667e-05 1.66667e-05 min 0.
Max flush time per shard   0.000266667 6.66667e-05 min -75.000019
Total Young Gen GC   3.587 1.014 s -71.731252
Total Old Gen GC   0.312 0 s -100.
Store size   419.119 418.928 GB -0.045571783
Translog size   6.65896e-06 31.8434 GB 478203700.
Heap used for segments   339.578 338.788 MB -0.23264169
Heap used for doc values   0.11544 0.128914 MB 11.671864
Heap used for terms   255.983 255.269 MB -0.27892477
Heap used for norms   0 0 MB 0
Heap used for points   37.485 37.505 MB 0.053354675
Heap used for stored fields   45.9951 45.8849 MB -0.23959074
Segment count   764 780   2.0942408
Min Throughput fieldstats_elasticlogs_q-* 0.11 0.09 ops/s -18.181818
Median Throughput fieldstats_elasticlogs_q-* 0.11 0.18 ops/s 63.636364
Max Throughput fieldstats_elasticlogs_q-* 0.11 0.27 ops/s 145.45455
50th percentile latency fieldstats_elasticlogs_q-* 9412.5 10921.9 ms 16.036122
100th percentile latency fieldstats_elasticlogs_q-* 9429.81 11023.1 ms 16.896311
50th percentile service time fieldstats_elasticlogs_q-* 9412.5 10921.9 ms 16.036122
100th percentile service time fieldstats_elasticlogs_q-* 9429.81 11023.1 ms 16.896311
error rate fieldstats_elasticlogs_q-* 0 0 % 0
Min Throughput relative-kibana-content_issues-dashboard_50% 0.03 0.03 ops/s 0.
Median Throughput relative-kibana-content_issues-dashboard_50% 0.03 0.03 ops/s 0.
Max Throughput relative-kibana-content_issues-dashboard_50% 0.16 0.1 ops/s -37.5
50th percentile latency relative-kibana-content_issues-dashboard_50% 211.759 198.267 ms -6.3713939
90th percentile latency relative-kibana-content_issues-dashboard_50% 815.589 1888.08 ms 131.49895
100th percentile latency relative-kibana-content_issues-dashboard_50% 6234.4 10013.7 ms 60.620108
50th percentile service time relative-kibana-content_issues-dashboard_50% 181.838 168.396 ms -7.3922942
90th percentile service time relative-kibana-content_issues-dashboard_50% 785.457 1859.92 ms 136.79463
100th percentile service time relative-kibana-content_issues-dashboard_50% 6234.4 10013.7 ms 60.620108
error rate relative-kibana-content_issues-dashboard_50% 0 0 % 0
Min Throughput relative-kibana-content_issues-dashboard_75% 0.01 0.01 ops/s 0.
Median Throughput relative-kibana-content_issues-dashboard_75% 0.01 0.01 ops/s 0.
Max Throughput relative-kibana-content_issues-dashboard_75% 0.14 0.1 ops/s -28.571429
50th percentile latency relative-kibana-content_issues-dashboard_75% 303.836 305.723 ms 0.62105873
90th percentile latency relative-kibana-content_issues-dashboard_75% 2023.73 798.621 ms -60.537176
100th percentile latency relative-kibana-content_issues-dashboard_75% 7344.46 10007.9 ms 36.264613
50th percentile service time relative-kibana-content_issues-dashboard_75% 213.77 218.01 ms 1.9834401
90th percentile service time relative-kibana-content_issues-dashboard_75% 1933.67 708.718 ms -63.348555
100th percentile service time relative-kibana-content_issues-dashboard_75% 7344.46 10007.9 ms 36.264613
error rate relative-kibana-content_issues-dashboard_75% 0 0 % 0
Min Throughput relative-kibana-traffic-dashboard_50% 0.03 0.03 ops/s 0.
Median Throughput relative-kibana-traffic-dashboard_50% 0.03 0.03 ops/s 0.
Max Throughput relative-kibana-traffic-dashboard_50% 0.23 0.14 ops/s -39.130435
50th percentile latency relative-kibana-traffic-dashboard_50% 360.103 351.977 ms -2.2565766
90th percentile latency relative-kibana-traffic-dashboard_50% 3812.55 5342.88 ms 40.139277
100th percentile latency relative-kibana-traffic-dashboard_50% 9897.44 12223 ms 23.496581
50th percentile service time relative-kibana-traffic-dashboard_50% 320.432 312.173 ms -2.5774579
90th percentile service time relative-kibana-traffic-dashboard_50% 3773.92 5304.03 ms 40.544315
100th percentile service time relative-kibana-traffic-dashboard_50% 9861.63 12190 ms 23.610397
error rate relative-kibana-traffic-dashboard_50% 0 0 % 0
Min Throughput relative-kibana-discover_50% 0.02 0.02 ops/s 0.
Median Throughput relative-kibana-discover_50% 0.02 0.02 ops/s 0.
Max Throughput relative-kibana-discover_50% 0.22 0.13 ops/s -40.909091
50th percentile latency relative-kibana-discover_50% 465.032 440.001 ms -5.3826403
90th percentile latency relative-kibana-discover_50% 879.546 918.561 ms 4.4358112
100th percentile latency relative-kibana-discover_50% 4616.18 7702.11 ms 66.850296
50th percentile service time relative-kibana-discover_50% 405.12 380.053 ms -6.1875494
90th percentile service time relative-kibana-discover_50% 819.774 858.726 ms 4.7515535
100th percentile service time relative-kibana-discover_50% 4616.18 7702.11 ms 66.850296
error rate relative-kibana-discover_50% 0 0 % 0

pmc

Metric Task CMS G1GC Unit Change %
Total indexing time   55.1542 62.5757 min 13.455911
Min indexing time per shard   8.31752 10.3166 min 24.034568
Median indexing time per shard   9.65123 11.5227 min 19.391000
Max indexing time per shard   14.6086 15.07 min 3.1584135
Total merge time   44.4351 44.8156 min 0.85630504
Min merge time per shard   7.9564 8.1999 min 3.0604293
Median merge time per shard   9.02675 8.90982 min -1.2953721
Max merge time per shard   9.31437 9.95788 min 6.9087872
Total merge throttle time   28.8702 27.4901 min -4.7803618
Min merge throttle time per shard   4.51987 5.09343 min 12.689746
Median merge throttle time per shard   6.11615 5.51017 min -9.9078669
Max merge throttle time per shard   6.54187 6.03607 min -7.7317342
Total refresh time   7.37102 7.20438 min -2.2607455
Min refresh time per shard   1.44172 1.3669 min -5.1896346
Median refresh time per shard   1.46093 1.4654 min 0.30596949
Max refresh time per shard   1.53483 1.50385 min -2.0184646
Total flush time   3.52218 4.50015 min 27.766043
Min flush time per shard   0.5386 0.6131 min 13.832157
Median flush time per shard   0.73755 0.778383 min 5.5363026
Max flush time per shard   0.828617 1.3403 min 61.751448
Total Young Gen GC   26.109 5.464 s -79.072351
Total Old Gen GC   1.143 0 s -100.
Store size   40.9117 40.9331 GB 0.052307775
Translog size   5.34441 5.33372 GB -0.20002208
Heap used for segments   22.9091 23.1308 MB 0.96773771
Heap used for doc values   0.0668411 0.0636253 MB -4.8111117
Heap used for terms   20.9177 21.1481 MB 1.1014595
Heap used for norms   0.0421143 0.0405884 MB -3.6232349
Heap used for points   0.00927925 0.00939941 MB 1.2949322
Heap used for stored fields   1.87319 1.86913 MB -0.21674256
Segment count   138 133   -3.6231884
Min Throughput index-append 974.09 1356.54 docs/s 39.262286
Median Throughput index-append 1119.11 1478.88 docs/s 32.147868
Max Throughput index-append 1155.76 1528.32 docs/s 32.235066
50th percentile latency index-append 2305.29 2374.97 ms 3.0226132
90th percentile latency index-append 4345.72 4902.01 ms 12.800871
99th percentile latency index-append 6800.68 7012.87 ms 3.1201292
100th percentile latency index-append 9696.25 7392.95 ms -23.754544
50th percentile service time index-append 2305.29 2374.97 ms 3.0226132
90th percentile service time index-append 4345.72 4902.01 ms 12.800871
99th percentile service time index-append 6800.68 7012.87 ms 3.1201292
100th percentile service time index-append 9696.25 7392.95 ms -23.754544
error rate index-append 0 0 % 0
Min Throughput default 20.02 20.02 ops/s 0.
Median Throughput default 20.03 20.03 ops/s 0.
Max Throughput default 20.03 20.03 ops/s 0.
50th percentile latency default 9.31461 10.0342 ms 7.7253905
90th percentile latency default 10.6594 12.0661 ms 13.196803
99th percentile latency default 11.2845 12.5932 ms 11.597324
100th percentile latency default 11.6693 12.7473 ms 9.2379149
50th percentile service time default 9.19263 9.90447 ms 7.7435946
90th percentile service time default 10.5448 11.9607 ms 13.427471
99th percentile service time default 11.1222 12.4939 ms 12.332992
100th percentile service time default 11.5191 12.6518 ms 9.8332335
error rate default 0 0 % 0
Min Throughput term 20.02 20.02 ops/s 0.
Median Throughput term 20.03 20.03 ops/s 0.
Max Throughput term 20.03 20.03 ops/s 0.
50th percentile latency term 9.8853 8.8444 ms -10.529777
90th percentile latency term 10.6476 9.36376 ms -12.057553
99th percentile latency term 12.7413 10.421 ms -18.210858
100th percentile latency term 29.2749 20.235 ms -30.879354
50th percentile service time term 9.77414 8.69404 ms -11.050589
90th percentile service time term 10.5436 9.24857 ms -12.282617
99th percentile service time term 12.6598 10.2819 ms -18.783077
100th percentile service time term 29.1359 20.079 ms -31.085019
error rate term 0 0 % 0
Min Throughput phrase 20.02 20.02 ops/s 0.
Median Throughput phrase 20.03 20.03 ops/s 0.
Max Throughput phrase 20.03 20.03 ops/s 0.
50th percentile latency phrase 10.0304 9.49498 ms -5.3379726
90th percentile latency phrase 10.6479 9.99144 ms -6.1651593
99th percentile latency phrase 14.5024 10.3404 ms -28.698698
100th percentile latency phrase 18.0932 10.4212 ms -42.402671
50th percentile service time phrase 9.89359 9.37132 ms -5.2788725
90th percentile service time phrase 10.524 9.87008 ms -6.2136070
99th percentile service time phrase 14.3045 10.2058 ms -28.653221
100th percentile service time phrase 17.9279 10.3087 ms -42.499121
error rate phrase 0 0 % 0
Min Throughput articles_monthly_agg_uncached 20.02 20.02 ops/s 0.
Median Throughput articles_monthly_agg_uncached 20.03 20.03 ops/s 0.
Max Throughput articles_monthly_agg_uncached 20.03 20.03 ops/s 0.
50th percentile latency articles_monthly_agg_uncached 12.4331 11.8155 ms -4.9673854
90th percentile latency articles_monthly_agg_uncached 13.4298 13.0478 ms -2.8444206
99th percentile latency articles_monthly_agg_uncached 16.9639 14.8695 ms -12.346218
100th percentile latency articles_monthly_agg_uncached 20.095 19.2801 ms -4.0552376
50th percentile service time articles_monthly_agg_uncached 12.2865 11.6596 ms -5.1023481
90th percentile service time articles_monthly_agg_uncached 13.2836 12.8601 ms -3.1881418
99th percentile service time articles_monthly_agg_uncached 16.8036 14.7171 ms -12.416982
100th percentile service time articles_monthly_agg_uncached 19.9565 19.1226 ms -4.1785884
error rate articles_monthly_agg_uncached 0 0 % 0
Min Throughput articles_monthly_agg_cached 20.03 20.03 ops/s 0.
Median Throughput articles_monthly_agg_cached 20.03 20.03 ops/s 0.
Max Throughput articles_monthly_agg_cached 20.04 20.04 ops/s 0.
50th percentile latency articles_monthly_agg_cached 4.65779 4.6134 ms -0.95302708
90th percentile latency articles_monthly_agg_cached 5.02153 4.8619 ms -3.1789116
99th percentile latency articles_monthly_agg_cached 5.63183 5.06305 ms -10.099382
100th percentile latency articles_monthly_agg_cached 5.94603 5.13005 ms -13.723106
50th percentile service time articles_monthly_agg_cached 4.5034 4.47007 ms -0.74010747
90th percentile service time articles_monthly_agg_cached 4.87476 4.72233 ms -3.1269232
99th percentile service time articles_monthly_agg_cached 5.46725 4.90648 ms -10.256893
100th percentile service time articles_monthly_agg_cached 5.80725 4.96033 ms -14.583839
error rate articles_monthly_agg_cached 0 0 % 0
Min Throughput scroll 25 24.97 pages/s -0.12
Median Throughput scroll 25.03 25.03 pages/s 0.
Max Throughput scroll 25.07 25.07 pages/s 0.
50th percentile latency scroll 872.236 879.935 ms 0.88267396
90th percentile latency scroll 901.938 950.875 ms 5.4257610
99th percentile latency scroll 943.159 1080.29 ms 14.539542
100th percentile latency scroll 1011.53 1111.04 ms 9.8375728
50th percentile service time scroll 871.527 879.643 ms 0.93123908
90th percentile service time scroll 901.688 937.052 ms 3.9219774
99th percentile service time scroll 942.98 1048.64 ms 11.204904
100th percentile service time scroll 1011.4 1052 ms 4.0142377
error rate scroll 0 0 % 0

4gb

200ms pause

Lap Metric Task Value Unit
All Total indexing time   4144.54 min
All Min indexing time per shard   112.155 min
All Median indexing time per shard   143.5 min
All Max indexing time per shard   156.004 min
All Total merge time   1914.94 min
All Min merge time per shard   45.6065 min
All Median merge time per shard   65.2066 min
All Max merge time per shard   74.0781 min
All Total merge throttle time   987.657 min
All Min merge throttle time per shard   22.1507 min
All Median merge throttle time per shard   33.2129 min
All Max merge throttle time per shard   42.4059 min
All Total refresh time   70.3105 min
All Min refresh time per shard   2.09607 min
All Median refresh time per shard   2.37322 min
All Max refresh time per shard   2.57018 min
All Total flush time   62.7995 min
All Min flush time per shard   1.84473 min
All Median flush time per shard   2.09957 min
All Max flush time per shard   2.24273 min
All Total Young Gen GC   517.084 s
All Total Old Gen GC   0 s
All Store size   440.962 GB
All Translog size   31.6654 GB
All Heap used for segments   339.795 MB
All Heap used for doc values   0.135124 MB
All Heap used for terms   256.074 MB
All Heap used for norms   0 MB
All Heap used for points   37.7344 MB
All Heap used for stored fields   45.8513 MB
All Segment count   830  
All Min Throughput create_elasticlogs_q_write 1.15 ops/s
All Median Throughput create_elasticlogs_q_write 1.15 ops/s
All Max Throughput create_elasticlogs_q_write 1.15 ops/s
All 100th percentile latency create_elasticlogs_q_write 869.54 ms
All 100th percentile service time create_elasticlogs_q_write 869.54 ms
All error rate create_elasticlogs_q_write 0 %
All Min Throughput index-append-1000-elasticlogs_q_write 2595.86 docs/s
All Median Throughput index-append-1000-elasticlogs_q_write 52003.4 docs/s
All Max Throughput index-append-1000-elasticlogs_q_write 62545.8 docs/s
All 50th percentile latency index-append-1000-elasticlogs_q_write 271.092 ms
All 90th percentile latency index-append-1000-elasticlogs_q_write 577.162 ms
All 99th percentile latency index-append-1000-elasticlogs_q_write 1540.29 ms
All 99.9th percentile latency index-append-1000-elasticlogs_q_write 2306.42 ms
All 99.99th percentile latency index-append-1000-elasticlogs_q_write 3149.27 ms
All 100th percentile latency index-append-1000-elasticlogs_q_write 5000.57 ms
All 50th percentile service time index-append-1000-elasticlogs_q_write 271.092 ms
All 90th percentile service time index-append-1000-elasticlogs_q_write 577.162 ms
All 99th percentile service time index-append-1000-elasticlogs_q_write 1540.29 ms
All 99.9th percentile service time index-append-1000-elasticlogs_q_write 2306.42 ms
All 99.99th percentile service time index-append-1000-elasticlogs_q_write 3149.27 ms
All 100th percentile service time index-append-1000-elasticlogs_q_write 5000.57 ms
All error rate index-append-1000-elasticlogs_q_write 0 %
All error rate rollover_elasticlogs_q_write_100M 0 %
All Min Throughput node_storage 1.66 ops/s
All Median Throughput node_storage 1.66 ops/s
All Max Throughput node_storage 1.66 ops/s
All 100th percentile latency node_storage 604.184 ms
All 100th percentile service time node_storage 604.184 ms
All error rate node_storage 0 %

and querying

Lap Metric Task Value Unit
All Total flush time   0.00095 min
All Min flush time per shard   0 min
All Median flush time per shard   1.66667e-05 min
All Max flush time per shard   0.000116667 min
All Total Young Gen GC   3.267 s
All Total Old Gen GC   0 s
All Store size   418.952 GB
All Translog size   6.14673e-06 GB
All Heap used for segments   335.18 MB
All Heap used for doc values   0.122974 MB
All Heap used for terms   251.634 MB
All Heap used for norms   0 MB
All Heap used for points   37.5549 MB
All Heap used for stored fields   45.8685 MB
All Segment count   781  
All Min Throughput fieldstats_elasticlogs_q-* 0.08 ops/s
All Median Throughput fieldstats_elasticlogs_q-* 0.08 ops/s
All Max Throughput fieldstats_elasticlogs_q-* 0.08 ops/s
All 50th percentile latency fieldstats_elasticlogs_q-* 12495.2 ms
All 100th percentile latency fieldstats_elasticlogs_q-* 12530.9 ms
All 50th percentile service time fieldstats_elasticlogs_q-* 12495.2 ms
All 100th percentile service time fieldstats_elasticlogs_q-* 12530.9 ms
All error rate fieldstats_elasticlogs_q-* 0 %
All Min Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_50% 0.07 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_50% 248.881 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_50% 1872.23 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_50% 13372.3 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_50% 219.016 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_50% 1855.43 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_50% 13372.3 ms
All error rate relative-kibana-content_issues-dashboard_50% 0 %
All Min Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_75% 0.07 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_75% 393.513 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_75% 2978.27 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_75% 13776.7 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_75% 303.39 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_75% 2888.4 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_75% 13776.7 ms
All error rate relative-kibana-content_issues-dashboard_75% 0 %
All Min Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-traffic-dashboard_50% 0.07 ops/s
All 50th percentile latency relative-kibana-traffic-dashboard_50% 388.102 ms
All 90th percentile latency relative-kibana-traffic-dashboard_50% 5748.8 ms
All 100th percentile latency relative-kibana-traffic-dashboard_50% 14644.5 ms
All 50th percentile service time relative-kibana-traffic-dashboard_50% 347.843 ms
All 90th percentile service time relative-kibana-traffic-dashboard_50% 5715.12 ms
All 100th percentile service time relative-kibana-traffic-dashboard_50% 14644.5 ms
All error rate relative-kibana-traffic-dashboard_50% 0 %
All Min Throughput relative-kibana-discover_50% 0.02 ops/s
All Median Throughput relative-kibana-discover_50% 0.02 ops/s
All Max Throughput relative-kibana-discover_50% 0.07 ops/s
All 50th percentile latency relative-kibana-discover_50% 425.685 ms
All 90th percentile latency relative-kibana-discover_50% 2441.25 ms
All 100th percentile latency relative-kibana-discover_50% 13634.1 ms
All 50th percentile service time relative-kibana-discover_50% 365.42 ms
All 90th percentile service time relative-kibana-discover_50% 2383.63 ms
All 100th percentile service time relative-kibana-discover_50% 13634.1 ms
All error rate relative-kibana-discover_50% 0 %

400ms pause

Lap Metric Task Value Unit
All Total indexing time   4220.18 min
All Min indexing time per shard   109.413 min
All Median indexing time per shard   145.484 min
All Max indexing time per shard   160.64 min
All Total merge time   1964.44 min
All Min merge time per shard   45.215 min
All Median merge time per shard   67.2583 min
All Max merge time per shard   72.8815 min
All Total merge throttle time   1027.47 min
All Min merge throttle time per shard   21.6697 min
All Median merge throttle time per shard   34.8937 min
All Max merge throttle time per shard   41.6726 min
All Total refresh time   72.0875 min
All Min refresh time per shard   2.10667 min
All Median refresh time per shard   2.43722 min
All Max refresh time per shard   2.52807 min
All Total flush time   63.637 min
All Min flush time per shard   1.75023 min
All Median flush time per shard   2.1079 min
All Max flush time per shard   2.40552 min
All Total Young Gen GC   527.336 s
All Total Old Gen GC   0 s
All Store size   438.028 GB
All Translog size   32.1362 GB
All Heap used for segments   340.766 MB
All Heap used for doc values   0.159756 MB
All Heap used for terms   256.993 MB
All Heap used for norms   0 MB
All Heap used for points   37.6405 MB
All Heap used for stored fields   45.9724 MB
All Segment count   847  
All Min Throughput create_elasticlogs_q_write 1.17 ops/s
All Median Throughput create_elasticlogs_q_write 1.17 ops/s
All Max Throughput create_elasticlogs_q_write 1.17 ops/s
All 100th percentile latency create_elasticlogs_q_write 852.275 ms
All 100th percentile service time create_elasticlogs_q_write 852.275 ms
All error rate create_elasticlogs_q_write 0 %
All Min Throughput index-append-1000-elasticlogs_q_write 823.91 docs/s
All Median Throughput index-append-1000-elasticlogs_q_write 50946.6 docs/s
All Max Throughput index-append-1000-elasticlogs_q_write 62455.5 docs/s
All 50th percentile latency index-append-1000-elasticlogs_q_write 281.014 ms
All 90th percentile latency index-append-1000-elasticlogs_q_write 580.994 ms
All 99th percentile latency index-append-1000-elasticlogs_q_write 1515.58 ms
All 99.9th percentile latency index-append-1000-elasticlogs_q_write 2298.29 ms
All 99.99th percentile latency index-append-1000-elasticlogs_q_write 3090.1 ms
All 100th percentile latency index-append-1000-elasticlogs_q_write 4639.73 ms
All 50th percentile service time index-append-1000-elasticlogs_q_write 281.014 ms
All 90th percentile service time index-append-1000-elasticlogs_q_write 580.994 ms
All 99th percentile service time index-append-1000-elasticlogs_q_write 1515.58 ms
All 99.9th percentile service time index-append-1000-elasticlogs_q_write 2298.29 ms
All 99.99th percentile service time index-append-1000-elasticlogs_q_write 3090.1 ms
All 100th percentile service time index-append-1000-elasticlogs_q_write 4639.73 ms
All error rate index-append-1000-elasticlogs_q_write 0 %
All error rate rollover_elasticlogs_q_write_100M 0 %
All Min Throughput node_storage 1.36 ops/s
All Median Throughput node_storage 1.36 ops/s
All Max Throughput node_storage 1.36 ops/s
All 100th percentile latency node_storage 735.847 ms
All 100th percentile service time node_storage 735.847 ms
All error rate node_storage 0 %

Querying

Lap Metric Task Value Unit
All Total flush time   0.0004 min
All Min flush time per shard   0 min
All Median flush time per shard   1.66667e-05 min
All Max flush time per shard   5e-05 min
All Total Young Gen GC   2.834 s
All Total Old Gen GC   0 s
All Store size   418.992 GB
All Translog size   6.14673e-06 GB
All Heap used for segments   337.708 MB
All Heap used for doc values   0.163212 MB
All Heap used for terms   254.087 MB
All Heap used for norms   0 MB
All Heap used for points   37.484 MB
All Heap used for stored fields   45.9743 MB
All Segment count   799  
All Min Throughput fieldstats_elasticlogs_q-* 0.09 ops/s
All Median Throughput fieldstats_elasticlogs_q-* 0.09 ops/s
All Max Throughput fieldstats_elasticlogs_q-* 0.09 ops/s
All 50th percentile latency fieldstats_elasticlogs_q-* 11518.8 ms
All 100th percentile latency fieldstats_elasticlogs_q-* 11531 ms
All 50th percentile service time fieldstats_elasticlogs_q-* 11518.8 ms
All 100th percentile service time fieldstats_elasticlogs_q-* 11531 ms
All error rate fieldstats_elasticlogs_q-* 0 %
All Min Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_50% 0.12 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_50% 219.815 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_50% 1534.2 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_50% 8514.19 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_50% 189.977 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_50% 1504.61 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_50% 8514.19 ms
All error rate relative-kibana-content_issues-dashboard_50% 0 %
All Min Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Median Throughput relative-kibana-content_issues-dashboard_75% 0.01 ops/s
All Max Throughput relative-kibana-content_issues-dashboard_75% 0.11 ops/s
All 50th percentile latency relative-kibana-content_issues-dashboard_75% 335.127 ms
All 90th percentile latency relative-kibana-content_issues-dashboard_75% 1451.88 ms
All 100th percentile latency relative-kibana-content_issues-dashboard_75% 8883.24 ms
All 50th percentile service time relative-kibana-content_issues-dashboard_75% 245.302 ms
All 90th percentile service time relative-kibana-content_issues-dashboard_75% 1361.97 ms
All 100th percentile service time relative-kibana-content_issues-dashboard_75% 8883.24 ms
All error rate relative-kibana-content_issues-dashboard_75% 0 %
All Min Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Median Throughput relative-kibana-traffic-dashboard_50% 0.03 ops/s
All Max Throughput relative-kibana-traffic-dashboard_50% 0.09 ops/s
All 50th percentile latency relative-kibana-traffic-dashboard_50% 388.058 ms
All 90th percentile latency relative-kibana-traffic-dashboard_50% 2643.55 ms
All 100th percentile latency relative-kibana-traffic-dashboard_50% 11206.9 ms
All 50th percentile service time relative-kibana-traffic-dashboard_50% 353.499 ms
All 90th percentile service time relative-kibana-traffic-dashboard_50% 2605.89 ms
All 100th percentile service time relative-kibana-traffic-dashboard_50% 11206.9 ms
All error rate relative-kibana-traffic-dashboard_50% 0 %
All Min Throughput relative-kibana-discover_50% 0.02 ops/s
All Median Throughput relative-kibana-discover_50% 0.02 ops/s
All Max Throughput relative-kibana-discover_50% 0.11 ops/s
All 50th percentile latency relative-kibana-discover_50% 457.725 ms
All 90th percentile latency relative-kibana-discover_50% 1072.09 ms
All 100th percentile latency relative-kibana-discover_50% 9038.17 ms
All 50th percentile service time relative-kibana-discover_50% 414.972 ms
All 90th percentile service time relative-kibana-discover_50% 1020.89 ms
All 100th percentile service time relative-kibana-discover_50% 9038.17 ms
All error rate relative-kibana-discover_50% 0 %

Long Running Benchmarks

These are benchmarks run over a period of 30 days, see:

https://github.com/elastic/rally-eventdata-track/blob/2965e869dbadaac3b7e2621fc113f2d138df81d2/README.md#7-elasticlogs-continuous-index-and-query

Invoked with 

./rally --track-path=~/.rally/external_data/rally-eventdata-track/eventdata --challenge=elasticlogs-continuous-index-and-query --track-params=./longrun-track-params.json --pipeline=benchmark-only --target-hosts=lr-1:9200,lr-2:9200,lr-3:9200

With longrun-track-params.json containing the following parameters: 

{
  "number_of_replicas": 1,
  "shard_count": 3,
  "p1_bulk_indexing_clients": 32,
  "p1_bulk_size": 1000,
  "p1_duration_secs": 7200,
  "p2_bulk_indexing_clients": 12,
  "p2_bulk_size": 1000,
  "p2_ops": 100,
  "max_rolledover_indices": 20,
  "rollover_max_size": "30gb"
}

CMS

Metric Task Value Unit
Total indexing time   19183 min
Min indexing time per shard   71.6433 min
Median indexing time per shard   365.058 min
Max indexing time per shard   400.495 min
Total merge time   8766.47 min
Min merge time per shard   26.59 min
Median merge time per shard   153.384 min
Max merge time per shard   162.753 min
Total merge throttle time   5539.83 min
Min merge throttle time per shard   15.0586 min
Median merge throttle time per shard   96.2832 min
Max merge throttle time per shard   100.804 min
Total refresh time   380.535 min
Min refresh time per shard   1.7326 min
Median refresh time per shard   6.53963 min
Max refresh time per shard   7.52868 min
Total flush time   112.721 min
Min flush time per shard   0.491467 min
Median flush time per shard   1.9902 min
Max flush time per shard   2.21713 min
Total Young Gen GC   417015 s
Total Old Gen GC   982.975 s
Store size   1160.17 GB
Translog size   25.6412 GB
Heap used for segments   961.925 MB
Heap used for doc values   0.333176 MB
Heap used for terms   727.31 MB
Heap used for norms   0 MB
Heap used for points   105.531 MB
Heap used for stored fields   128.75 MB
Segment count   1766  
Min Throughput create_elasticlogs_q_write 1.53 ops/s
Median Throughput create_elasticlogs_q_write 1.53 ops/s
Max Throughput create_elasticlogs_q_write 1.53 ops/s
100th percentile latency create_elasticlogs_q_write 652.052 ms
100th percentile service time create_elasticlogs_q_write 652.052 ms
error rate create_elasticlogs_q_write 0 %
Min Throughput index-append-elasticlogs_q_write-phase1 398.72 docs/s
Median Throughput index-append-elasticlogs_q_write-phase1 51613.6 docs/s
Max Throughput index-append-elasticlogs_q_write-phase1 57033.9 docs/s
50th percentile latency index-append-elasticlogs_q_write-phase1 501.756 ms
90th percentile latency index-append-elasticlogs_q_write-phase1 909.124 ms
99th percentile latency index-append-elasticlogs_q_write-phase1 1892.76 ms
99.9th percentile latency index-append-elasticlogs_q_write-phase1 2823.93 ms
99.99th percentile latency index-append-elasticlogs_q_write-phase1 3591.97 ms
100th percentile latency index-append-elasticlogs_q_write-phase1 5234.85 ms
50th percentile service time index-append-elasticlogs_q_write-phase1 501.754 ms
90th percentile service time index-append-elasticlogs_q_write-phase1 909.115 ms
99th percentile service time index-append-elasticlogs_q_write-phase1 1892.76 ms
99.9th percentile service time index-append-elasticlogs_q_write-phase1 2823.93 ms
99.99th percentile service time index-append-elasticlogs_q_write-phase1 3591.97 ms
100th percentile service time index-append-elasticlogs_q_write-phase1 5234.85 ms
error rate index-append-elasticlogs_q_write-phase1 0 %
Min Throughput rollover-indices-phase1 0.03 ops/s
Median Throughput rollover-indices-phase1 0.03 ops/s
Max Throughput rollover-indices-phase1 37.89 ops/s
50th percentile latency rollover-indices-phase1 138.942 ms
90th percentile latency rollover-indices-phase1 215.7 ms
99th percentile latency rollover-indices-phase1 336.975 ms
100th percentile latency rollover-indices-phase1 2168.3 ms
50th percentile service time rollover-indices-phase1 110.478 ms
90th percentile service time rollover-indices-phase1 185.676 ms
99th percentile service time rollover-indices-phase1 309.026 ms
100th percentile service time rollover-indices-phase1 2138.3 ms
error rate rollover-indices-phase1 0 %
Min Throughput delete-rolledover-indices-phase1 0.03 ops/s
Median Throughput delete-rolledover-indices-phase1 0.03 ops/s
Max Throughput delete-rolledover-indices-phase1 28.86 ops/s
50th percentile latency delete-rolledover-indices-phase1 389.232 ms
90th percentile latency delete-rolledover-indices-phase1 573.175 ms
99th percentile latency delete-rolledover-indices-phase1 1605.05 ms
100th percentile latency delete-rolledover-indices-phase1 1969.22 ms
50th percentile service time delete-rolledover-indices-phase1 361.761 ms
90th percentile service time delete-rolledover-indices-phase1 546.592 ms
99th percentile service time delete-rolledover-indices-phase1 1575.76 ms
100th percentile service time delete-rolledover-indices-phase1 1939.66 ms
error rate delete-rolledover-indices-phase1 0 %
Min Throughput index-append-elasticlogs_q_write-phase2 9259.16 docs/s
Median Throughput index-append-elasticlogs_q_write-phase2 27966.8 docs/s
Max Throughput index-append-elasticlogs_q_write-phase2 29205.3 docs/s
50th percentile latency index-append-elasticlogs_q_write-phase2 8.84277e+08 ms
90th percentile latency index-append-elasticlogs_q_write-phase2 1.59907e+09 ms
99th percentile latency index-append-elasticlogs_q_write-phase2 1.79813e+09 ms
99.9th percentile latency index-append-elasticlogs_q_write-phase2 1.81828e+09 ms
99.99th percentile latency index-append-elasticlogs_q_write-phase2 1.82063e+09 ms
100th percentile latency index-append-elasticlogs_q_write-phase2 1.82232e+09 ms
50th percentile service time index-append-elasticlogs_q_write-phase2 357.113 ms
90th percentile service time index-append-elasticlogs_q_write-phase2 693.34 ms
99th percentile service time index-append-elasticlogs_q_write-phase2 1247.84 ms
99.9th percentile service time index-append-elasticlogs_q_write-phase2 1863.79 ms
99.99th percentile service time index-append-elasticlogs_q_write-phase2 2562.98 ms
100th percentile service time index-append-elasticlogs_q_write-phase2 8649.53 ms
error rate index-append-elasticlogs_q_write-phase2 0 %
Min Throughput rollover-indices-phase2 0.03 ops/s
Median Throughput rollover-indices-phase2 0.03 ops/s
Max Throughput rollover-indices-phase2 9.71 ops/s
50th percentile latency rollover-indices-phase2 128.304 ms
90th percentile latency rollover-indices-phase2 209.183 ms
99th percentile latency rollover-indices-phase2 330.768 ms
99.9th percentile latency rollover-indices-phase2 724.107 ms
99.99th percentile latency rollover-indices-phase2 1578.88 ms
100th percentile latency rollover-indices-phase2 2109.85 ms
50th percentile service time rollover-indices-phase2 100.047 ms
90th percentile service time rollover-indices-phase2 180.673 ms
99th percentile service time rollover-indices-phase2 302.939 ms
99.9th percentile service time rollover-indices-phase2 694.107 ms
99.99th percentile service time rollover-indices-phase2 1550.5 ms
100th percentile service time rollover-indices-phase2 2108.8 ms
error rate rollover-indices-phase2 0 %
Min Throughput delete_rolled_over_indices-phase2 0.03 ops/s
Median Throughput delete_rolled_over_indices-phase2 0.03 ops/s
Max Throughput delete_rolled_over_indices-phase2 10.67 ops/s
50th percentile latency delete_rolled_over_indices-phase2 2055.94 ms
90th percentile latency delete_rolled_over_indices-phase2 3425.62 ms
99th percentile latency delete_rolled_over_indices-phase2 4688.44 ms
99.9th percentile latency delete_rolled_over_indices-phase2 5623.23 ms
99.99th percentile latency delete_rolled_over_indices-phase2 6886.52 ms
100th percentile latency delete_rolled_over_indices-phase2 7732.13 ms
50th percentile service time delete_rolled_over_indices-phase2 2029.3 ms
90th percentile service time delete_rolled_over_indices-phase2 3398.93 ms
99th percentile service time delete_rolled_over_indices-phase2 4662.08 ms
99.9th percentile service time delete_rolled_over_indices-phase2 5595.12 ms
99.99th percentile service time delete_rolled_over_indices-phase2 6858.84 ms
100th percentile service time delete_rolled_over_indices-phase2 7703.71 ms
error rate delete_rolled_over_indices-phase2 0 %
Min Throughput current-kibana-traffic-country-dashboard_60m-querying 0.07 ops/s
Median Throughput current-kibana-traffic-country-dashboard_60m-querying 0.1 ops/s
Max Throughput current-kibana-traffic-country-dashboard_60m-querying 0.19 ops/s
50th percentile latency current-kibana-traffic-country-dashboard_60m-querying 453.377 ms
90th percentile latency current-kibana-traffic-country-dashboard_60m-querying 1245.66 ms
99th percentile latency current-kibana-traffic-country-dashboard_60m-querying 4650.33 ms
99.9th percentile latency current-kibana-traffic-country-dashboard_60m-querying 8131.33 ms
99.99th percentile latency current-kibana-traffic-country-dashboard_60m-querying 11458.3 ms
100th percentile latency current-kibana-traffic-country-dashboard_60m-querying 15536 ms
50th percentile service time current-kibana-traffic-country-dashboard_60m-querying 430.89 ms
90th percentile service time current-kibana-traffic-country-dashboard_60m-querying 1066.41 ms
99th percentile service time current-kibana-traffic-country-dashboard_60m-querying 4262.78 ms
99.9th percentile service time current-kibana-traffic-country-dashboard_60m-querying 7517.92 ms
99.99th percentile service time current-kibana-traffic-country-dashboard_60m-querying 10583.9 ms
100th percentile service time current-kibana-traffic-country-dashboard_60m-querying 15029.3 ms
error rate current-kibana-traffic-country-dashboard_60m-querying 0 %
Min Throughput current-kibana-discover_30m-querying 0.08 ops/s
Median Throughput current-kibana-discover_30m-querying 0.1 ops/s
Max Throughput current-kibana-discover_30m-querying 0.1 ops/s
50th percentile latency current-kibana-discover_30m-querying 12055.1 ms
90th percentile latency current-kibana-discover_30m-querying 35708.3 ms
99th percentile latency current-kibana-discover_30m-querying 78523.7 ms
99.9th percentile latency current-kibana-discover_30m-querying 123237 ms
99.99th percentile latency current-kibana-discover_30m-querying 144474 ms
100th percentile latency current-kibana-discover_30m-querying 165604 ms
50th percentile service time current-kibana-discover_30m-querying 6492.36 ms
90th percentile service time current-kibana-discover_30m-querying 9728.82 ms
99th percentile service time current-kibana-discover_30m-querying 11446.8 ms
99.9th percentile service time current-kibana-discover_30m-querying 13370.3 ms
99.99th percentile service time current-kibana-discover_30m-querying 16442.2 ms
100th percentile service time current-kibana-discover_30m-querying 23343.6 ms
error rate current-kibana-discover_30m-querying 0 %
Min Throughput current-kibana-traffic-dashboard_30m-querying 0.06 ops/s
Median Throughput current-kibana-traffic-dashboard_30m-querying 0.1 ops/s
Max Throughput current-kibana-traffic-dashboard_30m-querying 0.1 ops/s
50th percentile latency current-kibana-traffic-dashboard_30m-querying 12090.1 ms
90th percentile latency current-kibana-traffic-dashboard_30m-querying 35715.8 ms
99th percentile latency current-kibana-traffic-dashboard_30m-querying 79355.4 ms
99.9th percentile latency current-kibana-traffic-dashboard_30m-querying 129831 ms
99.99th percentile latency current-kibana-traffic-dashboard_30m-querying 162978 ms
100th percentile latency current-kibana-traffic-dashboard_30m-querying 178505 ms
50th percentile service time current-kibana-traffic-dashboard_30m-querying 6490.4 ms
90th percentile service time current-kibana-traffic-dashboard_30m-querying 9734.3 ms
99th percentile service time current-kibana-traffic-dashboard_30m-querying 11448.6 ms
99.9th percentile service time current-kibana-traffic-dashboard_30m-querying 13418.2 ms
99.99th percentile service time current-kibana-traffic-dashboard_30m-querying 16198 ms
100th percentile service time current-kibana-traffic-dashboard_30m-querying 24632.9 ms
error rate current-kibana-traffic-dashboard_30m-querying 0 %
Min Throughput current-kibana-content_issues-dashboard_30m-querying 0.1 ops/s
Median Throughput current-kibana-content_issues-dashboard_30m-querying 0.1 ops/s
Max Throughput current-kibana-content_issues-dashboard_30m-querying 0.17 ops/s
50th percentile latency current-kibana-content_issues-dashboard_30m-querying 3547.31 ms
90th percentile latency current-kibana-content_issues-dashboard_30m-querying 8991.45 ms
99th percentile latency current-kibana-content_issues-dashboard_30m-querying 18752.5 ms
99.9th percentile latency current-kibana-content_issues-dashboard_30m-querying 30138.4 ms
99.99th percentile latency current-kibana-content_issues-dashboard_30m-querying 44543.8 ms
100th percentile latency current-kibana-content_issues-dashboard_30m-querying 85341.7 ms
50th percentile service time current-kibana-content_issues-dashboard_30m-querying 3103.14 ms
90th percentile service time current-kibana-content_issues-dashboard_30m-querying 5768.09 ms
99th percentile service time current-kibana-content_issues-dashboard_30m-querying 8859.5 ms
99.9th percentile service time current-kibana-content_issues-dashboard_30m-querying 10771.5 ms
99.99th percentile service time current-kibana-content_issues-dashboard_30m-querying 13463.2 ms
100th percentile service time current-kibana-content_issues-dashboard_30m-querying 17213.1 ms
error rate current-kibana-content_issues-dashboard_30m-querying 0 %

G1GC

Metric Task Value Unit
Total indexing time   13763.1 min
Min indexing time per shard   68.1738 min
Median indexing time per shard   249.735 min
Max indexing time per shard   288.699 min
Total merge time   7791.37 min
Min merge time per shard   31.4615 min
Median merge time per shard   135.078 min
Max merge time per shard   143.662 min
Total merge throttle time   4621.52 min
Min merge throttle time per shard   16.9749 min
Median merge throttle time per shard   80.0773 min
Max merge throttle time per shard   86.1202 min
Total refresh time   329.922 min
Min refresh time per shard   2.00225 min
Median refresh time per shard   5.7852 min
Max refresh time per shard   6.19802 min
Total flush time   131.45 min
Min flush time per shard   0.769683 min
Median flush time per shard   2.22353 min
Max flush time per shard   2.61475 min
Total Young Gen GC   10449.9 s
Total Old Gen GC   0 s
Store size   1166.8 GB
Translog size   32.1166 GB
Heap used for segments   980.643 MB
Heap used for doc values   0.354321 MB
Heap used for terms   745.778 MB
Heap used for norms   0 MB
Heap used for points   105.496 MB
Heap used for stored fields   129.015 MB
Segment count   1753  
Min Throughput create_elasticlogs_q_write 1.22 ops/s
Median Throughput create_elasticlogs_q_write 1.22 ops/s
Max Throughput create_elasticlogs_q_write 1.22 ops/s
100th percentile latency create_elasticlogs_q_write 822.537 ms
100th percentile service time create_elasticlogs_q_write 822.537 ms
error rate create_elasticlogs_q_write 0 %
Min Throughput index-append-elasticlogs_q_write-phase1 577.49 docs/s
Median Throughput index-append-elasticlogs_q_write-phase1 49978.3 docs/s
Max Throughput index-append-elasticlogs_q_write-phase1 52737.6 docs/s
50th percentile latency index-append-elasticlogs_q_write-phase1 480.633 ms
90th percentile latency index-append-elasticlogs_q_write-phase1 917.382 ms
99th percentile latency index-append-elasticlogs_q_write-phase1 1992.09 ms
99.9th percentile latency index-append-elasticlogs_q_write-phase1 3051.8 ms
99.99th percentile latency index-append-elasticlogs_q_write-phase1 4205.62 ms
100th percentile latency index-append-elasticlogs_q_write-phase1 6185.2 ms
50th percentile service time index-append-elasticlogs_q_write-phase1 480.643 ms
90th percentile service time index-append-elasticlogs_q_write-phase1 917.056 ms
99th percentile service time index-append-elasticlogs_q_write-phase1 1991.66 ms
99.9th percentile service time index-append-elasticlogs_q_write-phase1 3053.84 ms
99.99th percentile service time index-append-elasticlogs_q_write-phase1 4205.62 ms
100th percentile service time index-append-elasticlogs_q_write-phase1 6185.2 ms
error rate index-append-elasticlogs_q_write-phase1 0 %
Min Throughput rollover-indices-phase1 0.03 ops/s
Median Throughput rollover-indices-phase1 0.03 ops/s
Max Throughput rollover-indices-phase1 25.14 ops/s
50th percentile latency rollover-indices-phase1 94.3423 ms
90th percentile latency rollover-indices-phase1 122.251 ms
99th percentile latency rollover-indices-phase1 208.795 ms
100th percentile latency rollover-indices-phase1 1273.58 ms
50th percentile service time rollover-indices-phase1 65.7829 ms
90th percentile service time rollover-indices-phase1 93.6281 ms
99th percentile service time rollover-indices-phase1 178.813 ms
100th percentile service time rollover-indices-phase1 1243.57 ms
error rate rollover-indices-phase1 0 %
Min Throughput delete-rolledover-indices-phase1 0.03 ops/s
Median Throughput delete-rolledover-indices-phase1 0.03 ops/s
Max Throughput delete-rolledover-indices-phase1 16.73 ops/s
50th percentile latency delete-rolledover-indices-phase1 298.796 ms
90th percentile latency delete-rolledover-indices-phase1 444.972 ms
99th percentile latency delete-rolledover-indices-phase1 1016.28 ms
100th percentile latency delete-rolledover-indices-phase1 1726.89 ms
50th percentile service time delete-rolledover-indices-phase1 269.823 ms
90th percentile service time delete-rolledover-indices-phase1 415.723 ms
99th percentile service time delete-rolledover-indices-phase1 986.417 ms
100th percentile service time delete-rolledover-indices-phase1 1697.04 ms
error rate delete-rolledover-indices-phase1 0 %
Min Throughput index-append-elasticlogs_q_write-phase2 6090.03 docs/s
Median Throughput index-append-elasticlogs_q_write-phase2 30059 docs/s
Max Throughput index-append-elasticlogs_q_write-phase2 31284.8 docs/s
50th percentile latency index-append-elasticlogs_q_write-phase2 8.79076e+08 ms
90th percentile latency index-append-elasticlogs_q_write-phase2 1.57783e+09 ms
99th percentile latency index-append-elasticlogs_q_write-phase2 1.73483e+09 ms
99.9th percentile latency index-append-elasticlogs_q_write-phase2 1.75051e+09 ms
99.99th percentile latency index-append-elasticlogs_q_write-phase2 1.75225e+09 ms
100th percentile latency index-append-elasticlogs_q_write-phase2 1.75279e+09 ms
50th percentile service time index-append-elasticlogs_q_write-phase2 323.283 ms
90th percentile service time index-append-elasticlogs_q_write-phase2 636.159 ms
99th percentile service time index-append-elasticlogs_q_write-phase2 1238.91 ms
99.9th percentile service time index-append-elasticlogs_q_write-phase2 1891.77 ms
99.99th percentile service time index-append-elasticlogs_q_write-phase2 2629.17 ms
100th percentile service time index-append-elasticlogs_q_write-phase2 26975.1 ms
error rate index-append-elasticlogs_q_write-phase2 0 %
Min Throughput rollover-indices-phase2 0.03 ops/s
Median Throughput rollover-indices-phase2 0.03 ops/s
Max Throughput rollover-indices-phase2 21.62 ops/s
50th percentile latency rollover-indices-phase2 88.3668 ms
90th percentile latency rollover-indices-phase2 149.689 ms
99th percentile latency rollover-indices-phase2 244.085 ms
99.9th percentile latency rollover-indices-phase2 720.305 ms
99.99th percentile latency rollover-indices-phase2 1549.65 ms
100th percentile latency rollover-indices-phase2 2202.52 ms
50th percentile service time rollover-indices-phase2 58.7102 ms
90th percentile service time rollover-indices-phase2 120.066 ms
99th percentile service time rollover-indices-phase2 214.381 ms
99.9th percentile service time rollover-indices-phase2 689.79 ms
99.99th percentile service time rollover-indices-phase2 1519.58 ms
100th percentile service time rollover-indices-phase2 2172.28 ms
error rate rollover-indices-phase2 0 %
Min Throughput delete_rolled_over_indices-phase2 0.03 ops/s
Median Throughput delete_rolled_over_indices-phase2 0.03 ops/s
Max Throughput delete_rolled_over_indices-phase2 19.56 ops/s
50th percentile latency delete_rolled_over_indices-phase2 1770.59 ms
90th percentile latency delete_rolled_over_indices-phase2 3080.14 ms
99th percentile latency delete_rolled_over_indices-phase2 4298.78 ms
99.9th percentile latency delete_rolled_over_indices-phase2 5724.13 ms
99.99th percentile latency delete_rolled_over_indices-phase2 7984.33 ms
100th percentile latency delete_rolled_over_indices-phase2 11938.9 ms
50th percentile service time delete_rolled_over_indices-phase2 1743.51 ms
90th percentile service time delete_rolled_over_indices-phase2 3053.67 ms
99th percentile service time delete_rolled_over_indices-phase2 4270.9 ms
99.9th percentile service time delete_rolled_over_indices-phase2 5695.62 ms
99.99th percentile service time delete_rolled_over_indices-phase2 7957.38 ms
100th percentile service time delete_rolled_over_indices-phase2 11919.2 ms
error rate delete_rolled_over_indices-phase2 0 %
Min Throughput current-kibana-traffic-country-dashboard_60m-querying 0.08 ops/s
Median Throughput current-kibana-traffic-country-dashboard_60m-querying 0.1 ops/s
Max Throughput current-kibana-traffic-country-dashboard_60m-querying 0.44 ops/s
50th percentile latency current-kibana-traffic-country-dashboard_60m-querying 419.02 ms
90th percentile latency current-kibana-traffic-country-dashboard_60m-querying 1259.92 ms
99th percentile latency current-kibana-traffic-country-dashboard_60m-querying 4761.02 ms
99.9th percentile latency current-kibana-traffic-country-dashboard_60m-querying 8253.42 ms
99.99th percentile latency current-kibana-traffic-country-dashboard_60m-querying 10984.9 ms
100th percentile latency current-kibana-traffic-country-dashboard_60m-querying 17071.4 ms
50th percentile service time current-kibana-traffic-country-dashboard_60m-querying 396.009 ms
90th percentile service time current-kibana-traffic-country-dashboard_60m-querying 1081.59 ms
99th percentile service time current-kibana-traffic-country-dashboard_60m-querying 4340.72 ms
99.9th percentile service time current-kibana-traffic-country-dashboard_60m-querying 7711.39 ms
99.99th percentile service time current-kibana-traffic-country-dashboard_60m-querying 10360.1 ms
100th percentile service time current-kibana-traffic-country-dashboard_60m-querying 17069.8 ms
error rate current-kibana-traffic-country-dashboard_60m-querying 0 %
Min Throughput current-kibana-discover_30m-querying 0.07 ops/s
Median Throughput current-kibana-discover_30m-querying 0.1 ops/s
Max Throughput current-kibana-discover_30m-querying 0.1 ops/s
50th percentile latency current-kibana-discover_30m-querying 12657.8 ms
90th percentile latency current-kibana-discover_30m-querying 36617.3 ms
99th percentile latency current-kibana-discover_30m-querying 79959.4 ms
99.9th percentile latency current-kibana-discover_30m-querying 117979 ms
99.99th percentile latency current-kibana-discover_30m-querying 167513 ms
100th percentile latency current-kibana-discover_30m-querying 195818 ms
50th percentile service time current-kibana-discover_30m-querying 6582.26 ms
90th percentile service time current-kibana-discover_30m-querying 9737.67 ms
99th percentile service time current-kibana-discover_30m-querying 11421.1 ms
99.9th percentile service time current-kibana-discover_30m-querying 15397.7 ms
99.99th percentile service time current-kibana-discover_30m-querying 19357.9 ms
100th percentile service time current-kibana-discover_30m-querying 28185.7 ms
error rate current-kibana-discover_30m-querying 0 %
Min Throughput current-kibana-traffic-dashboard_30m-querying 0.07 ops/s
Median Throughput current-kibana-traffic-dashboard_30m-querying 0.1 ops/s
Max Throughput current-kibana-traffic-dashboard_30m-querying 0.11 ops/s
50th percentile latency current-kibana-traffic-dashboard_30m-querying 12681.5 ms
90th percentile latency current-kibana-traffic-dashboard_30m-querying 36753.8 ms
99th percentile latency current-kibana-traffic-dashboard_30m-querying 77536.2 ms
99.9th percentile latency current-kibana-traffic-dashboard_30m-querying 119055 ms
99.99th percentile latency current-kibana-traffic-dashboard_30m-querying 155367 ms
100th percentile latency current-kibana-traffic-dashboard_30m-querying 184506 ms
50th percentile service time current-kibana-traffic-dashboard_30m-querying 6579.2 ms
90th percentile service time current-kibana-traffic-dashboard_30m-querying 9735.43 ms
99th percentile service time current-kibana-traffic-dashboard_30m-querying 11398.4 ms
99.9th percentile service time current-kibana-traffic-dashboard_30m-querying 15134.2 ms
99.99th percentile service time current-kibana-traffic-dashboard_30m-querying 19513.4 ms
100th percentile service time current-kibana-traffic-dashboard_30m-querying 29241 ms
error rate current-kibana-traffic-dashboard_30m-querying 0 %
Min Throughput current-kibana-content_issues-dashboard_30m-querying 0.08 ops/s
Median Throughput current-kibana-content_issues-dashboard_30m-querying 0.1 ops/s
Max Throughput current-kibana-content_issues-dashboard_30m-querying 0.17 ops/s
50th percentile latency current-kibana-content_issues-dashboard_30m-querying 3236.13 ms
90th percentile latency current-kibana-content_issues-dashboard_30m-querying 8005.54 ms
99th percentile latency current-kibana-content_issues-dashboard_30m-querying 16026.2 ms
99.9th percentile latency current-kibana-content_issues-dashboard_30m-querying 24896 ms
99.99th percentile latency current-kibana-content_issues-dashboard_30m-querying 40832.9 ms
100th percentile latency current-kibana-content_issues-dashboard_30m-querying 69508.8 ms
50th percentile service time current-kibana-content_issues-dashboard_30m-querying 2859.44 ms
90th percentile service time current-kibana-content_issues-dashboard_30m-querying 5394.37 ms
99th percentile service time current-kibana-content_issues-dashboard_30m-querying 8426.07 ms
99.9th percentile service time current-kibana-content_issues-dashboard_30m-querying 10641.5 ms
99.99th percentile service time current-kibana-content_issues-dashboard_30m-querying 14927.1 ms
100th percentile service time current-kibana-content_issues-dashboard_30m-querying 26776.8 ms
error rate current-kibana-content_issues-dashboard_30m-querying 0 %

Comparison

Metric Task CMS G1GC Unit Change %
Total indexing time   19183 13763.1 min -28.253662
Min indexing time per shard   71.6433 68.1738 min -4.8427417
Median indexing time per shard   365.058 249.735 min -31.590323
Max indexing time per shard   400.495 288.699 min -27.914456
Total merge time   8766.47 7791.37 min -11.123063
Min merge time per shard   26.59 31.4615 min 18.320797
Median merge time per shard   153.384 135.078 min -11.934752
Max merge time per shard   162.753 143.662 min -11.730045
Total merge throttle time   5539.83 4621.52 min -16.576501
Min merge throttle time per shard   15.0586 16.9749 min 12.725619
Median merge throttle time per shard   96.2832 80.0773 min -16.831493
Max merge throttle time per shard   100.804 86.1202 min -14.566684
Total refresh time   380.535 329.922 min -13.300485
Min refresh time per shard   1.7326 2.00225 min 15.563315
Median refresh time per shard   6.53963 5.7852 min -11.536280
Max refresh time per shard   7.52868 6.19802 min -17.674546
Total flush time   112.721 131.45 min 16.615360
Min flush time per shard   0.491467 0.769683 min 56.609294
Median flush time per shard   1.9902 2.22353 min 11.723947
Max flush time per shard   2.21713 2.61475 min 17.933996
Total Young Gen GC   417015 10449.9 s -97.494119
Total Old Gen GC   982.975 0 s -100.
Store size   1160.17 1166.8 GB 0.57146798
Translog size   25.6412 32.1166 GB 25.253888
Heap used for segments   961.925 980.643 MB 1.9458898
Heap used for doc values   0.333176 0.354321 MB 6.3464955
Heap used for terms   727.31 745.778 MB 2.5392199
Heap used for norms   0 0 MB 0
Heap used for points   105.531 105.496 MB -0.033165610
Heap used for stored fields   128.75 129.015 MB 0.20582524
Segment count   1766 1753   -0.73612684
Min Throughput create_elasticlogs_q_write 1.53 1.22 ops/s -20.261438
Median Throughput create_elasticlogs_q_write 1.53 1.22 ops/s -20.261438
Max Throughput create_elasticlogs_q_write 1.53 1.22 ops/s -20.261438
100th percentile latency create_elasticlogs_q_write 652.052 822.537 ms 26.145921
100th percentile service time create_elasticlogs_q_write 652.052 822.537 ms 26.145921
error rate create_elasticlogs_q_write 0 0 % 0
Min Throughput index-append-elasticlogs_q_write-phase1 398.72 577.49 docs/s 44.835975
Median Throughput index-append-elasticlogs_q_write-phase1 51613.6 49978.3 docs/s -3.1683510
Max Throughput index-append-elasticlogs_q_write-phase1 57033.9 52737.6 docs/s -7.5328883
50th percentile latency index-append-elasticlogs_q_write-phase1 501.756 480.633 ms -4.2098151
90th percentile latency index-append-elasticlogs_q_write-phase1 909.124 917.382 ms 0.90834694
99th percentile latency index-append-elasticlogs_q_write-phase1 1892.76 1992.09 ms 5.2478920
99.9th percentile latency index-append-elasticlogs_q_write-phase1 2823.93 3051.8 ms 8.0692510
99.99th percentile latency index-append-elasticlogs_q_write-phase1 3591.97 4205.62 ms 17.083940
100th percentile latency index-append-elasticlogs_q_write-phase1 5234.85 6185.2 ms 18.154293
50th percentile service time index-append-elasticlogs_q_write-phase1 501.754 480.643 ms -4.2074403
90th percentile service time index-append-elasticlogs_q_write-phase1 909.115 917.056 ms 0.87348685
99th percentile service time index-append-elasticlogs_q_write-phase1 1892.76 1991.66 ms 5.2251738
99.9th percentile service time index-append-elasticlogs_q_write-phase1 2823.93 3053.84 ms 8.1414908
99.99th percentile service time index-append-elasticlogs_q_write-phase1 3591.97 4205.62 ms 17.083940
100th percentile service time index-append-elasticlogs_q_write-phase1 5234.85 6185.2 ms 18.154293
error rate index-append-elasticlogs_q_write-phase1 0 0 % 0
Min Throughput rollover-indices-phase1 0.03 0.03 ops/s 0.
Median Throughput rollover-indices-phase1 0.03 0.03 ops/s 0.
Max Throughput rollover-indices-phase1 37.89 25.14 ops/s -33.650040
50th percentile latency rollover-indices-phase1 138.942 94.3423 ms -32.099509
90th percentile latency rollover-indices-phase1 215.7 122.251 ms -43.323598
99th percentile latency rollover-indices-phase1 336.975 208.795 ms -38.038430
100th percentile latency rollover-indices-phase1 2168.3 1273.58 ms -41.263663
50th percentile service time rollover-indices-phase1 110.478 65.7829 ms -40.456109
90th percentile service time rollover-indices-phase1 185.676 93.6281 ms -49.574474
99th percentile service time rollover-indices-phase1 309.026 178.813 ms -42.136584
100th percentile service time rollover-indices-phase1 2138.3 1243.57 ms -41.843053
error rate rollover-indices-phase1 0 0 % 0
Min Throughput delete-rolledover-indices-phase1 0.03 0.03 ops/s 0.
Median Throughput delete-rolledover-indices-phase1 0.03 0.03 ops/s 0.
Max Throughput delete-rolledover-indices-phase1 28.86 16.73 ops/s -42.030492
50th percentile latency delete-rolledover-indices-phase1 389.232 298.796 ms -23.234472
90th percentile latency delete-rolledover-indices-phase1 573.175 444.972 ms -22.367165
99th percentile latency delete-rolledover-indices-phase1 1605.05 1016.28 ms -36.682346
100th percentile latency delete-rolledover-indices-phase1 1969.22 1726.89 ms -12.305888
50th percentile service time delete-rolledover-indices-phase1 361.761 269.823 ms -25.414016
90th percentile service time delete-rolledover-indices-phase1 546.592 415.723 ms -23.942721
99th percentile service time delete-rolledover-indices-phase1 1575.76 986.417 ms -37.400556
100th percentile service time delete-rolledover-indices-phase1 1939.66 1697.04 ms -12.508378
error rate delete-rolledover-indices-phase1 0 0 % 0
Min Throughput index-append-elasticlogs_q_write-phase2 9259.16 6090.03 docs/s -34.226971
Median Throughput index-append-elasticlogs_q_write-phase2 27966.8 30059 docs/s 7.4810132
Max Throughput index-append-elasticlogs_q_write-phase2 29205.3 31284.8 docs/s 7.1202830
50th percentile latency index-append-elasticlogs_q_write-phase2 8.84277e+08 8.79076e+08 ms -0.58816412
90th percentile latency index-append-elasticlogs_q_write-phase2 1.59907e+09 1.57783e+09 ms -1.3282721
99th percentile latency index-append-elasticlogs_q_write-phase2 1.79813e+09 1.73483e+09 ms -3.5203239
99.9th percentile latency index-append-elasticlogs_q_write-phase2 1.81828e+09 1.75051e+09 ms -3.7271487
99.99th percentile latency index-append-elasticlogs_q_write-phase2 1.82063e+09 1.75225e+09 ms -3.7558428
100th percentile latency index-append-elasticlogs_q_write-phase2 1.82232e+09 1.75279e+09 ms -3.8154660
50th percentile service time index-append-elasticlogs_q_write-phase2 357.113 323.283 ms -9.4731920
90th percentile service time index-append-elasticlogs_q_write-phase2 693.34 636.159 ms -8.2471803
99th percentile service time index-append-elasticlogs_q_write-phase2 1247.84 1238.91 ms -0.71563662
99.9th percentile service time index-append-elasticlogs_q_write-phase2 1863.79 1891.77 ms 1.5012421
99.99th percentile service time index-append-elasticlogs_q_write-phase2 2562.98 2629.17 ms 2.5825406
100th percentile service time index-append-elasticlogs_q_write-phase2 8649.53 26975.1 ms 211.86781
error rate index-append-elasticlogs_q_write-phase2 0 0 % 0
Min Throughput rollover-indices-phase2 0.03 0.03 ops/s 0.
Median Throughput rollover-indices-phase2 0.03 0.03 ops/s 0.
Max Throughput rollover-indices-phase2 9.71 21.62 ops/s 122.65705
50th percentile latency rollover-indices-phase2 128.304 88.3668 ms -31.127011
90th percentile latency rollover-indices-phase2 209.183 149.689 ms -28.441126
99th percentile latency rollover-indices-phase2 330.768 244.085 ms -26.206586
99.9th percentile latency rollover-indices-phase2 724.107 720.305 ms -0.52506052
99.99th percentile latency rollover-indices-phase2 1578.88 1549.65 ms -1.8513123
100th percentile latency rollover-indices-phase2 2109.85 2202.52 ms 4.3922554
50th percentile service time rollover-indices-phase2 100.047 58.7102 ms -41.317381
90th percentile service time rollover-indices-phase2 180.673 120.066 ms -33.545134
99th percentile service time rollover-indices-phase2 302.939 214.381 ms -29.232948
99.9th percentile service time rollover-indices-phase2 694.107 689.79 ms -0.62195022
99.99th percentile service time rollover-indices-phase2 1550.5 1519.58 ms -1.9941954
100th percentile service time rollover-indices-phase2 2108.8 2172.28 ms 3.0102428
error rate rollover-indices-phase2 0 0 % 0
Min Throughput delete_rolled_over_indices-phase2 0.03 0.03 ops/s 0.
Median Throughput delete_rolled_over_indices-phase2 0.03 0.03 ops/s 0.
Max Throughput delete_rolled_over_indices-phase2 10.67 19.56 ops/s 83.317713
50th percentile latency delete_rolled_over_indices-phase2 2055.94 1770.59 ms -13.879296
90th percentile latency delete_rolled_over_indices-phase2 3425.62 3080.14 ms -10.085182
99th percentile latency delete_rolled_over_indices-phase2 4688.44 4298.78 ms -8.3110800
99.9th percentile latency delete_rolled_over_indices-phase2 5623.23 5724.13 ms 1.7943424
99.99th percentile latency delete_rolled_over_indices-phase2 6886.52 7984.33 ms 15.941433
100th percentile latency delete_rolled_over_indices-phase2 7732.13 11938.9 ms 54.406354
50th percentile service time delete_rolled_over_indices-phase2 2029.3 1743.51 ms -14.083181
90th percentile service time delete_rolled_over_indices-phase2 3398.93 3053.67 ms -10.157903
99th percentile service time delete_rolled_over_indices-phase2 4662.08 4270.9 ms -8.3906754
99.9th percentile service time delete_rolled_over_indices-phase2 5595.12 5695.62 ms 1.7962081
99.99th percentile service time delete_rolled_over_indices-phase2 6858.84 7957.38 ms 16.016411
100th percentile service time delete_rolled_over_indices-phase2 7703.71 11919.2 ms 54.720258
error rate delete_rolled_over_indices-phase2 0 0 % 0
Min Throughput current-kibana-traffic-country-dashboard_60m-querying 0.07 0.08 ops/s 14.285714
Median Throughput current-kibana-traffic-country-dashboard_60m-querying 0.1 0.1 ops/s 0.
Max Throughput current-kibana-traffic-country-dashboard_60m-querying 0.19 0.44 ops/s 131.57895
50th percentile latency current-kibana-traffic-country-dashboard_60m-querying 453.377 419.02 ms -7.5780201
90th percentile latency current-kibana-traffic-country-dashboard_60m-querying 1245.66 1259.92 ms 1.1447747
99th percentile latency current-kibana-traffic-country-dashboard_60m-querying 4650.33 4761.02 ms 2.3802612
99.9th percentile latency current-kibana-traffic-country-dashboard_60m-querying 8131.33 8253.42 ms 1.5014764
99.99th percentile latency current-kibana-traffic-country-dashboard_60m-querying 11458.3 10984.9 ms -4.1315029
100th percentile latency current-kibana-traffic-country-dashboard_60m-querying 15536 17071.4 ms 9.8828527
50th percentile service time current-kibana-traffic-country-dashboard_60m-querying 430.89 396.009 ms -8.0951055
90th percentile service time current-kibana-traffic-country-dashboard_60m-querying 1066.41 1081.59 ms 1.4234675
99th percentile service time current-kibana-traffic-country-dashboard_60m-querying 4262.78 4340.72 ms 1.8283843
99.9th percentile service time current-kibana-traffic-country-dashboard_60m-querying 7517.92 7711.39 ms 2.5734512
99.99th percentile service time current-kibana-traffic-country-dashboard_60m-querying 10583.9 10360.1 ms -2.1145325
100th percentile service time current-kibana-traffic-country-dashboard_60m-querying 15029.3 17069.8 ms 13.576813
error rate current-kibana-traffic-country-dashboard_60m-querying 0 0 % 0
Min Throughput current-kibana-discover_30m-querying 0.08 0.07 ops/s -12.5
Median Throughput current-kibana-discover_30m-querying 0.1 0.1 ops/s 0.
Max Throughput current-kibana-discover_30m-querying 0.1 0.1 ops/s 0.
50th percentile latency current-kibana-discover_30m-querying 12055.1 12657.8 ms 4.9995438
90th percentile latency current-kibana-discover_30m-querying 35708.3 36617.3 ms 2.5456266
99th percentile latency current-kibana-discover_30m-querying 78523.7 79959.4 ms 1.8283652
99.9th percentile latency current-kibana-discover_30m-querying 123237 117979 ms -4.2665758
99.99th percentile latency current-kibana-discover_30m-querying 144474 167513 ms 15.946814
100th percentile latency current-kibana-discover_30m-querying 165604 195818 ms 18.244728
50th percentile service time current-kibana-discover_30m-querying 6492.36 6582.26 ms 1.3847045
90th percentile service time current-kibana-discover_30m-querying 9728.82 9737.67 ms 0.090966839
99th percentile service time current-kibana-discover_30m-querying 11446.8 11421.1 ms -0.22451690
99.9th percentile service time current-kibana-discover_30m-querying 13370.3 15397.7 ms 15.163459
99.99th percentile service time current-kibana-discover_30m-querying 16442.2 19357.9 ms 17.733028
100th percentile service time current-kibana-discover_30m-querying 23343.6 28185.7 ms 20.742730
error rate current-kibana-discover_30m-querying 0 0 % 0
Min Throughput current-kibana-traffic-dashboard_30m-querying 0.06 0.07 ops/s 16.666667
Median Throughput current-kibana-traffic-dashboard_30m-querying 0.1 0.1 ops/s 0.
Max Throughput current-kibana-traffic-dashboard_30m-querying 0.1 0.11 ops/s 10.
50th percentile latency current-kibana-traffic-dashboard_30m-querying 12090.1 12681.5 ms 4.8916055
90th percentile latency current-kibana-traffic-dashboard_30m-querying 35715.8 36753.8 ms 2.9062768
99th percentile latency current-kibana-traffic-dashboard_30m-querying 79355.4 77536.2 ms -2.2924716
99.9th percentile latency current-kibana-traffic-dashboard_30m-querying 129831 119055 ms -8.3000208
99.99th percentile latency current-kibana-traffic-dashboard_30m-querying 162978 155367 ms -4.6699555
100th percentile latency current-kibana-traffic-dashboard_30m-querying 178505 184506 ms 3.3618106
50th percentile service time current-kibana-traffic-dashboard_30m-querying 6490.4 6579.2 ms 1.3681745
90th percentile service time current-kibana-traffic-dashboard_30m-querying 9734.3 9735.43 ms 0.011608436
99th percentile service time current-kibana-traffic-dashboard_30m-querying 11448.6 11398.4 ms -0.43848156
99.9th percentile service time current-kibana-traffic-dashboard_30m-querying 13418.2 15134.2 ms 12.788601
99.99th percentile service time current-kibana-traffic-dashboard_30m-querying 16198 19513.4 ms 20.467959
100th percentile service time current-kibana-traffic-dashboard_30m-querying 24632.9 29241 ms 18.707095
error rate current-kibana-traffic-dashboard_30m-querying 0 0 % 0
Min Throughput current-kibana-content_issues-dashboard_30m-querying 0.1 0.08 ops/s -20.
Median Throughput current-kibana-content_issues-dashboard_30m-querying 0.1 0.1 ops/s 0.
Max Throughput current-kibana-content_issues-dashboard_30m-querying 0.17 0.17 ops/s 0.
50th percentile latency current-kibana-content_issues-dashboard_30m-querying 3547.31 3236.13 ms -8.7722810
90th percentile latency current-kibana-content_issues-dashboard_30m-querying 8991.45 8005.54 ms -10.964972
99th percentile latency current-kibana-content_issues-dashboard_30m-querying 18752.5 16026.2 ms -14.538328
99.9th percentile latency current-kibana-content_issues-dashboard_30m-querying 30138.4 24896 ms -17.394420
99.99th percentile latency current-kibana-content_issues-dashboard_30m-querying 44543.8 40832.9 ms -8.3309013
100th percentile latency current-kibana-content_issues-dashboard_30m-querying 85341.7 69508.8 ms -18.552361
50th percentile service time current-kibana-content_issues-dashboard_30m-querying 3103.14 2859.44 ms -7.8533357
90th percentile service time current-kibana-content_issues-dashboard_30m-querying 5768.09 5394.37 ms -6.4790945
99th percentile service time current-kibana-content_issues-dashboard_30m-querying 8859.5 8426.07 ms -4.8922625
99.9th percentile service time current-kibana-content_issues-dashboard_30m-querying 10771.5 10641.5 ms -1.2068885
99.99th percentile service time current-kibana-content_issues-dashboard_30m-querying 13463.2 14927.1 ms 10.873344
100th percentile service time current-kibana-content_issues-dashboard_30m-querying 17213.1 26776.8 ms 55.560590
error rate current-kibana-content_issues-dashboard_30m-querying 0 0 % 0

Notes

Interesting, the logging is different for CMS versus G1:

CMS: 

[2018-07-03T15:17:43,920][INFO ][o.e.e.NodeEnvironment    ] [_gTpwfP] heap size [7.8gb], compressed ordinary object pointers [true]
[2018-07-03T15:17:43,923][INFO ][o.e.n.Node               ] [_gTpwfP] JVM arguments [-Xms8g, -Xmx8g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInF
astThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.pL3V4IEM, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log,
-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Des.path.home=/home/dakrone/elasticsearch-6.4.0-SNAPSHOT, -Des.path.conf=/home/dakrone/elasticsearch-6.4.0-SNAPSHOT/config, -Des.distribution.flavor=default, -Des.distribution.type=zip]

G1GC: 

[2018-07-03T16:01:10,592][INFO ][o.e.e.NodeEnvironment    ] [TPmDtw6] heap size [8gb], compressed ordinary object pointers [true]
[2018-07-03T16:01:10,595][INFO ][o.e.n.Node               ] [TPmDtw6] JVM arguments [-Xms8g, -Xmx8g, -XX:+UseG1GC, -XX:InitiatingHeapOccupancyPercent=75, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.
noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.FNRo81tf, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.l
og:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Des.path.home=/home/dakrone/elasticsearch-6.4.0-SNAPSHOT, -Des.path.conf=/home/dakrone/elasticsearch-6.4.0-SNAPSHOT/config, -Des.distribution.flavor=default, -Des.distribution.type=zip]

Conclusion

Run with short term workloads, G1GC has a comparable advantage, showing far fewer young gen GCs and pushing the throughput for indexing up 14% for the logging use case. For the full-text use case indexing throughput increased 7% compared to CMS. On the query side (simulating Kibana dashboards) the latency and service times for the 50th - 100th percentiles all saw improvements when using G1GC, ranging from a 5% to ~75% drop in latency and service times. When using a smaller heap (8gb as opposed to 31gb) the indexing throughput increase is less pronounced but still exists, however the querying and aggregation levels show improvements for some queries and slowdowns for the long-tail latency in others. This leads us to believe G1GC performs better compared to CMS on larger heaps compared to smaller ones.

On the other hand, for the long term workloads (a continual indexing and querying over a period of 30 days) on large heaps we saw a slight decrease in indexing throughput when G1GC was used for the initial two hours of indexing, but an increase in indexing throughput for ongoing indexing while the cluster was loaded by queries. For aggregations simulating Kibana dashboards, the latency and service times were close enough (usually around 1-5% difference for the 50th percentile) to call them equivalent while the 99.99th and 100th percentile latency/service times saw a small increase (meaning they were slower) when G1GC was used.

Final Conclusion

These benchmarks show G1GC can in some situations outperform, and in some situations perform worse, than CMS. However, given that we saw no long term stability issues with G1GC during our 30 day testing, we are okay having official support for G1GC for JDK10+ and Elasticsearch 6.5.0+. CMS will still be the default garbage collector, but options will be added to the jvm.options file.

Pull Request adding these options: https://github.com/elastic/elasticsearch/pull/33685

Author: Lee Hinman

Created: 2018-11-14 Wed 09:58