Stress run with queue auto-adjustment

The behavior I'd like to see in this situation is that the stressed node (es5) would eventually lower its threadpool queue down to a level where it would start rejecting requests, causing other nodes in the cluster to serve them. This would lead to better overall performance since the load would be routed away from the overloaded node.

First I tried this branch as-is, I measured how the threadpools changed and whether the desired behavior was accomplished.

Spoiler alert: It was not.

My initial premise was based on the (deceptively simple) Little's Law formula of

L = λW

Where L is the optimal queue size, λ is the arrival rate of events, and W is the average time it takes to process a request.

I was measuring both λ and W, and calculating L to be used for the queue size.

I discovered a number of issues with this premise. First, by leaving both λ and W as "calculated" metrics, L is reported rather than targeted. This means that on es5, as W (avg response time) got higher due to the node being loaded, the estimated queue actually increased, which was not desired. I went back to the drawing board.

"Okay" I thought, I need to make the relationship between the optimal queue size and actual queue size inverse, so that as the "optimal" queue went up, the actual queue size went down. The second thing I tried was to take the maximum queue size and subtract the optimal queue size, so that as the optimal queue size went up (L, also known as the long-term average number of tasks in the queue), the actual queue would go down.

This sort of worked, but it had a number of issues -

It felt hacky, and not like an actual good implementation, being able to set a good maximum queue size would be almost impossible without knowing up front what your queue size already was at, set too high of a max and the adjustment would have no effect, too low and you'd be rejecting before you actually had stress on the node.

Additionally, it suffered the same problem as the previous solution, both λ and W were being measured, when they really should not be, which leads me to..

Back to the drawing board, I spent some time thinking and realized that we should not be measuring both λ and W, and determining L, rather, one variable should be configured, one measured, and one the resulting optimal value.

With this in mind, I implemented an alternative where W was set by the user as a target_response_rate, this is a level where the user would essentially configure a value saying "I am targeting my requests to all take less than N on average for a healthy node".

I then use λ to be the measured task rate, similar to how I did before, this means that with a target_response_rate of 5 seconds and task rate of 15 queries/second, there should be at most 15 * 5 = 75 events in the queue at peak usage.

I tested my theory; with the same setup, I set up a target_response_rate of 5 seconds and adjustment amount of 50 (the queue would go up or down by 50 each time, just makes it easier to read the logs). I knew that this was a particularly complex query that I would be throwing at the nodes, and 5 seconds was a reasonable average response time.

I am happy to report that with my initial testing, I was able to show that with this setup, the stressed node correctly routed traffic away from itself by lowering and subsequently rejecting search requests, routing them to the unstressed nodes, and improving average response times for the entire run.

Below are graphs of the response times with "None" and "Auto" queue adjustment:

I also captured the handled tasks per second for an unstressed and stressed node as well as the queue adjustment for each node, which you can check out also check out below

And you can also see the actual versus "optimal" queue in the graphs below.

Each JMeter thread ran a total of 40002 queries.

The max was a little high for the previous run, I think partially because stress was killing the disk so much in this case. Therefore, I did another run to see if the max would come down.

Results from run #2 of a stressed cluster with auto-adjustment, this time I adjusted the target_response_time to be 4 seconds instead of the 5 seconds I was using above.

Each JMeter thread ran a total of 40002 queries.

And after a second run

Here's a summary of the JMeter results

set size ratio 0.75
set style line 1 lc rgb "red"
set style line 2 lc rgb "blue"
set boxwidth 0.5
set style fill solid
set xlabel "Automatic Queue Adjustment"
set ylabel "Average response time (ms)"
plot [][0:10000] data every ::0::1 using 1:3:xtic(2) title "JMeter response times (no adjustment)" with boxes ls 1, \
     data every ::2::3 using 1:3:xtic(2) title "JMeter response times (adjustment)" with boxes ls 2

set xlabel "Tasks in 2k chunks"
set ylabel "Queue Size"
plot [][0:1000] data using 1:6 title "actual queue" with lines, data using 1:5 title "optimal queue" with lines

This is a graph of handled tasks per second

set xlabel "Tasks in 2k chunks"
set ylabel "Handled Tasks per second"
plot [][0:200] data using 1:4 with lines title "es1 handled tasks"

This node was stressed with stress -i 8 -c 8 -m 8 -d 8 (8 each of IO, CPU, Memory, and HDD workers)

set xlabel "Tasks in 2k chunks"
set ylabel "Queue Size"
plot [][0:1000] data using 1:6 title "actual queue" with lines, data using 1:5 title "optimal queue" with lines

This is a graph of handled tasks per second

set xlabel "Tasks in 2k chunks"
set ylabel "Handled Tasks per second"
plot [][0:200] data using 1:4 with lines title "es5 handled tasks"