Benchmarks

What the benchmarks show, and what they don't.

Real-work production flow, public traces through the real broker, and simulations with explicit limits. The latest studies test stronger baselines, isolate aging, and show where the original conclusions needed to narrow.

Summary

At a glance

−26%mean latency vs FIFOquiet laptop re-run
−31%mean latency vs FIFOAWS
−6% to −7%mean latency vs FIFO30-minute run with long backlogs
4 of 6real-broker trace cells passed with weights + pausing0 of 6 with default-aging routing

Every benchmark and how it came out
Benchmark	Where	Outcome	Headline
Production flow	Laptop, quiet re-run	Passed	Mean −25.6% vs FIFO
Production flow, replicated	AWS	Passed	Mean −30.8% vs FIFO
30-minute run, broker on its own host	AWS	3 of 8 checks failed	Mean −6% to −7% vs FIFO
Level weights + pausing aging, production flow	Laptop + AWS	Not passed	AWS mean −57.5% vs FIFO; P99 interval exceeds the guardrail
Size-binned keys	Laptop + AWS	Not passed	AWS mean −1.5% vs job-type keys; no measurable gain
Azure Functions 2021 trace	Real-broker replay	Weights: 2 of 3	Default-aging routing: 0 of 3; weighted mean −8% to −17%
Huawei 2025 R5 trace	Real-broker replay	Weights: 2 of 3	Default-aging routing: 0 of 3; weighted mean −2% to −47%
Huawei, aging off	Real-broker replay	2 of 2	Mean −37% / −45% vs FIFO, with tail and long-wait costs
Stronger baselines and ablations	AWS production flow	Descriptive	Per-type fair queueing beats predictive mean, with worse P99
CPU-work simulator validation	Laptop + AWS	Not passed	Simulated effects overstate measured gains by 11–12 points
Parameter sensitivity	Simulation	Incomplete	Variant confirmation through the real broker was triggered but not run
LPC-EGEE 2004 job log	Simulation	0 of 6	Predictor gave a prediction for 21% of jobs
DAS2-fs2 2003 job log	Simulation	0 of 6	Overload design made the mean worse (+72% to +89%)
Against RabbitMQ and Redis	AWS	Passed	Mean −21.7% and −25.5% vs their FIFO
Broker cost, empty handlers	AWS	No gate	Harbinger is the slowest of the three brokers

A gate pass means every applicable criterion passed. Descriptive studies have no overall gate, and incomplete registrations are labeled separately. Raw artifacts are hashed, but not all per-message files are committed.

Method

How we measured

Three evidence types. Production flow runs a real gRPC broker and CPU-work handlers on nine synthetic job types, a flash sale and a cost shift. Real-broker trace replay uses public durations with sleeping handlers. Simulations run scheduling code on simulated time; HPC results and parameter sensitivity remain simulation-only. These are different tests, not interchangeable replications.

Laptop

CPU: Intel Core i7-11800H, 8 cores / 16 threads
Settings: Turbo off, 2.3 GHz cap, performance profile
Machine: 14 GB RAM, Ubuntu 26.04.1, on AC power
Conditions: Sampled every 30 s: median load 1.04 on 16 threads

AWS

Hosts: Client c8i.2xlarge (8 vCPU), broker c8i.xlarge (4 vCPU)
Placement: us-east-1a, one cluster placement group
Software: Ubuntu 24.04, same Docker image on every host
Replication: Pre-registered, then run from commit 2da0e3a

Why effect sizes move between machines

Job sizes and arrival rates are fixed, so a slower CPU means a heavier load. The original laptop run had a CPU 1.6–1.8× faster than the quiet re-run, and AWS is faster again.

Image resize kernel (ms per megapixel, lower is faster)

AWS

6.60

Original laptop run

8.11

Quiet laptop re-run

13.15

FIFO mean latency on the same workload (s)

Original laptop run

8.0 s

Quiet laptop re-run

13.1 s

How to read the production-flow numbers

Quote an effect size together with its load. The mean gain barely moves across machines (−26% to −31%), but the short-job median gain swings from −58% to −95%, because heavier backlogs give aging more to promote.

Benchmark 1

Production flow: laptop and AWS

Seven arms, five seeds, about 35,000 messages per run. The gate needs every criterion to pass, with intervals paired by seed.

Mean latency reduction vs FIFO

Original laptop run

−27.9%

Quiet laptop re-run

−25.6%

AWS

−30.8%

Predictive routing against FIFO. The AWS point is the independent replication.

Every v2 criterion, laptop re-run against AWS. All ten passed on both.
Criterion	Laptop re-run	AWS
P1 mean vs FIFO (≤ −15%)	−25.6% [−29.4, −21.8]	−30.8% [−35.3, −27.1]
P2 short-job median vs FIFO (≤ −30%)	−57.7% [−70.0, −45.3]	−94.8% [−99.0, −86.5]
G1 all-message P99 vs FIFO (≤ 1.25×)	1.02× [1.00, 1.05]	1.03× [1.02, 1.05]
G2 export max wait vs FIFO (≤ 2.0×)	0.99× [0.95, 1.03]	0.99× [0.90, 1.05]
G3 no lost work	Pass	Pass
G4 throughput vs FIFO (≥ 0.98×)	1.00×	1.00× [1.00, 1.00]
S1 vs hand-tuned static map (≤ +10%)	−11.1%	−14.4%
S2 vs misconfigured static map	−30.1%	−36.8%
S3 shadow-mode overhead (±5%)	+0.1%	−0.1%
S4 producer-only keys vs FIFO (≤ −5%)	−10.5%	−15.7%

What it shows

Every verdict reproduced on a quiet machine and again on separate cloud hardware.
Mean latency gains were 26–31% in every environment, and the tuned static map was beaten by 11–14%.
The predictor is close to its ceiling. On AWS the oracle was 32% below FIFO and predictive 29%.
Shadow-mode overhead was not measurable in this benchmark: ±0.1% against FIFO.
The tail did not improve. All-message P99 was 1.02–1.03× FIFO, so the claim is about the mean and the median, not the tail.

Laptop re-run ↗AWS results ↗

Benchmark 2

Long backlogs: a 30-minute run

AWS, broker on its own host, two repetitions with a flash sale lasting six minutes. This run had no gate, only eight checks that had to hold in both repetitions. Three failed.

Predictive routing vs FIFO, mean latency by phase

Steady

−23%

Ramp

−54%

Flash sale

−7%

Recovery

−4%

Shifted

−18%

← better than FIFOworse than FIFO →

Mean latency by phase in milliseconds, median of two repetitions
Phase	FIFO	Static, tuned	Shadow	Predictive	Change
Steady (0–6 min)	59	50	59	46	−23%
Ramp (6–9)	471	284	466	215	−54%
Flash sale (9–15)	67,720	64,867	67,733	62,897	−7%
Recovery (15–21)	91,573	89,178	91,274	87,678	−4%
Shifted (21–30)	12,593	10,908	12,367	10,325	−18%

Why the overall gain shrinks to 6–7%

Outside overload the predictor still halves latency in the ramp. But the flash sale pushes FIFO's mean to 68 s, and 92 s in the recovery, and those phases dominate the average. With waits that long the default 5 s aging lifts almost every waiting job to the top level, so routing changes the order very little. This is the aging collapse from the Azure trace, seen on real infrastructure.

No guardrail moved: P99 and the export max wait match FIFO.
Shadow-mode overhead was not measurable, and predictive stayed ahead of the hand-tuned static map in this run.
After the mid-run cost shift, predictive was 15–23% ahead of FIFO with no drift alerts.

Benchmark 3 · v4

Real traces through the real broker

Azure Functions 2021 and Huawei Cloud 2025 R5, at three loads, on AWS. Gate A uses default-aging predictive routing; gate B adds level weights and pausing aging. Gate A passes 0 of 6 cells; gate B passes 4 of 6. All three Huawei repeats agree on every verdict.

v4 real-broker replay: window-average mean-latency change vs FIFO, with 95% intervals
Trace · load	Default-aging routing	Gate A	Weights + pausing	Gate B
Huawei R5 · 0.50	−2.5% [−4.6, +0.0]	Fail	−2.4% [−4.4, +0.0]	Fail
Huawei R5 · 0.80	−10.9% [−14.8, −1.6]	Fail	−37.0% [−66.0, −4.0]	Pass
Huawei R5 · 0.95	−8.4% [−11.5, −3.2]	Fail	−47.1% [−74.8, −17.9]	Pass
Azure 2021 · 0.50	−0.9% [−1.9, −0.2]	Fail	−8.0% [−11.5, −4.3]	Fail
Azure 2021 · 0.80	−1.6% [−3.0, −0.6]	Fail	−15.3% [−23.9, −9.0]	Pass
Azure 2021 · 0.95	−2.2% [−3.9, −1.1]	Fail	−17.2% [−25.8, −11.6]	Pass

28 equal-trace-time windows; moving-block bootstrap, block length 4, 10,000 resamples. Huawei pools three repeats; Azure has one run per cell. P1 requires a point estimate ≤ −15% and an interval below zero, not an entire interval below −15%. G1 and G2 require interval upper bounds ≤ 2.0× FIFO for P99 and long-class maximum wait. Valid runs also require all messages completed, no DLQ or lease expiry, and producer P99 lateness ≤ 10 ms in real time. Gate B's point ratios are at most 1.12× for P99 and 1.14× for long-class maximum wait across these cells.

Replay results and artifacts ↗Replay criteria and validity rules ↗

Historical simulations and the HPC boundary

The original v3 serverless simulation passed 1/6 routing cells and 5/6 weighted cells. Re-estimating with 28 windows and block bootstrap changes Huawei routing at 0.8 and Azure weighted at 0.5 to failures, yielding 0/6 and 4/6—the same verdicts as real-broker replay. The HPC logs below were not replayed through the real broker and are not pooled with those six cells.

Historical v3 simulation: mean latency vs FIFO under the original seven-window analysis. Filled cells passed then; serverless verdicts were revised in v4.
Trace	Arm	Load 0.5	Load 0.8	Load 0.95
Azure Functions 2021	Routing alone	−0.7%	−0.9%	−1.2%
	+ weights, pausing aging	−15.4%	−22.8%	−25.3%
	Perfect predictions + weights	−26.5%	−33.3%	−36.7%
Huawei 2025	Routing alone	−4.8%	−15.0%	−7.9%
	+ weights, pausing aging	−4.3%	−46.5%	−49.9%
	Perfect predictions + weights	−3.2%	−37.8%	−50.0%
LPC-EGEE 2004	Routing alone	0.0%	0.0%	0.0%
	+ weights, pausing aging	+1.0%	+1.9%	+8.5%
	Perfect predictions + weights	−33.1%	−52.1%	−53.8%
DAS2-fs2 2003	Routing alone	0.0%	0.0%	0.0%
	+ weights, pausing aging	+2.7%	+88.8%	+72.2%
	Perfect predictions + weights	−6.0%	−30.1%	−49.2%

This grid preserves the original v3 analysis, not the latest serverless verdicts. DAS2-fs2 also failed its tail and long-job guardrails.

Look closer

Azure Functions 2021 · real-broker replay

On the real broker, default-aging routing gained only 1–2% and failed all three cells. Weights and pausing aging passed at loads 0.8 and 0.95, not 0.5.
The load-0.8 pass is narrow: −15.3% against a −15% point-estimate bar. Azure ran once per cell, so repeat-based stability is unknown.
The gate averages equal-time windows. Whole-run weighted mean gains are only −7.9% and −9.3% at loads 0.8 and 0.95, because the final day's backlog holds most of the work.

The boundary

Recurring, predictable keys are useful but not sufficient for a scheduling gain. Default aging can erase the ordering, and prioritising short jobs can hurt long ones. Parameter variants were tested in serverless simulation; HPC time-scale tuning and real-broker confirmation of the sensitivity variants remain untested.

Serverless traces ↗HPC logs ↗

v4 · Baselines

Prediction is not the best scheduler here

Eleven arms, ten seeds, 110 production-flow runs; all messages completed. This is a descriptive study, with no overall pass gate. Predictive routing lowered mean latency 32.1% [29.6, 34.4] vs FIFO and 15.5% [13.3, 17.8] vs a tuned static map. But per-type fair queueing did better on the mean without predicting durations.

Median run metrics on the v4 synthetic workload; milliseconds, lower is better
Arm	Mean	Short-job P50	All-message P99
FIFO	6,927.7	4,965.1	21,099.4
Predictive	4,779.7	39.3	21,753.9
Per-type fair queueing	2,726.4	16.1	31,954.9

A stronger baseline, with a different trade-off

Predictive mean latency is 74.1% [64.5, 83.9] worse than per-type fair queueing. The fair scheduler's P99 is 51.7% [45.9, 58.1] worse than predictive. It requires operator-configured per-type queues/levels and was tested on nine near-deterministic types, not hundreds of trace keys. Harbinger avoids a per-endpoint priority map; it does not beat every scheduler.

What the predictor ablations establish

Removing decay/idle forgetting, the spread gate, staleness, or minimum observations changes the overall mean by only 0.0–0.4%; each interval is within ±3%. This workload does not demonstrate an overall benefit from those components.
Shifted-phase results are inconclusive, not proof that adaptation never helps. Censored-share handling was not ablated.
The median remains the better tested summary: the mean estimator is +3.6% [+1.4, +6.1] worse and P75 +3.8% [+1.6, +6.3] worse than default predictive routing.

Baseline and ablation results ↗

v4 · Mechanism

Aging's cost is now measured

Turning aging off is a separate intervention from adding weights with pausing aging. Both can preserve the predicted order, but neither removes the trade-off between short jobs and long waits.

Separate Huawei aging-off replay: window metrics vs each run's own FIFO, with 95% intervals
Metric	Load 0.8	Load 0.95
Default-aging mean change	−11.0% [−15.1, −1.7] · fail	−8.5% [−11.7, −3.2] · fail
Aging-off mean change	−37.4% [−66.0, −4.4] · pass	−45.4% [−73.6, −16.2] · pass
Aging-off P99 ratio	1.07× [1.01, 1.15]	1.12× [1.02, 1.18]
Aging-off long max-wait ratio	1.16× [1.02, 1.23]	1.14× [1.03, 1.21]

All three repeats agree at each load; all 18 runs completed without DLQ, submit errors or lost leases. Aging off also improves mean vs default-aging predictive by 34.1% and 41.8%. These runs use their own controls and different host sizes from the main replay. Runs of roughly 20–25 minutes do not establish starvation freedom over hours.

Synthetic production-flow isolation

Across ten seeds, aging off lowers mean latency another 37.4% [35.1, 39.8] vs predictive and 57.6% [57.1, 58.1] vs FIFO. Against FIFO, P99 rises 13.6% [12.6, 14.6] and long-class maximum wait 13.6% [12.0, 14.7]. Against predictive, those costs are 10.1% and 12.2%. The mechanism is measured; the cost is measured too.

AWS repeats the weighted-aging and size-bin misses

Weights with pausing aging lower mean latency 57.5% vs FIFO and 38.4% [34.7, 41.9] vs predictive. But P99 vs predictive is 1.100× [1.091, 1.107]; its upper bound exceeds the 1.10× limit, so the gate fails despite the rounded point estimate.
Size bins change mean latency −1.5% [−4.5, +0.4] vs job-type keys, failing the −3% criterion. Invoice mean rises 9.5% [4.4, 15.9]; resize intervals are too wide for a directional conclusion. Better prediction is not automatically better end-to-end latency.

Huawei aging-off replay ↗Synthetic aging isolation ↗AWS reruns ↗

v4 · Limits

Validate the model, not just the result

CPU-work production-flow validation: predictive mean change vs FIFO
Environment	Measured	Simulated	Effect gap
Quiet laptop	−25.6%	−37.0%	11.4 points · fail
AWS	−30.8%	−42.7%	11.8 points · fail

Both fail the 10-percentage-point effect-agreement criterion, though phase directions and policy-group order pass. Simulated absolute latencies are only 0.31–0.41× measured. Production-flow conversion also omits failed-attempt work for 64 retried local messages. The simulation does not faithfully model CPU contention.

Parameter sensitivity is still simulation-only

At load 0.8 under the original seven-window rule, P1 passes across 18 tested configurations (including default): Azure routing 0, weighted 16; Huawei routing 10, weighted 18. Azure weighted fails with min_samples=100 and with two levels. These are P1 sensitivity results, not full-gate robustness counts.
Huawei routing is sensitive to aging: thresholds of 1 s, 5 s and 30 s yield −3.3%, −15.0% and −40.7% mean changes. A parameter setting can change the verdict.
Using 28 windows and block bootstrap changes two original simulation passes to failures. The registered Part C—real-broker replay of verdict-flipping variants—was triggered but not run. Main replay of the default configuration does not substitute for that missing confirmation.

Simulator validation ↗Sensitivity and missing confirmation ↗

Conclusions

What the benchmarks support

Holds up

On the tested synthetic production flow, predictive routing beats FIFO and the hand-tuned map on mean and short-job latency, without a per-endpoint priority map.
The original production-flow result replicates: laptop, quiet laptop re-run and AWS agree on every v2 gate verdict.
Shadow mode changes no routing and had no measurable overhead in those runs.
The idea is not tied to Harbinger's broker. It also works beside RabbitMQ.

Holds with conditions

Keys must recur and predict well. They do on serverless traces and mostly do not on the two HPC logs.
Default-aging routing passes 0/6 real-broker serverless cells. Weights with pausing aging pass 4/6; aging off also restores gains on Huawei, with tail and long-wait costs.
Predictive is not universally better: per-type fair queueing beats its mean on the synthetic workload, and a history-built static map matches or beats it on Huawei.
Effect sizes depend on load relative to CPU speed.

Not claimed

Better tail latency or hours-long starvation freedom. Short-job gains can increase P99 and long-job waits.
Production readiness. All evidence is synthetic or trace-driven, and the broker is in-memory with no durability.
That learned models never help. The workloads tested had little for one to find.
Anything about other clouds, wide-area networks or durable queue modes.

When to turn it on

Start in shadow mode. It records predictions without changing routing. Measure its overhead on your workload.
Check that keys predict. Run a per-key predictability check on your own history. If most jobs sit on recurring keys with stable durations, move on.
Replay your workload against meaningful baselines. Include FIFO, a history-built static map and an appropriate fair scheduler. Predictability alone does not guarantee a gain.
Set tail and long-job budgets before enabling a policy. Compare default aging, weights with pausing, and aging off where relevant; use measured benefits and costs, not a blanket recommendation.

This sequence is our reading of the results, not a pre-registered result. The predictability check as a deployment guideline comes from the HPC-log findings, and an automatic safeguard is future work.

Limits of the evidence

The production flow is one scenario that we wrote. Serverless replay uses sleeping handlers and some reconstructed arrivals; HPC and sensitivity variants remain simulation-only.
One cloud provider and region, with experiment-specific host layouts. Results do not establish behaviour on other infrastructure.
Azure has one run per replay cell; two Huawei broker deaths remain unexplained. Roughly 5.8 GB of raw replay artifacts are hashed but not committed; run summaries are available.
The 30-minute run has two repetitions, so its checks carry no intervals.
The budgets of the first experiment were set after seeing earlier data, so that gate is less independent than the later ones.

Keep reading

ExperimentsHow each experiment steered the design, including the ones that failed.RabbitMQ and RedisSame workload, different broker: scheduling gains and raw broker cost.

Glossary

Ack: Short for acknowledgement. A consumer tells the broker it finished a message, and here it also reports how long the work took. Learn more ↗
Aging: A message that has waited too long is moved up one priority level, so low-priority work is delayed but never stuck forever. Learn more ↗
Backlog: Messages that have arrived but are still waiting because workers can't keep up.
Bootstrap: A statistics method that re-samples your own results many times to see how much they could vary by chance. Learn more ↗
Broker: The server in the middle. Producers hand messages to it, and consumers pull messages from it. Learn more ↗
Cold start: The period before the predictor has seen enough finished jobs for a key to trust its estimate. Cold keys use the default middle tier.
Confidence interval: A range that likely contains the true value. A 95% interval that sits entirely below zero means the improvement is unlikely to be luck. Learn more ↗
Consumer: A program that pulls messages from the queue and does the work they describe. Learn more ↗
Control: A test case designed so the thing being measured shouldn't help. If it still shows a gain, something is wrong. Learn more ↗
Dead-letter queue (DLQ): Where messages go after they fail too many times or expire, so they can be inspected instead of retried forever. Learn more ↗
Decision tree: A model that makes a prediction by answering a chain of yes/no questions about the input. Learn more ↗
EWMA: An exponentially weighted moving average. A running average that gives recent values more weight, so it adapts when things change. Learn more ↗
FIFO: First in, first out. Messages are served strictly in arrival order, like a single supermarket line. Learn more ↗
Gate: A pass/fail test agreed before running an experiment. Predictive routing is only allowed if every gate passes.
Gradient boosting: A heavier machine-learning model that combines many small decision trees. Used here as a stand-in for "a much more complex model". Learn more ↗
gRPC: A fast way for programs to call each other over a network. Harbinger's producers and consumers talk to the broker with it. Learn more ↗
Guardrail: A metric that must not get much worse, even if the main metric improves. Tail latency and lost work are guardrails here.
Head-of-line blocking: One slow item at the front of a line holds up everything behind it, however quick those items are. Learn more ↗
Histogram: A count of how many times each range of values occurred. Here: how many jobs took 1–2 ms, 2–4 ms, and so on. Learn more ↗
Key: What Harbinger groups messages by when learning: the producer, optionally plus a job_type label (and size). Each key gets its own duration history.
Latency: How long a message takes from being sent to being finished, including time spent waiting in the queue. Learn more ↗
Level weights: Instead of always serving the highest level first, each level gets a share of worker time (for example 8 : 3 : 1). Lower levels still make steady progress.
MAE: Mean absolute error: on average, how far a prediction is from the real value, in the same units (here, milliseconds). Learn more ↗
Mean: The ordinary average. It is pulled up by a few very slow items, so it shows the overall experience better than the median does. Learn more ↗
Median: The middle value: half the jobs are faster, half slower. It describes a typical job and ignores rare outliers. Learn more ↗
Message queue: Software that holds messages (jobs to do) until a worker is free to handle them, so senders don't have to wait. Learn more ↗
Multi-level queue: Several queues stacked by priority. The scheduler always serves the highest non-empty one first. Learn more ↗
Non-preemptive: Once a job starts it runs to the end. The scheduler can reorder waiting jobs but can't interrupt one that's running. Learn more ↗
Opt-in: Off by default. You have to switch it on deliberately.
Oracle: A reference that cheats by knowing each job's true cost in advance. It shows what perfect knowledge would give, not what's achievable.
P95 / P99 (tail latency): P99 is the time that 99% of messages beat. It describes the slowest 1%, the "tail", which matters because users notice the slow ones. Learn more ↗
Pausing aging: Aging that holds off promoting a message while the level above is still working through earlier promoted messages. It stops a long backlog from pushing everything to the top.
Pre-registration: Writing down the hypotheses, setup and pass/fail rules before running the experiment, so results can't be quietly reinterpreted afterwards. Learn more ↗
Predictor: The part of the broker that estimates how long a message will take, using what earlier messages with the same key took.
Producer: A program that creates messages and sends them to the queue. Learn more ↗
Quantile: A cut point that splits sorted values into equal groups. Harbinger uses them to decide where one tier ends and the next begins. Learn more ↗
Round-robin: Take turns between queues in a fixed rotation, regardless of what's in them. Learn more ↗
Seed: A starting number for a random generator. Same seed, same random-looking data, so a run can be repeated exactly. Learn more ↗
Serverless: A cloud style where you upload small functions and the provider runs them on demand. The Azure Functions trace records one such platform's invocations. Learn more ↗
Shadow mode: The predictor runs and learns, but its guesses are only recorded, never used. Safe way to try it on real traffic.
Shortest job first: Serve the quickest jobs first. It lowers the average wait but makes long jobs wait longer. Learn more ↗
Slowdown: Latency divided by the job's own work time. A 1 ms job that waits 1 s has a slowdown of 1000.
Spread gate: If a key's slowest runs (p99) are more than 16× its typical run (median), its cost is too erratic to trust, so it takes the default tier.
Static priority: A hand-written rule that fixes each job type's priority in advance and never changes with what's observed.
Throughput: How many messages get finished per second. Learn more ↗
Tier: A priority level in the queue. L0 is served first, then L1, then L2. Learn more ↗
Tier accuracy: How often a message's predicted tier matches the tier its real duration would have earned.
TTL: Time to live. After this long a message is considered expired and dropped. Learn more ↗
Worker: A process that takes one message at a time and runs the work for it.