Chronos-2 vs XGBoost: Benchmark Results on Real Many-Models Forecasting

From Framework to Evidence

A while back I wrote about when Chronos-2 beats a custom XGBoost pipeline, and later about the two forecasting philosophies behind SCOT and Chronos. Both posts were, honestly, arguments from architecture. I reasoned about breakeven points, cold-start behaviour, and operational cost. I did not have a clean head-to-head benchmark on real production data to point to.

Now I do.

A team I work with ran a proper evaluation: Amazon’s Chronos-2 foundation model against a tuned XGBoost many-models pipeline, on monthly order-quantity forecasting for a European manufacturer. One model per forecasting rank, thousands of series, real order-book data, the same preprocessing for every model. They even threw in a third contestant: the existing supply chain reference forecast, the one that humans review and adjust by hand.

The results are worth writing down, because a few of them contradict what I would have guessed.

The Setup

The forecasting problem is the classic many-models shape: predict next month’s requested order quantity at the grain of product x customer x plant. One business unit, eight sub-businesses, one year of monthly forecasts, a seven-month test horizon.

Every model ran through the identical pipeline: order-book loading, cleansing, harmonization, and feature engineering with the same internal and external drivers. Internal drivers included lead time and historical order behaviour. External drivers included weather, energy prices, exchange rates, GDP by country, and holidays.

That “same pipeline” detail matters. This is not one team’s XGBoost against another team’s foundation model with different features and a different cleaning step. Same inputs, same grain, same horizon. The only thing that changes is the model.

There is one fairness caveat, and it runs against the foundation model. Both XGBoost and Chronos-2 forecast month M from the order-book snapshot taken on day 9 of month M-1. The human-reviewed reference forecast uses a snapshot from day 27 of M-1, almost three weeks more recent, and then gets adjusted by supply chain experts. So the reference has both fresher data and human judgement on its side.

Keep that handicap in mind when you read the next table.

The Headline Numbers

The metric here is Forecast Accuracy (FA), where higher is better. I am reporting it exactly as the evaluation did rather than converting to WAPE, because the ranking is what matters and the conversion would just add noise.

Model	Mean FA	Median FA
Chronos-2	0.575	0.596
Supply chain reference (human-reviewed)	0.518	0.581
XGBoost	0.451	0.496

Chronos-2 wins. It beats the tuned XGBoost fleet by roughly 12 points of mean FA. That part I expected.

What I did not expect: Chronos-2 also edges out the human-reviewed reference forecast, in aggregate, despite the reference using a fresher snapshot and expert adjustment. The zero-shot-plus-fine-tuned foundation model beat the humans who had more recent data and got to override the machine.

At the observation level, Chronos-2 came out ahead on 90 of 112 measured observations. That is an 80% win rate on real data, not a benchmark leaderboard.

The Configuration That Won Is the Boring One

The team did not just run Chronos-2 once. They tested six configurations spanning the obvious axes: univariate versus multivariate, individual-series versus group-attention, zero-shot versus LoRA fine-tuning.

Configuration	Mean FA	Median FA
Fine-tuned univariate (LoRA)	0.576	0.595
Zero-shot univariate + covariates	0.561	0.590
Fine-tuned univariate + covariates	0.554	0.593
Group attention by product	0.550	0.575
Group attention by unit group (dict)	0.538	0.565
Group attention by unit group	0.537	0.565

The best configuration is a LoRA fine-tuned univariate model. No covariates fed to the model. No group attention. The simplest thing that could possibly work, plus a light fine-tune.

This is the result that made me sit up, because it contradicts the story I told in my earlier post. I spent paragraphs praising Chronos-2’s group attention mechanism, the one that shares information across related series. On this dataset, group attention did not help. Every group-attention variant scored below the plain univariate model.

Two counter-intuitive findings, stated plainly:

Group attention did not improve accuracy here. Grouping series by product or by unit group made forecasts slightly worse, not better.

Adding covariates during fine-tuning did not help either. The external drivers that the feature-engineering pipeline worked so hard to assemble, the weather and energy prices and GDP, did not move the needle once the model was fine-tuned on the target series itself.

I will come back to what I think this means. For now, the empirical takeaway is that the fancy features of the foundation model were not where the value came from. The value came from the pretrained temporal representation plus a cheap task-specific adapter.

Where Chronos-2 Pulled Ahead the Most

The accuracy advantage was not uniform. It was strongest on new prediction ranks, the cold-start cases where a product-customer-plant combination has little history.

This is the one result that lines up perfectly with my prior argument. Cold-start is exactly where a per-SKU custom model has nothing to learn from and a pretrained model leans on everything it has seen before. The benchmark confirms it: the thinner the history, the wider the gap in Chronos-2’s favour.

If your catalog is stable and every series has three years of clean history, this advantage shrinks. If you launch products constantly or expand into new markets, this is where the foundation model earns its keep.

The Operational Number That Ends the Argument

Accuracy is half the story. Here is the other half.

Dimension	XGBoost fleet	Chronos-2
Inference time (full run)	~30 min in AWS Batch	~2 min one-shot
Fine-tuning step	Not applicable	Up to ~30 min, done once per config
Models to maintain	One per rank (thousands)	One set of frozen weights + adapter

A roughly 15x inference speedup. Thirty minutes of AWS Batch orchestration collapses to about two minutes.

And the fine-tuning cost is bounded in a way people miss. The LoRA adapter is trained once per configuration, saved, and reused for subsequent months. You are not retraining thousands of models on a schedule. You fine-tune a single adapter, freeze the base weights so you avoid catastrophic forgetting, and reuse it. The config that won used a small adapter (rank 8) on the attention projections, a context window of about 1,000 steps, and roughly a thousand training steps with early stopping. That is a modest training job by any standard.

So the foundation model was both more accurate and dramatically cheaper to run. When that happens, the decision stops being interesting.

Why the Covariates and Group Attention Did Not Help

This is speculation, but informed speculation.

The external drivers probably did not help because their signal was already priced into the order book. The model forecasts from a snapshot that reflects orders customers have already started placing. If demand responds to energy prices or exchange rates, that response has partly shown up in the order behaviour the model already sees. The covariates were redundant with the target’s own recent history.

Group attention probably did not help because the series were not as correlated as the grouping assumed. Group attention shines when series genuinely coevolve, when knowing one tells you something real about another. Group by product code and you may be grouping series that share a label but not a demand pattern. The mechanism added parameters and noise without adding signal.

Both failures point to the same lesson: the architecture features that sound impressive in a model card are only worth using if your data actually has the structure they exploit. On this dataset, it did not, and the simpler configuration won.

What This Does and Does Not Prove

Let me be careful, because one benchmark on one business unit is evidence, not law.

What it supports: for M+1 monthly demand forecasting at this grain, a LoRA-fine-tuned Chronos-2 can replace a tuned XGBoost fleet, beat it on accuracy, beat a human-reviewed reference in aggregate, and run an order of magnitude faster. The cold-start advantage is real and measurable.

What it does not settle: longer horizons. The test was M+1. Nobody has yet shown me the same table for M+2, M+3, and beyond, where accuracy usually degrades and the ranking can change. It also covered one business unit. Whether the univariate-beats-everything result holds across other catalogs with different demand structure is an open question. Group attention and covariates might well earn their place on a dataset with genuinely coevolving series and drivers that are not already baked into the order book.

Those are exactly the next experiments to run, and I would not generalize the “keep it univariate” finding beyond this scope until they are done.

The Honest Recommendation

If you are running a many-models forecasting pipeline today and you have not benchmarked a fine-tuned foundation model against it, you are probably leaving both accuracy and money on the table. That is the uncomfortable version of this post.

But do not read this as “rip out XGBoost.” Read it as “wrap, do not rewrite.” The winning setup did not throw away the data pipeline. It reused the same order-book loading, cleansing, and harmonization, and swapped only the model at the end. The feature engineering that fed covariates turned out to be unnecessary for the model, but the cleaning and harmonization were not. Keep the plumbing. Replace the estimator.

Start with the plain univariate fine-tune before you reach for group attention or covariates. On this data the simplest configuration won outright, and the elaborate ones cost more for less. Add complexity only when a backtest proves it earns its place, and prove it on your horizon, not on M+1 alone.

And measure inference cost as a first-class result, not a footnote. A model that is one point more accurate but fifteen times slower to run is often the wrong choice in production. Here the foundation model happened to win on both axes at once, which is the easy case. When accuracy and cost point in different directions, decide deliberately, and write down why.

The teams that get this wrong are the ones still maintaining thousands of custom models out of habit, never having tested whether a single fine-tuned foundation model would beat the lot in a tenth of the runtime. On this benchmark, it did.