Verification & Skill

Over the 5 whole months, C3S multi-system scores highest at 4 of 5. The rest are split between NMME (1)

Over the 5 whole months, C3S multi-system scores highest at 5 of 5. A clean sweep.

This is the only question on the page that the difference in reforecast period cannot touch, because neither answer is compared with the other’s numbers: each route scores its own consensus against its own members, in the same pass, from the same reforecast. Both are exact.

And they disagree completely. C3S multi-system pools 10 systems and gains at 10 of 10; NMME pools 5 and gains at only 6. Read together they are not a contradiction but a condition: pooling pays in proportion to how much there is to pool. 10 systems from nine centres carry enough independent error to average away; 5 models that share more of their heritage do not, and their consensus mostly just dilutes whichever one was best.

Neither route can say this on its own page, because neither can see the other’s members. It is the whole reason this route exists.

Routes 03 and 02 both publish a tercile probability for the same six boxes and the same months, and they differ by as much as 12 points. Neither page can say which to believe. But each also publishes, for that same box, the share of it where its own reforecast beat climatology — so the disagreement can be settled with the only evidence either of them offers about itself.

Across all 60 region-months, C3S multi-system has the better reforecast in 50, NMME in 7, with 3 level. That is not close, and it points the same way as both results above it.

Route 03 benchmarks its consensus against one named member, and for the C3S archive that member is ecmwf-51 — ECMWF SEAS5, which is exactly the model route 10 publishes on its own. The two arrive by different roads: separate CDS ingests, separate store modules, and separate RPSS implementations that share no code beyond the region boxes.

They agree to 0.0008 at worst, across all 12 scorings. That is a statement about this site’s verification machinery and not about the model: the underlying forecast is the same C3S dataset in both, so this is a consistency check, not a second opinion. It is here because the rest of the page asks you to compare numbers from three pipelines, and this is the evidence that those pipelines agree when they should.

Three systems are in the table above. The site publishes a fourth forecast — the Analog Bench, route 09, its own — and it is deliberately not there.

It could not be. It scores rolling three-month seasons where these score months; it reports anomaly correlation against an 86-year library where these report RPSS against a 24-year reforecast; and a correlation and a skill score are not the same kind of number even before the periods differ. Putting 0.26 beside +0.056 in one row would produce a sentence that reads perfectly and means nothing.

The matched version of that comparison already exists and is worth more than a shared row would be: route 11 puts the analog engine and SEAS5 on identical footing over the same 24 years and the same targets. It found that doing so reverses the answer the two routes’ own published numbers imply. That reversal is the reason this page states the footing of every comparison it makes and refuses the ones it cannot make.