Testing automated forecasting on a public benchmark
A benchmark is useful evidence for a specific forecasting task, not a certificate of end-to-end planning value.
Book a PULSE workshop ↗Public forecasting data allows methods to be compared on a shared task. A useful report identifies the data split, evaluation metric and information available to each method. Without that context, a ranking is difficult to interpret.
Keep evaluation separate from tuning
Choose model settings using training and validation data. Preserve a separate final evaluation set. Repeatedly tuning on the final evaluation turns an independent check into another part of training.
State the boundary of the evidence
Forecast performance alone does not establish inventory, service or financial outcomes. Those depend on lead times, policies, constraints and the economics of mistakes. This page makes no unverified rank or client saving claim.
An illustrative decision
A method improves aggregate forecast error but performs worse on a small set of costly shortages. Investigate that distribution rather than relying only on the overall score. The business evaluation should match the decision the forecast serves.
How to use this in an evaluation
Agree the decision, inputs, objective and constraints before comparing results. Ask what is available for that scope today, what is illustrative and what is product direction. A demonstration should expose limitations as well as the favorable outcome.