Why Forecast Model Selection Fails and How to Fix It
Business forecasting challenges have grown significantly, with many organizations needing to forecast thousands of time series each cycle, amid ever-increasing uncertainty in their business environment. Evaluating dozens of forecast models for each SKU/Location time series, and adjusting model parameters, can be overwhelming. Such large-scale forecasting problems demand automation to ensure optimal performance consistently in a cost-effective manner.
Forecasting packages deal with this model assignment optimization problem in two major ways:
Human-driven model selection: manual assignment of a specific forecast model to a specific series based on time series analysis (if the series displays trend, seasonality, intermittency, etc.).
System-driven model selection: This is referred to as 'pick-best', where humans select a sub-set of forecast models, and the forecasting software runs forecasts for this sub-set of models being evaluated, then automatically picks the model with the least error.
Human-driven model selection approach does not scale across tens of thousands or more of time series and run the risk of model drift (the model assigned gradually stops to perform; either needs parameter fine-tuning or another model needs to be selected).
System-driven 'pick-best' model selection also does not work well for multiple reasons:
Overfitting Issue: The model that performed best on the training (in-sample) dataset may not perform the best on the validation (out-of-sample) dataset for a large proportion of time series forecasts. Reason is the model that best fits the actual sales is chosen, but the actual sales history consists of both an underlying signal and random noise. Of course, the noise factor changes during the validation period, and hence the 'best' model no longer delivers a good estimate of it.
Wisdom of many over one: A forecast developed by combining model specific forecasts is likely to perform better as errors in individual models cancel one another out.
Model selection is not time period (lag) specific: same model is used to develop short-term forecasts 1-3 periods out as well as mid-term forecasts 4-12 periods out.
Some software packages throw a composite forecasting capability to let Planners specify weights to be assigned each underlying model. However, there is no decision support provided to what these weights need to be: in reality, each series will have a unique profile in terms of which models and what weights; and also this profile will be dynamic across cycles, the numbers will continue to evolve and will need to be assessed when the change is significant enough to warrant taking a hit on forecast stability front as we have incremental overall value to gain due to higher accuracy. This is clearly not a problem humans can scale: this is where AI Optimal Forecasting comes in: it autonomously evaluates a rich library of forecast models, selects the right subset of models for a given series based on training phase (in-sample) performance, and then generates time-period specific (lag specific) optimal weights which minimize the relative error and relative bias overall while keeping the forecast deviation across cycles minimal.
Usually, business users (demand planners and their managers) struggle with automatic model selection because of its implied variability. A different model may be chosen every forecast cycle as different models get lucky in a given cycle based on random noise in time series. This model churn results in a high degree of variability in the 'best' forecast cycle to cycle: as different models will likely have quite different forecasts.
This Forecast Variability results in frustrated Supply Planners and high cost of forecast error. Supply Planners may see 'MRP Nervousness' with huge swings in requested quantity or timing: resulting in unnecessary Purchase Order Changes (Quantity Inflation or Deflation, Date Expedites or Delays, or outright Cancellations if lot size boundaries are crossed).
We need to balance Forecast Accuracy and Forecast Stability goals in a way that produces the best signal for a future month while keeping it as stable as possible across the cycles leading up to that month. This means optimizing the forecast signal for least error, least bias, and least variability across cycles. AI Optimal Forecasting aims to generate lag specific optimal weights to assign to each candidate forecast (from competing models). The resulting blend produces an optimal forecast from the lens of cost of forecast error and churn across cycles.
Now you can both identify the lowest achievable error for every series, benchmark current performance against such error floor, and then remove most of the avoidable error easily with such an AI Optimal Forecasting signal. Vyan.ai can deliver such a signal seamlessly into your current forecasting solution making it transparent to your business users: for accelerated value without the change management effort to get user adoption.
Join us to learn more in a webinar exclusively designed for Demand Planners / Demand Analysts / Data Scientists working in end-user organizations who are on point to produce the best possible demand forecast with the least possible effort in a way that delivers the least cost of forecast error. You can register here.
