Current Forecast Error: Do you see Angels or Demons?
Most companies know where they are at in terms of their current level of forecast error. However, few can tell you how much lower this forecast error could be or should be. This is critical when defining forecast error reduction targets. Is 32% error good, bad, or ugly? It depends on what the error floor (lowest achievable error) is: 32% error is awesome, if error floor is 30%, but it sucks if error floor is 10%. But how do you find out what the error floor is? You can neither benchmark nor improve your error without this insight.
Error ceiling is an equally important concept. You need to know the highest tolerable error: any error performance higher than your error ceiling is unpardonable, if you are investing any type of effort into your demand forecasting process: in terms of people's time or costs (forecasting software or forecasting as a managed service). Here again, is 68% error bad? It depends on what the forecastability band (error floor vs. error ceiling) is. 68% error is not so terrible if the time series is a highly intermittent one with forecastability bands at 52% floor | 88 % ceiling.
We have seen exception reports with SKU/Location combinations that are in the top 5% of forecast error ladder. This could be so pointless, if most of the intermittent (unforecastable) time series with high error show up in the top 5% of your overall portfolio. Any efforts to flog demand planners to improve the error in these combinations could be an exercise in futility, while the true culprits remain hidden, e.g., stable time series with 25-35% error, which really could be significantly lower.
Annual target setting for forecast error reduction amount to blind leading the blind in absence of clarity on SKU/Location combination specific performance bands: defining the lowest achievable error (what 'good' looks like that you should be able to achieve with sincere effort) and the highest tolerable error (what 'bad' looks like that you should not allow forecasters to get away with). Any error higher than the error ceiling falls in in the 'ugly' bucket: that is what should show up in your exception report.
Knowing granular combination specific performance bands enables proper benchmarking of current performance: % of combinations that are close to good vs. close to bad vs. ugly. It also helps in setting feasible targets for forecast error reduction. Such benchmarks and well-informed targets coupled with margin-based segmentation can provide very clear direction to all concerned: this is the starting point to make a significant dent in the cost of forecast error.
So, how do we determine the error floor and ceiling?
Error Ceiling: This one is relatively simple. We recommend generating a naive forecast based on a mix of n-period moving average (to account for any trend) and same-period-last-year value (to account for any seasonality). Please note that this is more refined than the classic definition of naive forecast (which is simply the last period actuals projected as next period forecast). We believe our way of naive forecast is simple enough that it can be calculated without any forecasting software and robust enough that we do not set the performance bar too low (or in this case inflate the error ceiling too high). The idea is that you should be able to achieve this level of error with NO effort whatsoever, so although this level of performance is bad (as we obviously have knowledgeable demand planners and forecasting software to help them), but any worse than this error ceiling is simply unpardonably ugly. If you find any combinations crossing the error ceiling, simply throw remove them from the forecasting process and switch to using the naive forecast.
Error Floor: Getting to lowest achievable error is not so straightforward without forecast optimization software. Essentially, you need to find out what the most optimal forecast looks like (an optimally blended forecast across all forecast models at all levels across human and machine signals). You need to then test this optimal forecast performance along with forecast performance of any of the competing forecasts: top down or bottoms up, machine or human, then select the forecast with the best performance (this should mostly be the optimal forecast, if the optimal forecasting engine is doing its job). We also recommend to use n-cycle average error when deciding who the 'winner' of this forecast competition is - as it is easy to have a really low or even zero error (forecast = actual) by fluke in a given forecast cycle, but averaging error across n-cycles (n should be minimum 3) takes this element of 'dumb-luck' away from the competition and ensures only the most stable performance forecast wins. This is our way of saying that we have been able to achieve x level of 'good' performance in the past, and technically it is possible to get back to this level (or stay really close to it). Obviously, this is assuming the forecast volatility of the time series has not changed materially (which you should be able to measure with a simple Coefficient of Variation COV (Standard Deviation divided by Mean). Do adjust the Error Floor as long-term average COV deviates materially from the short-term n-period COV.
vyan.ai provides Forecastability Bands out of the box. It also benchmarks current (baseline) performance against these bands to show the headroom available for higher accuracy (lower error). Confidently set targets and have a bottoms-up data-driven sense of your forecasting portfolio health - segment by segment. See how your Cost of Forecast Error is trending and understand Root Causes, in terms of actionability. You can send us an email at info@vyan.ai to see Forecastability Bands in action.
