2011 Forecasting Performance Benchmark Study
Possibility of a New Metric for Error
The flipside of the naïve forecast discussion is that while MAPE is a correct measurement of supply chain capability, it may not be the best measure of a demand planner’s work in that it does not provide a meaningful measurement of the forecast value add contributed by the demand planning team.
The key weakness of MAPE stems from the high level of noise in the data. For example, it is critical that promotions be reflected in the forecast. However, if a promotion is one out of a thousand items selling that week, entering the anticipated promotion lift in Demand Planning may have a negligible impact on MAPE. Moreover, it can even drive MAPE the wrong way if timing is off by a week.
Using MAPE to evaluate Demand Planning is a little like using days-on-hand to measure production schedule compliance. In theory, better compliance would lead to lower inventory but the mechanics are so complex that the connection is tenuous at best, particularly when measured across companies.
Jack Welsh, the former CEO of General Electric made famous the saying "If you can't measure it, you can't manage it. If you can't manage it, you can't improve it". Getting the right metric is a prerequisite to meaningful improvement; otherwise you end up with great KPI scores, but consistently fall short of realizing the true business potential. Going forward, Terra will endeavor to measure the effectiveness of different error methods and will share its results in future benchmarking studies.
