Solar forecasting needs a better accuracy metric
Solar

Solar forecasting needs a better accuracy metric

Contributed by Sean Kelly, CEO, Amperon

The utility industry uses a handful of metrics to determine the accuracy of demand forecasts. The most common are MAPE, nMAE, and RMSE. Each has distinct usefulness in calculating predictive error, but they also have distinct problems when applied to solar forecasting at the asset level, which is why we’re encouraging the adoption of a new metric, cnMAE.    

Solar forecasting — which predicts the amount of solar energy that will be available at a specific time and location — is becoming more important as solar becomes a larger part of the generation mix, alongside fossil fuels, nuclear, and other renewables. A recent Economist article contextualized solar’s blistering pace of growth, noting that installations continue to blast past even the most optimistic forecasts.  

For example, 15 years ago Greenpeace said global solar capacity would total 921 gigawatts (GW) by the year 2030. By comparison, the experts at the International Energy Agency (IEA) predicted a much more conservative amount of just 244 GW by 2030.  

But they were both way off. Last year, still seven years short of the 2030 deadline, the world’s solar capacity reached 1,419 (GW). The pace shows no signs of slowing and the variability of solar generation is already having significant effects on grid operations and wholesale markets, which is what makes accurate solar forecasting important.  

Independent power producers (IPPs) and energy traders are using short-term solar forecasts to determine how much power to sell into day-ahead markets, and whether to reserve any for sale on real-time markets. Solar forecasts also help them determine whether the market is over- or underpriced when making decisions about mid- and long-term hedging. 

Similarly, a growing number of utilities use solar forecasts to predict what their net demand will be. These utilities have solar generation capacity, as well as consumer load, and they need to know in advance how well their solar assets will perform to avoid buying additional power in expensive, real-time markets.  

In short, they all need to understand how accurate their solar forecasts are to keep from making bad bets.  

But the accuracy metrics are flawed 

Without getting too deep into the math, solar forecasting has unique aspects that disrupt the metrics commonly used for determining the accuracy of demand forecasts.  

Mean absolute percentage error (MAPE) is perhaps the most common accuracy metric for load forecasting. It’s calculated by first subtracting actual load or generation from the predicted amount to determine the size of the difference — or “error.” Then the (absolute) error value is divided by the actual observed amount to produce a percentage. But with solar generation, it’s not unusual for actual figures to be zero. In those instances, the standard MAPE calculation requires dividing the error amount by zero, which is mathematically impossible. Dividing by very small figures approaching zero also impairs the usefulness of the metric by hyper-inflating the percentage calculation. 

Normalized mean absolute error (nMAE) solves the mathematical problem described above by doing away with zero and near-zero figures. But it doesn’t account for seasonal variations in solar output capacity. So, in the winter, when average solar generation is minimal, it often exaggerates forecast errors, and when generation is at its highest in the summer, nMAE can depreciate errors. 

The other common metric for predictive error, Root Mean Squared Error (RMSE), also has only limited usefulness in solar forecasting. Like nMAE, RMSE overcomes the mathematical instability of dividing by zero or near-zero values. It’s also more consistent than nMAE across seasonal variations in solar capacity. However, RMSE doesn’t account for the size of the solar installation or the average yield from that asset over the period being examined. As a result, it can’t be used to compare forecasts across a range of installation sizes. 

cnMAE is a better alternative 

After considering numerous metrics, including many proposed at Solar Forecast Arbiter, we concluded the most useful measurement of predictive error for solar forecasting is nMAE with an additional normalization step for capacity: cnMAE, or capacity normalized mean absolute error.  

cnMAE normalizes for capacity by scaling the forecast error relative to a solar system’s rated nameplate capacity. Simply put, cnMAE = MAE/Capacity. 

  • norm: normalizing factor (with the same units as the forecasted and observed values) 
  • C: capacity 
  • F: forecasted value 
  • O: observed (actual) value 

For market participants, cnMAE maintains an unbiased context by providing capacity-normalized insights and reducing sensitivity to seasonality. This is important, because the impact of forecasting errors can be magnified by trading strategies — for instance, over-selling your solar generation in day-ahead markets, because your forecast error is unpredictable. At the same time, cnMAE is also ideal for power producers that have diversified solar portfolios, because it enables performance comparison between sites with different capacities. They can use the capacity-adjusted insights provided by cnMAE to better prioritize maintenance and operations. 

Solar forecasting is still nascent, so the faults with the common forecasting metrics haven’t proven disruptive on a wide scale. But we’re seeing rapidly increasing interest in our solar forecasts from both energy traders and power producers. Therefore, it’s time for the industry to coalesce around a single, well-suited measure of predictive error. In our experience, cnMAE is the best option. 

About the author

Sean Kelly is a veteran in the energy markets and Amperon’s CEO. Since 2005 Sean has been active in the evolving energy trading markets, working for Tenaska, Lehman, EDF, and E.On, as well as several family offices and proprietary trading firms. While at EDF, he led the transition of two nuclear plants (Nine Mile and Ginna) into the NYISO market, then led the buildout of E.On’s North American trading desk. 

Joined by a team of deep domain experts in energy trading and market risk management, he co-founded Bridge Energy Consulting (sold to Ablireo Energy in 2019), providing energy management services and solutions to clients looking for new ways to operate effectively in a fast-changing market. In 2017 he co-founded Amperon as a way of combining leading-edge thinking in energy forecasting with cutting-edge analytical technology.