Sunday, October 14, 2007

Bias in Forecasting?

Among weather enthusiasts there is discussion from time to time about bias in forecasting. The theory is that meteorologists have a bias or a tendency to forecast weather towards one side of the spectrum or another. Among weather enthusiasts, there is an opinion that Hurricane Schwartz at NBC 10 is biased towards forecasting warmer weather and that Rob Guarino at Fox 29 is biased towards forecasting colder weather.

Is this necessarily the case?

I took the four temperature challenges I put together over the last couple of months and analyzed the data with the goal of looking for particular tendencies in forecasting towards colder or warmer weather by each of the participants in the temperature challenges. The dates of the sample series are July 7-11, 2007; July 17-21, 2007; August 31-September 4, 2007; and October 9-13, 2007. This covers four five-day forecasts and a total of 20 sets of temperature forecasts but is based on forecasts made during the same time period of day (the 4 to 6 PM cycle when the evening news is on, the National Weather Service forecasts are issued, and I typically issue my forecasts here). Below is a graphic that provides an explanation of how I laid out the data.
In the case up above, CBS 3 averaged a warmth bias of 0.9 degrees over 20 days of forecasting. On a breakdown of a five day forecast, CBS 3 forecasted warmer than the actual temperature on Days 1 through 4 of a five day forecast, with only Day 5 being colder. Out of all 20 days, CBS 3 went warmer than expected on 11 and colder on 6, with 3 days being forecasted correct.

How did the other weather forecasting outlets do? The results may surprise you.

click on above table to see all data in full size

NBC 10, over this four forecast sample, averaged the coldest compared to actual, ending up averaging -0.35 degrees colder per day forecast, hardly significant compared to actual but what surprised me is that over this four forecast sample they were the only forecast outlet to average colder than actual. Much of this was due to the recent pattern change that cooled us off. NBC 10 was colder than forecasted on 4 of the 5 days in the five-day forecast from October 9th. However, removing the colder forecasts from that time frame from this sample yields NBC 10 at a near zero bias (they only had a total sum of 2 degrees warmer than forecast among the other three forecast samples, still the coldest of the TV stations).

The below table shows the total sum, per day average warm or cold, and the number of days each outlet forecasted warmer than actual (+), colder (-), or just right.


CBS 3 6 ABC NBC 10 FOX 29 NWS PHLWX
Total
18 14 -7 9 16 6
Avg. Day 0.9 0.7 -0.35 0.45 0.8 0.3
Days +
11 13 8 10 12 9
Days -
6 4 9 7 6 5
Perfect 3 3 3 3 2 6

The warmest forecasts were at CBS 3, which averaged nearly 1 degree warmer than average. Other forecasters that averaged over 1/2 degree warmer than forecast were Channel 6 (Accuweather) and the National Weather Service. Only NBC 10 and Phillyweather.net did not forecast over 50% of the 20 days to be warmer than actual. The highest percentage of warm days was Channel 6 at 65%. The highest percentage of colder than actual days was NBC 10, at 45%.

Keep in mind that a forecast sample of four five-day forecasts is not terribly large but it does provide a fair amount of insight into how TV stations typically forecast temperatures and can provide a reasonable assessment into who is going warm and going cold more often than not. To be honest, I was a bit surprised to see NBC 10 come out as cold as they did compared to everyone else (2/3rds of a degree colder than the next forecaster) and was also surprised to see how five of the six forecast outlets were warmer than what actually happened.

Based on the data, does your perception of Hurricane Schwartz being a 'heat miser' and a 'warm monger' change somewhat? Those aren't my words...those have been uttered by others in various places over the years. After a couple of cold season temperature forecast challenges, I will be tempted to see how everyone is doing and how the numbers have changed, so don't be surprised to see another analysis like this in the future.

Technorati Tags: , , , , , , , ,