Thursday, March 23, 2006

The Close of the 2005-2006 Forecast Challenge

The books have been closed on the 2005-2006 Philadelphia Weather Snow Forecast Challenge. Officially. Finally. En total. No mas.

"But, Tom, we really did not have a winter to begin with!!!"

Yes and no. It was not much of a winter outside of two or three decent events here in the Philadelphia area. Through the magic of the twelve inch snowfall on February 11th and 12th, however, the historical data will make it seem like it was a normal snowfall year here in the Delaware Valley at 19.4" (which is right around the winter average of 19.3"). We also had at least one inch of snow on the ground for twelve days, a little bit under the winter average of sixteen.

The Winter Forecast Challenge was a measure of two factors of accuracy -- how much and timing. While I did receive a couple of critiques stating that accuracy does not really matter when a snowfall starts at 3 AM. I would tend to disagree, it means that much more if it starts at 3 PM because of afterschool events and rush hour and even if it is forecasted to start at 3 AM and doesn't start until 7, schools may or may not cancel or delay because of the delay in the start of snowfall. Accuracy in amounts is quite obvious -- but for this winter I only used one measuring point (Philadelphia International Airport). Those factors are measured against each other based on the length of time range or snowfall range forecasted and points are assigned based on how far off the forecast is. The lower the score, much like golf, the better. We compared everyone on snowfall amounts and only the TV outlets on forecasted timing.

FINAL 2005-2006 Standings
FOX 29 2 2 5 7
CBS 3 2 14 0 14
6 ABC 0 3 14 17
PHILA WX 0 16 2 18
NBC 10 0 14 14 28
NWS PHI 0 11 N/A 11

The winner of our first Forecast Challenge for the Winter of 2005-2006 is FOX 29, who scored 7 total points for the winter. Rob Guarino and staff won mostly because of their accurate call on the February snowstorm, being the only TV outlet in Philadelphia to correctly forecast accumulations and timing on the event.

In second place, CBS 3, which was in the lead for most of the winter but dropped down into second place after the February snowstorm. To CBS' credit, they were the only outlet in Philadelphia to be perfect on forecasting WHEN snow would start for every snowfall we had this winter. Channel 6 finished in third because of timing. They were pretty close to FOX 29 on snowfall forecasts for the winter but were off on when snowfall would start, with the February snowstorm being one example. They were off about 3 hours.

I included myself for comparison and while I was accurate on timing, my snowfall totals varied. I busted badly on the December 9th event where Philadelphia only got 2.2" (forecasted 5-8") and that put me out of the running. In last place, NBC 10, which was in last for nearly the entire winter because of a bad timing call on the December 5th event (the Monday Night Football Snowfall). The Weather Service is in the middle of the pack when it comes to snowfall (finishing 3rd in that category) but since they do not give specific timing forecasts, they only get graded on snowfall. Like nearly everyone, the February 12th event was forecasted a little bit low so outside of that forecast, they did quite well on snow for the area.

In analyzing the first year of the Philadelphia Weather Forecast Challenge, I saw a few things that worked really well and a few things that I may adjust. I like the idea of comparing the media outlets against each other. Considering the claims you see on TV saying one channel is better than the other, there was not a grading system that compares the weight of their work. Since snowfall is something that impacts us more directly than say, being off by 3 degrees on a forecasted high, comparing their snowfall forecasts is one measure of seeing just how accurate a forecasting outlet (TV or government) really is. I would tend to argue that many of us do not care if a forecasted high of 75 is 3 degrees too high or low but do care if no snow was forecasted and 6" shows up on one's front door.

Additionally, I like the concept of weighing timing and accumulations equally. Like I stated before, I think that timing truly does matter in most hours of the day because it has an impact on a wide range of individuals -- from those involved with schools to the snowplow operators to those who work in Center City, a two or three hour variation from forecast to reality makes a big difference. I do see the point where a snowfall beginning at 3 AM impacts fewer people than one starting at 3 PM, perhaps a modification to the grading in timing can be made around when it starts, with additional weight given to rush hour (6 AM - 10 AM or 3 PM - 7 PM) snowfalls than to those outside of those time frames.

One thing that is difficult about this contest is using one location. I know in a couple of events one TV station had forecasted very high amounts for places east of Philadelphia and busted about 6" over what actually accumulated while that outlet had correctly forecasted the amount for the city. The contest's intent was to use Philadelphia as the base point and while snowfall may vary widely from town to town, using one location at least gives us a quick snapshot into how good a forecast may have been. I do believe a second forecast point, whether it be north, west, or even southeast of the city, may be one way to help improve the quality of this contest. It will never be a perfect science because it is using a finite number of geographic points, but adding a second location to the contest may help in holding accountability up as well as improving the quality of the contest.

Additionally, it would have been nice to see more snowfall events to track, not that I am a fan of snow despite my Midwest upbringing (I certainly am not a snow hound). Philadelphia only had three events this winter where over one inch of snow fell, with a three other events being less than 1" of accumulation. The lack of snowfall lead to high bust and loss potential for any one event for the course of this challenge. The pattern in place this winter just was NOT condusive to snowfall outside of the few events in December plus the big event in February. Hopefully, I will have better luck next year in that regard.

I want to thank everyone who followed along and want to thank those of you who contributed ideas early this winter. I appreciate the feedback greatly. If you have an idea or a suggestion for next winter's forecast challenge, feel free to drop me an email and let me know your thoughts and what you think could be improved. Like I said, this is an inexact science but it is a way for people to get a sense of how accurate one forecast outlet is in comparison to another. Congratulations to FOX 29 for winning the First (and not the last) Philadelphia Winter Forecast Challenge.