Philly has only dropped below 32 degrees in the month of May on two known occasions -- the first was May 1st, 1847 when temperatures bottomed out at 26 degrees according to old climate records. Prior to 1874, daily records aren't considered official since they aren't 100% accurate but there is the occasional report of temperatures thrown in with "icicles as long as arms, as thick as tree trunks" that tend to litter the diaries and weather almanacs of yore.
In the "modern" climate era, it's only happened once...and it is also the latest freeze on record. May 11th, 1966 featured a very cold late season high pressure center over the Mid Atlantic, buoyed by a departing strong low that was over Eastern Canada to help funnel cool air south one final time from Canada. The mid 1960's were a pretty cool era in general...and April and May 1966 were both among the ten coolest Aprils and Mays, respectively, on record in Philadelphia.
With high pressure overhead, light winds, and a cool airmass, temperatures bottomed out for many locations around the Delaware Valley. Record lows were set in a number of locations, including Philadelphia where the morning low was 28 degrees. This 28 is the modern era record for Philadelphia, as well as the record for the latest freeze on record in the city since records (official and unofficial) were first kept back in the 1800's.
Other morning lows around the region included 24 in Mount Pocono, 25 in AC, and 30's for a number of other locations. For the major climate sites such as Allentown, AC, Harrisburg, and Philly, these are record lows for the date.