We present a Spitzer MIPS study of the decay of debris disk excesses at 24 and 70{mu}m for 255 stars of types F4-K2. We have used multiple tests, including consistency between chromospheric and X-ray activity and placement on the H-R diagram, to assign accurate stellar ages. Within this spectral type range, at 24{mu}m, 13.6%+/-2.8% of the stars younger than 1 Gyr have excesses at the 3{sigma} level or more, whereas none of the older stars do, confirming previous work. At 70{mu}m, 22.5%+/-3.6% of the younger stars have excesses at >=3{sigma} significance, whereas only 4.7_-2.2_^+3.7^% of the older stars do. To characterize the far-infrared behavior of debris disks more robustly, we doubled the sample by including stars from the DEBRIS and DUNES surveys. For the F4-K4 stars in this combined sample, there is only a weak (statistically not significant) trend in the incidence of far-infrared excess with spectral type (detected fractions of 21.9_-4.3_^+4.8^%, late F; 16.5_-3.3_^+3.9^%, G; and 16.9_-5.0_^+6.3^%, early K). Taking this spectral type range together, there is a significant decline between 3 and 4.5 Gyr in the incidence of excesses, with fractional luminosities just under 10^-5^. There is an indication that the timescale for decay of infrared excesses varies roughly inversely with the fractional luminosity. This behavior is consistent with theoretical expectations for passive evolution. However, more excesses are detected around the oldest stars than are expected from passive evolution, suggesting that there is late-phase dynamical activity around these stars.