We explore the collisional decay of disk mass and infrared emission in debris disks. With models, we show that the rate of the decay varies throughout the evolution of the disks, increasing its rate up to a certain point, which is followed by a leveling off to a slower value. The total disk mass falls off {prop.to}t^-0.35^ at its fastest point (where t is time) for our reference model, while the dust mass and its proxy -the infrared excess emission- fades significantly faster ({prop.to}t^-0.8^). These later level off to a decay rate of M_tot_(t){prop.to}t^-0.08^ and M_dust_(t) or L_ir_(t){prop.to}t^-0.6^. This is slower than the {prop.to}t.^-1^ decay given for all three system parameters by traditional analytic models. We also compile an extensive catalog of Spitzer and Herschel 24, 70, and 100{mu}m observations. Assuming a log-normal distribution of initial disk masses, we generate model population decay curves for the fraction of stars harboring debris disks detected at 24{mu}m. We also model the distribution of measured excesses at the far-IR wavelengths (70-100{mu}m) at certain age regimes. We show general agreement at 24{mu}m between the decay of our numerical collisional population synthesis model and observations up to a Gyr. We associate offsets above a Gyr to stochastic events in a few select systems. We cannot fit the decay in the far-infrared convincingly with grain strength properties appropriate for silicates, but those of water ice give fits more consistent with the observations (other relatively weak grain materials would presumably also be successful). The oldest disks have a higher incidence of large excesses than predicted by the model; again, a plausible explanation is very late phases of high dynamical activity around a small number of stars. Finally, we constrain the variables of our numerical model by comparing the evolutionary trends generated from the exploration of the full parameter space to observations. Amongst other results, we show that erosive collisions are dominant in setting the timescale of the evolution and that planetesimals on the order of 100 km in diameter are necessary in the cascades for our population synthesis models to reproduce the observations.