We describe a joint high-contrast imaging survey for planets at the Keck and Very Large Telescope of the last large sample of debris disks identified by the Spitzer Space Telescope. No new substellar companions were discovered in our survey of 30 Spitzer-selected targets. We combine our observations with data from four published surveys to place constraints on the frequency of planets around 130 debris disk single stars, the largest sample to date. For a control sample, we assembled contrast curves from several published surveys targeting 277 stars that do not show infrared excesses. We assumed a double power-law distribution in mass and semimajor axis (SMA) of the form f(m,a)=Cm^{alpha}^{alpha}^{beta}^, where we adopted power-law values and logarithmically flat values for the mass and SMA of planets. We find that the frequency of giant planets with masses 5-20 M_Jup_ and separations 10-1000 au around stars with debris disks is 6.27% (68% confidence interval 3.68%-9.76%), compared to 0.73% (68% confidence interval 0.20%-1.80%) for the control sample of stars without disks. These distributions differ at the 88% confidence level, tentatively suggesting distinctness of these samples.