We determine probabilities of physical association for stars in blended Kepler Objects of Interest (KOIs), and find that 14.5%_-3.4%_^+3.8%^ of companions within ~4'' are consistent with being physically unassociated with their primary. This produces a better understanding of potential false positives in the Kepler catalog and will guide models of planet formation in binary systems. Physical association is determined through two methods of calculating multi-band photometric parallax using visible and near-infrared adaptive optics observations of 84 KOI systems with 104 contaminating companions within ~4''. We find no evidence that KOI companions with separations of less than 1'' are more likely to be physically associated than KOI companions generally. We also reinterpret transit depths for 94 planet candidates, and calculate that 2.6%+/-0.4% of transits have R>15R_{oplus}_, which is consistent with prior modeling work.