It is widely recognized that the irregular satellites of the giant planets were captured from initially heliocentric orbits. However, the mechanism of capture and the source region from which they were captured both remain unknown. We present an optical color survey of 43 irregular satellites of the outer planets conducted using the LRIS camera on the 10 m telescope at the Keck Observatory in Hawaii. The measured colors are compared to other planetary bodies in search for similarities and differences that may reflect upon the origin of the satellites. We find that ultrared matter (with color index B-R>=1.6), while abundant in the Kuiper Belt and Centaur populations, is depleted from the irregular satellites. We also use repeated determinations of the absolute magnitudes to make a statistical estimate of the average shape of the irregular satellites. The data provide no evidence that the satellites and the main-belt asteroids are differently shaped, consistent with collisions as the major agent shaping both.