Arcuate mid-infrared nebulae known as stellar bow shock nebulae (SBNe) have been previously hypothesized to be supported by the strong stellar winds and/or luminosity of massive early-type stars. We present an optical spectroscopic survey of 84 stars identified from mid-infrared images as candidate SBN-supporting stars. Eighty-one of 84 sources, 96%, are O or early-B spectral types. K-band luminosities for a larger sample of 289 stars at the centers of bow shock nebulae are overwhelmingly consistent with OB stars. This affirms both that SBNe are supported by massive stars and that arcuate mid-infrared nebulae are reliable indicators of the presence of a massive star. The radial velocity dispersions of these systems and detections of double-lined systems indicate that at least 27 of the 74 systems with multiple observations (>36%) are candidate multiple-star systems. This rate is consistent with observed multiplicity rates of field OB stars detected with similar radial velocity surveys and lower than, but not inconsistent with, the multiplicity rates of OB stars in clusters and associations.