Stream stars removed by tides from their progenitor satellite galaxy or globular cluster act as a group of test particles on neighbouring orbits, probing the gravitational field of the Milky Way. While constraints from individual streams have been shown to be susceptible to biases, combining several streams from orbits with various distances reduces these biases. We fit a common gravitational potential to multiple stellar streams simultaneously by maximizing the clustering of the stream stars in action space. We apply this technique to members of the GD-1, Palomar 5 (Pal 5), Orphan, and Helmi streams, exploiting both the individual and combined data sets. We describe the Galactic potential with a Stackel model, and vary up to five parameters simultaneously. We find that we can only constrain the enclosed mass, and that the strongest constraints come from the GD-1, Pal 5, and Orphan streams whose combined data set yields M(<20kpc)=2.96^+0.25^_-0.26_x10^11^M_{sun}_. When including the Helmi stream in the data set, the mass uncertainty increases to M(<20kpc)=3.12^+3.21^_-0.46_x10^11^M_{sun}_.