Castor is a system of six stars in which the two brighter objects, Castor A and B, revolve around each other every ~450yr and are both short-period spectroscopic binaries. They are attended by the more distant Castor C, which is also a binary. Here we report interferometric observations with the Center for High Angular Resolution Astronomy (CHARA) array that spatially resolve the companions in Castor A and B for the first time. We complement these observations with new radial velocity measurements of A and B spanning 30yr, with the Hipparcos intermediate data, and with existing astrometric observations of the visual AB pair obtained over the past three centuries. We perform a joint orbital solution to solve simultaneously for the three- dimensional orbits of Castor A and B as well as the AB orbit. We find that they are far from being coplanar: the orbit of A is nearly at right angles (92{deg}) relative to the wide orbit, and that of B is inclined about 59{deg} compared to AB. We determine the dynamical masses of the four stars in Castor A and B to a precision better than 1%. We also determine the radii of the primary stars of both subsystems from their angular diameters measured with the CHARA array, and use them together with stellar evolution models to infer an age for the system of 290Myr. The new knowledge of the orbits enables us to measure the slow motion of Castor C as well, which may assist future studies of the dynamical evolution of this remarkable sextuple system.