We present a large, uniform analysis of young (~10-150Myr) ultracool dwarfs, based on new high-precision infrared (IR) parallaxes for 68 objects. We find that low-gravity (VL-G) late-M and L dwarfs form a continuous sequence in IR color-magnitude diagrams, separate from the field population and from current theoretical models. These VL-G objects also appear distinct from young substellar (brown dwarf and exoplanet) companions, suggesting that the two populations may have a different range of physical properties. In contrast, at the L/T transition, young, old, and spectrally peculiar objects all span a relatively narrow range in near-IR absolute magnitudes. At a given spectral type, the IR absolute magnitudes of young objects can be offset from ordinary field dwarfs, with the largest offsets occurring in the Y and J bands for late-M dwarfs (brighter than the field) and mid-/late-L dwarfs (fainter than the field). Overall, low-gravity (VL-G) objects have the most uniform photometric behavior, while intermediate gravity (INT-G) objects are more diverse, suggesting a third governing parameter beyond spectral type and gravity class. We examine the moving group membership for all young ultracool dwarfs with parallaxes, changing the status of 23 objects (including 8 previously identified planetary-mass candidates) and fortifying the status of another 28 objects. We use our resulting age-calibrated sample to establish empirical young isochrones and show a declining frequency of VL-G objects relative to INT-G objects with increasing age. Notable individual objects in our sample include high-velocity (>~100km/s) INT-G objects, very red late-L dwarfs with high surface gravities, candidate disk-bearing members of the MBM20 cloud and {beta} Pic moving group, and very young distant interlopers. Finally, we provide a comprehensive summary of the absolute magnitudes and spectral classifications of young ultracool dwarfs, using a combined sample of 102 objects found in the field and as substellar companions to young stars.