We present a rest-frame ultraviolet morphological analysis of 108 z~2.1 Ly{alpha} emitters (LAEs) in the Extended Chandra Deep Field South and compare it to a similar sample of 171 LAEs at z~3.1. Using Hubble Space Telescope images from the Galaxy Evolution from Morphology and SEDs survey, Great Observatories Origins Deep Survey, and Hubble Ultradeep Field, we measure size and photometric component distributions, where photometric components are defined as distinct clumps of UV-continuum emission. At both redshifts, >80% of LAEs have observed half-light radii <2kpc, but the median half-light radius rises from 0.95+/-0.04kpc at z=3.1 to 1.41+/-0.14kpc at z=2.1. A similar evolution is seen in the sizes of individual rest-UV components, but there is no evidence for evolution in the number of multi-component systems. In the z=2.1 sample, we see clear correlations between the size of an LAE and other physical properties derived from its spectral energy distribution (SED). LAEs are found to be larger for galaxies with higher stellar mass, star formation rate, and dust obscuration, but there is no evidence for a trend between equivalent width and half-light radius at either redshift. The presence of these correlations suggests that a wide range of objects are being selected by LAE surveys at z~2, including a significant fraction of objects for which a massive and moderately extended population of old stars underlies the young starburst giving rise to the Ly{alpha} emission.