The Cosmic Dawn Survey Pre-Launch (PL) catalogues cover an effective 10.13deg^2^ area with uniform deep Spitzer/IRAC data (m~25mag, 5{sigma}), the largest area covered to these depths at infrared wavelengths. These data are used to gain new insight into the growth of stellar mass across cosmic history by characterising the evolution of the galaxy stellar mass function through 0.2<z<=6.5. The total volume (0.62Gpc^3^) represents an order of magnitude increase compared to previous works that have explored z>3 and significantly reduces cosmic variance, thus yielding strong constraints on the abundance of galaxies above the characteristic stellar mass (M*) across this ten billion year time period. The evolution of the galaxy stellar mass function is generally consistent with results from the literature but now provide firm estimates of number density where only upper limits were previously available. Contrasting the galaxy stellar mass function with the dark matter halo mass function suggests that massive galaxies (M~>10^11^M_{Sun}_) at z>3.5 required integrated star-formation efficiencies of M/(M_h_*f_b_)~>0.25-0.5, in excess of the commonlyheld view of 'universal peak efficiency' from studies on the stellar-to-halo mass relation. Such increased efficiencies imply an evolving peak in the stellar-to-halo mass relation at z>3.5 which can be maintained if feedback mechanisms from active galactic nuclei and stellar processes are ineffective at early times. In addition, a significant fraction of the most massive quiescent galaxies are observed to be in place already by z~2.5-3. The apparent lack in change of their number density by z~0.2 is consistent with relatively little mass growth from mergers. Utilising the unique volume, evidence for an environmental dependence of the galaxy stellar mass function is found all the way through z~3.5 for the first time, though a more careful characterisation of the density field is ultimately required for confirmation.