We present the rest-frame optical and UV surface brightness (SB) profiles for 149 galaxies with Mopt<-19.4mag at z=4-10 (29 of which are spectroscopically confirmed with JWST NIRSpec), securing high signal-to-noise ratios of 10-135 with deep JWST NIRCam 1-5um images obtained by the CEERS survey. We derive morphologies of our high-z galaxies, carefully evaluating the systematics of SB profile measurements with Monte Carlo simulations as well as the impacts of a) AGNs, b) multiple clumps including galaxy mergers, c) spatial resolution differences with previous HST studies, and d) strong emission lines, e.g., H{alpha} and [OIII], on optical morphologies with medium-band F410M images. Conducting Sersic profile fitting for our high-z galaxy SBs with GALFIT, we obtain effective radius ranges for optical re,opt and UV re,UV wavelengths of r_e,opt_=0.05-1.6kpc and r_e,UV_=0.03-1.7kpc that are consistent with previous results within large scatters in the size-luminosity relations. However, we find that the effective radius ratio, r_e,opt_/r_e,UV_, is almost unity, 1.01^+0.35^_-0.22_, over z=4-10 with no signatures of past inside- out star formation such as found at z~0-2. There are no spatial offsets exceeding 3{sigma} between the optical and UV morphology centers in cases of no mergers, indicative of major star-forming activity only being found near mass centers of galaxies at z>~4 probably experiencing the first phase of inside-out galaxy formation.