We studied the evolution of elliptical galaxies at z>~1 using a sample of 247 Extremely Red Objects (EROs) with R-Ks>3.35 (AB) and Ks<=22.1 (AB) constructed from BVRi'z'JHKs multicolor data of a 114arcmin^2^ area in the Subaru/XMM-Newton Deep Field taken with the Subaru Telescope and the UH 2.2m telescope. By fitting template spectra of old galaxies (OGs) and young, dusty starbursts (DSs) to the multicolor data, we classified EROs into these two classes and estimated their redshifts. We found that 58% of the EROs in our sample belong to the OG class and that these OGs have a wide range of colors at any redshift. We derived rest-frame B-band luminosity functions of OGs in our sample at z=1-1.5 and 1.5-2.5, and found that the number density of galaxies obeying passive evolution drops at z>~1.5 by a factor of three or more. The spatial correlation length of OGs is estimated to be r_0_>~8h^-1^Mpc, which is comparable to, or larger than, those of present-day early-type galaxies of similar luminosities. We also discussed properties of DSs in our data.