In the framework of the Standard Model Extension (SME), we present improved constraints on anisotropic Lorentz invariance and Charge-Parity-Time (CPT) violation by searching for astrophysical signals of cosmic vacuum birefringence with broadband optical polarimetry of high redshift astronomical sources, including Active Galactic Nuclei and Gamma-Ray Burst afterglows. We generalize Kislat (Constraints on lorentz invariance violation from optical polarimetry of astrophysical objects. Symmetry, 10(11), 2018. ISSN 2073-8994. doi:10.3390/sym10110596), which studied the SME mass dimension d=4 case, to arbitrary mass dimension for both the CPT-even and CPT-odd cases. We then present constraints on all 10, 16, and 42 anisotropic birefringent SME coefficients for dimension d=4, d=5, and d=6 models respectively, using 7554 observations for odd d and 7376 observations for even d of 1278 unique sources on the sky, which, to our knowledge comprises the most complete catalog of optical polarization from extragalactic sources in the literature to date. Compared to the smaller sample of 44 and 45 broadband optical polarimetry observations analyzed in Kislat (Constraints on lorentz invariance violation from optical polarimetry of astrophysical objects. Symmetry, 10(11), 2018. ISSN 2073-8994. doi:10.3390/sym10110596) and Kislat et al. (2017 Phys. Rev. D, 95(8):083013, doi: 10.1103/PhysRevD.95.083013), our dimension d=4 and d=5 average constraints are more sensitive by factors of 35 and 10, corresponding to a reduction in allowed SME parameter space volume for these studies of 15 and 16 orders of magnitude, respectively. Constraints from individual lines of sight can be significantly stronger using spectropolarimetry, due to the steep energy dependence of birefringence effects at increasing mass dimension. Nevertheless, due to the increased number of observations and lines of sight in our catalog, our average d=4 and d=5 broadband constraints are within factors of 2 and 12 of previous constraints using spectropolarimetry from Kislat (Constraints on lorentz invariance violation from optical polarimetry of astrophysical objects. Symmetry, 10(11), 2018. ISSN 2073-8994. doi:10.3390/sym10110596) and Kislat et al. (2017, Phys. Rev. D 95(8):083013, doi: 10.1103/PhysRevD.95.083013), respectively, using an independent data set and an improved analysis method. By contrast, our anisotropic constraints on all 42 birefringent SME coefficients for d=6 are the first to be presented in the literature.