J/A+A/633/A154ivo://CDS.VizieR/J/A+A/633/A154doi:10.26093/cds/vizier.36330154CDSLogan C.H.A.Fotopoulou S.2020-04-24T08:51:39Z2020-01-23T08:29:12ZCDS support team

CDS, Observatoire de Strasbourg, 11 rue de l'Universite, F-67000 Strasbourg, France

cds-question@unistra.frsurveysmorgan-keenan-classificationredshiftedphotometryClassification will be an important first step for upcoming surveys that will detect billions of new sources such as LSST and Euclid, as well as DESI, 4MOST and MOONS. The application of traditional methods of model fitting and colour-colour selections will face significant computational constraints, while machine-learning (ML) methods offer a viable approach to tackle datasets of that volume. While supervised learning methods can perform very well for classification tasks, the creation of representative and accurate training sets is a resource and time consuming task. We present a viable alternative using an unsupervised ML method to separate stars, galaxies and QSOs using photometric data. The heart of our work uses HDBSCAN to find the star, galaxy and QSO clusters in a multidimensional colour space. We optimized the hyperparameters and input attributes of three separate HDBSCAN runs, each to select a particular object class, and thus treat the output of each separate run as a binary classifier. We subsequently consolidate the output to give our final classifications, optimized on their F1 scores. We explore the use of Random Forest and PCA as part of the pre-processing stage for feature selection and dimensionality reduction. Using our dataset of ~50000 spectroscopically labelled objects we obtain an F1 score of 98.9, 98.9 and 93.13 respectively for star, galaxy and QSO selection using our unsupervised learning method. We find that careful attribute selection is a vital part of accurate classification with HDBSCAN. We applied our classification to a subset of the SDSS spectroscopic catalogue and demonstrate the potential of our approach in correcting misclassified spectra useful for DESI and 4MOST. Finally, we create a multiwavelength catalogue of 2.7 million sources using the KiDS, VIKING and ALLWISE surveys and publish corresponding classifications and photometric redshifts.2020A&A...633A.154Lhttps://cdsarc.cds.unistra.fr/viz-bin/cat/J/A+A/633/A154CatalogResearchIsServedByTAP VizieR generic serviceIsServedByConesearch servicerelated-toJ/A+A/619/A14 : Classification-aided zph estimation (Fotopoulou+, 2018)https://cds.unistra.fr/vizier-org/licences_vizier.htmlhttps://vizier.cds.unistra.fr/viz-bin/VizieR-2?-source=J/A+A/633/A154https://vizier.iucaa.in/viz-bin/VizieR-2?-source=J/A+A/633/A154http://vizieridia.saao.ac.za/viz-bin/VizieR-2?-source=J/A+A/633/A154https://vizier.cds.unistra.fr/viz-bin/votable?-source=J/A+A/633/A154https://vizier.iucaa.in/viz-bin/votable?-source=J/A+A/633/A154http://vizieridia.saao.ac.za/viz-bin/votable?-source=J/A+A/633/A154GETtext/xml+votablehttps://tapvizier.cds.unistra.fr/TAPVizieR/tapCone search capability for table J/A+A/633/A154/cpz (CPz catalogue with object classifications)https://vizier.cds.unistra.fr/viz-bin/conesearch/J/A+A/633/A154/cpz?https://vizier.iucaa.in/viz-bin/conesearch/J/A+A/633/A154/cpz?http://vizieridia.saao.ac.za/viz-bin/conesearch/J/A+A/633/A154/cpz?GETtext/xml+votable180.050000true34.41787-4.721270.005555555555555556Cone search capability for table J/A+A/633/A154/klabels (KiDSVW catalogue with object classifications)https://vizier.cds.unistra.fr/viz-bin/conesearch/J/A+A/633/A154/klabels?https://vizier.iucaa.in/viz-bin/conesearch/J/A+A/633/A154/klabels?http://vizieridia.saao.ac.za/viz-bin/conesearch/J/A+A/633/A154/klabels?GETtext/xml+votable180.050000true51.21843-31.500750.0055555555555555564/1028-1029 1033-1034 1365 1621-1622 1625-1626 1637 1641-1642 1685-1686 1689-1690 1706 2207 2223 2227 2229 2232-2234 2243-2244 2246 2248-2250 2911 2933-2934 2937 5/1024-1025 2050 4103 4107-4110 4120-4122 4124 4128 4130-4131 4140 4144 4160 4224 4420 4422 5459 5468-5469 6397-6399 6461-6463 6477-6479 6481-6483 6492-6494 6497-6499 6508-6510 6512-6514 6553-6558 6561-6563 6572-6574 6576-6578 6592-6594 6717 6719 6733-6735 6737-6739 6748-6750 6754-6755 6764 6766 6768-6770 6819 6828 6830 6848-6849 6912-6914 8823 8827 8887 8891 8903 8907 8913 8921-8922 8940 8944 8967 8971 8982-8983 8988-8990 9004 9008-9009 9088 10239 11262-11263 11639 11643 11719 11724-11725 11727-11730 11740 11745 11747 11753 11756 11758 11760 6/4104-4106 4108-4109 4112-4114 8192 8194-8195 8198 8204 8206 8224-8226 16406-16407 16409-16411 16421-16423 16425-16427 16445-16447 16492-16494 16500-16501 16504 16516-16517 16519 16564-16565 16568 16570 16580 16582 16584 16644 16648 16650 16900-16901 16904 17669 17671 17677 17679 17686 17692-17694 17712-17713 21829-21831 21833 21835 21860-21861 21863 21880-21881 21884-21885 25566-25567 25581-25583 25587 25823 25837 25839 25842-25843 25885-25887 25901-25903 25905 25907 25921-25923 25980-25982 26060-26062 26064-26066 26080-26082 26159 26173-26175 26189-26191 26209-26211 26236-26238 26241-26243 26300-26302 26316-26318 26320-26322 26336 26338 26380-26382 26384-26385 26400-26402 26847 26865 26867 26873 26911 26931 26945 26947 27004-27006 27013-27015 27060-27062 27084-27086 27088-27090 27104-27105 27221 27270-27271 27273-27275 27288 27290-27291 27316 27318-27319 27324-27325 27401 27404-27405 27408-27410 27660-27662 27664-27666 27680 27682 31402 34539-34540 34542-34543 34554 35287 35291 35303 35307 35543 35547 35559 35563 35611 35623 35627 35648-35649 35651 35660-35661 35663 35680 35682-35683 35692-35694 35696-35698 35700 35764 35768 35784 35863 35867 35879 35883 35920 35922-35923 35926 35964 36020-36021 36024 36040-36041 36044 36048 36128 36356 36360 40949-40951 40953-40955 45037-45039 45042-45043 45046 46549-46551 46553-46555 46565-46567 46569-46571 46869-46871 46873-46875 46885 46893-46895 46904-46906 46925-46927 46964 46968-46970 46979 46985 46987 47009 47021 47028 47036 47044-47045 47048https://cdsarc.cds.unistra.fr/viz-bin/moc/J/A+A/633/A154?format=asciiOpticaldefaultJ/A+A/633/A154/cpzCPz catalogue with object classificationsrecnoRecord number assigned by the VizieR team. Should Not be used for identification.meta.recordintidSpectroscopic redshift ID (G1)meta.id;meta.maincharRAJ2000Spectroscopic redshift right ascension (J2000) (G1)degpos.eq.ra;meta.mainindexedDEJ2000Spectroscopic redshift declination (J2000) (G1)degpos.eq.dec;meta.mainindexedzSpectroscopic redshift value (G1)src.redshiftdoubleHclass[0/3] Spectroscopic redshift classification (0=star, 1=galaxy, 3=QSO) (2)meta.code.classintumagu band total magnitudemagphot.mag;em.opt.Ufloate_umagu band total magnitude errormagstat.error;phot.mag;em.opt.Udoublegmagg band total magnitudemagphot.mag;em.opt.Bfloate_gmag? g band total magnitude errormagstat.error;phot.mag;em.opt.Bdoublenullablermagr band total magnitudemagphot.mag;em.opt.Rfloate_rmagr band total magnitude errormagstat.error;phot.mag;em.opt.Rdoubleimagi band total magnitudemagphot.mag;em.opt.Ifloate_imagi band total magnitude errormagstat.error;phot.mag;em.opt.Idoublezmagz band total magnitudemagphot.mag;em.opt.Ifloate_zmagz band total magnitude errormagstat.error;phot.mag;em.opt.IdoubleYmagY band total magnitudemagphot.mag;em.IR.Jfloate_YmagY band total magnitude errormagstat.error;phot.mag;em.opt.BfloatJmagJ band total magnitudemagphot.mag;em.IR.Jfloate_JmagJ band total magnitude errormagstat.error;phot.mag;em.IR.JfloatHmagH band total magnitudemagphot.mag;em.IR.Hfloate_HmagH band total magnitude errormagstat.error;phot.mag;em.IR.HfloatKmagK band total magnitudemagphot.mag;em.IR.Kfloate_KmagK band total magnitude errormagstat.error;phot.mag;em.IR.KfloatW1magW1 band total magnitudemagphot.mag;em.IR.3-4umfloate_W1magW1 band total magnitude errormagstat.error;phot.magfloatW2magW2 band total magnitudemagphot.mag;em.IR.4-8umfloate_W2magW2 band total magnitude errormagstat.error;phot.magfloatu3magu band 3 arcsecond magnitudemagphot.mag;em.opt.Ufloate_u3mag? u band 3 arcsecond magnitude errormagstat.error;phot.magdoublenullableg3magg band 3 arcsecond magnitudemagphot.mag;em.opt.Bfloate_g3magg band 3 arcsecond magnitude errormagstat.error;phot.magfloatr3magr band 3 arcsecond magnitudemagphot.mag;em.opt.Rfloate_r3magr band 3 arcsecond magnitude errormagstat.error;phot.magfloati3magi band 3 arcsecond magnitudemagphot.mag;em.opt.Ifloate_i3magi band 3 arcsecond magnitude errormagstat.error;phot.magfloatZ3magz band 3 arcsecond magnitudemagphot.mag;em.opt.Ifloate_Z3magz band 3 arcsecond magnitude errormagstat.error;phot.magfloatY3magY band 3 arcsecond magnitudemagphot.mag;em.IR.Jfloate_Y3magY band 3 arcsecond magnitude errormagstat.error;phot.magfloatJ3magJ band 3 arcsecond magnitudemagphot.mag;em.IR.Jfloate_J3magJ band 3 arcsecond magnitude errormagstat.error;phot.mag;em.IR.JfloatH3magH band 3 arcsecond magnitudemagphot.mag;em.IR.Hfloate_H3magH band 3 arcsecond magnitude errormagstat.error;phot.mag;em.IR.HfloatK3magK band 3 arcsecond magnitudemagphot.mag;em.IR.Kfloate_K3magK band 3 arcsecond magnitude errormagstat.error;phot.mag;em.IR.KfloatYhlr? Y band half light radius (HLR)arcsecphys.angSize;srcdoublenullableJhlr? J band half light radius (HLR)arcsecphys.angSize;srcdoublenullableHhlr? H band half light radius (HLR)arcsecphys.angSize;srcdoublenullableKhlr? K band half light radius (HLR)arcsecphys.angSize;srcdoublenullablePCAs1cPCA components, STAR HDBSCAN binary classifiers, 'optimal' method setup, colours used (pca_star_1_colours)meta.code.classdoublePCAs2cPCA components, STAR HDBSCAN binary classifiers, 'optimal' method setup, colours used (pca_star_2_colours)meta.code.classdoublePCAs3cPCA components, STAR HDBSCAN binary classifiers, 'optimal' method setup, colours used (pca_star_3_colours)meta.code.classdoublePCAg1cPCA components, GAL HDBSCAN binary classifiers, 'optimal' method setup, colours used (pca_gal_1_colours)meta.code.classdoublePCAg2cPCA components, GAL HDBSCAN binary classifiers, 'optimal' method setup, colours used (pca_gal_2_colours)meta.code.classdoublePCAg3cPCA components, GAL HDBSCAN binary classifiers, 'optimal' method setup, colours used (pca_gal_3_colours)meta.code.classdoublePCAq1cPCA components, QSO HDBSCAN binary classifiers, 'optimal' method setup, colours used (pca_qso_1_colours)meta.code.classdoublePCAq2cPCA components, QSO HDBSCAN binary classifiers, 'optimal' method setup, colours used (pca_qso_2_colours)meta.code.classdoublePCAq3cPCA components, QSO HDBSCAN binary classifiers, 'optimal' method setup, colours used (pca_qso_3_colours)meta.code.classdoubleClasscO[0/3] Consolidation method classification (0=outlier, 1=star, 2=gal, 3=QSO), colours used (hdbscanclass_optimal_method_colours)meta.code.classintCassscA[0/3] (0=outlier, 1=star, 2=gal, 3=QSO), colours used (hdbscanclass_alternative_method_colours)meta.code.classintdpc[0/1] Doubly positively classified objects, colours used (double_positives_colours)meta.codeintPoutcOutlier probability , colours used (outlier_probability_colours)stat.fit.goodnessfloatPstarcStar probability, colours used (star_probability_colours)stat.fit.goodnessfloatPgalcGalaxy probability, colours used (gal_probability_colours)stat.fit.goodnessfloatPqsocQSO probability, colours used (QSO_probability_colours)stat.fit.goodnessfloatLabelc[0/3] Final labels, 'highest probability' method (highest_probability_labels)meta.code.classintPCAs1cHLR? PCA components, STAR HDBSCAN binary classifiers, 'optimal' method setup, colours & HLR used (pca_star_1_colours+HLR)meta.code.classdoublenullablePCAs2cHLR? PCA components, STAR HDBSCAN binary classifiers, 'optimal' method setup, colours & HLR used (pca_star_2_colours+HLR)meta.code.classdoublenullablePCAs3cHLR? PCA components, STAR HDBSCAN binary classifiers, 'optimal' method setup, colours & HLR used (pca_star_3_colours+HLR)meta.code.classdoublenullablePCAg1cHLR? PCA components, GAL HDBSCAN binary classifiers, 'optimal' method setup , colours & HLR used (pca_gal_1_colours+HLR)meta.code.classdoublenullablePCAg2cHLR? PCA components, GAL HDBSCAN binary classifiers, 'optimal' method setup , colours & HLR used (pca_gal_2_colours+HLR)meta.code.classdoublenullablePCAg3cHLR? PCA components, GAL HDBSCAN binary classifiers, 'optimal' method setup , colours & HLR used (pca_gal_3_colours+HLR)meta.code.classdoublenullablePCAq1cHLR? PCA components, QSO HDBSCAN binary classifiers, 'optimal' method setup , colours & HLR used (pca_qso_1_colours+HLR)meta.code.classdoublenullablePCAq2cHLR? PCA components, QSO HDBSCAN binary classifiers, 'optimal' method setup , colours & HLR used (pca_qso_2_colours+HLR)meta.code.classdoublenullablePCAq3cHLR? PCA components, QSO HDBSCAN binary classifiers, 'optimal' method setup , colours & HLR used (pca_qso_3_colours+HLR)meta.code.classdoublenullableClasscHLRO[0/3]? Optimal consolidation method classification (0=outlier, 1=star, 2=gal, 3=QSO), colours & H LR used (hdbscanclass_optimal_method_colours+HLR)meta.code.classintnullableClasscHLRA[0/3]? Alternative consolidation method classification (0=outlier, 1=star, 2=gal, 3=QSO), colours & HLR used (hdbscanclass_alternative_method_colours+HLR)meta.code.classintnullabledpcHLR[0/1]? Doubly positively classified objects, colours & HLR used (double_positives_colours+HLR)meta.codeintnullablePoutcHLR? Outlier probability, colours & HLR used (outlier_probability_colours+HLRs)stat.fit.goodnessdoublenullablePstarcHLR? Star probability, colours & HLR used (star_probability_colours+HLRs)stat.fit.goodnessdoublenullablePgalcHLR? Galaxy probability, colours & HLR used (gal_probability_colours+HLRs)stat.fit.goodnessdoublenullablePqsocHLR? QSO probability, colours & HLR used (QSO_probability_colours+HLRs)stat.fit.goodnessdoublenullableLabelcHLR[0/3]? Final labels for when the 'highest probability' method is used, colours & HLR used (highest_probability_labels_colours+HLRs)meta.code.classintnullableJ/A+A/633/A154/klabelsKiDSVW catalogue with object classificationsidSpectroscopic redshift ID (G1)meta.id;meta.mainchare_imagi band total magnitude errormagstat.error;phot.mag;em.opt.IfloatrecnoRecord number assigned by the VizieR team. Should Not be used for identification.meta.recordintKiDSuKiDSVW identifier (serialid_kids_u_dr4v3)meta.idintRAJ2000Spectroscopic redshift right ascension (J2000)degpos.eq.ra;meta.mainindexedDEJ2000Spectroscopic redshift declination (J2000)degpos.eq.dec;meta.mainindexedzClass14Spectroscopic redshift classificationmeta.code.classcharzSDSS14? Spectroscopic redshift value from SDSS DR14src.redshift;pos.heliocentricdoublenullablezSDSSQ14? Spectroscopic redshift value from DR14Q (1)src.redshift;pos.heliocentricdoublenullableumagu band total magnitudemagphot.mag;em.opt.Ufloate_umagu band total magnitude errormagstat.error;phot.mag;em.opt.Ufloatgmagg band total magnitudemagphot.mag;em.opt.Bfloate_gmagg band total magnitude errormagstat.error;phot.mag;em.opt.Bfloatrmagr band total magnitudemagphot.mag;em.opt.Rfloate_rmagr band total magnitude errormagstat.error;phot.mag;em.opt.Rfloatimagi band total magnitudemagphot.mag;em.opt.Ifloatzmagz band total magnitudemagphot.mag;em.opt.Ifloate_zmagz band total magnitude errormagstat.error;phot.mag;em.opt.IdoubleYmagY band total magnitudemagphot.mag;em.IR.Jfloate_YmagY band total magnitude errormagstat.error;phot.mag;em.opt.BdoubleJmagJ band total magnitudemagphot.mag;em.IR.Jfloate_JmagJ band total magnitude errormagstat.error;phot.mag;em.IR.JdoubleHmagH band total magnitudemagphot.mag;em.IR.Hfloate_HmagH band total magnitude errormagstat.error;phot.mag;em.IR.HdoubleKmagK band total magnitudemagphot.mag;em.IR.Kfloate_KmagK band total magnitude errormagstat.error;phot.mag;em.IR.KdoubleW1magW1 band total magnitudemagphot.mag;em.IR.3-4umfloate_W1magW1 band total magnitude errormagstat.error;phot.magfloatW2magW2 band total magnitudemagphot.mag;em.IR.4-8umfloate_W2magW2 band total magnitude errormagstat.error;phot.magfloatu3magu band 3 arcsecond magnitudemagphot.mag;em.opt.Ufloate_u3mag? u band 3 arcsecond magnitude errormagstat.error;phot.magdoublenullableg3magg band 3 arcsecond magnitudemagphot.mag;em.opt.Bfloate_g3magg band 3 arcsecond magnitude errormagstat.error;phot.magfloatr3magr band 3 arcsecond magnitudemagphot.mag;em.opt.Rfloate_r3magr band 3 arcsecond magnitude errormagstat.error;phot.magfloati3magi band 3 arcsecond magnitudemagphot.mag;em.opt.Ifloate_i3magi band 3 arcsecond magnitude errormagstat.error;phot.magfloatz3magz band 3 arcsecond magnitudemagphot.mag;em.opt.Ifloate_z3magz band 3 arcsecond magnitude errormagstat.error;phot.magfloatY3magY band 3 arcsecond magnitudemagphot.mag;em.IR.Jfloate_Y3magY band 3 arcsecond magnitude errormagstat.error;phot.magfloatJ3magJ band 3 arcsecond magnitudemagphot.mag;em.IR.Jfloate_J3magJ band 3 arcsecond magnitude errormagstat.error;phot.mag;em.IR.JfloatH3magH band 3 arcsecond magnitudemagphot.mag;em.IR.Hfloate_H3magH band 3 arcsecond magnitude errormagstat.error;phot.mag;em.IR.HfloatK3magK band 3 arcsecond magnitudemagphot.mag;em.IR.Kfloate_K3magK band 3 arcsecond magnitude errormagstat.error;phot.mag;em.IR.KfloatPCAs1PCA components, STAR HDBSCAN binary classifiers, 'optimal' method setup (pca_star_1_)meta.code.classdoublePCAs2PCA components, STAR HDBSCAN binary classifiers, 'optimal' method setup (pca_star_2)meta.code.classdoublePCAs3PCA components, STAR HDBSCAN binary classifiers, 'optimal' method setup (pca_star_3)meta.code.classdoublePCAg1PCA components, GAL HDBSCAN binary classifiers, 'optimal' method setup (pca_gal_1)meta.code.classdoublePCAg2PCA components, GAL HDBSCAN binary classifiers, 'optimal' method setup (pca_gal_2)meta.code.classdoublePCAg3PCA components, GAL HDBSCAN binary classifiers, 'optimal' method setup (pca_gal_3)meta.code.classdoublePCAq1PCA components, QSO HDBSCAN binary classifiers, 'optimal' method setup (pca_qso_1)meta.code.classdoublePCAq2PCA components, QSO HDBSCAN binary classifiers, 'optimal' method setup (pca_qso_2)meta.code.classdoublePCAq3PCA components, QSO HDBSCAN binary classifiers, 'optimal' method setup (pca_qso_3)meta.code.classdoubleClassO[0/3] Optimal consolidation method classification (0=outlier, 1=star, 2=gal, 3=QSO), colours used (hdbscanclass_optimal_method)meta.code.classintCasssA[0/3] Alternative consolidation method classification (0=outlier, 1=star, 2=gal, 3=QSO), colours used (hdbscanclass_alternative_method)meta.code.classintdpc[0/3] Doubly positively classified objects, colours used (double_positives)meta.codeintPoutcOutlier probability, colours used (outlier_probability)stat.fit.goodnessfloatPstarcStar probability, colours used (star_probability)stat.fit.goodnessfloatPgalcGalaxy probability, colours used (gal_probability)stat.fit.goodnessfloatPqsocQSO probability, colours used (QSO_probability)stat.fit.goodnessfloatLabelc[0/3] Final labels, 'highest probability' method (highest_probability_labels)meta.code.classintzPredGPhotometric redshift predictions (GAL_PHOTOZ_PREDICTOR)stat.fit.paramdoublezPredQPhotometric redshift predictions (QSO_PHOTOZ_PREDICTOR)stat.fit.paramdoublephztrain[0/10]? Training/validation/test set labels for the photometric redshift predictions (see Sect. 7.3) (2)meta.codeintnullable