310.1. Photo-z access#

310_1_Photoz_access

310.1. Photo-z access and visualization¶

For the Rubin Science Platform at data.lsst.cloud.
Data Release: Data Preview 1
Container Size: Large
LSST Science Pipelines version: Release r29.2.0
Last verified to run: 2025-09-02
Repository: github.com/lsst/tutorial-notebooks

Learning objective: How to access, cross-match, and visualize the LSDB-formatted photo-z.

LSST data products: Object

Packages: lsdb

Credit: Originally developed by the Rubin Community Science team. Please consider acknowledging them if this notebook is used for the preparation of journal articles, software releases, or other notebooks.

Get Support: Everyone is encouraged to ask questions or raise issues in the Support Category of the Rubin Community Forum. Rubin staff will respond to all questions posted there.

1. Introduction¶

As documented in the SITCom (Systems Integration, Testing, and Commissioning) technical note "Initial studies of photometric redshifts with LSSTComCam from DP1" (SITCOMTN-154; Charles et al. 2025), members of the Rubin Commissioning Science Unit for photometric redshifts have generated photometric redshift (photo-z; PZ) estimates for Data Preview 1 (DP1) objects using a variety of PZ estimators.

The Science Unit team members have made their preliminary PZ results for DP1 objects available in LSDB-formatted files. LSDB (Large Scale Database) is a python tool for scalable analysis of large catalogs (query and cross-match). Built on top of Dask, LSDB uses the HATS (Hierarchical Adaptive Tiling Scheme) data format to efficiently perform spatial operations. Find LSDB tutorials for accessing Rubin data at lsdb.io/dp1. Additional information and explanations of different DP1 LSDB data products are available at data.lsdb.io. See also the photo-z page in the DP1 documentation.

These PZ estimates are not necessarily representative of the future LSST-provided PZ in the object table. This is due to several reasons.

The DP1 observations - and thus the DP1 Object photometry - are not fully representative of the future LSST observations (i.e., non-uniform depth across the field, different depths per filter, missing filters for some fields).
The input spectroscopic training sets (and other PZ estimator inputs) will be improved and refined in the future.
In the future, PZ results will be available as columns in the Object table; for DP1 they are only available as separate LSDB catalogs.
The choice of which PZ estimator(s) will be used to generate PZ for the LSST Object table remains to be determined.

See also "A Roadmap to Photometric Redshifts for the LSST Object Catalog" (DMTN-049).

This tutorial only demonstrates how to access and visualize the PZ estimates with LSDB. It does not provide instruction regarding the different PZ estimators or demonstrate how to generate PZ estimates.

Related tutorials: The 100-level tutorial on how to access LSDB-formatted files. The 100-level tutorials on the TAP (Table Access Protocol) service, the Butler, and displaying images with Firefly.

1.1. Import packages¶

Import the LSDB package to work with LSDB-formatted files, along with standard astronomy packages and LSST software for data access.

In [1]:

import lsdb
from astropy import units as u
from astropy.coordinates import SkyCoord
import matplotlib.pyplot as plt
import warnings

from lsst.rsp import get_tap_service
import lsst.afw.display as afwDisplay
from lsst.daf.butler import Butler

1.2. Define parameters and functions¶

Create an instance of the TAP service.

In [2]:

service = get_tap_service("tap")
assert service is not None

Suppress the user warning that will otherwise be printed by dask:

/opt/lsst/software/stack/conda/envs/lsst-scipipe-10.0.0/lib/python3.12/site-packages/dask/dataframe/dask_expr/_collection.py:1435: UserWarning: Dask currently has limited support for converting pandas extension dtypes to arrays. Converting double[pyarrow] to object dtype.

In [3]:

warnings.filterwarnings('ignore', category=UserWarning)

Set the backend for afwDisplay to be Firefly, and open the Firefly tab.

In [4]:

afwDisplay.setDefaultBackend("firefly")
afw_display = afwDisplay.Display(frame=1)

Create an instance of the Butler.

In [5]:

butler = Butler("dp1", collections="LSSTComCam/DP1")
assert butler is not None

2. Explore the PZ catalog¶

Open the read-only LSDB PZ catalog, which has been added by Rubin staff to the /rubin directory.

In [6]:

pz_cat = lsdb.open_catalog("/rubin/lsdb_data/object_photoz")

Display the results.

In [7]:

pz_cat

Out[7]:

lsdb Catalog object_photoz:

	coord_dec	coord_ra	g_cModelMag	g_cModelMagErr	g_gaap1p0Mag	g_gaap1p0MagErr	g_gaap3p0Mag	g_gaap3p0MagErr	g_kronMag	g_kronMagErr	g_psfMag	g_psfMagErr	g_sersicMag	g_sersicMagErr	i_cModelMag	i_cModelMagErr	i_gaap1p0Mag	i_gaap1p0MagErr	i_gaap3p0Mag	i_gaap3p0MagErr	i_kronMag	i_kronMagErr	i_psfMag	i_psfMagErr	i_sersicMag	i_sersicMagErr	objectId	r_cModelMag	r_cModelMagErr	r_gaap1p0Mag	r_gaap1p0MagErr	r_gaap3p0Mag	r_gaap3p0MagErr	r_kronMag	r_kronMagErr	r_psfMag	r_psfMagErr	r_sersicMag	r_sersicMagErr	u_cModelMag	u_cModelMagErr	u_gaap1p0Mag	u_gaap1p0MagErr	u_gaap3p0Mag	u_gaap3p0MagErr	u_kronMag	u_kronMagErr	u_psfMag	u_psfMagErr	u_sersicMag	u_sersicMagErr	y_cModelMag	y_cModelMagErr	y_gaap1p0Mag	y_gaap1p0MagErr	y_gaap3p0Mag	y_gaap3p0MagErr	y_kronMag	y_kronMagErr	y_psfMag	y_psfMagErr	y_sersicMag	y_sersicMagErr	z_cModelMag	z_cModelMagErr	z_gaap1p0Mag	z_gaap1p0MagErr	z_gaap3p0Mag	z_gaap3p0MagErr	z_kronMag	z_kronMagErr	z_psfMag	z_psfMagErr	z_sersicMag	z_sersicMagErr	lephare_z_median	lephare_z_mean	lephare_z_mode	lephare_z_err95_low	lephare_z_err95_high	lephare_z_err68_low	lephare_z_err68_high	knn_z_median	knn_z_mode	knn_z_err95_low	knn_z_err95_high	knn_z_err68_low	knn_z_err68_high	tpz_z_median	tpz_z_mean	tpz_z_mode	tpz_z_err95_low	tpz_z_err95_high	tpz_z_err68_low	tpz_z_err68_high	cmnn_z_median	cmnn_z_mean	cmnn_z_mode	cmnn_z_err95_low	cmnn_z_err95_high	cmnn_z_err68_low	cmnn_z_err68_high	gpz_z_median	gpz_z_mean	gpz_z_mode	gpz_z_err95_low	gpz_z_err95_high	gpz_z_err68_low	gpz_z_err68_high	bpz_z_median	bpz_z_mean	bpz_z_mode	bpz_z_err95_low	bpz_z_err95_high	bpz_z_err68_low	bpz_z_err68_high	dnf_z_median	dnf_z_mean	dnf_z_mode	dnf_z_err95_low	dnf_z_err95_high	dnf_z_err68_low	dnf_z_err68_high	fzboost_z_median	fzboost_z_mean	fzboost_z_mode	fzboost_z_err95_low	fzboost_z_err95_high	fzboost_z_err68_low	fzboost_z_err68_high
npartitions=4
Order: 3, Pixel: 2	double[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	int64[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]
Order: 5, Pixel: 4471	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
Order: 2, Pixel: 80	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
Order: 0, Pixel: 8	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...

130 out of 130 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema

Lazily loaded catalogs: note the message under the displayed table above, that all of the columns have been loaded "lazily". This is always the default for LSDB catalogs, and it means that only the metadata is loaded at first. This way, LSDB can plan how tasks will be executed in the future without actually doing any computation.

2.1. Column names¶

The columns of the LSDB PZ catalog include:

the objectId, coord_ra, coord_dec
photometry measurements for each filter f_ (ugrizy):
- f_cModelMag and f_cModelMagErr
- f_gaap1p0Mag and f_gaap1p0MagErr
- f_gaap3p0Mag and f_gaap3p0MagErr
- f_kronMag and f_kronMagErr
- f_psfMag and f_psfMagErr
standardized outputs from the PZ estimators

Option to display all 130 column names.

In [8]:

# pz_cat.all_columns

Print the names of each of the PZ estimators.

In [9]:

for col in pz_cat.columns:
    if col.find('z_median') >= 0:
        print(col.split('_')[0])

lephare
knn
tpz
cmnn
gpz
bpz
dnf
fzboost

The same standardized PZ point-estimates are provided for all PZ codes.

For a given PZ estimator, show all columns associated with it (use LePhare for this example).

In [10]:

for col in pz_cat.columns:
    if col.find('lephare') >= 0:
        print(col)

lephare_z_median
lephare_z_mean
lephare_z_mode
lephare_z_err95_low
lephare_z_err95_high
lephare_z_err68_low
lephare_z_err68_high

2.2. Sky partitions¶

When pz_cat is displayed (above) there are four rows, one for each of the partitions of the LSDB PZ catalog. These partitions are how the LSDB-formatted files are stored, with each partition typically having about the same number of objects.

Show the four partitions of pz_cat on the sky. This is not necessarily the same as a sky coverage map, but rather shows the file's polygonal partition boundaries.

In [11]:

fig = pz_cat.plot_pixels(plot_title="Sky Partition Map")

No description has been provided for this image

Figure 1: The sky partitions of the lazily-loaded PZ catalog.

2.3. Table data¶

Show the first 10 rows of the PZ catalog.

In [12]:

pz_cat.head(n=10)

Out[12]:

	coord_dec	coord_ra	g_cModelMag	g_cModelMagErr	g_gaap1p0Mag	g_gaap1p0MagErr	g_gaap3p0Mag	g_gaap3p0MagErr	g_kronMag	g_kronMagErr	g_psfMag	g_psfMagErr	g_sersicMag	g_sersicMagErr	i_cModelMag	i_cModelMagErr	i_gaap1p0Mag	i_gaap1p0MagErr	i_gaap3p0Mag	i_gaap3p0MagErr	i_kronMag	i_kronMagErr	i_psfMag	i_psfMagErr	i_sersicMag	i_sersicMagErr	objectId	r_cModelMag	r_cModelMagErr	r_gaap1p0Mag	r_gaap1p0MagErr	r_gaap3p0Mag	r_gaap3p0MagErr	r_kronMag	r_kronMagErr	r_psfMag	r_psfMagErr	r_sersicMag	r_sersicMagErr	u_cModelMag	u_cModelMagErr	u_gaap1p0Mag	u_gaap1p0MagErr	u_gaap3p0Mag	u_gaap3p0MagErr	u_kronMag	u_kronMagErr	u_psfMag	u_psfMagErr	u_sersicMag	u_sersicMagErr	y_cModelMag	y_cModelMagErr	y_gaap1p0Mag	y_gaap1p0MagErr	y_gaap3p0Mag	y_gaap3p0MagErr	y_kronMag	y_kronMagErr	y_psfMag	y_psfMagErr	y_sersicMag	y_sersicMagErr	z_cModelMag	z_cModelMagErr	z_gaap1p0Mag	z_gaap1p0MagErr	z_gaap3p0Mag	z_gaap3p0MagErr	z_kronMag	z_kronMagErr	z_psfMag	z_psfMagErr	z_sersicMag	z_sersicMagErr	lephare_z_median	lephare_z_mean	lephare_z_mode	lephare_z_err95_low	lephare_z_err95_high	lephare_z_err68_low	lephare_z_err68_high	knn_z_median	knn_z_mode	knn_z_err95_low	knn_z_err95_high	knn_z_err68_low	knn_z_err68_high	tpz_z_median	tpz_z_mean	tpz_z_mode	tpz_z_err95_low	tpz_z_err95_high	tpz_z_err68_low	tpz_z_err68_high	cmnn_z_median	cmnn_z_mean	cmnn_z_mode	cmnn_z_err95_low	cmnn_z_err95_high	cmnn_z_err68_low	cmnn_z_err68_high	gpz_z_median	gpz_z_mean	gpz_z_mode	gpz_z_err95_low	gpz_z_err95_high	gpz_z_err68_low	gpz_z_err68_high	bpz_z_median	bpz_z_mean	bpz_z_mode	bpz_z_err95_low	bpz_z_err95_high	bpz_z_err68_low	bpz_z_err68_high	dnf_z_median	dnf_z_mean	dnf_z_mode	dnf_z_err95_low	dnf_z_err95_high	dnf_z_err68_low	dnf_z_err68_high	fzboost_z_median	fzboost_z_mean	fzboost_z_mode	fzboost_z_err95_low	fzboost_z_err95_high	fzboost_z_err68_low	fzboost_z_err68_high
_healpix_29
9743840754662961	7.384259	37.691623	24.47764	0.099679	24.495476	0.10725	24.706463	0.324693	24.424726	0.191889	24.620298	0.100349	24.405224	0.097277	23.635216	0.081684	23.707817	0.090896	24.194178	0.354615	<NA>	-0.74688	23.78896	0.08238	23.599329	0.082486	648368263004160627	23.779001	0.070325	23.785612	0.074787	23.627234	0.16011	23.574217	0.124147	23.916664	0.070193	23.683573	0.067429	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	23.254601	0.195918	23.410389	0.229063	<NA>	-2.071935	23.328369	0.264754	23.409246	0.200028	23.216286	0.198179	0.472849	1.006352	0.26	0.025956	2.928256	0.189219	2.262542	0.354946	0.35	0.279976	1.448482	0.323445	0.416408	0.929378	1.15502	0.76	0.285381	2.759384	0.428183	1.962157	0.346	0.346	0.35	-1.86976	2.56176	-0.778245	1.470245	1.053337	1.053337	1.05	-1.058609	3.165283	-0.018235	2.124908	0.3664	0.427425	0.37	0.095007	1.29707	0.217904	0.510234	0.376226	0.632773	0.35	0.250174	1.930399	0.307735	1.8138	1.470184	1.688171	1.45	0.190596	2.899186	1.213509	2.837311
9790364208974412	7.384505	37.825903	24.323572	0.122887	24.426447	0.118917	24.009132	0.200507	24.064924	0.105405	24.801682	0.14209	24.040373	0.122416	22.049297	0.031457	22.446346	0.03327	22.07099	0.059301	22.047583	0.061915	22.662476	0.032939	21.972223	0.031251	648368263004160629	23.167933	0.067988	23.545584	0.069262	23.168234	0.121379	23.146521	0.122287	23.708218	0.069962	23.054098	0.067247	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	21.436895	0.057024	21.866301	0.059338	21.593569	0.109709	21.498554	0.08717	21.961704	0.05829	21.360699	0.056965	0.797163	0.79826	0.79	0.608637	0.971621	0.706346	0.888711	0.870491	0.83	0.584325	1.347395	0.787742	1.300438	0.856566	0.850866	0.87	0.671112	1.057048	0.751866	0.93037	0.822401	0.822401	0.82	0.416927	1.227875	0.616669	1.028133	0.845662	0.845662	0.85	0.222868	1.468456	0.529665	1.161659	0.851417	0.900758	0.83	0.718107	1.469	0.790875	0.934627	0.88185	1.137106	0.84	0.749974	2.416718	0.805785	2.290113	0.909653	0.932351	0.89	0.77628	1.305105	0.846703	1.019736
9790638736070111	7.384602	37.735987	23.970366	0.105929	24.406008	0.111449	24.471415	0.292834	24.404013	0.180138	24.458973	0.10539	24.182455	0.106072	22.512117	0.038572	22.779631	0.042455	22.64044	0.093108	22.613405	0.052021	22.959467	0.041555	22.580866	0.038756	648368263004160630	22.92972	0.051049	23.229658	0.053703	23.201448	0.127753	23.098104	0.086705	23.35681	0.052465	23.022087	0.050369	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	22.398613	0.134553	22.818617	0.152102	22.333656	0.236013	22.70363	0.208627	22.972351	0.1448	22.598732	0.131582	0.422185	0.403456	0.5	0.11698	0.65061	0.222251	0.540888	3.419299	0.47	0.400706	3.777948	0.454588	3.503128	0.5644	0.576293	0.55	0.369873	0.917966	0.457975	0.671103	0.369522	0.369522	0.37	-1.716102	2.455146	-0.688694	1.427738	1.199712	1.199712	1.2	-1.098293	3.497717	0.033737	2.365687	0.367587	0.370837	0.34	0.191886	0.555351	0.266064	0.478066	0.5184	0.540249	0.5	0.390801	0.790178	0.447903	0.637092	0.489836	1.124163	0.48	0.053658	2.590033	0.118107	2.546751
9790627285516776	7.384733	37.755	23.1555	0.035596	23.304375	0.038716	23.136683	0.082227	23.129242	0.064071	23.412075	0.035951	23.139175	0.035402	20.933046	0.009113	21.115335	0.010333	20.94533	0.022117	20.913446	0.015869	21.246553	0.009288	20.946211	0.009019	648368263004160632	21.652712	0.013236	21.794811	0.014329	21.653646	0.031114	21.641132	0.023259	21.939974	0.013548	21.644455	0.013149	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	20.698534	0.023066	20.843676	0.022691	20.837675	0.054656	20.749979	0.038387	20.914526	0.023367	20.657722	0.022809	0.444585	0.439153	0.45	0.324077	0.543099	0.373301	0.495726	0.493137	0.52	0.294289	0.978441	0.409479	0.892939	0.53489	0.533961	0.54	0.450159	0.610709	0.494682	0.573567	0.408933	0.408933	0.41	-4.994295	5.812161	-2.332588	3.150454	0.512443	0.512443	0.51	-0.309976	1.334863	0.095159	0.929727	0.414421	0.416193	0.41	0.300257	0.531833	0.350979	0.485581	0.485808	0.486254	0.48	0.372991	0.601612	0.427819	0.544872	0.512022	0.501902	0.5	0.251083	0.66819	0.454493	0.587153
9790279296366971	7.384423	37.809984	24.615427	0.172167	24.928499	0.188339	24.570839	0.338234	24.653318	0.345533	25.057758	0.17673	24.446444	0.179831	23.529343	0.122173	23.789467	0.130543	23.367281	0.223613	23.094378	0.226578	24.024485	0.127906	23.335236	0.12472	648368263004160633	23.973753	0.123115	24.069269	0.117049	23.649416	0.196319	23.3545	0.198215	24.382648	0.133568	23.622393	0.118455	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	23.935442	0.454849	24.155045	0.479088	23.188274	0.467378	23.151632	0.505068	24.256157	0.4698	23.75399	0.510389	0.613002	0.992934	0.21	0.048328	2.897712	0.1882	2.08063	3.514192	0.67	0.645225	3.737598	3.120424	3.603326	0.863754	1.175022	0.55	0.230593	2.671155	0.494291	2.323624	3.2135	3.2135	3.0	1.304249	5.122751	2.244773	4.182227	1.951961	1.951961	1.95	-0.548775	4.452697	0.683123	3.220799	0.393436	0.464736	0.37	0.097529	1.39132	0.227324	0.575535	0.496043	0.50075	0.48	0.374829	0.638435	0.430447	0.573635	2.418674	2.330559	2.42	0.214242	2.990911	2.359628	2.482334
9790616261652049	7.384337	37.751818	25.832487	0.470591	25.951944	0.44628	24.928364	0.430948	24.77503	0.489589	26.337721	0.532336	25.623697	0.537655	24.35647	0.207523	24.58601	0.216237	25.09276	0.863069	24.364311	0.211174	24.78595	0.214844	23.971445	0.204806	648368263004160636	24.283621	0.165027	24.748137	0.209428	26.476265	2.535438	<NA>	<NA>	24.637386	0.160398	23.996178	0.175366	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	28.302862	22.364232	25.161079	1.07498	<NA>	-2.791267	24.75342	1.505622	<NA>	-1.520818	26.415966	6.004965	0.744231	1.292686	0.48	0.048539	2.960606	0.250115	2.681197	3.559888	0.67	0.637026	3.79985	3.190044	3.728092	2.085772	1.866416	2.11	0.400563	2.978456	1.06306	2.824998	2.7685	2.7685	2.77	0.909484	4.627516	1.825262	3.711738	2.808421	2.808421	2.81	0.412661	5.204181	1.592847	4.023996	0.490088	0.689687	0.42	0.113198	2.844637	0.271608	1.169013	2.991679	4.266103	3.0	2.961297	2.999584	2.980267	2.997337	2.763767	2.554834	2.78	0.685746	2.991619	2.072407	2.818326
9790617836385051	7.385155	37.749581	24.37381	0.08859	24.328342	0.093959	23.606466	0.1198	23.733315	0.152092	24.400892	0.088177	24.132309	0.087374	23.134724	0.048888	23.105486	0.05495	22.860153	0.109708	22.852711	0.102266	23.160114	0.048328	22.889015	0.048677	648368263004160640	23.727949	0.071234	23.701004	0.079158	23.538275	0.167949	23.572071	0.116552	23.763842	0.071115	23.487066	0.070322	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	23.241261	0.177326	23.311413	0.208128	23.657236	0.693455	<NA>	<NA>	23.300556	0.18045	23.055115	0.183131	0.581044	0.586582	0.63	0.081766	1.710872	0.185411	0.756478	0.615067	0.61	0.417676	3.7329	0.549609	0.759773	0.608772	0.655854	0.7	0.191081	1.475139	0.430759	0.786609	0.6763	0.6763	0.68	0.526736	0.825864	0.600413	0.752187	1.586832	1.586832	1.59	-0.478088	3.651752	0.539121	2.634543	0.640628	0.605225	0.66	0.111541	0.942731	0.40044	0.729276	0.627654	0.633005	0.61	0.457661	0.809421	0.540863	0.733827	0.716682	1.018554	0.66	0.575277	2.819124	0.627577	1.854836
9790622415457070	7.385424	37.75799	24.574924	0.118248	24.713509	0.132519	24.543406	0.281766	24.62748	0.254634	24.788721	0.118956	24.281572	0.123912	24.150602	0.149332	24.280024	0.171959	24.000349	0.333331	22.933916	0.405029	24.409283	0.152889	23.798534	0.14811	648368263004160658	24.599567	0.190138	24.809601	0.199543	23.910538	0.215586	23.537251	0.214479	24.878414	0.180055	24.326792	0.182068	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	23.999357	0.486642	24.231352	0.51067	24.933985	2.361857	24.219395	0.617193	24.215994	0.445176	23.716112	0.465094	1.637913	1.563318	0.07	0.067268	2.862164	0.754262	2.346194	1.035881	0.98	0.881812	2.612356	0.936167	1.64532	1.148833	1.295246	1.05	0.137335	2.268187	0.87005	1.962945	1.310033	1.310033	1.31	-0.18043	2.800496	0.553793	2.066273	1.614456	1.614456	1.61	-0.37422	3.603133	0.60543	2.623482	1.552825	1.442473	1.65	0.139099	2.427413	0.822388	1.989213	1.002449	1.044105	0.97	0.83942	1.390764	0.909621	1.262529	1.210635	1.408404	0.95	0.123849	2.453285	0.932286	2.014589
9790343387330549	7.386252	37.837035	22.19738	0.020662	22.62575	0.02131	22.514151	0.047693	22.475031	0.022535	22.953592	0.024043	22.242985	0.020655	20.62064	0.008053	21.051832	0.008189	20.80867	0.016387	21.52709	0.133611	21.402493	0.009335	20.665747	0.008046	648368263004160663	21.36224	0.013176	21.774237	0.013252	21.519627	0.026121	22.996588	0.424257	22.152771	0.015651	21.400103	0.013108	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	20.417246	0.026422	20.847218	0.026578	20.648401	0.051643	23.193117	2.417434	21.048414	0.02915	20.457617	0.026484	0.652092	0.650277	0.66	0.582235	0.709732	0.617806	0.681616	0.659332	0.67	0.579854	0.727309	0.621115	0.705149	0.686002	0.6744	0.69	0.525945	0.75776	0.630787	0.72337	0.623962	0.623962	0.62	0.163239	1.084686	0.390198	0.857727	0.515723	0.515723	0.52	-0.463822	1.495269	0.018716	1.012731	0.635304	0.638973	0.62	0.570169	0.71987	0.599518	0.680132	0.664494	0.663854	0.67	0.553116	0.771057	0.608293	0.719209	0.656243	0.656171	0.66	0.604107	0.707484	0.625943	0.686394
9790296873661101	7.386119	37.788965	24.485449	0.134638	24.79792	0.153138	24.471313	0.281977	24.861954	0.210167	24.862902	0.135645	24.473808	0.135871	23.496103	0.089996	23.665995	0.095832	23.318987	0.175301	23.36842	0.157012	23.902096	0.095898	23.414433	0.089416	648368263004160664	23.824345	0.094061	24.072723	0.101228	23.510471	0.149238	23.559093	0.196361	24.204102	0.096309	23.779049	0.093401	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	23.226875	0.259967	23.529587	0.266157	24.254639	1.255442	<NA>	<NA>	23.653784	0.260904	23.328705	0.273549	0.643132	0.938137	0.25	0.042531	2.801782	0.200192	1.79175	0.538206	0.56	0.100845	0.618302	0.201477	0.577386	0.680599	0.919958	0.65	0.306137	2.685081	0.574863	1.366323	0.559472	0.559472	0.56	-0.796591	1.915534	-0.128575	1.247518	1.146525	1.146525	1.15	-1.258746	3.551796	-0.073876	2.366925	0.405467	0.474019	0.28	0.076247	1.376711	0.193282	0.676165	0.579402	0.679277	0.55	0.456545	1.414136	0.51203	0.709834	1.445796	1.419607	2.5	0.050229	2.979194	0.398084	2.513348

10 rows × 130 columns

2.4. Plot N(pz)¶

$N(pz)$ refers to a histogram of the number of objects, $N$, as a function of photo-z, $pz$.

In order to find out and visualize which fields have PZ, and get an initial sense of how the PZ results vary by estimator, for each DP1 field plot the histogram N(pz) for $0<pz<3$ for each PZ estimator.

Define the field names and central coordinates in degrees.

In [13]:

field_centers = {"47 Tuc": [6.02, -72.08],
                 "SV 38 7": [37.86, 6.98],
                 "Fornax": [40.00, -34.45],
                 "ECDFS": [53.13, -28.10],
                 "EDFS": [59.10, -48.73],
                 "SV 95 -25": [95.00, -25.00],
                 "Seagull": [106.23, -10.51]}

Retrieve the names of the 8 PZ estimators with a median PZ column and store them in array all_pz_codes.

In [14]:

all_pz_codes = []
for col in pz_cat.columns:
    if col.find('z_median') >= 0:
        all_pz_codes.append(col.split('_')[0])

Create the histograms. Plots with no data (a flat histogram) represent fields without PZ.

In [15]:

for field in field_centers.keys():
    coords = field_centers[field]
    pz_cat_cone = pz_cat.cone_search(ra=coords[0], dec=coords[1],
                                     radius_arcsec=2.0 * 3600.0)
    fig, ax = plt.subplots(2, 4, figsize=(10, 4))
    ze = 0
    for i in range(2):
        for j in range(4):
            pz_code = all_pz_codes[ze]
            tx = (pz_cat_cone[pz_code+'_z_median'] > 0) & \
                 (pz_cat_cone[pz_code+'_z_median'] < 3)
            ax[i, j].hist(pz_cat_cone[pz_code+'_z_median'][tx],
                          bins=100, histtype='step', color='black')
            ax[i, j].set_title(all_pz_codes[ze])
            ax[i, j].set_xlim([0, 3])
            ze += 1
            del pz_code, tx
    plt.suptitle(field)
    plt.tight_layout()
    del coords, pz_cat_cone, fig

Figure 2: Histograms of the median PZ for each field, and each PZ estimator. The Seagull Nebula, the Fornax dwarf galaxy, and the 47 Tuc globular cluster fields do not have PZ estimates for their objects, but the rest do. The N(z) distributions for each PZ estimator are different.

Clean up before Section 3.

In [16]:

del field_centers, pz_cat

3. Abell 360¶

Use the known rich galaxy cluster Abell 360 in the Low Ecliptic Latitude Field (also called Rubin SV 38 7) to demonstrate addition functionality of the LSDB PZ catalogs.

Define the central coordinates (RA, Dec) of the brightest cluster galaxy for Abell 360 (in degrees), and its known redshift ($z=0.22$).

In [17]:

ra_a360 = 37.865
dec_a360 = 6.982
z_a360 = 0.22

3.1. Select columns¶

Select 12 of the 130 columns to retrieve.

In [18]:

use_columns = ["objectId", "r_cModelMag", "i_cModelMag", "z_cModelMag"]
for code in all_pz_codes:
    use_columns.append(code + '_z_median')
use_columns

Out[18]:

['objectId',
 'r_cModelMag',
 'i_cModelMag',
 'z_cModelMag',
 'lephare_z_median',
 'knn_z_median',
 'tpz_z_median',
 'cmnn_z_median',
 'gpz_z_median',
 'bpz_z_median',
 'dnf_z_median',
 'fzboost_z_median']

Open the catalog using only the selected columns.

In [19]:

pz_cat = lsdb.open_catalog("/rubin/lsdb_data/object_photoz",
                           columns=use_columns)

In [20]:

pz_cat

Out[20]:

lsdb Catalog object_photoz:

	objectId	r_cModelMag	i_cModelMag	z_cModelMag	lephare_z_median	knn_z_median	tpz_z_median	cmnn_z_median	gpz_z_median	bpz_z_median	dnf_z_median	fzboost_z_median	coord_ra	coord_dec
npartitions=4
Order: 3, Pixel: 2	int64[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]
Order: 5, Pixel: 4471	...	...	...	...	...	...	...	...	...	...	...	...	...	...
Order: 2, Pixel: 80	...	...	...	...	...	...	...	...	...	...	...	...	...	...
Order: 0, Pixel: 8	...	...	...	...	...	...	...	...	...	...	...	...	...	...

14 out of 130 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema

3.2. Cone search¶

Execute a cone search on pz_cat using the coordinates of Abell 360 and a radius of 0.1 deg.

In [21]:

pz_cat_a360 = pz_cat.cone_search(ra=ra_a360, dec=dec_a360,
                                 radius_arcsec=0.1 * 3600.0)

This table has only 1 partition, compared to 4 as in Section 3.1 because of the spatial limitation of 0.1 deg.

In [22]:

pz_cat_a360

Out[22]:

lsdb Catalog object_photoz:

	objectId	r_cModelMag	i_cModelMag	z_cModelMag	lephare_z_median	knn_z_median	tpz_z_median	cmnn_z_median	gpz_z_median	bpz_z_median	dnf_z_median	fzboost_z_median	coord_ra	coord_dec
npartitions=1
Order: 3, Pixel: 2	int64[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]

14 out of 130 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema

Option to show the data for the first 10 rows.

In [23]:

# pz_cat_a360.head(n=10)

3.3. Visualizations¶

Use the plot_pixels method to display the sky partition map, as in Section 2.2.

In [24]:

fig = pz_cat_a360.plot_pixels(plot_title="Abell 360 Sky Partition Map")

Figure 3: Similar to Figure 1 in Section 2.2, but for the one sky partition overlapping Abell 360.

Use the plot_points method to mark the location of each object in pz_cat_a360, and color the points by the photo-z knn_z_median.

In [25]:

center = SkyCoord(ra_a360 * u.deg, dec_a360 * u.deg)
fig, ax = pz_cat_a360.plot_points(fov=(0.2 * u.deg, 0.25 * u.deg), center=center,
                                  s=3, cmap='viridis', vmin=0.0, vmax=1,
                                  color_col="knn_z_median")

Figure 4: The default coordinate scatter plot with a PZ colorbar.

Same as above, but color the points by the i-band apparent magnitude i_cModelMag.

In [26]:

fig, ax = pz_cat_a360.plot_points(fov=(0.2 * u.deg, 0.25 * u.deg),
                                  center=center, s=3, cmap='viridis',
                                  color_col="i_cModelMag")

Figure 5: The default coordinate scatter plot with an i-band magnitude colorbar.

Similar to Figure 2 in Section 2.4, plot the N(pz) histograms for each of the eight PZ estimators, but only for galaxies within 0.1 deg of Abell 360.

In [27]:

fig, ax = plt.subplots(2, 4, figsize=(10, 4))
ze = 0
for i in range(2):
    for j in range(4):
        pz_code = all_pz_codes[ze]
        tx = (pz_cat_a360[pz_code+'_z_median'] > 0) & \
             (pz_cat_a360[pz_code+'_z_median'] < 3)
        ax[i, j].hist(pz_cat_a360[pz_code+'_z_median'][tx],
                      bins=100, histtype='step', color='black')
        ax[i, j].axvline(0.22, lw=1, ls='solid', color='dodgerblue')
        ax[i, j].set_title(all_pz_codes[ze])
        ax[i, j].set_xlim([0, 3])
        ze += 1
        del pz_code, tx
plt.suptitle('Abell 360')
plt.tight_layout()

Figure 6: Similar to Figure 2, but only for the Abell 360 region. The vertical blue line marks $z=0.22$, the known redshift of Abell 360.

3.4. Red sequence galaxies¶

The tutorial notebook in this series (304 Cosmology) that focuses on the red sequence (RS) of Abell 360 demonstrates that the RS has a color of $r-i \sim 0.5$ mag and $i-z \sim 0.25$ mag. As in that tutorial, use a RS width of $\pm 0.1$ mag, and restrict to galaxies brighter than $i=23$ mag to define the RS-like galaxies.

In [28]:

pz_cat_a360_rs = pz_cat_a360.query("r_cModelMag - i_cModelMag > 0.4 "
                                   "and r_cModelMag - i_cModelMag < 0.6 "
                                   "and i_cModelMag - z_cModelMag > 0.15 "
                                   "and i_cModelMag - z_cModelMag < 0.35 "
                                   "and i_cModelMag < 23")

In [29]:

# pz_cat_a360_rs

Plot the PZ histogram for all galaxies within 0.1 deg of Abell 360, and only those that have RS-like colors and magnitudes.

In [30]:

fig = plt.figure(figsize=(6, 4))
plt.axvline(z_a360, lw=5, alpha=0.5, color='grey', label='z=0.22')
plt.hist(pz_cat_a360['knn_z_median'], bins=300, histtype='step',
         ls='dashed', color='black', label='all galaxies')
plt.hist(pz_cat_a360_rs['knn_z_median'], bins=25, histtype='step',
         color='darkorange', label='RS-like galaxies')
plt.xlim([0, 2])
plt.xlabel('photo-z')
plt.ylabel('number of galaxies')
plt.legend(loc='upper right')
plt.show()

Figure 7: The photo-z distribution of all galaxies within a 0.1 deg radius of Abell 360 (black dashed) compared to only those with r-i and i-z colors consistent with the red sequence of Abell 360 (orange solid), with the known redshift of Abell 360, $z=0.22$, as a thick vertical grey line.

3.5. Merge with a TAP table¶

An LSDB catalog can be merged with a table of DP1 data obtained via the TAP service so long as they have an identifier column, such as objectId, in common.

Use the TAP service to retrieve all objects within 0.1 deg of Abell 360. This will include stars and galaxies.

Include the objectId column, plus the sky and pixel coordinates and the extendedness parameter.

In [31]:

query = "SELECT objectId, coord_ra, coord_dec, x, y, refExtendedness " \
        "FROM dp1.Object " \
        "WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), " \
        "CIRCLE('ICRS', %f, %f, %f)) = 1 "%(ra_a360, dec_a360, 0.1)
print(query)

SELECT objectId, coord_ra, coord_dec, x, y, refExtendedness FROM dp1.Object WHERE CONTAINS(POINT('ICRS', coord_ra, coord_dec), CIRCLE('ICRS', 37.865000, 6.982000, 0.100000)) = 1

In [32]:

job = service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()

Job phase is COMPLETED

In [33]:

assert job.phase == 'COMPLETED'
obj_cat = job.fetch_result().to_table().to_pandas()
assert len(obj_cat) == 6464
print(len(obj_cat))

Option to show obj_cat.

In [34]:

# obj_cat

The lsdb.from_dataframe method requires that columns named "ra" and "dec" exist, so rename the coordinate columns and then convert to an lsdb data frame.

In [35]:

obj_cat_rename = obj_cat.rename(columns={'coord_ra': 'ra', 'coord_dec': 'dec'})
obj_cat_lsdb = lsdb.from_dataframe(obj_cat_rename)
del obj_cat_rename

In [36]:

obj_cat_lsdb

Out[36]:

lsdb Catalog from_lsdb_dataframe:

	objectId	ra	dec	x	y	refExtendedness
npartitions=1
Order: 5, Pixel: 34	int64[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	float[pyarrow]

6 out of 6 available columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema

Option to display the first 10 rows of obj_cat_lsdb.

In [37]:

# obj_cat_lsdb.head(n=10)

Use the merge method to do an "inner" merge on the shared column objectId. An "inner" merge means the resulting catalog, merge_cat, only contains objects that appear in both obj_cat_lsdb and pz_cat_a360.

In [38]:

merge_cat = pz_cat_a360.merge(obj_cat_lsdb, how='inner', on='objectId')

Option to display the first 10 rows of merge_cat.

In [39]:

# merge_cat.head(n=10)

3.6. Mark galaxies on image¶

Use the pixel coordinates x and y, retrieved from the Object table with the TAP service, to mark RS-like galaxies on a deep_coadd image retrieved using the Butler.

Do a query on merge_cat for RS-like galaxies.

In [40]:

merge_cat_rs = merge_cat.query("r_cModelMag - i_cModelMag > 0.4 "
                               "and r_cModelMag - i_cModelMag < 0.6 "
                               "and i_cModelMag - z_cModelMag > 0.15 "
                               "and i_cModelMag - z_cModelMag < 0.35 "
                               "and i_cModelMag < 23")

Do a query on merge_cat_rs for RS-like galaxies that have a KNN PZ consistent with the histogram for Abell 360 in Figure 7.

In [41]:

merge_cat_rs_pz = merge_cat_rs.query("knn_z_median > 0.2 and knn_z_median < 0.3")

Retrieve the i-band deep_coadd image for the coordinates of Abell 360 from the Butler.

In [42]:

band = "i"
dataset_refs = butler.query_datasets("deep_coadd",
                                     where="band.name = band AND \
                                     patch.region OVERLAPS POINT(ra, dec)",
                                     bind={"band": band,
                                           "ra": ra_a360,
                                           "dec": dec_a360})
deep_coadd = butler.get(dataset_refs[0])

Display the deep_coadd image in the Firefly window, which will automatically open when the following cell is executed.

In [43]:

afw_display.mtv(deep_coadd)
afw_display.setMaskTransparency(100)

Get the bounding box (bbox) for the deep_coadd image.

In [44]:

bbox = deep_coadd.getBBox()

For objects in merge_cat_rs with x and y values within the bbox, mark with a red circle.

In [45]:

tx = merge_cat_rs["x"] < (bbox.getMaxX()-100)
tx &= merge_cat_rs["y"] < (bbox.getMaxY()-100)
xvals = merge_cat_rs["x"][tx]
yvals = merge_cat_rs["y"][tx]
with afw_display.Buffering():
    for xval, yval in zip(xvals, yvals):
        afw_display.dot("o", xval, yval,
                        size=20, ctype="red")
del tx, xvals, yvals

For objects in merge_cat_rs_pz with x and y values within the bbox, mark with a red X.

In [46]:

tx = merge_cat_rs_pz["x"] < (bbox.getMaxX()-100)
tx &= merge_cat_rs_pz["y"] < (bbox.getMaxY()-100)
xvals = merge_cat_rs_pz["x"][tx]
yvals = merge_cat_rs_pz["y"][tx]
with afw_display.Buffering():
    for xval, yval in zip(xvals, yvals):
        afw_display.dot("x", xval, yval,
                        size=20, ctype="red")
del tx, xvals, yvals

Zoom in on the Firefly window to better visualize which objects are marked with both circles and Xs.

Screenshot 2025-09-02 at 6.27.21 PM.png

Figure 8: A screenshot of the Firefly window with galaxies marked, zoomed in on the brightest cluster galaxy of Abell 360.

4. Exercises for the learner¶

Do a query on merge_cat for objects that are point-like (have a refExtendedness == 0). Mark them on the image with yellow circles of size 10. Find that there is at least one point-like (not extended) object with a RS-like color and photo-z (i.e., is marked with a red circle and x and a yellow circle). Hint: 2h 31m 32.0s, +6d 59m 13.4s.

In [ ]: