103.3. TAP image access via ObsCore

103.3. TAP image access via the ObsCore table¶

For the Rubin Science Platform at data.lsst.cloud.
Data Release: Data Preview 1
Container Size: medium
LSST Science Pipelines version: r29.1.1
Last verified to run: 2025-06-21
Repository: github.com/lsst/tutorial-notebooks

Learning objective: How to use the ObsCore table and the Table Access Protocol (TAP) service to access image data.

LSST data products: deep_coadd, visit_image

Packages: lsst.rsp, lsst.afw.display, lsst.afw.image

Credit: Originally developed by the Rubin Community Science team. Please consider acknowledging them if this notebook is used for the preparation of journal articles, software releases, or other notebooks.

Get Support: Everyone is encouraged to ask questions or raise issues in the Support Category of the Rubin Community Forum. Rubin staff will respond to all questions posted there.

1. Introduction¶

TAP provides standardized access to catalog data for discovery, search, and retrieval. Full documentation for TAP is provided by the International Virtual Observatory Alliance (IVOA).

ADQL (Astronomy Data Query Language) is similar to SQL (Structured Query Langage). The documentation for ADQL includes more information about syntax and keywords.

Not all ADQL functionality is supported yet in the preview-era RSP, such as the INTERSECTS operator.

When to use the TAP service.

Use the TAP service to find, retrieve, and display whole images, based on their spatial, temporal, and spectral coverage, and their LSST image subtype. TAP is good for quick access to images' pixel and header data, for display and examination by eye. TAP can be used to obtain the access_url to pass to the image cutout service.

For science cases that involve source detection or image analysis with the LSST Science Pipelines, the Butler should instead be used to find and retrieve images.

Related tutorials: Other 100-level tutorials demonstrate how to access images via other services, such as SIA and the Butler, and how to use the image cutout service. The 100-level tutorials on image display have more on using Firefly. The 200-level tutorials on coadded and visit images have more on these data products (e.g., the pixel data and metadata).

1.1. DAL service errors¶

The following error message might be encountered.

DALServiceError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Typically this is fixed by rerunning the cell with the TAP service fetch statement, or by clearing the notebook, restarting the kernel, and re-executing the cells.

1.2. Import packages¶

Import numpy.

Import pyvo packages for working with Datalinker.

Import LSST Science Pipelines packages for image display, image type conversion, and utilities for remote data access.

In [1]:

import numpy as np
from pyvo.dal.adhoc import DatalinkResults

import lsst.afw.display as afwDisplay
from lsst.afw.image import ExposureF
from lsst.rsp import get_tap_service
from lsst.rsp.utils import get_pyvo_auth

1.3. Define parameters and functions¶

Instantiate the TAP service.

In [2]:

service = get_tap_service("tap")

Define search coordinates right ascension (target_ra) and declination (target_dec), in degrees, to use in all queries. These coordinates are near the center of the DP1 ECDFS field.

In [3]:

target_ra = 53.076
target_dec = -28.110

Set afwDisplay to use Firefly, and define afw_display to show images in frame 1.

In [4]:

afwDisplay.setDefaultBackend("firefly")
afw_display = afwDisplay.Display(frame=1)

2. The ObsCore table¶

The ObsCore table schema conforms to the IVOA (International Virtual Observatory Alliance) standards.

The LSST ObsCore table is essentially a view into the images stored in the LSST's data Butler.

Retrieve ObsCore schema: the names of the columns and their data types, descriptions and units, and display it. Notice that the ObsCore table is in the ivoa collection, not the DP1 collection like the data tables.

In [5]:

query = "SELECT column_name, datatype, description, unit " \
        "FROM tap_schema.columns " \
        "WHERE table_name = 'ivoa.ObsCore'"
results = service.search(query).to_table()
results

Out[5]:

Table length=36

column_name	datatype	description	unit
str64	str64	str512	str64
access_format	char	Content format of the dataset
access_url	char	URL used to access dataset
calib_level	int	Calibration level of the observation: in {0, 1, 2, 3, 4}
dataproduct_subtype	char	Data product specific type
dataproduct_type	char	Data product (file content) primary type
em_max	double	stop in spectral coordinates	m
em_min	double	start in spectral coordinates	m
em_res_power	double	Value of the resolving power along the spectral axis (R)
em_xel	long	Number of elements along the spectral axis
facility_name	char	The name of the facility, telescope, or space craft used for the observation
instrument_name	char	The name of the instrument used for the observation
lsst_band	char	Abstract filter band designation
lsst_detector	long	Identifier for CCD within the LSSTCam focal plane
lsst_filter	char	Physical filter designation from the LSSTCam filter set
lsst_patch	long	Lower level of LSST coadd skymap hierarchy
lsst_tract	long	Upper level of LSST coadd skymap hierarchy
lsst_visit	long	Identifier for a specific LSSTCam pointing
o_ucd	char	Nature of the observable axis
obs_collection	char	Name of the data collection
obs_id	char	Internal ID given by the ObsTAP service
obs_publisher_did	char	ID for the Dataset given by the publisher
obs_title	char	Brief description of dataset in free format
pol_xel	long	Number of elements along the polarization axis
s_dec	double	Central Spatial Position in ICRS; Declination	deg
s_fov	double	Estimated size of the covered region as the diameter of a containing circle	deg
s_ra	double	Central Spatial Position in ICRS; Right ascension	deg
s_region	char	Sky region covered by the data product (expressed in ICRS frame)
s_resolution	double	Spatial resolution of data as FWHM of PSF	arcsec
s_xel1	long	Number of elements along the first coordinate of the spatial axis
s_xel2	long	Number of elements along the second coordinate of the spatial axis
t_exptime	double	Total exposure time	s
t_max	double	Stop time in MJD	d
t_min	double	Start time in MJD	d
t_resolution	double	Temporal resolution FWHM	s
t_xel	long	Number of elements along the time axis
target_name	char	Object of interest

In [6]:

del query, results

3. Query for images¶

It is recommended to tightly constrain image queries, so that they return only the image data products needed for a given scientific analysis.

The main constraints on ObsCore TAP queries are:

• image region overlaps point
• image type (calibration level: raw, processed, coadded)
• band (filter, spectral coverage)
• time (date and time of acquisition)

3.1. Position¶

Query by position.

To query only by position is not recommended because it would return every image that overlaps that region, including raw, direct, difference, and coadded images. Typically, for scientific analyses, a specific image type or band is desired and should be included in the query, as demonstrated in following sections.

In [7]:

query = """SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', {},{}), s_region) = 1
        """.format(target_ra, target_dec)
print(query)

SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', 53.076,-28.11), s_region) = 1

Run the TAP query.

In [8]:

job = service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()
assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
print(len(results))

Job phase is COMPLETED

2361

Option to show the results table.

In [9]:

# results

Print the unique data product subtypes and how many of each were returned by the query.

In [10]:

values, counts = np.unique(results['dataproduct_subtype'],
                           return_counts=True)
for value, count in zip(values, counts):
    print(value, count)

lsst.deep_coadd 12
lsst.difference_image 779
lsst.raw 779
lsst.template_coadd 12
lsst.visit_image 779

In [11]:

job.delete()
del query, results, values, counts

3.2. Image type¶

Constrain the query by image type.

The calibration levels and data product subtypes are:

• 1 : raw
• 2 : visit_image
• 3 : deep_coadd, template_coadd, difference_image

With the TAP service, constraining on calib_level is not necessary.

For example, query for visit_images by requiring that the dataproduct_subtype = lsst.visit_image.

In [12]:

query = """SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', {},{}), s_region) = 1
        AND dataproduct_subtype='lsst.visit_image'
        """.format(target_ra, target_dec)
print(query)

SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', 53.076,-28.11), s_region) = 1
        AND dataproduct_subtype='lsst.visit_image'

In [13]:

job = service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()
assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
print(len(results))

Job phase is COMPLETED
779

In [14]:

job.delete()
del query, results

3.3. Band¶

Query by band.

Add a constraint that the lsst_band be "r".

In [15]:

query = """SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', {},{}), s_region) = 1
        AND dataproduct_subtype='lsst.visit_image'
        AND lsst_band = 'r'
        """.format(target_ra, target_dec)
print(query)

SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', 53.076,-28.11), s_region) = 1
        AND dataproduct_subtype='lsst.visit_image'
        AND lsst_band = 'r'

In [16]:

job = service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()
assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
print(len(results))

Job phase is COMPLETED
214

In [17]:

job.delete()
del query, results

3.4. Time¶

Query by the date and time of the observation.

In the ObsCore table, t_min and t_max are the start and end times of the observation. Add a query constraint for images that started after MJD 60646.04 and ended before MJD 60646.09.

In [18]:

query = """SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', {},{}), s_region) = 1
        AND dataproduct_subtype='lsst.visit_image'
        AND lsst_band = 'r'
        AND t_min > 60646.04 AND t_max < 60646.09
        ORDER BY t_min ASC
        """.format(target_ra, target_dec)
print(query)

SELECT * FROM ivoa.ObsCore
        WHERE CONTAINS(POINT('ICRS', 53.076,-28.11), s_region) = 1
        AND dataproduct_subtype='lsst.visit_image'
        AND lsst_band = 'r'
        AND t_min > 60646.04 AND t_max < 60646.09
        ORDER BY t_min ASC

In [19]:

job = service.submit_job(query)
job.run()
job.wait(phases=['COMPLETED', 'ERROR'])
print('Job phase is', job.phase)
if job.phase == 'ERROR':
    job.raise_if_error()
assert job.phase == 'COMPLETED'
results = job.fetch_result().to_table()
print(len(results))

Job phase is COMPLETED
15

Option to print just the start times of the images.

In [20]:

# results['t_min']

Delete the job and query but keep the results to use in the next section.

In [21]:

job.delete()
del query

4. Retrieve and display an image¶

Extract the access URL from the first row of the results as datalink_url.

Retrieve the datalink VOTable document as dl_result. At this point the service requires authorization credentials, which are passed with the utility function get_pyvo_auth() from the lsst.rsp package.

Get the URL for the specific image as image_url.

In [22]:

datalink_url = results['access_url'][0]
dl_result = DatalinkResults.from_result_url(datalink_url, session=get_pyvo_auth())
image_url = dl_result.getrecord(0).get('access_url')

Option to display the URLs.

In [23]:

# datalink_url

In [24]:

# dl_result

In [25]:

# image_url

Read the entire image into an ExposureF object. The ExposureF format is designed to work with seamlessly with afw.display and so is recommended for all Rubin images.

In [26]:

visit_image = ExposureF(image_url)

Display the exposure in the Firefly tab.

In [27]:

afw_display.image(visit_image)

Set the mask transparency to 100% transparent.

In [28]:

afw_display.setMaskTransparency(100)

5. Exercises for the learner¶

The approximate central coordinates of the Rubin_SV_95_-25 field for Data Preview 1 are RA, Dec = $95, -25$. Show that the number of visit images obtained on MJD 60634 that overlap these coordinates was 43.

In [ ]: