SKA SRC DM Product Streamer API

This service streams data products held on a site’s Rucio Storage Element directly to users over HTTPS.

[TOC]

Overview

The Product Streamer (PSAPI) is a site-local service. It runs next to a site’s StoRM Rucio Storage Element (RSE) and reads product files straight from a read-only mount of its deterministic storage area. There is no staging, copy or external transfer step: the bytes go back on the same HTTP response.

Users find PSAPI through DataLink. DataLink asks the Data Management API (DMAPI) which services are co-located with a product’s replicas, and DMAPI answers from the Site Capabilities API (SCAPI), where each site registers its PSAPI against the RSE it mounts. The client then sends DataLink’s VOTable, or an explicit list of files, to PSAPI. PSAPI is exposed through its own Ingress, not the site’s Gatekeeper, and authorises every data request itself against the Permissions API (PAPI).

flowchart LR client["Client"] -->|"1. links"| datalink["DataLink"] datalink -->|"2. locate"| dmapi["Data Management API"] dmapi --> scapi["Site Capabilities API"] client -->|"3. stream"| psapi["Product Streamer"] psapi -->|"authorise"| papi["Permissions API"] rse[("Site RSE")] -->|"read"| psapi

Architecture

PSAPI is a single component with no workers, queues or caches. A request is authorised, resolved to paths on the storage mount and streamed back.

sequenceDiagram participant Client participant DL as DataLink participant PS as PSAPI participant PAPI as Permissions API participant FS as RSE mount Client->>DL: GET /v1/links?id=scope:name DL-->>Client: VOTable with PSAPI access URL Client->>PS: POST /v1/data/product (VOTable, token) PS->>PAPI: authorise route + scope of first DID PAPI-->>PS: authorised PS->>FS: read PS-->>Client: 200/206 file, or 200 TAR

A client without a suitable token gets an IVOA ivoa_bearer 401 challenge pointing to IAM discovery, then a second one pointing to the Auth API’s token exchange for the product-streamer-api audience. PSAPI never calls IAM or the Auth API itself; it only advertises their URLs.

Endpoints

Method

Path

Purpose

POST

/v1/data/product

Stream one or more products; requires a product-streamer-api token

GET

/v1/ping

Liveness and service version

GET

/v1/health

Readiness, including PAPI reachability

GET

/v1/metrics

Prometheus metrics

The request body is either a DataLink VOTable (Content-Type: application/xml), exactly as DataLink returned it, or a JSON list of objects, one per file:

[
  {
    "did": "srcnet_test.comm:eb-test-20250101-00001.product-1/image/image.fits",
    "path": "/sa/srcnet_test.comm/eb-test-20250101-00001.product-1/image/image.fits"
  }
]

did is the Rucio file DID, <scope>:<name>. path is the file’s absolute location on the storage mount; for a VOTable it is derived by appending each IVOID’s storage path to STORAGE_BASE_PATH. Rucio’s identity LFN-to-PFN algorithm is used, so this is STORAGE_BASE_PATH/<scope>/<name>, with no hash directories. Every path must resolve, after following symlinks, inside STORAGE_BASE_PATH, or the request is rejected with a 400.

Data delivery

  • One product that is a file: streamed as raw bytes with Content-Length, ETag and HTTP Range support, so interrupted downloads can be resumed.

  • Several products, or a dataset directory: streamed as an uncompressed TAR archive whose exact Content-Length is computed before streaming starts.

Chunk sizes adapt to the client’s measured bandwidth. Foreseeable errors (400, 401, 403, 404, 416) are returned before any bytes are sent; an I/O error mid-stream closes the connection short of Content-Length.

Implementation

Python 3.13, managed with Poetry.

  • API: FastAPI, versioned with fastapi-versionizer (API routes under /v1), served by Uvicorn. The request model is Pydantic v2 in strict mode with extra="forbid".

  • Streaming: aiofiles for file reads, with blocking filesystem work run in thread pools. TAR archives are written out by hand, file by file, so memory use stays flat.

  • Authorisation: the ska-src-permissions-api client, called with the caller’s own token.

  • Observability: ska-src-logging provides structured logging and OpenTelemetry traces, metrics and logs. Prometheus metrics are served at /v1/metrics.

Source layout under src/ska_src_dm_product_service/:

Package

Contents

rest/

FastAPI app, the product and status routers, the PAPI dependency, OpenAPI customisation

models/

Pydantic request and response models

common/

DataLink VOTable parsing, 401 challenge and other exceptions

One image is published, psapi-core (images/psapi-core/Dockerfile), which runs Uvicorn on port 8080. The Helm chart in etc/helm/ deploys it. Development and integration testing run in the Deployment Stack.

Further reading