# Product Streamer — Operator Guide

How to deploy, register, and verify the Product Streamer (PSAPI) at an SRC site.

## Purpose

PSAPI streams data products held on the site's StoRM Rucio Storage Element (RSE) to users over HTTPS. It
runs next to the RSE with a read-only mount of its deterministic storage area and reads product files directly
from it: a single file is returned as raw bytes with HTTP `Range` support, and several files or a dataset as an
uncompressed TAR archive.

PSAPI is exposed through its own Ingress, not through the site's Gatekeeper, and authorises every request
itself by passing the caller's token to the Permissions API (PAPI). Users do not address it directly: DataLink
asks the Data Management API (DMAPI) for services co-located with a product's replicas, DMAPI finds PSAPI
through the Site Capabilities API (SCAPI), and DataLink returns its access URL to the user.

See the [README](https://gitlab.com/ska-telescope/src/src-dm/ska-src-dm-product-streamer-api/-/blob/main/README.md) for the service architecture.

## Prerequisites

PSAPI depends on the SRCNet global services — IAM, PAPI, the Auth API (AAPI), SCAPI, DMAPI and DataLink —
but these are already running and need no action from a site operator. What must exist at the site before
deploying:

| Requirement | Notes |
|:------------|:------|
| Kubernetes cluster access | Namespace (conventionally `product-streamer`) with permission to create Deployments, Services, Ingresses, Secrets and PVCs |
| ingress-nginx | The chart's Ingress uses class `nginx` and has no TLS block, so TLS termination is the site's responsibility |
| A StoRM RSE | Registered in SCAPI as an `rse`-type storage area. The area's `name` must be the Rucio RSE name |
| RSE volume | A PVC bound to the RSE's storage, mounted read-only |
| CA bundle | Mounted at `/etc/ssl/certs/SKALocalCA.pem` |

The chart does not render the RSE volume or the CA bundle mount, so the deployment overlay must add both to the
`core` Deployment:

- **RSE volume.** Mount the PVC at the deterministic storage root and set `STORAGE_BASE_PATH` to that mount
  point. If it is unset the service falls back to `/sa`.
- **CA bundle.** The chart always points `REQUESTS_CA_BUNDLE` and `SSL_CERT_FILE` at
  `/etc/ssl/certs/SKALocalCA.pem`. If nothing is mounted there, every call to PAPI fails, and so does every
  data request.

No IAM client is needed at runtime: PSAPI makes no calls under its own identity.

## Site Capabilities Registration

DataLink offers a product-streamer access URL only for replicas whose RSE has a co-located `product_streamer`
service. PSAPI must therefore be registered as a local service under the site's compute element **and** linked
to the RSE's storage area. Register once the deployment passes the [Testing](#testing) checks up to "Stream a
file", and deregister before undeploying.

### SCAPI service entry

Add the service to the site's compute element as a local service, using the SCAPI node editor or the SCAPI
integration client (`ska_src_site_capabilities_api.client.integration register_service`):

| Field | Value |
|:------|:------|
| `id` | A fixed UUID, so re-registering updates the entry rather than duplicating it |
| `type` | `product_streamer` — must match exactly; DMAPI and DataLink look the service up by this value |
| `prefix` / `host` / `port` | PSAPI's **own** public scheme, Ingress host and port, e.g. `https` / `product-streamer.<site-domain>` / `443` |
| `path` | `/v1/data/product` |
| `associated_storage_area_id` | UUID of the RSE storage area PSAPI mounts |

DataLink builds the URL it hands to users as `prefix://host:port/path`, so these fields must describe an
address a user's client can reach. DMAPI matches the associated storage area's `name` against the Rucio RSE
names of a product's replicas: if the field is missing or the name does not match, the service never appears
in DataLink results.

To check discovery, ask DataLink for a product known to be on the RSE and look for the
`product-streamer` resource:

```bash
curl -s "$DATALINK_URL/v1/links?id=<scope>:<name>" | grep -A3 'ID="product-streamer"'
```

## Access Groups

`POST /v1/data/product` is authorised against the `product-streamer-api` v1 `route` policy in PAPI, and AAPI
issues tokens for the service under the matching `exchange` policy. Both policies must be enabled. The status
routes are unauthenticated.

**Issuer and audience.** The token must be issued by the IAM named in the policy and carry the
`product-streamer-api` audience. A raw IAM token fails the audience check, and PSAPI turns that failure into a
`401` challenge telling the client to exchange its token at AAPI.

**Group membership.** The caller must belong to both:

```
<roles-root>/user    or  <roles-root>/oper
data/namespaces/<scope>/viewer  or  .../editor  or  .../owner
```

`<roles-root>` differs per environment (e.g. `roles/prod`, `roles/staging`). `<scope>` is the Rucio scope of
the request's **first** DID only, so a request that mixes scopes is authorised against the first one alone.

**Service tokens.** A token carrying the scope `product-streamer-api-service` is authorised with no group
check and can read every scope on the RSE. Grant it only to services that need unrestricted read access.

**Token exchange.** AAPI exchanges a token for the `product-streamer-api` audience only if the user is in
`<roles-root>/user` or `<roles-root>/oper`.

Users request these groups through the usual IAM group-membership process for the data they need.

## Deployment

The Helm chart, `psapi`, is in `etc/helm/`. It deploys one image, `psapi-core`, as the `core` Deployment, with
a Service on port `8080` and an Ingress routing the configured host to it.

### Values to set

```yaml
svc:
  api:
    image:
      tag: <release>
    permissions_api_url: https://permissions.<env-domain>/api/v1
    iam_wellknown_endpoint: https://iam.<env-domain>/.well-known/openid-configuration
    aapi_url: https://authn.<env-domain>/api

ing:
  api:
    host: product-streamer.<site-domain>
```

### Key configuration

| Variable | Required | Notes |
|:---------|:---------|:------|
| `STORAGE_BASE_PATH` | yes | Mount point of the RSE's deterministic storage root, i.e. the directory that directly contains the scope directories. SRCNet Rucio uses the `identity` LFN-to-PFN algorithm, so a file is at `STORAGE_BASE_PATH/<scope>/<name>` with no hash directories. **Not set by the chart** — see [Prerequisites](#prerequisites) |
| `PERMISSIONS_API_URL` | yes | PAPI base URL, including `/api/v1`. The service does not start without it |
| `PERMISSIONS_SERVICE_NAME`, `PERMISSIONS_SERVICE_VERSION` | yes | `product-streamer-api` and `1` (the chart defaults). Must name an enabled PAPI policy |
| `IAM_WELLKNOWN_ENDPOINT` | yes | IAM OIDC discovery URL, advertised in the missing-token challenge. The chart default is not the staging or dev IAM, so always set it |
| `AAPI_URL` | yes | AAPI base URL, used to build the token-exchange challenge |
| `UVICORN_NWORKERS` | — | Worker processes; chart default `4` |
| `DISABLE_AUTHENTICATION` | — | `yes` skips PAPI entirely. Development only — never set it in a deployed environment |
| `LOG_LEVEL`, `LOG_FORMAT`, `OTEL_*` | — | Structured logging and OpenTelemetry export |

## Testing

PSAPI's integration tests run in the
[Deployment Stack](https://gitlab.com/ska-telescope/src/src-api/ska-src-api-deployment-stack)
(`stack test product-streamer`), against its dev and staging environments. The checks below are for verifying
a deployment at a site.

**Check the service is up and can reach PAPI.**

```bash
curl -s https://product-streamer.<site-domain>/v1/health
```

Expect `"dependent_services": {"permissions-api": {"status": "UP"}}`. A `500` reporting `DOWN` means PSAPI
cannot reach PAPI — check `PERMISSIONS_API_URL` and the CA bundle mount. Every data request will fail until it
is fixed. The pod's liveness probe does not check this.

**Stream a file.** Exchange a token for the service audience, then request a file you know is on the RSE, in a
scope you can read:

```bash
TOKEN=$(curl -s "https://authn.<env-domain>/api/v1/token/exchange/product-streamer-api?access_token=$RAW_TOKEN" \
  | jq -r .access_token)

curl -s -X POST https://product-streamer.<site-domain>/v1/data/product \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '[{"did": "<scope>:<name>", "path": "<STORAGE_BASE_PATH>/<scope>/<name>"}]' \
  -o out.bin -D -
```

Expect `200` with a `Content-Length` matching the file size. A `400` "Path outside allowed storage area" means
the path does not resolve under `STORAGE_BASE_PATH`; a `404` means it is under it but does not exist.

**End-to-end through DataLink.** Once registered, forward DataLink's VOTable as-is. This exercises discovery
and the `STORAGE_BASE_PATH` mapping together:

```bash
curl -s "$DATALINK_URL/v1/links?id=<scope>:<name>" -o datalink.xml
curl -s -X POST https://product-streamer.<site-domain>/v1/data/product \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/xml' \
  --data-binary @datalink.xml -o product.out -D -
```