Kubernetes Deployment Guide

This guide covers Helm-first Kubernetes deployment and key operational constraints.

For reverse proxy and tunnel routing details, see REVERSE_PROXY_AND_TUNNELS.md.

Helm Chart (Recommended)

The chart supports:

• All-in-One mode (single pod)
• Individual mode (separate services)

Published Helm chart reference:

• Repository URL: https://soundspan.github.io/soundspan
• Chart name: soundspan
• Chart reference: soundspan/soundspan

helm repo add soundspan https://soundspan.github.io/soundspan
helm repo update
helm search repo soundspan/soundspan

All-in-One install

helm repo add soundspan https://soundspan.github.io/soundspan
helm install soundspan soundspan/soundspan \
  --set music.persistence.existingClaim=my-music-pvc

Individual mode install

helm repo add soundspan https://soundspan.github.io/soundspan
helm install soundspan soundspan/soundspan \
  --set deploymentMode=individual \
  --set music.persistence.existingClaim=my-music-pvc \
  --set tidalSidecar.enabled=true \
  --set ytmusicStreamer.enabled=true

For HA-oriented defaults without configuring many individual switches:

helm repo add soundspan https://soundspan.github.io/soundspan
helm install soundspan soundspan/soundspan \
  --set deploymentMode=individual \
  --set haMode.enabled=true \
  --set music.persistence.existingClaim=my-music-pvc

See full values reference in ../charts/soundspan/README.md.

Manual Deployment Notes

Images are published to GHCR.

Notes:

• The backend worker image is only used in deploymentMode: individual.
• The DCLAP provider is the individual-mode vibe embedding engine. It remains default-off; set vibeProviderDclap.enabled=true to enable it and automatic backend wiring.
• AIO mode still uses the single ghcr.io/soundspan/soundspan image.

Storage Class Guidance (RWX vs RWO)

RWX support typically requires NFS/CephFS/Longhorn/EFS class support.

Security Context

The chart separates pod-level and container-level settings:

global:
    podSecurityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
            type: RuntimeDefault
    securityContext:
        allowPrivilegeEscalation: false
        capabilities:
            drop: [ALL]
        readOnlyRootFilesystem: false

readOnlyRootFilesystem defaults to false. Enable it only after you provide writable mounts for every path used by the selected component. The AIO image runs as root internally due to supervisord orchestration; individual images are non-root.

Process and Health Check Notes

Prometheus Scraping

The backend and the dedicated worker expose process-local Prometheus data at GET /metrics. The endpoint requires Authorization: Bearer <METRICS_TOKEN> and returns 401 when the token is missing, invalid, or unset. Each process has its own registry. Prometheus must scrape every backend and worker pod.

The chart can create Prometheus Operator ServiceMonitor resources:

metrics:
    serviceMonitor:
        enabled: true
        interval: 30s
        scrapeTimeout: 10s

secrets:
    # Prefer an externally managed Secret in production.
    metricsToken: "replace-with-a-long-random-token"

metrics.serviceMonitor.enabled defaults to false. With secrets.existingSecret, add a METRICS_TOKEN key to that Secret before enabling the ServiceMonitor. The chart-generated Secret creates and preserves the token automatically.

Do not set metrics.public=true or METRICS_PUBLIC=true on an internet-routed service. That opt-out is intended only for isolated private networks. The starter Grafana dashboard covers request latency and errors, queue state, cache results, federation syncs, per-peer federation health, and process resources. See the federation alert pack for example PrometheusRule definitions and the API-versus-worker metric ownership map.

Resource Starting Point

Treat these as baseline estimates, then tune per library size and workload. The chart sets no CPU requests or limits by default. Add them through <component>.resources, such as backend.resources or audioAnalyzer.resources. The AIO container bundles the backend, frontend, Postgres, Redis, the MusicCNN analyzer, and the CPU-first DCLAP ONNX provider in one pod. The DCLAP provider vendors a few hundred MB of artifacts instead of the former 2.35 GB torch checkpoint. The AIO memory limit must cover both analysis runtimes; lower aio.resources.limits.memory only if you disable analysis (config.features.audioAnalysis: false).

Rollout Hardening (Individual Mode)

Use per-component values to reduce disruption during rolling updates:

• backend.strategy, frontend.strategy, backendWorker.strategy
• backend.pdb, frontend.pdb, backendWorker.pdb
• backend.topologySpreadConstraints, frontend.topologySpreadConstraints, backendWorker.topologySpreadConstraints

Recommended starting point:

• API/frontend replicas >=2
• maxUnavailable: 0 for API/frontend rolling updates
• pdb.enabled: true with minAvailable: 1 for API/frontend
• Use external HA Redis (or Redis Sentinel/cluster) before scaling API/worker replicas; single-pod Redis remains a SPOF.
• Redis HA deployment is operator-managed; the chart/app are designed to consume a provided HA endpoint.
• Keep LISTEN_TOGETHER_REDIS_ADAPTER_ENABLED=true on backend API pods for cross-pod Listen Together socket fanout.
• Keep LISTEN_TOGETHER_STATE_SYNC_ENABLED=true on backend API pods for cross-pod Listen Together in-memory state alignment.
• Keep LISTEN_TOGETHER_STATE_STORE_ENABLED=true on backend API pods for authoritative shared Listen Together state in Redis.
• Keep LISTEN_TOGETHER_MUTATION_LOCK_ENABLED=true on backend API pods for per-group hot-path mutation serialization.
• Keep LISTEN_TOGETHER_ALLOW_POLLING=false on backend/frontend unless sticky sessions are guaranteed.
• Set LISTEN_TOGETHER_RECONNECT_SLO_MS to your reconnect target (default 5000).
• Set SCHEDULER_CLAIM_SKIP_WARN_THRESHOLD to your tolerated skip burst before warning (default 3).
• Keep READINESS_REQUIRE_DEPENDENCIES=true so readiness fails when PostgreSQL/Redis are unhealthy.
• Tune READINESS_DEPENDENCY_CHECK_INTERVAL_MS and READINESS_DEPENDENCY_CHECK_TIMEOUT_MS for probe cadence/latency budgets.

Backend cache/log volumes in individual mode default to PVCs with ReadWriteOnce. For backend.replicas > 1, use one of:

• ReadWriteMany storage
• explicit shared existingClaim
• type: emptyDir for ephemeral per-pod cache/log storage
• disable backend cache/log persistence

The chart now fails render early if replicas are scaled with incompatible default PVC access mode.

Rollback Runbook (Scale-Out Incidents)

If rollout causes elevated errors or reconnect churn:

1. Scale API/frontend back to single replica:
   • backend.replicas=1
   • frontend.replicas=1
2. Disable dedicated worker split temporarily (optional):
   • backendWorker.enabled=false
   • backend returns to BACKEND_PROCESS_ROLE=all (chart default behavior)
3. Keep Redis adapter enabled unless specifically debugging:
   • config.listenTogetherRedisAdapterEnabled=true
4. Re-run Helm upgrade with known-good image tags and values.

Staged Rollout Sequence (Recommended)

Use this order to reduce blast radius when moving from single-replica to HA topology:

1. Deploy role-split capable images with existing replica counts unchanged.
2. Enable dedicated worker deployment:
   • backendWorker.enabled=true
   • backend.replicas=1
   • backendWorker.replicas=1
3. Confirm scheduler ownership logs are clean:
   • no sustained SchedulerClaim/SLO warnings.
4. Scale backend API to 2 replicas and validate:
   • readiness stable,
   • Listen Together reconnect latency within target.
5. Scale frontend to 2 replicas and re-check SLOs.
6. Increase backend/frontend replicas further only after SLO gates pass.

Rollout SLO Gate (Required)

Use these SLOs as hard rollout gates for multi-replica backend changes.

Suggested checks during rollout:

# 1) Ensure rollout completed
kubectl -n soundspan rollout status deploy/soundspan-backend --timeout=10m
kubectl -n soundspan rollout status deploy/soundspan-frontend --timeout=10m
kubectl -n soundspan rollout status deploy/soundspan-backend-worker --timeout=10m

# 2) Inspect SLO warning channels (last 15m)
kubectl -n soundspan logs deploy/soundspan-backend --since=15m | grep -E "ListenTogether/SLO.*exceeded target|SchedulerClaim/SLO.*skipped"
kubectl -n soundspan logs deploy/soundspan-backend-worker --since=15m | grep -E "SchedulerClaim/SLO.*skipped"

# 3) Verify ready pods are stable
kubectl -n soundspan get pods -l app.kubernetes.io/component=backend
kubectl -n soundspan get pods -l app.kubernetes.io/component=frontend
kubectl -n soundspan get pods -l app.kubernetes.io/component=backend-worker

Or run the packaged helper from repo root:

./scripts/k8s-rollout-slo-check.sh --namespace soundspan

For full HA live validation (preflight + SLO gate + optional rollout restarts):

# Non-disruptive validation
./scripts/k8s-ha-live-validation.sh \
  --context <your-cluster-context> \
  --namespace soundspan

# Active restart drill (disruptive): restart backend and frontend, then re-check SLOs
./scripts/k8s-ha-live-validation.sh \
  --context <your-cluster-context> \
  --namespace soundspan \
  --restart-backend \
  --restart-frontend

See also

• Deployment Guide — Docker run/compose deployment modes and release channels
• Reverse Proxy and Tunnels — Edge routing for split deployments
• Configuration and Security — Environment config and security hardening
• Environment Variables — Complete env var reference by container