# SCENES — Scene & Composition Design Contract Status: **authoritative spec** for Phase 3. Backend (`t2`) and frontend (`t3`) implementation tasks consume this document and must not make design decisions that contradict it. - Scope: `services/multistreaming` (CommonJS backend in `src/`, React/TS/shadcn frontend in `web/src/`). - Constraint: no runtime tests. Verification is `node --check` (backend) and `npm run build` (web = `tsc -b && vite build`). --- ## 1. Overview Today a room is a passthrough router: each published feed is remuxed (`-c copy`) to every enabled output that routes that feed. Scenes add a second programming mode for a room: 1. A **scene** is a named, persisted composition for a room: a layout (grid or picture-in-picture) that places live feeds, plus text/image overlays. 2. A scene has **scene outputs** (destinations) — one per platform account the composed program is pushed to — each with its own **audio routing**. 3. Activating a scene switches the room from passthrough mode to composed mode. In composed mode the room pushes the composed program (re-encoded) instead of raw per-feed passthrough. Deactivating the scene restores passthrough. 4. **Passthrough mode (default) is unchanged**: with no composed scene active, fan-out stays `-c copy` exactly as today. Security invariants are unchanged and extended in §8. --- ## 2. Terminology | Term | Meaning | |------|---------| | feed | A single OBS ingest publishing to `rtmp://:1935/live/`. | | live feed | A feed currently publishing (tracked from RTMP publish events). | | output | Existing passthrough routing record (`feedId → accountId`). Unchanged. | | scene | A named composition definition for a room. | | scene output | A destination for the composed program (`sceneId → accountId`) with an audio mode. | | slot | A position in the layout, referenced by index; holds a `feedId` or `null`. | | overlay | A text or image element drawn on top of the composed video. | | program audio | The mixed audio of all live feeds participating in the scene. | | canvas | The composed video frame size (default 1920×1080). | --- ## 3. Persisted data model `Store` gains three collections, exactly like the existing ones, normalized in `_normalize` and written atomically by `_save`: ```js // store.js state shape additions { scenes: [ Scene ], sceneOutputs: [ SceneOutput ], images: [ RoomImage ], } ``` ### 3.1 `Scene` ```jsonc { "id": "uuid", "roomId": "uuid", "ownerId": "uuid", // creator; informational only "name": "Main show", // trimmed, 1..64 chars "active": false, // at most one true per room "layout": { "type": "grid", // "grid" | "pip" "columns": 2, // grid ONLY: integer 2..4 "slots": ["feedId", null] // Array; see slot rules }, "overlays": [ // text overlay { "id": "uuid", "kind": "text", "text": "LIVE", "x": 32, "y": 32, "fontSize": 48, "color": "#ffffff", "bold": true }, // image overlay { "id": "uuid", "kind": "image", "imageId": "uuid", "x": 16, "y": 844, "width": 200, "height": 112, "opacity": 1.0 } ], "createdAt": 1700000000000, "updatedAt": 1700000000000 } ``` **Slot rules** - `slots` is an array indexed by slot number. Each element is a `feedId` string (must be a feed in the same room) or `null` (empty). - `grid`: `slots.length` ∈ 1..6 (the cell order, row-major). `columns` ∈ 2..4. - `pip`: `slots.length` == 2 — index 0 is the **main** (fullscreen) feed, index 1 is the **PiP** window feed. - Unknown `feedId`, duplicate `feedId`, or out-of-range slot length is rejected with HTTP 400 (see §6.2 validation). **Layout is always valid JSON with no secrets.** A scene never stores stream keys. ### 3.2 `SceneOutput` ```jsonc { "id": "uuid", "roomId": "uuid", "sceneId": "uuid", // must be a scene in the same room "accountId": "uuid", // must be an account in the room (accountsInRoom) "enabled": true, "audio": { "mode": "program" } // | { mode: "silent" } // | { mode: "feed", "feedId": "uuid" } } ``` - `audio.mode`: - `"program"` — the mixed program audio (§5.4). - `"silent"` — no audio stream (`-an`). - `"feed"` — one specific feed's audio (`feedId` required, feed in the same room). - Enabled scene outputs are the push destinations when the scene is active. ### 3.3 `RoomImage` ```jsonc { "id": "uuid", "roomId": "uuid", "name": "logo.png", // sanitized, 1..64 chars "mime": "image/png", // sniffed: image/png | image/jpeg | image/webp "size": 48231, // decoded bytes "path": "uploads/.", // server-generated; never user input "createdAt": 1700000000000 } ``` - Files live under `/uploads/`. The stored basename matches `/^[a-f0-9-]{36}\.(png|jpg|webp)$/` (UUID + sniffed extension). The client never chooses the path. ### 3.4 Validation (server-side, authoritative) - Scene `name`: trimmed string, 1..64 chars. - `layout.type`: `"grid"` or `"pip"` (else 400). - Grid: `columns` integer 2..4; `slots` array length 1..6. - PiP: `slots` array length exactly 2. - Every non-null slot `feedId` must exist in `store.listFeeds(roomId)`. - `overlays` array length ≤ 8. - text: `text` trimmed non-empty, ≤ 256 chars; `x` ∈ [0,1920) integer, `y` ∈ [0,1080) integer; `fontSize` ∈ 8..144 integer; `color` matches `/^#[0-9a-fA-F]{6}$/`; `bold` optional boolean. - image: `imageId` must be a `RoomImage` in the same room; `x`,`y`,`width`, `height` integers with `width`,`height` ∈ 16..1920/1080 respectively; `opacity` optional ∈ [0,1]. - `sceneOutput`: `sceneId` in room; `accountId` in `store.accountsInRoom(roomId)`; `audio.mode` one of the three; `mode === "feed"` requires a valid room `feedId`. - Image upload: decoded size ≤ `SCENE_MAX_IMAGE_BYTES` (default 1 MiB); type sniffed from magic bytes (PNG `89 50 4E 47`, JPEG `FF D8 FF`, WebP `RIFF....WEBP`). Client-declared MIME is **not** trusted. ### 3.5 Pruning (deletion hooks) - `removeFeed(id)` additionally: in every scene of the same room set each slot element equal to `id` to `null`; for every sceneOutput whose `audio.mode === "feed"` and `audio.feedId === id`, set `audio = { mode: "silent" }`. - `removeAccount(id)` additionally: delete sceneOutputs with `accountId === id` (mirrors existing output pruning). - `removeScene(id)`: delete sceneOutputs with `sceneId === id`. If it was the active scene, no scene is active; reconcile stops the scene fanout. - `removeImage(id)`: delete the file, the record, and every overlay item (`kind === "image"`) across the room's scenes that references `imageId`. - `_normalize`: add the three arrays; defensively strip scene slots referencing unknown feeds, sceneOutputs referencing unknown scenes/accounts, and overlay image items referencing unknown images; if more than one scene is `active` per room, keep the first and clear the rest. --- ## 4. Composition engine — `src/compose.js` New module. It contains **all** FFmpeg knowledge and produces argument arrays; `fanout.js` stays lifecycle-only. Exports: - `CANVAS` — `{ width, height, fps, videoBitrate, audioBitrate }` resolved from config (§9). - `escapeDrawtext(text)` and `escapeFilterPath(path)` — escaping helpers (§5.5). - `buildScenePlan({ scene, liveFeeds, getInputUrl, destinations, config })` → a plan object or `null` (null when zero live feeds, i.e. nothing to compose). ```js // buildScenePlan output (passthrough to fanout, never persisted) { inputs: [ { feedId, url } ], // live feeds in slot order outputs: [ { outputId, url, args, safeArgs } ] // one ffmpeg argv per destination } ``` `fanout.js` spawns **one ffmpeg child per `outputs[]` entry**, each reading all `inputs[]` and applying its own audio `-map` (see §5.4). This mirrors the existing per-output child model and keeps restart/reconcile reuseable. It means video is re-encoded once per destination — an accepted, documented trade-off for a small self-hosted tool (per-output audio routing cannot be shared through a single `tee` muxer without identical audio). ### 4.1 Inputs For a scene with `slots`, the live inputs are the slots' feeds that are currently live, **in slot order** (null/non-live slots skipped). Input index `i` (0-based) corresponds to `inputs[i]`. Source URL for feed with `streamKey`: `rtmp://127.0.0.1:/live/`. Destination URLs come from `destinations` (resolved by `index.js` from grants, exactly like today) — **never stored**. ### 4.2 Layouts Canvas = `SCENE_WIDTH` × `SCENE_HEIGHT` (default 1920×1080), fps = `SCENE_FPS` (default 30). `n` = number of live inputs. Normalization chain for input `i` to a target cell `Wi×Hi`: ``` [i:v] scale=Wi:Hi:force_original_aspect_ratio=decrease, pad=Wi:Hi:(ow-iw)/2:(oh-ih)/2, settb=AVTB, fps=30, format=yuv420p [v_i] ``` (`fps` uses the canvas fps constant.) **Grid, n > 1** — `c = min(columns, n)`, `rows = ceil(n / c)`, `cellW = 1920 / c`, `cellH = 1080 / rows`. For input `i`: `x_i = (i % c) * cellW`, `y_i = floor(i / c) * cellH`. ``` [v_0][v_1]...[v_{n-1}] xstack=inputs=n:layout=x_0_y_0|x_1_y_1|...|x_{n-1}_y_{n-1}:fill=black [comp] ``` **Grid, n == 1** — single fullscreen tile (no xstack): the normalization chain targets `1920x1080` and its label is `[comp]`. **PiP, both slots live** (`m` = main input index, `p` = pip input index): ``` [m:v] scale=1920:1080:force_original_aspect_ratio=decrease, pad=1920:1080:(ow-iw)/2:(oh-ih)/2, settb=AVTB, fps=30, format=yuv420p [main] [p:v] scale=480:270:force_original_aspect_ratio=decrease, pad=480:270:(ow-iw)/2:(oh-ih)/2, settb=AVTB, fps=30, format=yuv420p [pip] [main][pip] overlay=x=main_w-overlay_w-16:y=16:format=auto [comp] ``` **PiP, only one slot live** — promote that feed to fullscreen (same chain as grid n==1) and label it `[comp]`. **PiP, no slots live** — `buildScenePlan` returns `null`. ### 4.3 Overlays Overlays are applied to `[comp]` in array order, chaining labels `[comp] → [ov_1] → [ov_2] → … → [vout]`. If there are no overlays, `[comp]` is `[vout]`. **Text overlay k** (`prev` = previous label, `fontpath` = `SCENE_FONT` or `SCENE_FONT_BOLD` when `bold`): ``` [prev] drawtext=text='':x=:y=:fontsize=: fontcolor=:fontfile='':expansion=none [ov_k] ``` **Image overlay k**: ``` movie=filename='', scale=: [img_k]; [prev][img_k] overlay=x=:y=:format=auto [ov_k] ``` - `absImagePath` = `path.join(config.dataDir, image.path)`. - Image scale is exact `scale=w:h` (no aspect preservation — the editor controls both dimensions). ### 4.4 Audio routing Program audio (built **only if** at least one destination uses `mode: "program"` and `n ≥ 1`): ``` n == 1: [0:a] aresample=48000, aformat=sample_fmts=fltp:channel_layouts=stereo [aout] n >= 2: [0:a][1:a]...[n-1:a] amix=inputs=n:duration=first:dropout_transition=2, aresample=48000, aformat=sample_fmts=fltp:channel_layouts=stereo [aout] ``` **Assumption (documented):** every participating feed publishes H.264 video + AAC audio (OBS defaults). Video-only or audio-only feeds are out of scope for composition; the `-map :a?` form still tolerates a missing audio stream for `mode: "feed"`. Per-destination audio args: | mode | live? | args | |------|-------|------| | `program` | `[aout]` exists | `-map [vout] -map [aout] -c:a aac -b:a -ar 48000` | | `program` | `[aout]` absent | `-map [vout] -an` | | `silent` | — | `-map [vout] -an` | | `feed` (feed live at input index `j`) | yes | `-map [vout] -map j:a? -c:a aac -b:a -ar 48000` | | `feed` (feed not live) | no | `-map [vout] -an` | ### 4.5 Full ffmpeg argv (per destination) ``` ffmpeg -hide_banner -loglevel warning -i -i ... -i -filter_complex -map [vout]