vellum-ai / watch-together

Watch Together

Real-time co-watching. An editor model watches playback continuously and wakes
the assistant only at moments worth reacting to. Each wake includes dialogue
and story-critical frames. Wakes arrive through signal files, so do not poll.

Choose a mode

Source mode (preferred)

watch-file.py plays a local media file in mpv and reads that file directly.
It does not capture the screen or require a loopback audio driver. Frames come
from the source at full quality. Sidecar .srt subtitles or the first embedded
text subtitle track provide exact dialogue. Without subtitles, the editor
transcribes audio. Pause stops the flow and seeks resynchronize it.

Use source mode whenever the user has a local media file. The bundled script
does not accept stream URLs.

Screen mode (fallback)

capture_live.py records supported desktops through ffmpeg and feeds completed
segments to the same editor pipeline.

• macOS: Uses AVFoundation. It auto-detects BlackHole for audio and records
  video only when BlackHole is unavailable.

• Windows: Uses ffmpeg gdigrab. It records video only by default. System
  audio requires an existing DirectShow playback source such as Stereo Mix or
  a configured virtual audio cable, passed explicitly as the audio device.

• Linux: Uses X11 capture and $DISPLAY. It records video only by default.
  System audio requires an existing PulseAudio or PipeWire-Pulse monitor source
  passed explicitly. Native Wayland screen capture is not supported by this
  script.

Screen capture cannot bypass DRM or protected-video blanking. If browser video
is black in the recording, explain that screen mode is unsupported for that
content. Do not claim the fallback works around content protection.

How it works

1. Playback is divided into 60-second windows.

2. editor.py reviews each window, selects story-critical frames, collects
   dialogue, and decides whether to wake now or hold a developing moment.

3. A wake extracts the selected frames at 720p and sends a [WATCH] message to
   the active conversation.

4. The assistant looks at the frames and dialogue and reacts only when useful.

The editor chooses when the assistant looks and what evidence it receives. The
assistant decides whether to speak and how to react. A hard cap controlled by
WATCH_MAX_HOLD prevents indefinite holds.

When a [WATCH] message arrives

1. Inspect the attached frames with vision.
2. Read the dialogue as the primary source for the window.

3. React naturally. A short line, a stage direction, or silence can all be
   appropriate. Reserve longer reactions for moments that earn them.

4. Track theories, callbacks, and running jokes in replies so memory can carry
   them between sessions.

If a moment needs closer inspection, extract dense 720p frames from the source
named in the wake.

macOS or Linux:

python3 "$VELLUM_WORKSPACE_DIR/watch-together/scripts/rewind.py" \
  <source-file> <output-dir> <start-seconds> <end-seconds>

The existing rewind.sh command remains a POSIX compatibility wrapper.

Windows PowerShell:

py -3 "$env:VELLUM_WORKSPACE_DIR\watch-together\scripts\rewind.py" `
  <source-file> <output-dir> <start-seconds> <end-seconds>

If the Python launcher is unavailable but python --version reports Python 3,
replace py -3 with python.

Start a session

Choose a stable lowercase session ID such as example-show-s1e1. Do not include
personal data in it.

macOS or Linux

Create the session directory:

SESSION_ID="example-show-s1e1"
mkdir -p "$VELLUM_WORKSPACE_DIR/watch-together/sessions/$SESSION_ID"

Source mode:

python3 "$VELLUM_WORKSPACE_DIR/watch-together/scripts/watch-file.py" \
  <local-media-file> \
  "$VELLUM_WORKSPACE_DIR/watch-together/sessions/$SESSION_ID" \
  <conversation-id>

Screen mode:

python3 "$VELLUM_WORKSPACE_DIR/watch-together/scripts/capture_live.py" \
  "$VELLUM_WORKSPACE_DIR/watch-together/sessions/$SESSION_ID" \
  <conversation-id>

On macOS, the optional positional arguments are chunk seconds, AVFoundation
screen index, and AVFoundation audio index. The default screen index remains
2, and BlackHole is auto-detected when no audio index is supplied.

On Linux, $DISPLAY is used by default. To add system audio, pass chunk seconds,
the X11 display, and a Pulse monitor source:

python3 "$VELLUM_WORKSPACE_DIR/watch-together/scripts/capture_live.py" \
  "$VELLUM_WORKSPACE_DIR/watch-together/sessions/$SESSION_ID" \
  <conversation-id> 60 "$DISPLAY" <pulse-monitor-source>

Windows PowerShell

Create the session directory:

$SessionId = "example-show-s1e1"
$SessionDir = Join-Path $env:VELLUM_WORKSPACE_DIR "watch-together\sessions\$SessionId"
New-Item -ItemType Directory -Force $SessionDir | Out-Null

Source mode:

py -3 "$env:VELLUM_WORKSPACE_DIR\watch-together\scripts\watch-file.py" `
  <local-media-file> $SessionDir <conversation-id>

Screen mode, video only:

py -3 "$env:VELLUM_WORKSPACE_DIR\watch-together\scripts\capture_live.py" `
  $SessionDir <conversation-id>

Screen mode with a configured DirectShow playback source:

ffmpeg -list_devices true -f dshow -i dummy
py -3 "$env:VELLUM_WORKSPACE_DIR\watch-together\scripts\capture_live.py" `
  $SessionDir <conversation-id> 60 desktop "<DirectShow-audio-device>"

Do not describe the optional Windows audio path as automatic loopback. The user
must already have a playback capture source exposed through DirectShow.

After giving the matching command, tell the user to start the show. When mpv
exits or screen capture stops, the final window is flushed.

The conversation ID is the bare UUID from the conversation database record,
not the timestamped conversation folder name. Using the folder name routes the
wake incorrectly.

Cost setup

Watching produces many small turns. Recommend normal inference mode and an
inference profile with a reduced context ceiling so the conversation compacts
regularly. Example:

"llm": {
  "profiles": {
    "watch-mode": {
      "contextWindow": { "maxInputTokens": 200000 }
    }
  }
}

Environment variables

• GEMINI_API_KEY: Enables editor verdicts and transcription. Without it,
  fixed-cadence wakes use evenly spaced frames and no editor transcription.

• GEMINI_MODEL: Editor model. Defaults to gemini-3.7-flash.
• WATCH_MAX_HOLD: Maximum seconds between wakes. Defaults to 240.
• WATCH_MAX_FRAMES: Maximum frames attached per wake. Defaults to 8.

• WATCH_MPV_ARGS: Extra mpv arguments for source mode. Quoted arguments use
  the current platform's command-line rules.

Prerequisites

All modes require Python 3 and ffmpeg. Source mode also requires mpv. Confirm
each command is on PATH before starting.

macOS:

brew install ffmpeg mpv
brew install blackhole-2ch  # optional screen-mode system audio

Configure a Multi-Output Device in Audio MIDI Setup when using BlackHole.

Windows: Install current Python 3, ffmpeg, and mpv builds, add them to
PATH, then verify:

py -3 --version
ffmpeg -version
ffprobe -version
mpv --version

Linux: Install the distro packages. For Debian or Ubuntu:

sudo apt install python3 ffmpeg mpv

For Fedora:

sudo dnf install python3 ffmpeg mpv

File locations

• Scripts: $VELLUM_WORKSPACE_DIR/watch-together/scripts/

• Sessions: $VELLUM_WORKSPACE_DIR/watch-together/sessions/<session-id>/

  • chunks/: Recorded segments from screen mode
  • editor/verdicts/: Per-window editor output
  • wakes/wake-NNN/: Frames attached to each wake
  • editor-state.json: Held-window state
  • subs.srt: Extracted source-mode subtitles
  • mpv.sock: POSIX source-mode IPC socket; Windows uses a named pipe

• Signals: JSON files at
  $VELLUM_WORKSPACE_DIR/signals/user-message.<requestId> with optional image
  attachments