galaxy/specs/APP-3805/TECH.md

TECH.md — Client/Server Version Skew for Remote Server

Linear: APP-3805

1. Problem

The Warp remote server binary is installed at a single, unversioned path per channel (e.g. ~/.warp/remote-server/oz). The existence check is test -x {bin} and we never inspect the binary's version before talking to it. When the client auto-updates to a new version, it happily reuses the old remote server binary, which can drift arbitrarily far from the protocol/behaviour the client expects. The InitializeResponse already carries server_version, but the client ignores it.

We need a version-gated install flow: connecting from a client at version V always ends up talking to a server binary also at V. Any local cargo run workflow (where the client has no GIT_RELEASE_TAG) keeps working with script/deploy_remote_server: a deployed binary at the unversioned path always wins, and when one is missing the client falls back to installing latest-for-channel at the same unversioned path so the dev loop self-heals.

2. Requirements

R1. Exact version match. A connected client at version V must only communicate with a remote daemon spawned from a binary at the same version V. No silent skew.

R2. Automatic reinstall on drift. When the installed binary is the wrong version (or missing), the client reinstalls the correct version as part of the connect flow. No manual rm -rf step.

R3. Cheap happy path. When the correct binary is already installed, connecting should require no extra downloads and no extra SSH round-trips beyond today's single test -x check.

R4. Local dev workflow preserved. Channel::Local clients (the default cargo run) must keep working with script/deploy_remote_server. When a deployed binary exists at the unversioned path, the client uses it without re-downloading from the CDN. When no binary exists, the client auto-installs latest-for-channel at the same unversioned path so a stale install directory self-heals. The unversioned filename is reserved for Channel::Local and Channel::Oss (which has no release-pinned CDN artifact and is treated identically); every other channel always uses a versioned filename, so versioned-channel builds can never silently overwrite a script/deploy_remote_server-owned slot.

R5. Clear failure surface. If install or version validation fails, the user sees an actionable setup-failed state rather than a confusing protocol error.

R6. Defense-in-depth. A hand-placed or half-written binary at the expected path shouldn't be trusted blindly — the version reported at the handshake must also be verified.

R7. No regression in storage behaviour. We don't introduce unbounded disk growth, and we don't break existing installs at the current unversioned path during rollout.

3. Current connect flow

The install + handshake path this spec mutates spans four files. The sequence below shows today's behaviour; §4 slots version-aware steps into the marked decision points.

sequenceDiagram
    participant UI as Client UI
    participant Mgr as RemoteServerManager
    participant Tx as SshTransport
    participant Remote as Remote host
    participant Daemon as oz daemon

    UI->>Mgr: connect_session
    Mgr->>Tx: setup(session_id)
    Note over Tx: today: path = {dir}/{binary_name}<br/>(no version in filename)
    Tx->>Remote: ssh: test -x {path}
    alt binary missing
        Tx->>Remote: ssh: install_remote_server.sh<br/>(curl → tar → mv → {path})
    end
    Tx-->>Mgr: SetupReady
    Mgr->>Tx: connect()
    Tx->>Remote: ssh: `{path} remote-server-proxy`
    Remote->>Daemon: spawn (if not already running)
    Mgr->>Daemon: Initialize
    Daemon-->>Mgr: InitializeResponse { server_version, host_id }
    Note over Mgr: today: server_version is ignored
    Mgr->>Mgr: mark_session_connected

Key pieces of today's behaviour the diagram elides:

• setup::remote_server_binary() resolves the path purely from the channel; there is no version in the filename.
• install_remote_server.sh pulls from {server_root_url}/download/cli?package=tar&os=...&arch=...&channel={channel} with no version pin. The warp-server /download/cli endpoint already accepts a version= query parameter that pins the artifact to an exact release (and falls back to latest-for-channel when omitted), so this spec only needs client-side changes.
• ChannelState::app_version() returns option_env!("GIT_RELEASE_TAG") — Some("v0.…") on release builds, None on cargo run. This is the signal we'll thread through in §4.
• script/deploy_remote_server is the developer escape hatch: it rsyncs a locally-built binary into ~/.warp-local/remote-server/oz-local and assumes the client won't try to download on top of it.

4. Proposed solution

Encode the client's expected version into the installed binary's filename, so version drift turns into a missing-file miss that naturally re-triggers the existing install flow. Layer a handshake-level version check on top as a safety net, and reserve the unversioned filename for Channel::Local and Channel::Oss.

4.1 Channel-keyed binary paths

In crates/remote_server/src/setup.rs, the path resolution is keyed strictly off [Channel]:

• remote_server_binary():
  • When ChannelState::channel() is Channel::Local or Channel::Oss → {dir}/{binary_name} (the unversioned deploy_remote_server slot; Oss has no release-pinned CDN artifact and follows the same convention).
  • For every other channel (Stable, Preview, Dev, Integration) → {dir}/{binary_name}-{v}, where v = ChannelState::app_version().unwrap_or(env!("CARGO_PKG_VERSION")).
• binary_check_command() keeps the single test -x {remote_server_binary()} contract. Because the version is part of the filename on every versioned channel, any drift resolves to a miss on the existing code path and re-runs install_script. The CARGO_PKG_VERSION fallback is intentionally not expected to point at a real release artifact; see §4.4 for the failure shape on versioned-channel + no-tag builds.

4.2 Install script pins the exact version (or latest, for Local/Oss)

warp-server's /download/cli already honours a version= query parameter (pins the redirect to the exact versioned artifact when present; falls back to latest-for-channel when absent), so no server-side change is needed. In install_remote_server.sh, add {version_query} and {version_suffix} placeholders used in two places:

• Download URL query string: ...&channel={channel}{version_query} (e.g. &version=v0.…, or empty).
• Final install path: mv "$bin" "$install_dir/{binary_name}{version_suffix}" (e.g. -v0.…, or empty). In setup.rs::install_script(), substitute based on ChannelState::channel():
• Channel::Local and Channel::Oss → both substitutions are empty strings. These channels download latest-for-channel and install at the unversioned {dir}/{binary_name} path that script/deploy_remote_server also uses.
• Every other channel → version_query = "&version={v}", version_suffix = "-{v}", where v resolves via app_version().unwrap_or(CARGO_PKG_VERSION). Release-tagged clients pin the exact published version; versioned-channel builds without a release tag pin CARGO_PKG_VERSION, which deliberately doesn't map to a real /download/cli artifact. Unknown versions (rolled-back releases, or the CARGO_PKG_VERSION versioned-channel fallback) surface as a GCS 404 on the redirected URL, which curl -fSL turns into a non-zero exit and SetupFailed for the user.

4.3 Handshake version validation

In crates/remote_server/src/manager.rs::connect_session, immediately after client.initialize().await returns Ok(resp):

• Let client_v = ChannelState::app_version() and server_v = resp.server_version.
• If both are Some (or server returned a non-empty string) and they differ → log the skew, delete the versioned binary on the remote host (see below), then emit RemoteServerManagerEvent::SessionDisconnected and call mark_session_disconnected.
• If both sides are unknown (client None and server reports an empty string) → accept and log a warning. This is the cargo run (Local/Oss) + script/deploy_remote_server dev loop where neither side carries a release tag.
• Any other shape (one side has a version, the other doesn't) → treated as incompatible. This is defense-in-depth: in normal operation the path-based check (§4.1) would already have triggered a reinstall to bring both sides into agreement, so a mixed shape at the handshake means something unusual (hand-placed binary, partial download) and we prefer to tear the session down and reinstall.

Why delete the binary on mismatch? The handshake check only fires when filename-based detection (§4.1) said we had the right version but the running daemon disagrees. That means the file at the versioned path is wrong — partial download, hand-placed, or corrupted. If we only disconnect, the next reconnect will re-run ensure_binary_installed, see test -x {path} succeed (the file still exists), skip install, respawn the daemon, and hit the same mismatch — a reconnect loop. Deleting the binary forces the next ensure_binary_installed to miss and reinstall, breaking the loop in a single extra SSH command (ssh rm -f {path}).

Factor the comparison into a pure helper (fn version_is_compatible(client: Option<&str>, server: &str) -> bool) so it's unit-testable without wiring up a client.

4.4 Unversioned channels (Local / Oss) and versioned-channel-without-tag

Channel::Local (the default cargo run) and Channel::Oss take the unversioned branch of remote_server_binary() / install_script() (see §4.1, §4.2). Concretely:

• If script/deploy_remote_server has put a binary at the unversioned path, binary_check_command succeeds and we connect without touching the CDN.
• If the unversioned binary is missing but the install directory exists (e.g. a previous install ran here), the controller's auto-update branch fires and install_script runs with empty version_query/version_suffix — i.e. it pulls latest-for-channel and installs at the same unversioned path. Future deploy_remote_server runs simply overwrite this in place.
• If the install directory is also missing (truly fresh host), we fall through to the normal install-mode path (AlwaysAsk → user modal, etc.). script/deploy_remote_server is not touched; its target path (~/.warp-local/remote-server/oz-local on the local channel) is exactly what the Channel::Local branch of remote_server_binary() returns. Versioned-channel builds without a release tag (e.g. cargo run --bin dev, --bin preview) are deliberately unsupported for SSH remote-server installs. The CARGO_PKG_VERSION fallback (§4.1) keeps the path deterministic but the resulting &version=CARGO_PKG_VERSION query 404s against /download/cli, surfacing a clean BinaryInstallComplete::Err(_) and the existing failed-banner path. Developers who need to test against a real channel should either build with GIT_RELEASE_TAG set or use cargo run (Local) + script/deploy_remote_server.

4.5 Rollout / backwards compatibility

The first release with this change will, for every user, look for a versioned path that does not yet exist on their remote host. That's fine — it falls through to the normal install path and overwrites into the new versioned filename. The legacy unversioned binary ({dir}/{binary_name}) is simply orphaned on disk. We accept that (see §4.6) and can sweep it later.

4.6 Deliberate non-goals

• Cleanup of old versioned binaries. Accept accumulation for v1; easy to add later as a post-install step that keeps the current + previous version.
• Upload-over-SSH fallback when the remote can't reach the CDN. Reasonable future work, not required here.
• Protocol change to send client version in Initialize. We only read server_version off the response; the reverse direction can be added if/when the server wants to reject proactively.

5. How the solution maps to the requirements

• R1 (exact match). Versioned filename (§4.1) + handshake check (§4.3) give us two independent enforcements of exact match.
• R2 (automatic reinstall). A new version turns into a path miss (§4.1), which re-enters the existing install_script path (§4.2). No manual step.
• R3 (cheap happy path). Still a single test -x SSH command on the hot path; handshake check is a string compare on a response we already receive.
• R4 (local dev workflow). Channel::Local and Channel::Oss short-circuit to the unversioned path for both the install location and the existence check (§4.1). A deployed binary always wins via test -x; if missing but the install directory exists, the install script runs with empty version_query/version_suffix and pulls latest-for-channel to the same unversioned path (§4.2, §4.4). deploy_remote_server is unchanged and overwrites in place. Versioned-channel builds without a release tag fail cleanly via the CARGO_PKG_VERSION 404 path so they can never silently overwrite a Local/Oss-owned slot.
• R5 (clear failure). Install errors (CDN 404, network failure, etc.) propagate via BinaryInstallComplete { result: Err(_) } and the controller surfaces them through the existing show_ssh_remote_server_failed_banner path. Handshake version mismatches reach the same banner via SessionConnectionFailed { phase: Initialize } (§4.3).
• R6 (defense-in-depth). Handshake check catches stale or hand-placed binaries even when the filename is right (§4.3).
• R7 (no regression). Existing unversioned install is simply orphaned after the first upgrade; no code assumes its absence or presence (§4.5).

6. Testing and validation

Covers each requirement with a concrete check:

• Unit tests in crates/remote_server/src/setup_tests.rs (uses the existing test-util ChannelState::set_app_version hook):
  • remote_server_binary() returns a {name}-{version} path on every versioned channel (Stable, Preview, Dev, Integration), whether app_version() is Some or falls back to CARGO_PKG_VERSION. (R1, R2)
  • remote_server_binary() returns the bare {name} path on Channel::Local and Channel::Oss, regardless of app_version(). (R4)
  • install_script() substitutes {version_query} / {version_suffix} into both the URL and the install path on every versioned channel. (R1, R2)
  • install_script() substitutes empty strings for both placeholders on Channel::Local and Channel::Oss, so the script downloads latest-for-channel and installs at the unversioned path. (R4)
• Unit tests for version_is_compatible covering:
  • Matching Some("v…") on both sides → compatible. (R1)
  • Differing Some/Some → incompatible. (R1, R6)
  • Both unknown (client None, server "") → compatible with warning. (R4)
  • Mixed shape (Some/empty, None/non-empty) → incompatible. (R6)
• Manual validation (captured in the PR description):
  • Connect once at version V; confirm the binary lands at {dir}/{name}-{V} and the session comes up. (R1, R2, R3)
  • Bump the client to version V+1 (or stub the version in ChannelState::set_app_version), reconnect; confirm a new install runs and the connection succeeds. (R1, R2)
  • Corrupt the versioned binary (e.g. replace its contents with a stub) so the handshake reports a different version; confirm the session is torn down with SessionDisconnected and a user-visible error. (R6)
  • cargo run (Local) against a host where the install directory exists but the unversioned binary is missing; confirm the controller auto-installs latest-for-channel at the unversioned path and the session comes up. (R4)
  • After running script/deploy_remote_server, reconnect with cargo run (Local) and confirm test -x succeeds and we connect without re-downloading. (R4)
• Existing remote-server manager tests continue to pass unchanged (no behavioural change for the None/"" path).

7. Risks and mitigations

• Churn on every release. Each stable/preview/dev release will pull a fresh binary on every host the user connects to the first time. Expected; bandwidth impact is minor.
• Orphaned binaries accumulate. See §4.6; acceptable for v1, cleanup is a straightforward follow-up.
• Repeat handshake mismatches. Addressed by deleting the offending binary on mismatch (§4.3) so the next reconnect reinstalls rather than looping. Worth explicit test coverage (§6).

8. Open questions

• Should Initialize (client → server) also carry the client's expected version so the server can reject proactively, or is client-side enforcement enough? Leaning "enough" for now.
• Do we want to surface the "version-skew detected, retrying" state as a distinct UI affordance, or is the standard reconnect flow sufficient? Current plan: reconnect-only, revisit if it surfaces as a usability issue.