galaxy/specs/APP-4281/TECH.md

APP-4281: Gate remote-server install on a remote-side preinstall check

Linear: APP-4281

Context

The prebuilt Linux oz CLI is built on the namespace-profile-ubuntu-20-04 runner (.github/workflows/create_release.yml:513, :668, :839). That toolchain links against glibc 2.31, so the resulting binary carries glibc 2.29-era symbol versions in its dynamic table. When the install script (crates/remote_server/src/install_remote_server.sh) drops that binary onto a Linux host whose runtime glibc is older than ~2.29, the dynamic loader refuses to launch it:

/lib64/libm.so.6: version `GLIBC_2.29' not found (required by /home/wasp-dev/.warp-preview/remote-server/oz-preview)

This affects long-lived enterprise distros — RHEL/CentOS 7 (glibc 2.17), RHEL/CentOS 8 (glibc 2.28), Amazon Linux 2 (glibc 2.26), Ubuntu 18.04 (glibc 2.27), Debian 10 (glibc 2.28) — as well as non-glibc systems like Alpine (musl) and Termux (bionic).

Today the setup pipeline does not consult the remote host's capabilities. RemoteServerController::on_binary_check_complete (app/src/terminal/writeable_pty/remote_server_controller.rs:202) decides between install, auto-update, prompt, and fall-back purely from Result<bool, String> (binary present?) and has_old_binary. Once install_binary succeeds the controller advances to connect_session, the SSH proxy spawns oz remote-server-proxy, and the loader's GLIBC_… error surfaces only at connect time as an opaque SetupFailed. The product spec (specs/APP-4281/PRODUCT.md) calls for surfacing no install UI in that case — the user should land directly in the legacy SSH flow.

The legacy SSH/RemoteCommandExecutor flow is already a first-class outcome of the controller's state machine: it is reached today via SshExtensionInstallMode::NeverInstall and via the Err(_) arm of on_binary_check_complete (remote_server_controller.rs:281, :286), both of which call flush_stashed_bootstrap to release the stashed bootstrap so Sessions::initialize_bootstrapped_session wires up the ControlMaster-backed RemoteCommandExecutor. We reuse that path for unsupported hosts.

Goals

Run a single preinstall check script over the existing SSH connection — before any install UI surfaces — that decides whether the host can run the prebuilt remote-server binary, and gate every user-visible install affordance (choice block, AlwaysInstall auto-install, has_old_binary auto-update) on its result. When the host is positively unsupported, fall back silently to the legacy SSH flow. When the check is inconclusive, fail open and proceed as today.

Make the script the single source of truth for "can this host run the binary?" so future capability checks (additional shared libs, kernel version, free disk, presence of curl/tar) are an additive, script-only change.

Non-goals

This spec does not lower the glibc floor of the prebuilt binary, ship multiple Linux artifacts targeting different glibc versions, or expose user-visible UI explaining the fall-back. Those are tracked as follow-ups.

Relevant code

• crates/remote_server/src/setup.rs — RemoteServerSetupState, RemotePlatform/RemoteOs/RemoteArch, parse_uname_output (:94), binary_check_command, install_script, CHECK_TIMEOUT.
• crates/remote_server/src/install_remote_server.sh — existing install script template; the new preinstall script lives next to it.
• crates/remote_server/src/transport.rs — RemoteTransport trait (:63).
• crates/remote_server/src/manager.rs — RemoteServerManager::check_binary (:452), RemoteServerManagerEvent::BinaryCheckComplete (:311), session state machine.
• crates/remote_server/src/ssh.rs — run_ssh_command, run_ssh_script (used by install_binary; reused for the preinstall check).
• app/src/remote_server/ssh_transport.rs — SshTransport impl of RemoteTransport.
• app/src/terminal/writeable_pty/remote_server_controller.rs — SshInitState, on_binary_check_complete (:202), flush_stashed_bootstrap (:150).
• app/src/terminal/prompt_render_helper.rs:248 and app/src/terminal/view.rs:11491 — UI surfaces that match on RemoteServerSetupState.

Proposed changes

1. New preinstall_check.sh

A new shell script template alongside install_remote_server.sh. It runs before any install UI surfaces and emits a structured, machine-parseable summary on stdout. The script is the only place we encode "can this host run the prebuilt binary?" — the client side just reads the verdict.

Output format: one key=value pair per line. Unknown keys are ignored on the client, so the script can grow new checks without a coordinated client release. Required keys for v1:

status=supported|unsupported|unknown
reason=<short identifier when unsupported, omitted otherwise>
libc_family=glibc|musl|bionic|uclibc|unknown
libc_version=<major.minor when libc_family=glibc, omitted otherwise>
required_glibc=<major.minor>

Script (lives at crates/remote_server/src/preinstall_check.sh):

#!/usr/bin/env bash
# Preinstall check for the Warp remote-server binary.
#
# Emits a structured key=value summary on stdout. Exits 0 on success.
# A non-zero exit indicates a probe-level failure; the client treats
# those as `status=unknown` (fail open).

set -u

# The minimum glibc the prebuilt Linux CLI requires. The Linux CLI is
# built on Ubuntu 20.04 (see `.github/workflows/create_release.yml`),
# which ships glibc 2.31. Bump this when the runner image is bumped.
required_glibc="2.31"
echo "required_glibc=${required_glibc}"

# 1. Detect libc family and (when glibc) its version.
libc_family="unknown"
libc_version=""

if version=$(getconf GNU_LIBC_VERSION 2>/dev/null); then
    # Output: "glibc 2.31"
    libc_family="glibc"
    libc_version="${version##* }"
elif ldd_out=$(ldd --version 2>&1 | head -n1); then
    case "$ldd_out" in
        *musl*)   libc_family="musl"   ;;
        *uClibc*) libc_family="uclibc" ;;
        *)
            v=$(printf '%s\n' "$ldd_out" | grep -oE '[0-9]+\.[0-9]+' | head -n1)
            if [ -n "$v" ]; then
                libc_family="glibc"
                libc_version="$v"
            fi
            ;;
    esac
fi

echo "libc_family=${libc_family}"
[ -n "$libc_version" ] && echo "libc_version=${libc_version}"

# 2. Decide status from the gathered facts.
status="unknown"
reason=""

if [ "$libc_family" = "glibc" ] && [ -n "$libc_version" ]; then
    have_major="${libc_version%%.*}"
    have_minor="${libc_version#*.}"
    have_minor="${have_minor%%.*}"
    req_major="${required_glibc%%.*}"
    req_minor="${required_glibc#*.}"
    if [ "$have_major" -gt "$req_major" ] \
       || { [ "$have_major" -eq "$req_major" ] && [ "$have_minor" -ge "$req_minor" ]; }; then
        status="supported"
    else
        status="unsupported"
        reason="glibc_too_old"
    fi
elif [ "$libc_family" = "musl" ] || [ "$libc_family" = "bionic" ] || [ "$libc_family" = "uclibc" ]; then
    status="unsupported"
    reason="non_glibc"
fi

echo "status=${status}"
[ -n "$reason" ] && echo "reason=${reason}"

The script is loaded with include_str! — no templating, since the floor is hardcoded into the script itself for now:

pub const PREINSTALL_CHECK_SCRIPT: &str = include_str!("preinstall_check.sh");

Keeping the floor inside the script keeps the bash side self-contained and avoids splitting the supported-glibc value across two source files. If we later need to template the script (e.g. to inject the artifact's actual symbol-version floor at release time), we can switch back to a replace helper without touching the trait or controller. The required_glibc value still rides on the script's stdout (required_glibc=2.31) so the Rust parser can populate UnsupportedReason::GlibcTooOld { required } directly from the script's report rather than a separate Rust constant.

2. PreinstallCheckResult and parser

Add to setup.rs:

#[derive(Clone, Debug, PartialEq, Eq)]
pub struct PreinstallCheckResult {
    pub status: PreinstallStatus,
    pub libc: RemoteLibc,
    /// Verbatim script stdout (trimmed). Forwarded to telemetry for
    /// diagnostics on hosts that report `Unknown`.
    pub raw: String,
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum PreinstallStatus {
    Supported,
    Unsupported { reason: UnsupportedReason },
    Unknown,
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum UnsupportedReason {
    GlibcTooOld { detected: (u32, u32), required: (u32, u32) },
    NonGlibc { name: String },
}

#[derive(Clone, Debug, PartialEq, Eq)]
pub enum RemoteLibc {
    Glibc { major: u32, minor: u32 },
    NonGlibc { name: String },
    Unknown,
}

impl PreinstallCheckResult {
    pub fn is_supported(&self) -> bool {
        match self.status {
            // Fail open on Unknown — see §3.
            PreinstallStatus::Supported | PreinstallStatus::Unknown => true,
            PreinstallStatus::Unsupported { .. } => false,
        }
    }
}

pub fn parse_preinstall_output(stdout: &str) -> PreinstallCheckResult { /* … */ }

Parser rules:

• Treat stdout as a list of key=value lines; lines without = and unknown keys are ignored (forward-compatibility).
• status=supported → PreinstallStatus::Supported.
• status=unsupported + reason=glibc_too_old → Unsupported { GlibcTooOld { detected, required } } populated from libc_version and required_glibc. Missing/malformed numbers → PreinstallStatus::Unknown.
• status=unsupported + reason=non_glibc → Unsupported { NonGlibc { name: libc_family } }.
• Any other / missing status → PreinstallStatus::Unknown.
• libc_family + libc_version populate the libc field independently of status, so telemetry has the underlying signal even on Unknown.

3. Fail-open semantics

is_supported() returns true for both Supported and Unknown. Hosts where the script could not classify the libc (no getconf, weird ldd output, exotic distro, busybox-only environments) keep today's install-and-try behavior. Only positive detection of an incompatible libc — a glibc version below the script's hardcoded floor, or any non-glibc libc — triggers the silent fall-back.

The supported-glibc floor lives in preinstall_check.sh itself (currently required_glibc="2.31", matching the Ubuntu 20.04 build runner). Bumping the build image is a script-only change. The Rust side does not duplicate the value; it reads required_glibc back out of the script's stdout when constructing telemetry and UnsupportedReason::GlibcTooOld { required }.

4. RemoteTransport::run_preinstall_check

Extend RemoteTransport with a single new method (matching the boxed-future style of the existing probes — see transport.rs:68-96):

fn run_preinstall_check(
    &self,
) -> Pin<Box<dyn Future<Output = Result<PreinstallCheckResult, String>> + Send>>;

SshTransport::run_preinstall_check pipes setup::PREINSTALL_CHECK_SCRIPT through the existing ControlMaster socket via remote_server::ssh::run_ssh_script (the same helper install_binary already uses), with CHECK_TIMEOUT. On success it parses stdout into PreinstallCheckResult via parse_preinstall_output. On SSH-level failure (timeout, broken pipe, non-zero exit with no parseable summary) it returns Err(_), which the manager logs and treats as inconclusive (the controller then falls into the existing fail-open path).

5. Run before deciding install UI

RemoteServerManager::check_binary (manager.rs:452) keeps the existing futures::join! over the three concurrent probes and adds the preinstall script call afterwards, gated on Linux:

let (platform_result, check_result, old_binary_result) = futures::join!(
    transport.detect_platform(),
    transport.check_binary(),
    transport.check_has_old_binary(),
);
let preinstall = match &platform_result {
    Ok(p) if matches!(p.os, RemoteOs::Linux) => match transport.run_preinstall_check().await {
        Ok(r)  => Some(r),
        Err(e) => {
            log::warn!("preinstall check failed for session {session_id:?}: {e}");
            None
        }
    },
    _ => None,
};

The result rides on BinaryCheckComplete:

BinaryCheckComplete {
    session_id: SessionId,
    result: Result<bool, String>,
    remote_platform: Option<RemotePlatform>,
    preinstall_check: Option<PreinstallCheckResult>,
    has_old_binary: bool,
}

RemoteServerManager also caches the preinstall result per-session (mirroring session_platforms at manager.rs:414) so it is available on later events for telemetry.

Sequencing the preinstall check after detect_platform (instead of folding it into the join!) keeps macOS hosts at zero extra round-trips. Linux hosts pay one additional ControlMaster channel — same cost class as check_has_old_binary, multiplexed through the existing socket.

6. Controller: gate the install UI on the preinstall result

This is the user-visible point of the change. RemoteServerController::on_binary_check_complete (remote_server_controller.rs:202) now performs the preinstall gate before any of the existing branches that surface the choice block (AlwaysAsk → request_remote_server_block), force install (AlwaysInstall), auto-update (has_old_binary), or connect:

if let Some(check) = preinstall_check.as_ref() {
    if !check.is_supported() {
        log::info!(
            "Preinstall check returned {:?} for {session_id:?}; \
             falling back to legacy SSH",
            check.status,
        );
        send_telemetry_from_ctx!(
            TelemetryEvent::RemoteServerHostUnsupported { /* … */ },
            ctx,
        );
        RemoteServerManager::handle(ctx).update(ctx, |mgr, ctx| {
            mgr.mark_setup_unsupported(session_id, check.clone(), ctx);
        });
        // Best-effort cleanup of any prior install on this host —
        // it cannot launch and would otherwise force the auto-update
        // path on every reconnect.
        if let Ok(true) = result {
            mgr.schedule_remove_remote_server_binary(session_id, &transport, ctx);
        }
        self.flush_stashed_bootstrap(session_info, ctx);
        return;
    }
}
// Existing branches unchanged below this point:
//   Ok(true)                   -> connect_session
//   Ok(false) + has_old_binary -> install_binary(is_update=true)
//   Ok(false) + AlwaysAsk      -> request_remote_server_block
//   Ok(false) + AlwaysInstall  -> install_binary(is_update=false)
//   Ok(false) + NeverInstall   -> flush_stashed_bootstrap
//   Err(_)                     -> flush_stashed_bootstrap

Because this branch runs before request_remote_server_block, an unsupported host never sees the choice block — exactly what the product spec requires for the "ideal path." The legacy SSH flow is reached via the existing flush_stashed_bootstrap exit, so no new fall-back code path is added — only a new entry point into the existing one.

7. New Unsupported setup state

Extend RemoteServerSetupState (setup.rs:8) with a non-error terminal variant so the controller can distinguish "the remote is incompatible, fall back silently" from "the install failed, surface a real error":

pub enum RemoteServerSetupState {
    Checking,
    Installing { progress_percent: Option<u8> },
    Updating,
    Initializing,
    Ready,
    Failed { error: String },
    /// Preinstall check classified the host as incompatible. Treated
    /// as a clean fall-back to the legacy ControlMaster-backed SSH flow.
    Unsupported { reason: UnsupportedReason },
}

is_terminal() and is_in_progress() are extended so Unsupported behaves like Failed for downstream code that asks "is this still in flight?" The two existing UI sites (prompt_render_helper.rs:255 and view.rs:11492) already fall through to a _ => arm and need no changes; the user sees the same prompt they see on a NeverInstall legacy SSH session today.

8. Stale-install cleanup

Hosts that connected before this change may have an oz binary on disk that can no longer launch. When the controller takes the unsupported branch with result == Ok(true), it asks the manager to schedule a best-effort transport.remove_remote_server_binary() (already implemented at ssh_transport.rs:235). The call is fire-and-forget: failure is logged and does not block the legacy fall-back. Without this, check_has_old_binary would return true on every reconnect and the controller would silently re-enter the auto-update path against a host that now reports Unsupported.

9. Telemetry

Replace the previously-proposed RemoteServerLibcUnsupported with a script-shaped event:

TelemetryEvent::RemoteServerHostUnsupported {
    remote_os: Option<String>,
    remote_arch: Option<String>,
    status: String,                 // "unsupported" | "unknown"
    reason: Option<String>,         // "glibc_too_old" | "non_glibc"
    detected_libc: String,          // "glibc 2.28", "musl", "unknown"
    required_glibc: String,         // "2.31"
    had_old_binary: bool,
    /// First 256 bytes of the script's stdout, for diagnosing
    /// `Unknown` outcomes on exotic distros.
    script_stdout_preview: String,
}

Also extend the existing RemoteServerSetupDuration (sent from on_session_connected at remote_server_controller.rs:372) with remote_libc: Option<String> so we can correlate setup latency with libc distribution on supported hosts and watch for regressions after future required_glibc bumps in the script.

End-to-end flow

Supported Linux host (e.g. Ubuntu 22.04, glibc 2.35)

1. SSH session opens; on_ssh_init_shell_requested schedules check_binary.
2. Manager runs uname -sm, test -x …, test -d …, then on Linux pipes preinstall_check.sh through the SSH socket.
3. Script emits status=supported, libc_family=glibc, libc_version=2.35.
4. BinaryCheckComplete { preinstall_check: Some(Supported), … } arrives at the controller.
5. The fail-open gate is bypassed; existing branches drive install/auto-update/prompt/connect as today.

Unsupported Linux host (e.g. RHEL 7, glibc 2.17)

1. SSH session opens; check_binary runs as above.
2. Script emits status=unsupported, reason=glibc_too_old, libc_version=2.17.
3. BinaryCheckComplete { preinstall_check: Some(Unsupported { GlibcTooOld { … } }), … } arrives.
4. Controller logs, emits RemoteServerHostUnsupported, asks the manager to mark the session Unsupported, optionally schedules remove_remote_server_binary, and calls flush_stashed_bootstrap.
5. Sessions::initialize_bootstrapped_session wires up RemoteCommandExecutor against the existing ControlMaster socket. The user sees the legacy SSH prompt with no modal, no banner, no error.

Inconclusive Linux host (e.g. minimal busybox container)

1. Script emits status=unknown (no getconf, ldd output unparseable).
2. is_supported() returns true; existing branches run as today, including the choice block under AlwaysAsk. If the install or connect later fails, the existing Failed banner path takes over.

macOS host

1. detect_platform returns RemoteOs::MacOs; the preinstall check is skipped.
2. BinaryCheckComplete { preinstall_check: None, … } arrives.
3. The unsupported branch is bypassed entirely (the early return is gated on Some(check)); existing logic applies unchanged.

Diagram

stateDiagram-v2
    [*] --> Checking: on_ssh_init_shell_requested
    Checking --> Connecting: binary present + supported (or macOS / Unknown)
    Checking --> Installing: binary missing + supported (or macOS / Unknown)
    Checking --> Updating: binary missing + has_old_binary + supported
    Checking --> Unsupported: preinstall returned Unsupported
    Installing --> Connecting: install ok
    Updating --> Connecting: update ok
    Connecting --> Ready: initialize ok
    Unsupported --> [*]: flush_stashed_bootstrap → legacy SSH executor
    Installing --> Failed: install error
    Connecting --> Failed: connect/initialize error

Testing and validation

Unit

• Bash-level: a small shell harness (crates/remote_server/src/preinstall_check_test.sh) that runs preinstall_check.sh against a stubbed getconf / ldd (PATH-injected) for each scenario — Ubuntu 20.04, RHEL 7, RHEL 8, Alpine musl, busybox-no-getconf-no-ldd — and asserts the emitted key/value lines exactly. This catches script regressions independently of the Rust parser.
• setup_tests.rs: table-driven tests for parse_preinstall_output covering each of the script's golden outputs, plus malformed/partial inputs (missing status, unknown reason, garbled libc_version).
• setup_tests.rs: truth table for PreinstallCheckResult::is_supported against Supported, Unsupported { GlibcTooOld { … } }, Unsupported { NonGlibc { "musl" } }, and Unknown — the last must be reported as supported per the fail-open rule.
• Manager tests: a mock RemoteTransport::run_preinstall_check that returns each variant; assert BinaryCheckComplete.preinstall_check carries the value, the per-session cache is populated, and the corresponding setup state is reached.
• Controller tests: drive on_binary_check_complete with Some(Unsupported { … }) and assert flush_stashed_bootstrap was called, no request_remote_server_block/install_binary/connect_session was issued, and RemoteServerHostUnsupported was emitted. Repeat with result = Ok(true) to assert remove_remote_server_binary is scheduled.

Manual

• Ubuntu 22.04 / Debian 12 (glibc 2.35+): unchanged install / auto-update / connect path. Choice block still appears under AlwaysAsk for first-time hosts.
• RHEL 7 (glibc 2.17), RHEL 8 (glibc 2.28), Amazon Linux 2 (glibc 2.26), Ubuntu 18.04 (glibc 2.27): SSH lands in the legacy flow with no choice block, modal, or error block; Warp.log shows the unsupported-host telemetry line.
• Alpine 3.x (musl): legacy fall-back, telemetry tagged reason=non_glibc.
• Busybox-only minimal container: status=unknown, choice block still appears under AlwaysAsk. Confirm that today's install-then-fail behavior is preserved (regression check for fail-open).
• Host with a pre-existing incompatible binary (simulate by scp-ing a Linux binary onto an Alpine VM): legacy fall-back; ssh <host> 'ls ~/.warp-*/remote-server' afterwards shows the binary was removed.
• macOS remote: unchanged.

Presubmit

./script/presubmit (cargo fmt, clippy, tests). The bash harness from §Unit runs as part of cargo test via a [[bin]] test wrapper invoked from setup_tests.rs.

Risks and mitigations

Script behaves badly on an exotic distro

status=unknown falls open and the user keeps today's install-and-try path. The script-level set -u plus explicit case arms keep the interpreter from emitting partial/garbage output that would confuse the parser; truly unparseable stdout maps to PreinstallStatus::Unknown in the parser too.

Script is the new install bottleneck

The script is templated and shipped in-tree under crates/remote_server/src/preinstall_check.sh, exactly like install_remote_server.sh. Updates ride the normal release train. Because the parser tolerates unknown keys, we can extend the script without coordinating a client release — only behavior that depends on the new keys requires a client bump.

Hardcoded required_glibc drifts from the build environment

The floor is hardcoded in preinstall_check.sh and documented to track the runner image in .github/workflows/create_release.yml. Bumping the runner means editing the script in the same PR. Follow-up: derive the value at release time from objdump -T <oz> | grep GLIBC_ | sort -V | tail -1 and inject it into the script during script/bundle so the source of truth is the artifact itself.

Extra round-trip on every Linux SSH connect

One additional ControlMaster channel multiplexed through the existing socket — same cost class as check_has_old_binary, well under the human perceptual threshold. macOS hosts pay nothing because the probe is gated on RemoteOs::Linux.

Users on marginal hosts lose the install prompt entirely

Intentional for the prebuilt artifact: installing a binary that crashes on launch is worse than skipping the install. Follow-up considers shipping a lower-glibc artifact and lowering the script's required_glibc per-artifact to broaden the supported set.

Removing a stale binary fails

Best-effort and logged. The session still falls back to legacy SSH; the worst case is a leftover binary on disk that subsequent runs detect again and skip again.

Follow-ups

• The preinstall script is now the natural place to add additional host capability checks: CPU instruction set requirements, free disk space in ~/.warp-XXXX/remote-server, presence of curl/tar. Each new check is an additive script-only change plus a parser key.
• Derive the script's required_glibc from the released binary at bundle time instead of a hardcoded value.
• Ship a second prebuilt Linux CLI built against an older glibc (Ubuntu 18.04 / glibc 2.27) and pick the right artifact based on the script's reported libc version; this is what closes APP-4281's stated goal of "support glibc 2.28."
• Once telemetry sizes the affected population, surface a one-time, dismissible explanation in the SSH choice area on unsupported hosts so users understand they are on the legacy SSH path by design.