Files

T

David Stern 0dbd3d567a Initial public release of Warp.

Repo-Sync-Origin: warpdotdev/warp-internal@12af1d983b

2026-04-28 08:43:33 -05:00

13 KiB

Raw Blame History

Remote Server SSH: Technical Spec

Problem

The current SSH wrapper flow creates a RemoteCommandExecutor that runs every generator/completion command by opening a new SSH channel through a ControlMaster socket. This is unreliable and stateless. We want to replace it with a persistent remote server binary (~/.warp/remote-server/oz) on the remote machine that communicates over stdin/stdout using length-prefixed protobuf messages. The binary is the Oz CLI, installed from the /download/cli endpoint if not already present.

The challenge is that this introduces two independent async conditions that must both complete before the session is ready: (1) the shell Bootstrapped DCS hook, and (2) the remote server InitializeResponse. Today the bootstrap path is fully synchronous — Bootstrapped DCS immediately triggers initialize_bootstrapped_session(). We need to gate that call on both conditions without breaking non-SSH or flag-off flows.

Relevant code

warp_core/src/features.rs — FeatureFlag enum and DOGFOOD_FLAGS array
app/src/terminal/model/terminal_model.rs:2808-2846 — bootstrapped() handler that takes pending_session_info and emits HandlerEvent::Bootstrapped
app/src/terminal/model/terminal_model.rs:2848-2860 — pre_interactive_ssh_session() and ssh() handlers
app/src/terminal/model/terminal_model.rs:2862-2908 — init_shell() handler
app/src/terminal/model_events.rs:89-110 — ModelEventDispatcher handler for HandlerEvent::Bootstrapped that synchronously calls sessions.initialize_bootstrapped_session()
app/src/terminal/model/session.rs:220-332 — Sessions::initialize_bootstrapped_session() which creates the command executor and emits SessionBootstrapped
app/src/terminal/model/session.rs:410-414 — IsLegacySSHSession enum
app/src/terminal/model/session.rs:448-542 — SessionInfo struct and create_pending()
app/src/terminal/model/session/command_executor.rs:190-296 — executor selection logic in new_command_executor_for_local_tty_session()
app/src/terminal/model/session/command_executor/remote_command_executor.rs — current RemoteCommandExecutor using ControlMaster
app/src/terminal/prompt_render_helper.rs:231-254 — prompt_working_dir() and bootstrapping_shell_message() that drive the status text
app/src/terminal/view.rs:10255-10267 — ModelEvent::PreInteractiveSSHSession (no-op) and ModelEvent::SSH handlers
app/src/terminal/view.rs:10860-11024 — handle_session_bootstrapped()
remote_server/proto/remote_server.proto — protobuf definitions for Initialize/InitializeResponse
remote_server/src/lib.rs — generated proto bindings
app/assets/bundled/bootstrap/bash_body.sh and zsh_body.sh — shell-side SSH wrapper functions

Current state

DCS hook sequence for SSH wrapper flow

PreInteractiveSSHSession — no-op marker
SSH — carries socket_path (ControlMaster) and remote_shell. Stored in pending_legacy_ssh_session.
InitShell — creates SessionInfo::create_pending(), consuming pending_legacy_ssh_session to populate IsLegacySSHSession::Yes { socket_path }. Calls reinit_shell() to reset the block list. Warp input becomes visible with "Starting shell...".
Bootstrapped — terminal_model.bootstrapped() merges pending session info, emits HandlerEvent::Bootstrapped. The ModelEventDispatcher synchronously calls sessions.initialize_bootstrapped_session(), which creates the RemoteCommandExecutor from the socket path, stores the session, and emits SessionBootstrapped. The view shows the normal prompt.

Key constraint

initialize_bootstrapped_session() is called synchronously inside the ModelEventDispatcher event handler for HandlerEvent::Bootstrapped (model_events.rs:98-106). This is the only place session initialization happens. All downstream logic (view, history, telemetry) flows from the SessionBootstrapped event it emits.

Proposed changes

1. New feature flag

Add RemoteServerSSH to the FeatureFlag enum in warp_core/src/features.rs. Add it to DOGFOOD_FLAGS for initial rollout.

2. Remote server setup state machine

Create a new module app/src/terminal/ssh/remote_server_setup.rs (or similar location alongside the existing app/src/terminal/ssh/ module) that encapsulates the install → launch → initialize flow.

enum RemoteServerSetupState {
    /// Checking if the binary exists on remote.
    Checking,
    /// Downloading and installing the binary.
    Installing { progress_percent: Option<u8> },
    /// Binary is launched, waiting for InitializeResponse.
    Initializing,
    /// Handshake complete. Ready.
    Ready,
    /// Something failed. Fall back to ControlMaster.
    Failed { error: String },
}

The setup runs as an async task, using the existing SSH ControlMaster socket (from IsLegacySSHSession::Yes { socket_path }) to execute remote commands. The steps are:

Check: ssh -o ControlPath={socket} placeholder@placeholder 'test -x ~/.warp/remote-server/oz && ~/.warp/remote-server/oz --version'
Install (if check fails): pipe the following script into bash -s over the control socket SSH connection (mirroring how RemoteCommandExecutor runs commands today):
```
set -e
arch=$(uname -m)
case "$arch" in
  x86_64)  pkg=oz-linux-x86_64.tar.gz ;;
  aarch64|arm64) pkg=oz-linux-aarch64.tar.gz ;;
  *) echo "unsupported arch: $arch" >&2; exit 2 ;;
esac
mkdir -p "$HOME/.warp/remote-server"
curl -fSL "$WARP_GET_URL?package=$pkg" -o "$HOME/.warp/remote-server/oz.tar.gz"
tar -xzf "$HOME/.warp/remote-server/oz.tar.gz" -C "$HOME/.warp/remote-server"
chmod +x "$HOME/.warp/remote-server/oz"
```
$WARP_GET_URL is substituted at runtime from the configured server root URL (SERVER_ROOT_URL env var or its compiled-in default), pointing at /download/cli. Exit code 2 is mapped to ErrorReason::UnsupportedPlatform. The script is shipped as a constant &str in the install module. Parse curl stderr for download progress if available.
Launch: ssh -o ControlPath={socket} placeholder@placeholder '~/.warp/remote-server/oz' — keep the SSH channel open, forwarding stdin/stdout.
Initialize: Send a ClientMessage { request_id, initialize: Initialize {} } (length-prefixed protobuf) to the process's stdin. Read a ServerMessage { initialize_response } from stdout. Timeout after 10 seconds.

The async task communicates state changes back to the UI via an event channel.

3. New event: `RemoteServerReady`

Add a new Event variant:

Event::RemoteServerReady {
    session_id: SessionId,
    result: Result<(), RemoteServerSetupError>,
}

This is sent by the async setup task when it completes (success or failure). The ModelEventDispatcher receives this alongside the existing HandlerEvent::Bootstrapped.

4. Gate session initialization on both conditions

This is the core change. In ModelEventDispatcher (model_events.rs), when the feature flag is enabled and the session is a legacy SSH session:

When HandlerEvent::Bootstrapped arrives: store the BootstrappedEvent payload in a new field pending_bootstrapped_event: Option<BootstrappedEvent> on ModelEventDispatcher. Do NOT call initialize_bootstrapped_session() yet.
When Event::RemoteServerReady arrives: store the result.
After either event, check if both are present. If so, call initialize_bootstrapped_session() as normal.
If the remote server setup failed, call initialize_bootstrapped_session() anyway (fallback to ControlMaster executor).

When the flag is disabled, or the session is not a legacy SSH session, the existing synchronous path is unchanged.

5. Dynamic bootstrapping message

Extend bootstrapping_shell_message() in prompt_render_helper.rs to show stage-specific messages. The Sessions model (or a new model) needs to expose the current RemoteServerSetupState for the pending session. The render helper checks this state:

RemoteServerSetupState::Checking → "Starting shell..." (unchanged)
RemoteServerSetupState::Installing { progress_percent: Some(p) } → "Installing Warp SSH tools... ({p}%)"
RemoteServerSetupState::Installing { progress_percent: None } → "Installing Warp SSH tools..."
RemoteServerSetupState::Initializing → "Initializing..."
No remote server state (flag off, non-SSH) → existing behavior

The state is stored on Sessions keyed by SessionId and updated via events from the async setup task. The prompt re-renders on each state change via ctx.notify().

6. Command executor selection

In new_command_executor_for_local_tty_session() (command_executor.rs:245-258), when the flag is enabled and the remote server is ready, create a new RemoteServerCommandExecutor instead of RemoteCommandExecutor. This new executor sends commands to the running remote server process via stdin/stdout protobuf messages instead of opening SSH channels.

For the initial iteration, the new executor can wrap the same interface but communicate through the persistent process. If the remote server setup failed, the existing RemoteCommandExecutor (ControlMaster) is used as fallback.

7. Cleanup on SSH exit

The remote server process is spawned as an SSH channel via ControlMaster. When the SSH session exits:

The ControlMaster connection is torn down, which kills the remote process.
The Arc<Session> holding the executor is dropped via Rust's RAII.
No explicit cleanup is needed.

End-to-end flow

User runs ssh user@host.
PreInteractiveSSHSession DCS → no-op.
SSH DCS → stores socket path in pending_legacy_ssh_session.
InitShell DCS → creates SessionInfo::create_pending() with IsLegacySSHSession::Yes { socket_path }. Warp input appears with "Starting shell...". If RemoteServerSSH flag is enabled: kicks off the async remote server setup task using the socket path from the pending session info.
Setup task transitions: Checking → Installing (if needed) → Initializing. Each state change updates the Sessions model and triggers a prompt re-render.
Bootstrapped DCS → ModelEventDispatcher stores the BootstrappedEvent instead of immediately initializing the session.
Setup task sends Event::RemoteServerReady { session_id, Ok(()) }.
ModelEventDispatcher sees both conditions met, calls initialize_bootstrapped_session() which creates the RemoteServerCommandExecutor.
SessionBootstrapped event fires. View transitions to normal prompt with working directory.

If Bootstrapped arrives after RemoteServerReady, step 6 triggers immediate initialization. The order doesn't matter.

Risks and mitigations

Risk: Setup task blocks on slow network. The download could take a long time on a slow connection. Mitigation: the shell bootstrap runs in parallel, so only the delta matters. Show progress percentage to set expectations. Timeout the entire setup after a generous limit (e.g. 60 seconds) and fall back.

Risk: ControlMaster socket gone before setup completes. If the SSH session drops during setup, the task will fail. Mitigation: the task uses the existing error handling — any SSH command failure transitions to Failed state, and the session falls back.

Risk: Remote host has no curl or wget. Mitigation: try curl first, then wget. If neither exists, transition to Failed and fall back to ControlMaster.

Risk: Race between Bootstrapped and RemoteServerReady events. Mitigation: the ModelEventDispatcher stores whichever arrives first and processes initialization when both are present. This is a simple two-flag check, not a complex synchronization problem.

Risk: Regression for non-SSH sessions. Mitigation: the gating logic in ModelEventDispatcher only applies when the feature flag is enabled AND is_legacy_ssh_session is Yes. All other sessions use the existing synchronous path.

Testing and validation

Unit tests: Test RemoteServerSetupState transitions, uname output parsing, download URL construction.
Unit tests: Test the two-condition gate in ModelEventDispatcher — verify initialize_bootstrapped_session is called only when both conditions are met, in both arrival orders.
Unit tests: Test fallback behavior when setup fails — verify RemoteCommandExecutor is created instead of RemoteServerCommandExecutor.
Integration test: SSH into a Docker container (using the existing SSH testing setup in the repo). Verify binary installation, launch, and Initialize handshake.
Manual testing: SSH into fresh Linux x86_64 and aarch64 hosts. Verify prompt message transitions. SSH again to verify installation is skipped.
Feature flag off: Verify no behavioral change with the flag disabled.

Follow-ups

Version checking: Compare installed binary version to client version and re-install on mismatch.
Auto-update: Silently update the binary when a newer version is available.
Richer remote server capabilities: File watching, codebase indexing, cached completions via the persistent process.
Replace ControlMaster entirely: Once the remote server is stable, remove the ControlMaster-based RemoteCommandExecutor code path.
Progress reporting: Improve download progress by parsing curl/wget output more reliably.

13 KiB Raw Blame History