galaxy/specs/APP-4234/TECH.md

Remote Server Integration Tests

Context

The SshRemoteServer feature flag gates a new SSH session flow where a persistent binary (remote-server-proxy) runs on the remote host, replacing the legacy ControlMaster-based command execution. The feature has unit test coverage at the protocol layer (crates/remote_server/src/client_tests.rs) but no integration test coverage exercising the full client ↔ server lifecycle over a real SSH connection.

Current integration test infra

The existing SSH integration tests (crates/integration/src/test/ssh.rs) cover the legacy wormholing flow:

• Connect to a GCP-hosted Ubuntu VM (ubuntu-14-04) via IAP tunnel with password auth
• Helper steps in app/src/integration_testing/subshell/ — setup_gcloud_sdk(), enter_ssh_command(), enter_ssh_password(), wait_for_password_prompt()
• Builder pattern: new_builder().with_step(TestStep) with assertion callbacks
• Feature flag gating via set_should_run_test(|| FeatureFlag::X.is_enabled())

Remote server flow

When SshRemoteServer is enabled for a legacy SSH session (app/src/terminal/writeable_pty/remote_server_controller.rs):

1. RemoteServerController intercepts SshInitShell, stashes the bootstrap script
2. Runs check_binary via RemoteServerManager → if missing, install_binary → then connect_session
3. connect_session (crates/remote_server/src/manager.rs:368) spawns the proxy over SSH, performs the proto Initialize handshake → emits SessionConnected { host_id }
4. Bootstrap is flushed; RemoteServerCommandExecutor (app/src/terminal/model/session/command_executor/remote_server_executor.rs) is wired as the session's CommandExecutor
5. On CWD change, navigate_to_directory fires → returns is_git flag + triggers RepoMetadataSnapshot push

Key config: SshExtensionInstallMode::AlwaysInstall (setting in app/src/terminal/wormhole/settings.rs:85) bypasses the choice block UI, needed for deterministic test flow.

Binary deployment problem

The production install script (crates/remote_server/src/install_remote_server.sh) downloads from the CDN (app.warp.dev/download/cli). This fetches a published binary, not one built from the developer's branch. For integration tests, the binary must come from the current codebase so that changes to the remote-server protocol or logic are tested. The existing script/deploy_remote_server already solves this for local development — it cross-compiles for x86_64-unknown-linux-musl and uploads via rsync.

Proposed changes

1. CI step: cross-compile and deploy binary to test VM

Add script/deploy_remote_server_to_test_vm that:

1. Cross-compiles the Oz CLI for the test VM target:

   cargo build -p warp --bin warp --target x86_64-unknown-linux-musl \
     --profile dev-remote \
     --features release_bundle,crash_reporting,standalone,agent_mode_debug
   

   Same build command as script/deploy_remote_server.

2. Uploads the binary to ubuntu-14-04 at ~/.warp-dev/remote-server/oz-dev via sshpass + scp through the GCP IAP tunnel (the test VM uses password auth; sshpass provides it non-interactively). Uses the same proxy command as the SSH integration tests (app/src/integration_testing/subshell/util.rs:2).

CI calls this script once before launching the integration test suite. Since check_binary will find the binary already present, the RemoteServerController flow becomes check_binary → Ok(true) → connect_session, skipping the CDN-based install.

2. Assertion helpers — app/src/integration_testing/remote_server.rs

New module with reusable test steps and action helpers:

• wait_for_remote_server_ready(tab_idx) — TestStep that polls until Sessions::remote_server_setup_states for the active session reaches RemoteServerSetupState::Ready.
• assert_remote_server_connected(tab_idx) — reads RemoteServerManager singleton, asserts the active session is in RemoteSessionState::Connected.
• assert_command_executor_is_remote_server(tab_idx) — downcasts the session's CommandExecutor via as_any().downcast_ref::<RemoteServerCommandExecutor>().
• assert_remote_server_has_navigated(tab_idx) — asserts that host_id_for_session is populated for the active session.
• write_file_via_remote_server(tab_idx, path, content) — action callback that calls RemoteServerClient::write_file on a background thread (async → sync bridge via tokio::runtime::Runtime::block_on).
• load_repo_metadata_directory_via_remote_server(tab_idx, repo_path, dir_path) — action callback that calls RemoteServerManager::load_remote_repo_metadata_directory through the model handle.

Also adds Session::command_executor() accessor gated on #[cfg(any(test, feature = "integration_tests"))] (app/src/terminal/model/session.rs).

Registered in app/src/integration_testing/mod.rs.

3. Integration tests — crates/integration/src/test/remote_server.rs

All tests gated on FeatureFlag::SshRemoteServer.is_enabled() and configured with with_user_defaults setting SshExtensionInstallMode to AlwaysInstall.

Test A — Connection and handshake (test_remote_server_connect_bash / _zsh)

Validates the core flow: SSH → binary check → proto handshake → executor wiring.

Steps:

1. wait_until_bootstrapped_single_pane_for_tab(0) — local shell ready
2. setup_gcloud_sdk()
3. enter_ssh_command(shell) + wait_for_password_prompt + enter_ssh_password
4. wait_for_remote_server_ready(0) — covers check → connect → handshake
5. wait_until_bootstrapped_single_pane_for_tab(0) — remote shell bootstrapped
6. assert_remote_server_connected(0) — manager has Connected state with a HostId
7. assert_command_executor_is_remote_server(0) — session uses RemoteServerCommandExecutor

Test B — Repo metadata (test_remote_server_navigate_to_repo)

Validates the full navigate-to-directory flow: create git repo on remote → cd into it → NavigatedToDirectory response received → session has host_id tracked.

After Test A setup, plus: 8. Create a git repo on the remote via execute_command("mkdir -p /tmp/warp-test-repo && cd /tmp/warp-test-repo && git init -b main ...") 9. execute_command("cd /tmp/warp-test-repo") — triggers CWD change → navigate_to_directory 10. assert_remote_server_has_navigated(0) — host_id present for active session 11. assert_remote_server_connected(0) — session still healthy after navigation

Test C — Completions routing (test_remote_server_completions)

Validates that completions run through RemoteServerCommandExecutor::execute_command (the RunCommand proto path) rather than falling back to the legacy RemoteCommandExecutor.

After Test A setup, plus: 8. Run a command to trigger completions loading 9. assert_command_executor_is_remote_server(0) — confirm executor type

Test D — File write via proto client API (test_remote_server_file_operations)

Validates WriteFile through the RemoteServerClient proto API. Uses write_file_via_remote_server helper to dispatch the async write from an action callback, then reads the file back via a shell command (which goes through RemoteServerCommandExecutor::run_command) to confirm content integrity.

After Test A setup, plus: 8. write_file_via_remote_server(0, "/tmp/warp-rs-test-file.txt", "hello from proto") — writes via proto 9. execute_command("cat /tmp/warp-rs-test-file.txt") — reads back via RunCommand, asserts "hello from proto" 10. Clean up and verify executor type

Test E — Lazy loading repo metadata (test_remote_server_lazy_load_directory)

Validates the LoadRepoMetadataDirectory proto round-trip: navigate to a git repo → create a subdirectory → call load_remote_repo_metadata_directory for that subdirectory → verify the response flows through without error.

After Test A setup, plus: 8. Create a git repo with a subdir/nested file on the remote 9. cd /tmp/warp-lazy-repo → triggers NavigatedToDirectory and full indexing 10. load_repo_metadata_directory_via_remote_server(0, repo_path, "subdir") — triggers lazy-load proto request 11. assert_remote_server_connected(0) — connection still healthy 12. execute_command("cat subdir/nested") — verifies subdirectory content accessible

4. Wire into the test runner

• Add mod remote_server; + pub use remote_server::*; in crates/integration/src/test.rs
• Register test functions in crates/integration/tests/integration/shell_integration_tests.rs
• Register in crates/integration/src/bin/integration.rs for the manual runner

Testing and validation

The tests in this spec are the validation — they verify the remote-server feature works end-to-end over a real SSH connection. Specifically:

• Test A proves the install-check → handshake → executor-wiring pipeline works against a real remote host, catching protocol mismatches or connection failures that unit tests with mock streams cannot.
• Test B proves the NavigatedToDirectory → repo metadata pipeline works through the proto, catching serialization issues or git-detection regressions on the remote.
• Test C proves completions are routed through the remote-server binary rather than silently falling back to the legacy ControlMaster executor, which would mask remote-server regressions.
• Test D proves the WriteFile proto message works end-to-end by writing via the client API and reading back via RunCommand, catching serialization or FileModel regressions invisible to shell-only tests.
• Test E proves the LoadRepoMetadataDirectory lazy-loading proto round-trip works, catching regressions in the subdirectory expansion path that is distinct from the initial NavigatedToDirectory indexing.

All tests run against a binary built from the current branch (via the CI deploy step), ensuring protocol changes are tested before merge.

Parallelization

• CI script (step 1) and assertion helpers (step 2) can be built in parallel — they touch disjoint files.
• Test module (step 3) depends on the assertion helpers but the test groups (A–E) are independent functions that can be written in parallel once the shared helpers exist.
• Test runner wiring (step 4) is trivial and can be done alongside any other step.