galaxy/specs/APP-4217/TECH.md

TECH.md — Remote Server Reliability Telemetry (APP-4217)

Context

The SSH remote server feature (gated behind FeatureFlag::SshRemoteServer) has no telemetry. Before releasing beyond dogfood we need client-side reliability metrics covering the full lifecycle: binary check → install → connect → initialize → steady-state operation → disconnection.

Auth state is not yet passed to the remote server, so all tracking is client-side via the existing TelemetryEvent infrastructure.

Relevant code

• crates/remote_server/src/manager.rs — RemoteServerManager singleton; owns session lifecycle, emits RemoteServerManagerEvent
• crates/remote_server/src/client/mod.rs — RemoteServerClient; protocol I/O, reader/writer tasks, request/response correlation
• crates/remote_server/src/transport.rs — RemoteTransport trait (check_binary, install_binary, connect)
• app/src/remote_server/ssh_transport.rs — SshTransport implementation
• app/src/terminal/writeable_pty/remote_server_controller.rs — RemoteServerController; per-pane orchestrator for the SSH init flow
• app/src/terminal/view.rs (4170-4217) — TerminalView subscription to RemoteServerManagerEvent; converts manager events into terminal events and UI
• app/src/server/telemetry/events.rs — TelemetryEvent enum; each variant needs definition, name(), description(), payload(), contains_ugc(), enablement_state()

Current lifecycle

stateDiagram-v2
    [*] --> CheckBinary: SshInitShell
    CheckBinary --> Connect: found
    CheckBinary --> ShowChoiceBlock: not found (AlwaysAsk)
    CheckBinary --> Install: not found (AlwaysAllow)
    CheckBinary --> Skipped: not found (AlwaysSkip)
    CheckBinary --> Skipped: check error
    ShowChoiceBlock --> Install: user clicks Install
    ShowChoiceBlock --> Skipped: user clicks Skip
    Install --> Connect: success
    Install --> Skipped: failure
    Connect --> Initialize: transport.connect() ok
    Connect --> Skipped: transport.connect() error
    Initialize --> Connected: handshake ok
    Initialize --> Skipped: handshake error
    Connected --> Disconnected: reader EOF / error

Every Skipped path falls back to ControlMaster-based command execution. Today none of these transitions emit telemetry.

Reliability Metrics

Seven telemetry events covering every stage of the lifecycle:

1. Binary check outcome — RemoteServerBinaryCheck

Tracks whether the remote binary was found, not found, or the check itself failed (SSH timeout, unreachable host). This is the entry point for every remote server session.

field	type	description
found	bool	true if binary exists and is executable
error	Option<String>	set when check_binary returns Err

Signals: check failure rate (SSH-level issues), install-needed rate.

2. Installation outcome — RemoteServerInstallation

Tracks whether the install script succeeded or failed.

field	type	description
succeeded	bool
error	Option<String>	set on failure

Signals: install success rate, common failure reasons.

3. Initialization outcome — RemoteServerInitialization

Tracks the two-phase connection flow: transport.connect() (SSH/process spawn) and client.initialize() (protocol handshake). Distinguishing the phase helps diagnose whether failures are transport-level or protocol-level.

field	type	description
succeeded	bool
phase	&'static str	"connect" or "initialize"
error	Option<String>	set on failure

Signals: connection success rate, breakdown of failure by phase.

4. Session disconnection — RemoteServerDisconnection

Tracks when an established connection drops (reader EOF or fatal error). This is the most important stability signal — it tells us how reliable connections are after successful setup.

No fields.

Signals: disconnection rate, connection stability.

5. Client request error — RemoteServerClientRequestError

Tracks when a request to the remote server fails (timeout, disconnect, unexpected response, server error).

field	type	description
operation	&'static str	e.g. "navigate_to_directory", "run_command"
error_type	&'static str	e.g. "timeout", "disconnected", "server_error"

Signals: per-operation failure rate, error distribution.

6. Server message decoding error — RemoteServerMessageDecodingError

Tracks when the reader task receives a malformed server message with no parseable request_id (messages with a request_id surface as ClientError::Protocol to the caller and are captured by metric 5).

No fields.

Signals: protocol-level corruption or version mismatch.

7. End-to-end setup duration — RemoteServerSetupDuration

Tracks wall-clock time from check_binary start to SessionConnected. Users see a shimmer loading footer during this window; tracking duration flags when setup is too slow.

field	type	description
duration_ms	u64	elapsed time
installed_binary	bool	whether install was needed (affects expected duration)

Signals: setup latency, install vs no-install performance.

Proposed Changes

1. TelemetryEvent variants (app/src/server/telemetry/events.rs)

Add seven new variants to the TelemetryEvent enum, all gated behind EnablementState::Flag(FeatureFlag::SshRemoteServer).

Each variant needs entries in:

• name() (discriminant → human-readable name)
• description() (discriminant → doc string)
• payload() (variant → json!({...}))
• contains_ugc() (all return false)
• enablement_state() (all → Flag(FeatureFlag::SshRemoteServer))

2. Propagate decode errors from RemoteServerClient

File: crates/remote_server/src/client/mod.rs

• Add ClientEvent::MessageDecodingError variant.
• In reader_task, on ProtocolError::Decode(_, None) (no parseable request_id), send the new event through the event channel alongside the existing log::warn!.

3. New manager events

File: crates/remote_server/src/manager.rs

• Add RemoteServerManagerEvent::ClientRequestFailed { session_id, operation, error }
• Add RemoteServerManagerEvent::ServerMessageDecodingError { session_id }
• Emit ClientRequestFailed at existing log::error! sites in navigate_to_directory and load_repo_metadata_directory.
• Forward ClientEvent::MessageDecodingError in forward_client_event.

4. Propagate failure phase in SessionConnectionFailed

File: crates/remote_server/src/manager.rs

Add phase: &'static str field to RemoteServerManagerEvent::SessionConnectionFailed ("connect" or "initialize"), set at the two Err arms in connect_session (lines 359 and 342). Update the single downstream consumer in terminal/view.rs.

5. Track telemetry in TerminalView subscription

File: app/src/terminal/view.rs (4170-4217)

Expand the existing RemoteServerManagerEvent match arms to add send_telemetry_from_ctx! calls:

• BinaryCheckComplete { Ok(true) } → RemoteServerBinaryCheck { found: true, error: None }
• BinaryCheckComplete { Ok(false) } → RemoteServerBinaryCheck { found: false, error: None }
• BinaryCheckComplete { Err(e) } → RemoteServerBinaryCheck { found: false, error: Some(e) }
• BinaryInstallComplete { Ok(()) } → RemoteServerInstallation { succeeded: true, error: None }
• BinaryInstallComplete { Err(e) } → RemoteServerInstallation { succeeded: false, error: Some(e) }
• SessionConnected → RemoteServerInitialization { succeeded: true, phase: "initialize", error: None }
• SessionConnectionFailed { phase } → RemoteServerInitialization { succeeded: false, phase, error: None }
• SessionDisconnected → RemoteServerDisconnection
• ClientRequestFailed → RemoteServerClientRequestError { operation, error_type }
• ServerMessageDecodingError → RemoteServerMessageDecodingError

6. Track setup duration in RemoteServerController

File: app/src/terminal/writeable_pty/remote_server_controller.rs

• Add setup_start: Option<Instant> and did_install: bool to SshInitState or as fields on the controller.
• Record Instant::now() at the start of on_ssh_init_shell_requested.
• Set did_install = true when entering the install path.
• Subscribe to RemoteServerManagerEvent::SessionConnected (new subscription). On match, compute duration_ms and emit RemoteServerSetupDuration.

Testing and Validation

1. Unit tests for telemetry payload correctness: verify each new TelemetryEvent variant produces the expected name(), payload(), and enablement_state(). Follow the pattern of existing telemetry tests.

2. Manual validation on dogfood: SSH to a remote host and verify each metric fires by checking the Rudderstack live event stream (use the telemetry stream workflow). Scenarios to exercise:

   • Clean install (binary not present) — expect BinaryCheck(not found), Installation(success), Initialization(success), SetupDuration.
   • Binary already present — expect BinaryCheck(found), Initialization(success), SetupDuration (shorter).
   • Kill the remote server process mid-session — expect Disconnection.
   • SSH to an unreachable host — expect BinaryCheck(error).

3. Compile check: run cargo check --workspace to confirm the new variants are exhaustively handled in all match arms (name(), payload(), contains_ugc(), enablement_state(), description()).

Parallelism

Steps 1 (telemetry variants) and 2-3 (client/manager event plumbing) touch independent files and can be worked on in parallel. Steps 4-6 depend on both.

Follow-ups

• Server-side reliability telemetry once auth state is forwarded.
• Per-request latency tracking (p50/p99 for navigate_to_directory, run_command, etc.).
• Reconnect success/failure tracking once client-side reconnect is implemented (APP-4068 follow-up).