public/galaxy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first pass of merging in warp (doesn't build)

Browse Source
This commit is contained in:
Ryan Ward
2026-07-01 16:08:58 -05:00
parent 2f64909469
commit 4770ac06b5
3662 changed files with 414574 additions and 89772 deletions
Download Patch File Download Diff File Expand all files Collapse all files
specs/REMOTE-1894/PRODUCT.md
+84
View File
@@ -0,0 +1,84 @@
# Resume cloud agent runs on transient transport failures
Linear: [REMOTE-1894](https://linear.app/warpdotdev/issue/REMOTE-1894/oz-cloud-runs-error-mid-execution-from-connectivitytransport-failures)
## Summary
Oz cloud runs (and local agent conversations) automatically recover from transient network/transport failures mid-turn instead of dying with a terminal error. While recovery is in flight the run visibly reports that it is reconnecting; recovery is bounded so a persistent outage still produces a clean terminal failure.
## Problem
Emily reported blank BDR outreach emails from the two-stage Oz pipeline (#feedback-platform): when either stage's cloud run errors, the email body is never written to HubSpot and the rep sees a blank template. Roland traced the blank emails to Oz runs on oz-dev/staging being killed mid-execution by transient transport failures, with broken recovery on the client:
- `ConnectionReset` mid-response killed runs with no retry at all ("MultiAgent request failed after 0 retries").
- A run that did log "retrying (attempt 1/3)" still failed immediately — the retry never actually recovered the run.
- `peer closed connection without sending TLS close_notify` (UnexpectedEof) killed a run the same way.
These are transient transport blips: a fresh request would almost certainly have succeeded, so the runs should have survived.
## Goals / Non-goals
Goals:
- A single transient transport failure mid-run never produces a failed run.
- Recovery state is visible (not silent) and bounded (no infinite retry loops).
Non-goals:
- Session-share initialization failures (bb8 pool timeouts) — tracked separately in [REMOTE-1878](https://linear.app/warpdotdev/issue/REMOTE-1878).
- Silent conversation death with no error emitted — tracked in [REMOTE-1881](https://linear.app/warpdotdev/issue/REMOTE-1881).
- Server-side LLM/turn retries (landed separately in warp-server #11754/#11755/#11770).
## Behavior
"User" below means both the operator observing a cloud run (Oz UI, run state API, pipeline webhooks) and the Warp user watching a local agent conversation.
### Recovery on transient failure
1. When the agent response stream fails mid-turn from a transient network/server failure (connection reset, TLS close_notify EOF, truncated response, 5xx, request timeout), the conversation automatically recovers and continues. A single such failure never produces a failed run.
2. If the failure happens before the agent has streamed any actions for the turn, recovery is invisible: the request is re-sent (up to 3 times) and, if an attempt succeeds, the user sees a normal uninterrupted turn.
3. If the failure happens after actions have streamed, the conversation resumes from the server's authoritative state. Work that already executed (commands, tool calls) is never re-executed by the recovery.
4. A run that recovers completes indistinguishably from one that never failed: final state SUCCEEDED (or the normal terminal state), full output present, downstream consumers (e.g. the BDR pipeline's webhook/HubSpot write) observe a normal completion.
### Visible recovery state
5. While recovery is pending, the conversation shows a non-terminal "Reconnecting" status with an in-progress treatment (spinner-style icon, not an error icon).
6. While recovery is pending, the cloud run's task state remains IN_PROGRESS with the status message "Connection lost while receiving the agent response; attempting to resume." The run state never flaps through a terminal ERROR that would tear down the execution mid-recovery.
7. Run lists, the orchestration pill bar, and parent-agent aggregations count a reconnecting conversation as working/in-progress, not failed.
8. No error notification, desktop notification, or conversation-ended tombstone is shown while recovery is pending; stale notifications for the conversation are cleared, as they are when a turn starts. A notification fires only on the eventual terminal outcome.
### Bounded failure
9. Recovery is bounded: at most 3 in-request retries before actions have streamed, and at most one automatic resume after actions have streamed. A resumed request does not auto-resume again.
10. If recovery is exhausted (the resume also hits a transient failure, or pre-action retries run out while online), the run ends with a terminal error and the message "Warp lost connection while receiving the agent response. This is usually temporary." There is no retry storm: a persistent outage produces exactly one resume attempt before the terminal failure.
11. A cloud run held open for recovery waits at most 120 seconds; if recovery has not restored progress by then, the run ends with the last recorded error.
12. Application-level failures are never auto-recovered: out-of-credits and server-overload failures end the turn immediately with their specific messages (a recovery attempt would fail identically or add load the server shed). Non-transient errors (4xx, malformed responses) likewise fail immediately.
### Offline behavior
13. If the client is offline when a pre-action failure occurs, the retry waits for connectivity to return instead of failing, showing the "Reconnecting" state while parked. The retry fires automatically when the client comes back online.
14. An automatic resume likewise waits for connectivity before sending.
### Cancellation and interaction during recovery
15. Cancelling a conversation while recovery is pending takes effect immediately: the conversation shows Cancelled, and no recovery attempt fires afterward.
16. Sending a new message to a conversation with a pending resume replaces the recovery: the pending resume is dropped and the new request proceeds normally. (While a retry is parked the original request is still logically active, so new messages queue as usual.)
17. Passive background requests (e.g. automatic code-diff suggestions) never auto-resume; their failures are silent and terminal as before.
### Limits and adjacent surfaces
18. Recovery does not survive an app restart: a conversation restored from disk mid-recovery restores with a terminal Error status.
19. Recovering conversations are treated as in-progress for queued prompts and follow-up gating; when a recovering conversation reaches a terminal state, the same finished-conversation handling runs as for any in-progress conversation (queued prompts fail/clear appropriately).
specs/REMOTE-1894/TECH.md
+87
View File
@@ -0,0 +1,87 @@
# Resume cloud agent runs on transient transport failures — tech spec
Linear: [REMOTE-1894](https://linear.app/warpdotdev/issue/REMOTE-1894/oz-cloud-runs-error-mid-execution-from-connectivitytransport-failures) · Product spec: [PRODUCT.md](./PRODUCT.md) · PR: [warp#12431](https://github.com/warpdotdev/warp/pull/12431)
References are pinned to `84276e07` (the implementation branch); behavior invariants `I<n>` refer to PRODUCT.md.
## Context
Every agent turn is one streaming MAA request (`POST /ai/multi-agent`) wrapped by `ResponseStream`, which emits events to `BlocklistAIController`; the controller updates the conversation in `BlocklistAIHistoryModel`, whose status drives the headless driver (cloud runs), `LocalAgentTaskSyncModel` (run-state reporting), and the UI.
On master, recovery was broken in four compounding ways (the failure modes in Roland's run links):
- Retry required `is_online` and was abandoned otherwise; the "attempt 1/3" path could fail without ever recovering.
- A stream that ended in a clean EOF without a `StreamFinished` event (how mid-body `ConnectionReset`/`close_notify` aborts surface through reqwest) was hardcoded as a non-resumable terminal error in the controller.
- When a resume *was* scheduled, the sync model still reported terminal ERROR immediately — tearing down the cloud execution before the resume could land — and the conversion in `ambient_agents` dropped the structured error (and its resume flag) in favor of the stringly `status_error_message`.
- Recovery state was smuggled through a `will_attempt_resume` rendering flag on the error rather than modeled in the conversation state machine.
Key files:
- [`app/src/ai/blocklist/controller/response_stream.rs:41 @ 84276e07`](https://github.com/warpdotdev/warp/blob/84276e0732860798fa49eb7372e6ee90cdf1728b/app/src/ai/blocklist/controller/response_stream.rs#L41) — request lifecycle, retry/resume decision
- [`app/src/ai/blocklist/controller.rs:2783 @ 84276e07`](https://github.com/warpdotdev/warp/blob/84276e0732860798fa49eb7372e6ee90cdf1728b/app/src/ai/blocklist/controller.rs#L2783) — stream-event handling, resume spawn, cancellation
- [`app/src/ai/agent/conversation.rs:4205 @ 84276e07`](https://github.com/warpdotdev/warp/blob/84276e0732860798fa49eb7372e6ee90cdf1728b/app/src/ai/agent/conversation.rs#L4205) — conversation status state machine
- [`app/src/ai/agent_sdk/driver.rs:2793 @ 84276e07`](https://github.com/warpdotdev/warp/blob/84276e0732860798fa49eb7372e6ee90cdf1728b/app/src/ai/agent_sdk/driver.rs#L2793) — cloud-run lifecycle decisions
- [`app/src/ai/blocklist/local_agent_task_sync_model.rs:322 @ 84276e07`](https://github.com/warpdotdev/warp/blob/84276e0732860798fa49eb7372e6ee90cdf1728b/app/src/ai/blocklist/local_agent_task_sync_model.rs#L322) — conversation → run-state mapping
## Implementation
### `ConversationStatus::TransientError` (client-only state)
[`conversation.rs:4205`](https://github.com/warpdotdev/warp/blob/84276e0732860798fa49eb7372e6ee90cdf1728b/app/src/ai/agent/conversation.rs#L4205): new non-terminal status ("Reconnecting", in-progress icon treatment — I5). `mark_request_completed_with_error` takes `recovery_pending: bool` and sets `TransientError` vs `Error`; the exchange itself is still marked finished-with-error so the structured error is preserved for rendering and restore. Consumers updated exhaustively (no wildcard arms):
- Driver ([`driver.rs:2793`](https://github.com/warpdotdev/warp/blob/84276e0732860798fa49eb7372e6ee90cdf1728b/app/src/ai/agent_sdk/driver.rs#L2793)): `TransientError` → `end_run_after(AUTO_RESUME_TIMEOUT = 120s)` with the last structured error (I11); a recovery flips status back to `InProgress`, cancelling the deadline. The old `will_attempt_resume` check in the Error arm is gone — `Error` is always terminal now.
- Sync model ([`local_agent_task_sync_model.rs:322`](https://github.com/warpdotdev/warp/blob/84276e0732860798fa49eb7372e6ee90cdf1728b/app/src/ai/blocklist/local_agent_task_sync_model.rs#L322)): `TransientError` → `IN_PROGRESS` with no status message (I6); `task_update_for_conversation_error` ignores the `will_attempt_resume` rendering hint (terminal classification only).
- `ambient_agents::conversation_output_status_from_conversation`: early `None` for `TransientError`; for terminal `Error` it now prefers the structured exchange error over `status_error_message`.
- Run lists / pill bar / aggregation / notifications / queued-prompt gating treat it as working (I7, I8, I19).
- Restore-from-disk derives status from exchanges, so `TransientError` restores as terminal `Error` (I18, accepted).
### Strict retry/resume split
[`response_stream.rs:41`](https://github.com/warpdotdev/warp/blob/84276e0732860798fa49eb7372e6ee90cdf1728b/app/src/ai/blocklist/controller/response_stream.rs#L41): pure `recovery_action(has_received_client_actions, is_recoverable, has_retry_budget, can_attempt_resume_on_error, is_online) -> {RetryNow, RetryWhenOnline, Resume, Fail}`:
- No actions yet + retryable + budget (3) → retry, verbatim re-send (I2). Offline parks the retry (`RetryWhenOnline`).
- Actions received + transient + resume-eligible → one-shot `ResumeConversation` (I3); resumes run with `can_attempt_resume_on_error = false`, bounding recovery (I9).
- Everything else → `Fail` (terminal). Recovery eligibility (retry and resume) uses `AIApiError::is_recoverable()` ([`server_api.rs:341`](https://github.com/warpdotdev/warp/blob/84276e0732860798fa49eb7372e6ee90cdf1728b/app/src/server/server_api.rs#L341)) (I12).
The controller passes `recovery_pending = should_resume_conversation_after_stream_finished()` into the history model; `will_attempt_resume` on `RenderableAIError` remains rendering-only.
### Silent-EOF synthesis (`AIApiError::UnexpectedEof`)
The server always sends `StreamFinished`, but a transport cut between chunks reaches the stream layer as a clean EOF with no error event — empirically confirmed (both e2e scenarios below exercised only this path; the `Err`-event arm never fired). `ResponseStream` tracks `stream_finished_received` / `error_event_emitted` and, on completion without either, synthesizes `UnexpectedEof` (retryable + transient) and runs the same `recovery_action` decision ([`response_stream.rs:388`](https://github.com/warpdotdev/warp/blob/84276e0732860798fa49eb7372e6ee90cdf1728b/app/src/ai/blocklist/controller/response_stream.rs#L388)). This replaces the controller's hardcoded-terminal EOF fallback (now a defensive warn). HTTP send failures do not take this path — they arrive as in-stream error events; request-conversion failures (no stream ever created) are surfaced terminally with the original error instead of being synthesized as `UnexpectedEof`. Non-retried failures from both paths report to Sentry via a shared helper with classification tags.
### Offline parking
`defer_retry_until_online` parks a retry on `NetworkStatus::wait_until_online()`, suppressing completion of the failed attempt (`deferred_retry_pending`) and emitting `WaitingForNetwork { waiting }`; the controller mirrors it onto the conversation (`TransientError` ↔ `InProgress`) without finishing any exchange (I13). The resume spawn in `AfterStreamFinished` also waits for connectivity (I14). Parked work is invalidated by `current_request_id` on cancellation/supersession.
### Cancellation and replacement
`cancel_conversation_progress` aborts the parked resume handle and, when no active stream remains, flips a `TransientError` conversation to `Cancelled` directly (I15). `send_request_input` aborts the pending resume for the conversation before sending (I16) and forces `can_attempt_resume_on_error = false` for passive requests (I17).
## Testing and validation
Unit (all in-tree, `cargo nextest run -p warp --lib`):
- `response_stream_tests.rs` — exhaustive `recovery_action` matrix: retry/park/fail pre-actions (I2, I9, I13), resume gating post-actions (I3, I9), budget exhaustion and non-retryable → fail (I10, I12).
- `server_api_tests.rs` — `is_recoverable` classification: 5xx/timeout/transport and app-level (quota/overload/misc/JSON) recoverable, other 4xx not (I12); `UnexpectedEof` recoverable (I1).
- `history_model_tests.rs` — `recovery_pending` → `TransientError` and no terminal derived outcome (I5, I6 upstream); structured exchange error preserved through conversion; non-recoverable error stays terminal.
- `local_agent_task_sync_model_tests.rs` — `TransientError` → `IN_PROGRESS` with no status message (I6); `will_attempt_resume` hint ignored for terminal classification (I10, I12).
E2e (oz-local + warp-server `TransportReset` LLM mock, `simulate_maa_transport_reset: true`):
- Recovery (I1, I3–I6, I11): single mid-stream reset → client log shows the synthesized truncation → conversation `TransientError` → driver "automatic recovery pending — waiting up to 120s" → resume fires → run completes with the mocked `finish_task`. Verified 2026-06-10.
- Bounded failure (I9, I10): cycling resets → exactly one resume, then terminal ERROR with the friendly lost-connection message; 2 LLM calls total, no retry storm. Verified 2026-06-10.
- The pre-action retry paths (I2) are not reachable through this mock on cloud runs (the user-message-append ClientActions always precede the LLM call), so they are covered by the unit matrix plus the request-send-failure path.
Lint/build gates: `cargo fmt`, `cargo clippy -p warp --lib --tests -- -D warnings`, `cargo check -p warp --lib --features crash_reporting`.
Not covered by automated tests (manual/review only): controller cancellation-during-recovery (I15, I16), notification suppression (I8), and the offline `wait_until_online` integration (I13, I14) — exercised via the unit matrix decision but not a live network flap.
## Risks and mitigations
- **Stuck "Reconnecting"**: any path that sets `TransientError` and never delivers a recovery would hang the UI state. Mitigated by the driver's 120s deadline (cloud), the one-shot resume contract, and the cancellation flip to `Cancelled`; restore-from-disk degrades it to `Error`.
- **Behavior change**: quota/overload failures no longer set the resume flag (master did). Intentional — a resume would fail identically — but visible to anyone who relied on the old flag.
- **Sentry volume**: synthesized truncations now report (with `will_attempt_resume`/`is_recoverable` tags) — expect new `UnexpectedEof` events that previously appeared as the generic EOF fallback.
## Parallelization
Not proposed: the change is a single tightly-coupled state machine (status variant + decision function + consumers), already implemented and validated on one branch (PR #12431). Splitting it across agents would have created merge conflicts in `controller.rs`/`response_stream.rs` without wall-clock benefit.