public/galaxy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first pass of merging in warp (doesn't build)

Browse Source
This commit is contained in:
Ryan Ward
2026-07-01 16:08:58 -05:00
parent 2f64909469
commit 4770ac06b5
3662 changed files with 414574 additions and 89772 deletions
Download Patch File Download Diff File Expand all files Collapse all files
specs/REMOTE-1465/PRODUCT.md
+56
View File
@@ -0,0 +1,56 @@
# Oz File-Edit Hooks for Snapshotting Non-Git-Tracked Files — Product Spec
Linear: [REMOTE-1465](https://linear.app/warpdotdev/issue/REMOTE-1465)
Figma: none provided
## Summary
When the Oz agent creates or edits a file during a cloud run, the file's absolute path is automatically added to the end-of-run snapshot declarations so the file is uploaded alongside the existing repo-diff snapshot. This closes the handoff gap for files the agent writes outside any git repo (or inside one but not visible to `git diff HEAD`), so the next execution of the same run can see them.
## Problem
The REMOTE-1332 snapshot pipeline captures workspace state via `git diff --binary HEAD` plus `git ls-files --others --exclude-standard` for every declared `.git` directory under the workspace. That captures tracked changes and untracked non-gitignored files inside a declared repo, but it silently loses:
- Files the agent creates or edits outside any git repo (e.g. `/tmp`, `$HOME`, or the workspace root before `git init`).
- Files inside a repo but below a `.gitignore` entry, which the git-diff path deliberately ignores.
After a cloud → cloud handoff, the next execution never sees those files, so agent work that deposits logs, scratch scripts, or generated artifacts outside a repo is effectively thrown away.
## Behavior
1. During a cloud Oz run, every time the agent successfully completes a file-edit tool call that creates or modifies a file, the file's absolute path is recorded in the snapshot declarations file as a `file` entry. The entry lands in the declarations file before the end-of-run upload reads it.
2. Each entry uses the existing REMOTE-1332 schema: `{"version":1,"kind":"file","path":"<absolute path>"}`. The declarations file is the same per-run file the existing pipeline reads (resolved via `$OZ_SNAPSHOT_DECLARATIONS_FILE`, otherwise `/tmp/oz/<task-id>/snapshot-declarations.jsonl`).
3. All recorded paths are absolute. If a tool-call result surfaces a relative path, it is resolved against the driver's `working_dir`. Paths that still cannot be made absolute are dropped with a WARN log and not recorded.
4. Within a single run, the same absolute path is never written more than once to the declarations file. Multiple edits to the same file, or a create followed by an edit, still produce only one `file` entry.
5. Appends happen incrementally as each tool call completes, not in a single batch at end of run. If the run crashes after some edits but before the upload step, any `file` entries that already landed on disk are still present for any recovery tooling that reads the declarations file.
6. File deletions are not recorded in v1. `RequestFileEditsResult::Success.deleted_files` is observed but ignored. Deletions of tracked files inside a declared repo are still captured by `git diff HEAD` in the existing pipeline.
7. Before upload, the snapshot pipeline drops `file` entries whose absolute path falls under any declared `repo` entry's path. This prevents double-uploading files that the repo's diff already captures. If the agent creates files and then later `git init`s a directory containing them, the end-of-run script emits a `repo` entry for the new directory, which causes the earlier `file` entries to be filtered out at gather time.
8. A `file` entry whose path is not under any declared repo is uploaded in full, exactly like an operator-authored `file` entry already would be today per REMOTE-1332.
9. The feature is inert when any of the following are true, matching the existing REMOTE-1332 gating:
- `FeatureFlag::OzHandoff` is disabled.
- The run has no associated task ID (purely local runs).
- The run was started with `--no-snapshot`.
In these cases, no declarations are written and no file-edit observations have any user-visible effect.
10. The hook only runs for the Warp Oz SDK driver. Third-party harnesses (e.g. Claude Code) do not participate in this mechanism; their file writes go through their own tools and are not observed (this can and will be fixed in follow-up PRs, using each agent's hook system to track file edits).
11. Writer failures (declarations file not writable, path normalization error, file system I/O error) are logged at WARN level and absorbed. The agent run, the current tool call, and the end-of-run snapshot upload all continue as if no file entry were recorded for that call.
12. Shell-driven writes (`cat > file`, `echo >>`, `tee`, `touch`, etc.) are not captured by this mechanism. Those files only appear in the snapshot when they happen to sit under a declared repo and the git-diff path catches them.
13. A gitignored file edited by the agent inside a declared repo is not captured by v1: it is dropped by the repo-overlap filter in (7) and git's diff path ignores it. This is a known v1 limitation; see "Open questions" below.
## Non-goals
- Capturing file deletions. The existing `file` declaration kind uploads bytes and cannot represent a tombstone. Tracked-file deletions are still covered by `git diff HEAD`.
- Capturing shell-driven file writes. Parsing arbitrary shell commands to infer file effects is brittle and out of scope.
- Adding start-of-run scanning. End-of-run repo scanning plus gather-time overlap filtering is enough to handle the "agent created files before `git init`" case; a redundant start-of-run scan is not added.
- Extending this behavior to third-party harnesses (Claude Code, etc.). Harness-specific hook support is a separate follow-up.
- Changing the existing snapshot declarations schema, the `snapshot-declarations.sh` script, or the Docker image.
## Open questions
- **Gitignored files inside a declared repo:** v1 drops any `file` entry whose path falls under a declared repo path, which means gitignored files inside that repo are silently lost. Should a follow-up tighten the overlap filter (e.g. use `git check-ignore -q` to keep `file` entries whose paths the repo's diff will not carry)?
- **Per-run size caps:** REMOTE-1332's `MAX_SNAPSHOT_FILES_PER_RUN = 100` already bounds total blobs. A very chatty agent could plausibly touch more than 99 distinct non-repo files in one run; any excess ends up as `skipped` in the manifest. Do we need a separate earlier-than-cap warning for operators when the tool-call writer approaches that ceiling?
specs/REMOTE-1465/TECH.md
+132
View File
@@ -0,0 +1,132 @@
# Oz File-Edit Hooks for Snapshotting Non-Git-Tracked Files — Tech Spec
Product spec: `specs/REMOTE-1465/PRODUCT.md`
Linear: [REMOTE-1465](https://linear.app/warpdotdev/issue/REMOTE-1465)
## Context
This feature extends the REMOTE-1332 end-of-run snapshot pipeline (`specs/REMOTE-1332/TECH.md:408` explicitly records it as a follow-up). The declarations file already supports two entry kinds (`repo` and `file`), but no code path ever emits `file` entries automatically — the docker-image `snapshot-declarations.sh` script only emits `repo` lines, and operator-authored `file` entries are the only current producer. That means files the agent creates or edits outside any declared git repo never make it into the snapshot upload.
The Oz SDK driver already has every hook point we need. It subscribes to `BlocklistAIHistoryEvent` for the driver's own terminal view, observes every `AIAgentExchange` as it completes, and already runs the end-of-run snapshot pipeline with the correct gates. The work here is to observe successful file-edit tool results inside that subscription and append matching `file` lines to the declarations file the pipeline already reads, then filter out any `file` entries that would double-upload against the scanned `repo` entries.
### Relevant code
- `app/src/ai/agent_sdk/driver/snapshot.rs` — the end-of-run snapshot pipeline. The JSONL declarations format is documented in the module-level doc, and `DeclarationEntry` (in the `Declarations file parsing` section) already understands both `Repo` and `File` kinds. `resolve_declarations_path` resolves the per-run declarations path (env override → `/tmp/oz/<task-id>/snapshot-declarations.jsonl` → `/tmp/oz/snapshot-declarations.jsonl`). `upload_snapshot_from_declarations_file` is where the parsed entries are consumed.
- `build_repo_patch` in the same file generates the per-repo patch via `git diff --binary HEAD` plus `git ls-files --others --exclude-standard`. This is the path that already covers tracked changes and untracked non-gitignored files inside a declared repo.
- `app/src/ai/agent_sdk/driver.rs::AgentDriver::execute_run` — the existing `BlocklistAIHistoryEvent` subscription. The `AppendedExchange` arm already calls `write_exchange_inputs` once a new exchange is appended. Every `AIAgentActionResultType::RequestFileEdits(RequestFileEditsResult::Success { updated_files, deleted_files, .. })` reaches the next exchange as an `AIAgentInput::ActionResult` input, and the paths in `updated_files[i].file_context.file_name` are the absolute paths we need.
- `AgentDriverOptions` and the `AgentDriver` struct in the same file already own `working_dir: PathBuf`, `task_id: Option<AmbientAgentTaskId>`, and `snapshot_disabled: bool`. These are the inputs the writer needs.
- `AgentDriver::run_snapshot_upload` defines the existing gate (`FeatureFlag::OzHandoff.is_enabled()`, `task_id.is_some()`, `!snapshot_disabled`). The writer reuses this gate verbatim.
- `crates/ai/src/agent/action_result/mod.rs::RequestFileEditsResult` is the variant we pattern-match. `UpdatedFileContext.file_context.file_name` carries the absolute file path populated by `apply_create_file` / `apply_search_replace` / `apply_v4a_update` in `app/src/ai/blocklist/action_model/execute/request_file_edits/diff_application.rs` (they call `host_native_absolute_path` before constructing diffs, so the path reaching the executor is already absolute in practice).
- `warp-agent-docker/snapshot-declarations.sh` — the existing script's dedup step reads only `repo` JSONL lines into its seen-set, so `file` lines written by the driver are left intact across repeated script invocations.
## Proposed changes
### 1. Observe successful file edits in the SDK driver
Extend the existing `BlocklistAIHistoryEvent` handler in `AgentDriver::execute_run` so that on every `AppendedExchange` event, the driver walks `exchange.input` for `AIAgentInput::ActionResult` entries and extracts every `AIAgentActionResultType::RequestFileEdits(RequestFileEditsResult::Success { updated_files, .. })` result's paths. Action results from the prior exchange flow back as inputs on the next exchange, so scanning inputs on each newly-appended exchange captures every completed file edit. `deleted_files` is deliberately ignored (see product invariant 6).
Gate the observer at construction time: `AgentDriver::new` only constructs a `DeclarationsWriterHandle` when `FeatureFlag::OzHandoff.is_enabled() && task_id.is_some() && !snapshot_disabled`. When that gate fails the field stays `None` and the observer's `if let Some(writer) = me.snapshot_file_writer.as_ref()` short-circuits before touching any exchange data.
The subscription closure runs on the driver's model-context thread and must not touch the filesystem inline. It only collects path strings from each successful `RequestFileEdits` result and hands the resulting `Vec<String>` off to the writer task introduced in section 2 via a non-blocking `DeclarationsWriterHandle::append` call. Path normalization (joining relative paths against `working_dir`, dropping non-absolute or non-UTF-8 paths), the on-write repo preempt, directory creation, and JSONL append writes all happen on the writer task — never on the subscription thread.
Any error surfaced by the writer task is logged and absorbed. The observer never fails the subscription, the exchange, or the overall run.
### 2. Dedicated async writer task for `file` declarations
Introduce a `DeclarationsWriterHandle` in `app/src/ai/agent_sdk/driver/snapshot.rs`, owned by `AgentDriver` for the lifetime of the run:
```rust
enum WriterCommand {
Append(Vec<String>),
Flush(oneshot::Sender<()>),
}
pub(super) struct DeclarationsWriterHandle {
tx: tokio::sync::mpsc::UnboundedSender<WriterCommand>,
}
impl DeclarationsWriterHandle {
pub(super) fn new(
task_id: AmbientAgentTaskId,
working_dir: PathBuf,
background: &Background,
) -> Self;
pub(super) fn append(&self, paths: Vec<String>);
pub(super) async fn flush(&self);
}
```
`new` spawns a single long-lived task on the application's `Background` tokio runtime that owns the `seen: HashSet<String>` plus the resolved declarations path. The task loops on `rx.recv()` and processes commands sequentially, which gives us in-process serialization for free — two subscription events cannot race on either the set or the file because there is exactly one writer. `AgentDriver::new` runs in a sync `add_singleton_model` callback, so the writer is spawned via `ctx.background_executor()` rather than `tokio::spawn` (the foreground executor has no current tokio runtime).
`WriterCommand::Append` handling, per path:
1. Normalize: join non-absolute paths against `working_dir`, then drop the path with a `log::warn!` if it is still not absolute or is not valid UTF-8.
2. Skip if the normalized path is already in `seen` (in-memory dedup, product invariant 4).
3. Walk the path's ancestors and `tokio::fs::try_exists` each `<ancestor>/.git`. If any ancestor is already a repo at enqueue time, skip with `log::debug!` and add the path to `seen` so future appends short-circuit. This is the cheap on-write preempt for the common "agent edits inside an existing repo" case and avoids populating the declarations file with entries the gather-time filter would just drop anyway.
4. Otherwise `tokio::fs::create_dir_all` the declarations file's parent, open the file with `tokio::fs::OpenOptions::new().append(true).create(true).open(...)`, write one JSONL line built from the `FileDeclaration` struct via `serde_json::to_string`, `flush().await`, and then `seen.insert(path)`.
5. Per-path write failures are logged at `log::warn!` (the `seen` set is not advanced) and the task continues with the next path and the next command.
`WriterCommand::Flush` simply acks the provided oneshot. Because the writer task drains the mpsc queue in order, the ack fires only after every previously-enqueued `Append` has finished its fs writes — callers that await `flush()` can rely on every prior `append()` being on disk by the time `flush` returns.
`DeclarationsWriterHandle::append` is a sync, non-blocking send from the subscription thread. `DeclarationsWriterHandle::flush` is called once from `run_snapshot_upload` immediately before `snapshot::run_declarations_script`, so no driver-side write is in flight when the script starts its own appends.
Visibility is the minimum needed: the handle, its command enum, and `new`/`append`/`flush` are `pub(super)` so the driver module can construct and drive them; the writer task's internal helpers stay `fn`.
### 3. Drop `file` entries covered by declared repos at gather time
Add a pure helper in `app/src/ai/agent_sdk/driver/snapshot.rs`:
```rust
fn drop_files_covered_by_repos(entries: Vec<DeclarationEntry>) -> Vec<DeclarationEntry>;
```
It:
- Collects every `EntryKind::Repo` path.
- For each `EntryKind::File` entry, drops it if the file path, treated as a filesystem path, `starts_with` any repo path (per `std::path::Path::starts_with` on normalized components). Logs an `info!` noting which repo covers which file so operators can trace elided entries.
- Leaves every other entry untouched.
Invocation site: `upload_snapshot_from_declarations_file`, immediately after `read_and_parse_declarations` returns and before the `repo_count` / `file_count` log line so the count reflects what will actually be uploaded.
This gather-time filter and the on-write ancestor-repo check in section 2 are two layers with different jobs:
- The on-write check preempts `file` declarations when the path already sits inside an existing repo at edit time, so common-case edits inside `/workspace/existing-repo/` never hit the declarations file at all. This is the fast path for the majority of agent edits and keeps the end-of-run gather step cheap.
- The gather-time filter covers the "agent writes files, then later `git init`s the parent directory" case: at write time no `.git` existed yet so the path was recorded as a `file` entry, but once the end-of-run script emits the new `repo` entry, this filter elides the earlier `file` entries so the file is not uploaded twice (once as a raw blob, once inside the patch's untracked-files section).
### 4. No changes to `snapshot-declarations.sh` or `warp-agent-docker`
The existing script's dedup step only tracks `repo` lines (`warp-agent-docker/snapshot-declarations.sh:42`), so driver-written `file` lines survive re-invocation. Nothing in the Docker image or `entrypoint.sh` needs to change.
### 5. Feature flag and rollout
No new flags. Reuses `FeatureFlag::OzHandoff` so the whole mechanism is in lockstep with REMOTE-1332's rollout.
## Testing and validation
Product-spec invariants in `specs/REMOTE-1465/PRODUCT.md` map to tests as follows. New unit tests live in `app/src/ai/agent_sdk/driver/snapshot_tests.rs` next to the existing REMOTE-1332 coverage.
- Invariant 1, 2, 4, 5 — `DeclarationsWriterHandle` unit test: construct a handle against a synthetic task id and a tmp `working_dir`, `append` a sequence of absolute paths that includes a repeat, `flush().await`, and assert the handle created the parent directory, wrote one JSONL line per unique path, and a second `append+flush` of the same paths is a no-op. Assert exact JSONL shape by round-tripping through `serde_json::from_str::<DeclarationLine>` so any schema drift surfaces here.
- Invariant 3 — same handle test extended: relative paths passed to `append` are resolved against `working_dir`; paths that cannot be made absolute are dropped with a WARN log and not written.
- On-write repo preempt (section 2 step 2) — writer unit test: set up a tmp dir containing a `.git` subdirectory, `append` a path under it, `flush().await`, and assert no JSONL line was written. Repeat with a path outside any `.git` ancestor and assert the line is written.
- Flush semantics (section 2) — writer unit test: `append` many paths followed immediately by `flush().await`, and assert every expected JSONL line is on disk by the time `flush` returns; a follow-up `append+flush` after the first flush still works.
- Invariant 6 — covered by structural observation: the observer pattern-matches `RequestFileEditsResult::Success { updated_files, .. }` and never reads `deleted_files`. Any future regression here would surface in the driver-side observer code rather than as a behavioral difference, so no dedicated unit test is added.
- Invariant 7, 8 — end-to-end pipeline test in `snapshot_tests.rs` (`e2e_repo_plus_inside_and_outside_files_filters_overlap`): pre-seed a declarations file with one `repo` path and two `file` paths (one inside the repo, one outside). Run `upload_snapshot_from_declarations_file` against a `mockito::Server` harness and assert the inside-repo file is not uploaded and does not appear in the manifest's `files` list, while the outside-repo file is uploaded normally.
- Invariant 9 — covered structurally: `AgentDriver::new` returns `snapshot_file_writer: None` whenever any of the three gate conditions fail, and the observer closure no-ops on `None`. Driver-level harness is not exercised here because the gate logic is co-located with the construction site and is small enough to verify by inspection.
- Invariant 10 — no explicit test; enforced structurally because the observer lives on `AgentDriver`, which only runs for Oz SDK runs. Third-party harnesses don't subscribe to `BlocklistAIHistoryEvent` for file edits.
- Invariant 11 — error-path test (`declarations_writer_continues_after_per_path_write_failures`): pre-create a directory at the declarations file path so the first `append`'s open call fails, then remove it and assert the next `append` succeeds. Verifies the writer task absorbs the failure and keeps servicing commands.
- Invariant 12, 13 — documented as product-level limitations; no explicit tests. The existing REMOTE-1332 untracked-files coverage continues to validate the underlying git path.
Manual validation:
- Run a cloud Oz run (`./script/oz-local` per Warp Drive notebook `zOJarbIZgXHJDXS7dF9u82`) that asks the agent to create a file at `/tmp/oz-handoff-check.txt`. Confirm the declarations file picks up a `file` line, and confirm the end-of-run snapshot manifest includes the file with `"status": "uploaded"`.
- Run a cloud Oz run where the agent edits a file inside a pre-existing git repo under the workspace. Confirm the end-of-run pipeline logs `drop_files_covered_by_repos` electing not to upload the file as a standalone blob, and confirm the manifest still shows the repo's patch containing the change.
- Run a cloud Oz run where the agent creates files first, then runs `git init`. Confirm the manifest shows one `repo` entry for the initialized directory, and no separate `file` entries for the pre-existing files under it.
- Repeat the first manual case with `--no-snapshot`. Confirm no declarations file is written even though the agent edits a file.
## Risks and mitigations
- **Double-uploading when the overlap filter misses a nested repo root.** `Path::starts_with` is strict, so `/workspace/my-proj` and `/workspace/my-proj/sub/.git` both get their own `repo` entry after scanning; a `file` under `sub/` is correctly caught by the `sub` repo. The only real failure mode is comparing paths with differing trailing separators or symlinks; normalize both sides with a helper that strips trailing `/` and canonicalizes where possible before comparing.
- **Writer contention on a single declarations file.** In-process serialization comes from the single writer task owning both the `seen` set and the file handle — two subscription events cannot race on either, because `mpsc::UnboundedSender::send` from the subscription is non-blocking and the writer task processes commands sequentially. Cross-process serialization against `snapshot-declarations.sh` relies on two pieces: (1) `run_snapshot_upload` awaits `DeclarationsWriterHandle::flush` before spawning the script, so no driver-side write is queued when the script starts; and (2) `O_APPEND` atomicity on POSIX guarantees that any write smaller than `PIPE_BUF` (4096 on Linux, 512 on macOS, safely larger than our JSONL lines) cannot interleave with another writer's output. `flock(2)` is not added; the drain plus atomic appends are sufficient for correctness.
- **Gitignored files inside a declared repo are silently dropped.** Documented in product invariant 13 and the Open questions section. Mitigation deferred to a follow-up: fold `git check-ignore -q <path>` into `drop_files_covered_by_repos` so `file` entries whose paths git would ignore are preserved.
- **Exchange input walker missing new tool variants.** The observer explicitly pattern-matches `AIAgentInput::ActionResult` carrying `AIAgentActionResultType::RequestFileEdits(RequestFileEditsResult::Success { .. })` and ignores other variants. If future file-writing tools (e.g. `CreateDocuments`, `EditDocuments`) start writing to the local filesystem, they must either go through `RequestFileEdits` or get their own observer branch. The existing `CreateDocuments` / `EditDocuments` variants target Warp Drive documents rather than local files, so this is not a regression today.
- **Last-exchange edits without a follow-up.** Action results from the most recent exchange only land in the next exchange's inputs, so file edits performed in the very last exchange of a run (e.g. when the agent stops without producing a follow-up) are not observed. In practice the agent always emits a final response after tool calls, but if this becomes a measurable gap we can also poll `exchange.input` from the `UpdatedConversationStatus` arm before the run completes.
## Follow-ups
- Use `git check-ignore -q` (or `git ls-files --error-unmatch`) to keep `file` entries whose paths fall inside a declared repo but would not be carried by the repo's diff.
- Add a tombstone-style `deleted_file` declaration kind plus manifest support so the snapshot can represent deletion of files outside any declared repo.
- Wire equivalent hooks for third-party harnesses (Claude Code) via their hook system so this mechanism also works for non-Oz runs, per the Linear issue's explicit follow-up line.
- Surface a WARN earlier (e.g. at 75% of `MAX_SNAPSHOT_FILES_PER_RUN`) when the tool-call writer approaches the per-run cap.