galaxy/specs/REMOTE-1465/TECH.md

Oz File-Edit Hooks for Snapshotting Non-Git-Tracked Files — Tech Spec

Product spec: specs/REMOTE-1465/PRODUCT.md Linear: REMOTE-1465

Context

This feature extends the REMOTE-1332 end-of-run snapshot pipeline (specs/REMOTE-1332/TECH.md:408 explicitly records it as a follow-up). The declarations file already supports two entry kinds (repo and file), but no code path ever emits file entries automatically — the docker-image snapshot-declarations.sh script only emits repo lines, and operator-authored file entries are the only current producer. That means files the agent creates or edits outside any declared git repo never make it into the snapshot upload.

The Oz SDK driver already has every hook point we need. It subscribes to BlocklistAIHistoryEvent for the driver's own terminal view, observes every AIAgentExchange as it completes, and already runs the end-of-run snapshot pipeline with the correct gates. The work here is to observe successful file-edit tool results inside that subscription and append matching file lines to the declarations file the pipeline already reads, then filter out any file entries that would double-upload against the scanned repo entries.

Relevant code

• app/src/ai/agent_sdk/driver/snapshot.rs — the end-of-run snapshot pipeline. The JSONL declarations format is documented in the module-level doc, and DeclarationEntry (in the Declarations file parsing section) already understands both Repo and File kinds. resolve_declarations_path resolves the per-run declarations path (env override → /tmp/oz/<task-id>/snapshot-declarations.jsonl → /tmp/oz/snapshot-declarations.jsonl). upload_snapshot_from_declarations_file is where the parsed entries are consumed.
• build_repo_patch in the same file generates the per-repo patch via git diff --binary HEAD plus git ls-files --others --exclude-standard. This is the path that already covers tracked changes and untracked non-gitignored files inside a declared repo.
• app/src/ai/agent_sdk/driver.rs::AgentDriver::execute_run — the existing BlocklistAIHistoryEvent subscription. The AppendedExchange arm already calls write_exchange_inputs once a new exchange is appended. Every AIAgentActionResultType::RequestFileEdits(RequestFileEditsResult::Success { updated_files, deleted_files, .. }) reaches the next exchange as an AIAgentInput::ActionResult input, and the paths in updated_files[i].file_context.file_name are the absolute paths we need.
• AgentDriverOptions and the AgentDriver struct in the same file already own working_dir: PathBuf, task_id: Option<AmbientAgentTaskId>, and snapshot_disabled: bool. These are the inputs the writer needs.
• AgentDriver::run_snapshot_upload defines the existing gate (FeatureFlag::OzHandoff.is_enabled(), task_id.is_some(), !snapshot_disabled). The writer reuses this gate verbatim.
• crates/ai/src/agent/action_result/mod.rs::RequestFileEditsResult is the variant we pattern-match. UpdatedFileContext.file_context.file_name carries the absolute file path populated by apply_create_file / apply_search_replace / apply_v4a_update in app/src/ai/blocklist/action_model/execute/request_file_edits/diff_application.rs (they call host_native_absolute_path before constructing diffs, so the path reaching the executor is already absolute in practice).
• warp-agent-docker/snapshot-declarations.sh — the existing script's dedup step reads only repo JSONL lines into its seen-set, so file lines written by the driver are left intact across repeated script invocations.

Proposed changes

1. Observe successful file edits in the SDK driver

Extend the existing BlocklistAIHistoryEvent handler in AgentDriver::execute_run so that on every AppendedExchange event, the driver walks exchange.input for AIAgentInput::ActionResult entries and extracts every AIAgentActionResultType::RequestFileEdits(RequestFileEditsResult::Success { updated_files, .. }) result's paths. Action results from the prior exchange flow back as inputs on the next exchange, so scanning inputs on each newly-appended exchange captures every completed file edit. deleted_files is deliberately ignored (see product invariant 6).

Gate the observer at construction time: AgentDriver::new only constructs a DeclarationsWriterHandle when FeatureFlag::OzHandoff.is_enabled() && task_id.is_some() && !snapshot_disabled. When that gate fails the field stays None and the observer's if let Some(writer) = me.snapshot_file_writer.as_ref() short-circuits before touching any exchange data.

The subscription closure runs on the driver's model-context thread and must not touch the filesystem inline. It only collects path strings from each successful RequestFileEdits result and hands the resulting Vec<String> off to the writer task introduced in section 2 via a non-blocking DeclarationsWriterHandle::append call. Path normalization (joining relative paths against working_dir, dropping non-absolute or non-UTF-8 paths), the on-write repo preempt, directory creation, and JSONL append writes all happen on the writer task — never on the subscription thread.

Any error surfaced by the writer task is logged and absorbed. The observer never fails the subscription, the exchange, or the overall run.

2. Dedicated async writer task for file declarations

Introduce a DeclarationsWriterHandle in app/src/ai/agent_sdk/driver/snapshot.rs, owned by AgentDriver for the lifetime of the run:

enum WriterCommand {
    Append(Vec<String>),
    Flush(oneshot::Sender<()>),
}

pub(super) struct DeclarationsWriterHandle {
    tx: tokio::sync::mpsc::UnboundedSender<WriterCommand>,
}

impl DeclarationsWriterHandle {
    pub(super) fn new(
        task_id: AmbientAgentTaskId,
        working_dir: PathBuf,
        background: &Background,
    ) -> Self;
    pub(super) fn append(&self, paths: Vec<String>);
    pub(super) async fn flush(&self);
}

new spawns a single long-lived task on the application's Background tokio runtime that owns the seen: HashSet<String> plus the resolved declarations path. The task loops on rx.recv() and processes commands sequentially, which gives us in-process serialization for free — two subscription events cannot race on either the set or the file because there is exactly one writer. AgentDriver::new runs in a sync add_singleton_model callback, so the writer is spawned via ctx.background_executor() rather than tokio::spawn (the foreground executor has no current tokio runtime).

WriterCommand::Append handling, per path:

1. Normalize: join non-absolute paths against working_dir, then drop the path with a log::warn! if it is still not absolute or is not valid UTF-8.
2. Skip if the normalized path is already in seen (in-memory dedup, product invariant 4).
3. Walk the path's ancestors and tokio::fs::try_exists each <ancestor>/.git. If any ancestor is already a repo at enqueue time, skip with log::debug! and add the path to seen so future appends short-circuit. This is the cheap on-write preempt for the common "agent edits inside an existing repo" case and avoids populating the declarations file with entries the gather-time filter would just drop anyway.
4. Otherwise tokio::fs::create_dir_all the declarations file's parent, open the file with tokio::fs::OpenOptions::new().append(true).create(true).open(...), write one JSONL line built from the FileDeclaration struct via serde_json::to_string, flush().await, and then seen.insert(path).
5. Per-path write failures are logged at log::warn! (the seen set is not advanced) and the task continues with the next path and the next command.

WriterCommand::Flush simply acks the provided oneshot. Because the writer task drains the mpsc queue in order, the ack fires only after every previously-enqueued Append has finished its fs writes — callers that await flush() can rely on every prior append() being on disk by the time flush returns.

DeclarationsWriterHandle::append is a sync, non-blocking send from the subscription thread. DeclarationsWriterHandle::flush is called once from run_snapshot_upload immediately before snapshot::run_declarations_script, so no driver-side write is in flight when the script starts its own appends.

Visibility is the minimum needed: the handle, its command enum, and new/append/flush are pub(super) so the driver module can construct and drive them; the writer task's internal helpers stay fn.

3. Drop file entries covered by declared repos at gather time

Add a pure helper in app/src/ai/agent_sdk/driver/snapshot.rs:

fn drop_files_covered_by_repos(entries: Vec<DeclarationEntry>) -> Vec<DeclarationEntry>;

It:

• Collects every EntryKind::Repo path.
• For each EntryKind::File entry, drops it if the file path, treated as a filesystem path, starts_with any repo path (per std::path::Path::starts_with on normalized components). Logs an info! noting which repo covers which file so operators can trace elided entries.
• Leaves every other entry untouched.

Invocation site: upload_snapshot_from_declarations_file, immediately after read_and_parse_declarations returns and before the repo_count / file_count log line so the count reflects what will actually be uploaded.

This gather-time filter and the on-write ancestor-repo check in section 2 are two layers with different jobs:

• The on-write check preempts file declarations when the path already sits inside an existing repo at edit time, so common-case edits inside /workspace/existing-repo/ never hit the declarations file at all. This is the fast path for the majority of agent edits and keeps the end-of-run gather step cheap.
• The gather-time filter covers the "agent writes files, then later git inits the parent directory" case: at write time no .git existed yet so the path was recorded as a file entry, but once the end-of-run script emits the new repo entry, this filter elides the earlier file entries so the file is not uploaded twice (once as a raw blob, once inside the patch's untracked-files section).

4. No changes to snapshot-declarations.sh or warp-agent-docker

The existing script's dedup step only tracks repo lines (warp-agent-docker/snapshot-declarations.sh:42), so driver-written file lines survive re-invocation. Nothing in the Docker image or entrypoint.sh needs to change.

5. Feature flag and rollout

No new flags. Reuses FeatureFlag::OzHandoff so the whole mechanism is in lockstep with REMOTE-1332's rollout.

Testing and validation

Product-spec invariants in specs/REMOTE-1465/PRODUCT.md map to tests as follows. New unit tests live in app/src/ai/agent_sdk/driver/snapshot_tests.rs next to the existing REMOTE-1332 coverage.

• Invariant 1, 2, 4, 5 — DeclarationsWriterHandle unit test: construct a handle against a synthetic task id and a tmp working_dir, append a sequence of absolute paths that includes a repeat, flush().await, and assert the handle created the parent directory, wrote one JSONL line per unique path, and a second append+flush of the same paths is a no-op. Assert exact JSONL shape by round-tripping through serde_json::from_str::<DeclarationLine> so any schema drift surfaces here.
• Invariant 3 — same handle test extended: relative paths passed to append are resolved against working_dir; paths that cannot be made absolute are dropped with a WARN log and not written.
• On-write repo preempt (section 2 step 2) — writer unit test: set up a tmp dir containing a .git subdirectory, append a path under it, flush().await, and assert no JSONL line was written. Repeat with a path outside any .git ancestor and assert the line is written.
• Flush semantics (section 2) — writer unit test: append many paths followed immediately by flush().await, and assert every expected JSONL line is on disk by the time flush returns; a follow-up append+flush after the first flush still works.
• Invariant 6 — covered by structural observation: the observer pattern-matches RequestFileEditsResult::Success { updated_files, .. } and never reads deleted_files. Any future regression here would surface in the driver-side observer code rather than as a behavioral difference, so no dedicated unit test is added.
• Invariant 7, 8 — end-to-end pipeline test in snapshot_tests.rs (e2e_repo_plus_inside_and_outside_files_filters_overlap): pre-seed a declarations file with one repo path and two file paths (one inside the repo, one outside). Run upload_snapshot_from_declarations_file against a mockito::Server harness and assert the inside-repo file is not uploaded and does not appear in the manifest's files list, while the outside-repo file is uploaded normally.
• Invariant 9 — covered structurally: AgentDriver::new returns snapshot_file_writer: None whenever any of the three gate conditions fail, and the observer closure no-ops on None. Driver-level harness is not exercised here because the gate logic is co-located with the construction site and is small enough to verify by inspection.
• Invariant 10 — no explicit test; enforced structurally because the observer lives on AgentDriver, which only runs for Oz SDK runs. Third-party harnesses don't subscribe to BlocklistAIHistoryEvent for file edits.
• Invariant 11 — error-path test (declarations_writer_continues_after_per_path_write_failures): pre-create a directory at the declarations file path so the first append's open call fails, then remove it and assert the next append succeeds. Verifies the writer task absorbs the failure and keeps servicing commands.
• Invariant 12, 13 — documented as product-level limitations; no explicit tests. The existing REMOTE-1332 untracked-files coverage continues to validate the underlying git path.

Manual validation:

• Run a cloud Oz run (./script/oz-local per Warp Drive notebook zOJarbIZgXHJDXS7dF9u82) that asks the agent to create a file at /tmp/oz-handoff-check.txt. Confirm the declarations file picks up a file line, and confirm the end-of-run snapshot manifest includes the file with "status": "uploaded".
• Run a cloud Oz run where the agent edits a file inside a pre-existing git repo under the workspace. Confirm the end-of-run pipeline logs drop_files_covered_by_repos electing not to upload the file as a standalone blob, and confirm the manifest still shows the repo's patch containing the change.
• Run a cloud Oz run where the agent creates files first, then runs git init. Confirm the manifest shows one repo entry for the initialized directory, and no separate file entries for the pre-existing files under it.
• Repeat the first manual case with --no-snapshot. Confirm no declarations file is written even though the agent edits a file.

Risks and mitigations

• Double-uploading when the overlap filter misses a nested repo root. Path::starts_with is strict, so /workspace/my-proj and /workspace/my-proj/sub/.git both get their own repo entry after scanning; a file under sub/ is correctly caught by the sub repo. The only real failure mode is comparing paths with differing trailing separators or symlinks; normalize both sides with a helper that strips trailing / and canonicalizes where possible before comparing.
• Writer contention on a single declarations file. In-process serialization comes from the single writer task owning both the seen set and the file handle — two subscription events cannot race on either, because mpsc::UnboundedSender::send from the subscription is non-blocking and the writer task processes commands sequentially. Cross-process serialization against snapshot-declarations.sh relies on two pieces: (1) run_snapshot_upload awaits DeclarationsWriterHandle::flush before spawning the script, so no driver-side write is queued when the script starts; and (2) O_APPEND atomicity on POSIX guarantees that any write smaller than PIPE_BUF (4096 on Linux, 512 on macOS, safely larger than our JSONL lines) cannot interleave with another writer's output. flock(2) is not added; the drain plus atomic appends are sufficient for correctness.
• Gitignored files inside a declared repo are silently dropped. Documented in product invariant 13 and the Open questions section. Mitigation deferred to a follow-up: fold git check-ignore -q <path> into drop_files_covered_by_repos so file entries whose paths git would ignore are preserved.
• Exchange input walker missing new tool variants. The observer explicitly pattern-matches AIAgentInput::ActionResult carrying AIAgentActionResultType::RequestFileEdits(RequestFileEditsResult::Success { .. }) and ignores other variants. If future file-writing tools (e.g. CreateDocuments, EditDocuments) start writing to the local filesystem, they must either go through RequestFileEdits or get their own observer branch. The existing CreateDocuments / EditDocuments variants target Warp Drive documents rather than local files, so this is not a regression today.
• Last-exchange edits without a follow-up. Action results from the most recent exchange only land in the next exchange's inputs, so file edits performed in the very last exchange of a run (e.g. when the agent stops without producing a follow-up) are not observed. In practice the agent always emits a final response after tool calls, but if this becomes a measurable gap we can also poll exchange.input from the UpdatedConversationStatus arm before the run completes.

Follow-ups

• Use git check-ignore -q (or git ls-files --error-unmatch) to keep file entries whose paths fall inside a declared repo but would not be carried by the repo's diff.
• Add a tombstone-style deleted_file declaration kind plus manifest support so the snapshot can represent deletion of files outside any declared repo.
• Wire equivalent hooks for third-party harnesses (Claude Code) via their hook system so this mechanism also works for non-Oz runs, per the Linear issue's explicit follow-up line.
• Surface a WARN earlier (e.g. at 75% of MAX_SNAPSHOT_FILES_PER_RUN) when the tool-call writer approaches the per-run cap.