# APP-3792 codebase indexing persistence PR tech spec
## Problem statement
This PR makes remote codebase indexing survive daemon restarts and reconnects by restoring daemon-owned codebase index metadata and snapshots at startup, keeping the client synchronized with daemon status snapshots and updates, and exposing remote indexed codebases to the agent context in the same broad shape as local indexed codebases.
The changes are intentionally scoped to the persistence/bootstrap and protocol plumbing needed for APP-3792. They do not redesign the local indexing product flow, move retrieval fully into the daemon, or remove the current remote `ResyncCodebase` protocol path.
## Current state
Local codebase indexing is owned in-process by `CodebaseIndexManager` in `crates/ai/src/index/full_source_code_embedding/manager.rs`. The normal app path constructs it with app-scoped persisted metadata, app-default snapshot storage, a `BulkFilesystemWatcher`, and a client-side `StoreClient`. Local settings can trigger a manual resync through `CodeSettingsPageAction::ManualResync`, which calls `CodebaseIndexManager::try_manual_resync_codebase` directly because the settings UI and index manager live in the same process.
Remote codebase indexing splits those responsibilities across the Warp client and the remote-server daemon. The daemon owns remote filesystem walking, snapshot files, indexing work, and backend sync. The client owns session context, settings/speedbump decisions, agent tool advertisement, and retrieval orchestration. The client and daemon communicate through `crates/remote_server/proto/remote_server.proto`, so operations that are direct method calls locally become client-to-daemon messages remotely.
Before this PR, remote daemon indexing state was too transient: a reconnect or daemon restart did not have a narrow restore path for known remote codebase index metadata and daemon-scoped snapshots. The client also needed a bootstrap status snapshot from the daemon so the active remote repo and agent context could reflect already-indexed remote codebases without waiting for a new indexing run.
## Goals
Restore remote codebase index metadata for the remote-server daemon while keeping a clear startup boundary between full persistence reads and the subset of restored data the daemon is allowed to consume.
Give the daemon an identity-scoped persistence root and snapshot directory so long-lived remote indexing data does not mix with normal app `warp.sqlite` state.
Reuse the existing `CodebaseIndexManager` implementation for both app and daemon paths by injecting snapshot storage rather than forking indexing logic.
Push a full remote codebase index status snapshot after daemon initialize/reconnect, then keep the client current with incremental status updates.
Avoid automatic reindex requests when navigating to a repo that is already ready, stale with a last ready index, queued, or indexing.
Expose ready remote codebases to agent context as stable `(name, path)` entries.
Keep the current `ResyncCodebase` protocol in this PR while documenting why it exists and how it compares with local resync.
## Non-goals
Do not remove or fold `ResyncCodebase` into `IndexCodebase` in this PR.
Do not change local codebase indexing behavior, local persistence schema semantics beyond the shared metadata reuse, or local settings UI behavior.
Do not build a daemon-only indexing implementation separate from `CodebaseIndexManager`.
Do not make the daemon consume or initialize app-only state such as panes, cloud objects, command history, user profiles, MCP servers, or projects.
Do not introduce a client-initiated per-repo status fetch path; daemon-pushed snapshots and deltas remain the synchronization mechanism.
## Proposed design
### Daemon-scoped persistence restore
`persistence::initialize` accepts a `PersistenceScope` so startup can choose between the normal app scope and the remote-server daemon scope. `PersistenceScope::App` uses the normal app database path. `PersistenceScope::RemoteServerDaemon { identity_key }` uses a daemon-specific database path derived from the remote-server identity. Both scopes read the same `PersistedData` shape through the existing SQLite restore helper and both scopes receive writer handles for subsequent updates.
The boundary between app and daemon restore lives at startup initialization rather than in the SQLite reader. `initialize_app` maps the full restored `PersistedData` payload directly into the startup variables used by singleton registration, matching normal app startup. Immediately after that mapping, `initialize_app` applies the launch-mode boundary. Normal app startup consumes the full app restore payload. `LaunchMode::RemoteServerDaemon` preserves `persisted_workspaces` from `codebase_indices` for indexing restore and defaults the app-only startup fields. This keeps a single persistence read path and makes daemon consumption explicit and auditable at initialization.
This split is intentional. The daemon needs the same Diesel/SQLite open, migrate, writer, and full-schema read path as the app so the persistence layer does not fork into app-shaped and daemon-shaped restore contracts. But the daemon should not retain unrelated app-scoped state after initialization. Keeping the direct `PersistedData` startup mapping and filtering in `initialize_app` makes the daemon boundary explicit where startup models are registered while still reusing the existing persistence infrastructure.
The full app restore payload includes state that is meaningful only inside the interactive app process: window/session restoration, cloud object caches, command history, user profiles, workspace language-server settings, MCP server installations, project rules, ignored suggestions, and other UI or app-lifecycle state. Synchronizing all of that into the remote daemon would create two problems. First, the daemon would spend startup time reading and allocating data it will never use. Second, the daemon would become another consumer of app-owned invariants and migrations, so future app persistence changes could accidentally affect a headless remote process.
The daemon only consumes enough persisted data to reconstruct codebase-indexing state:
- repo metadata used to seed `CodebaseIndexManager`,
- the daemon snapshot root used to validate or discard serialized snapshots,
- identity-scoped status/enablement decisions represented by restored codebase metadata,
- writer handles for subsequent codebase-index metadata updates.
That narrow startup consumption is still “syncing app data” in the sense that it reuses the same `codebase_indices` model and full persisted data shape that local app startup uses, but it is not letting the daemon initialize the entire app object graph. This is the intended boundary: share the data model and persistence infrastructure for codebase indexing, then select launch-mode-appropriate fields before registering startup models.
The daemon database and snapshot directories should remain owner-only, matching the remote-server socket/cache privacy model. The database is identity-scoped because enablement/status/backend authorization decisions are Warp-user-specific. Snapshot files are injected separately so the daemon can use its remote-server data root instead of the app default.
### Shared index manager with injected snapshot storage
`CodebaseIndexManager::new` remains the default app constructor. In `local_fs` builds it migrates old app snapshots if needed and passes `SnapshotStorage::app_default()` into `new_with_snapshot_storage`.
`CodebaseIndexManager::new_with_snapshot_storage` is the daemon-compatible seam. It accepts the same persisted `WorkspaceMetadata` and indexing configuration as the app constructor, but lets startup choose the snapshot root. The remote-server daemon passes daemon-scoped storage; the app passes app-default storage.
This keeps indexing behavior shared:
- snapshot validity checks use the same code,
- invalid metadata emits the same `RemoveExpiredIndexMetadata` event,
- valid metadata feeds the same persisted build queue,
- rebuild/resync/drop paths continue to use `CodebaseIndexManager` and `CodebaseIndex`.
The only difference is where metadata and serialized snapshots are restored from and written to.
### Startup and status synchronization
On app startup, `initialize_app` selects a persistence scope from `LaunchMode`. The normal app, CLI, proxy, and tests use `PersistenceScope::App`; `LaunchMode::RemoteServerDaemon { identity_key }` uses `PersistenceScope::RemoteServerDaemon`.
`initialize_app` normalizes the full restored `PersistedData` payload into the startup fields consumed later by singleton registration. It then clears app-only restored fields for daemon launch while preserving `persisted_workspaces` from `codebase_indices`. That lets the existing `CodebaseIndexManager` constructor receive daemon-restored index metadata without pretending the daemon has a full app session restore.
When a client initializes with the daemon, `RemoteServerModel` pushes `CodebaseIndexStatusesSnapshot`. The client-side `RemoteCodebaseIndexModel` applies that snapshot by replacing statuses for the connected host, then applies subsequent `CodebaseIndexStatusUpdated` deltas. This makes reconnect a full-resync boundary and keeps steady-state updates lightweight.
### Navigation and automatic indexing
`RemoteCodebaseIndexModel` records the active repo for a host when it receives `NavigatedToDirectory`. If the navigated directory is a git repo and remote auto-indexing is enabled, it calls `should_request_auto_index_for_navigated_git_repo` before sending an indexing request.
That guard mirrors the local product expectation: navigating into an already-known repo should not immediately trigger another indexing run. It returns false when the current status is ready, stale with a usable last root, queued, or indexing. It returns true when the repo is missing from the status map or has an unusable state such as failed/unavailable/missing root hash.
This preserves automatic indexing for first discovery and recovery while avoiding repeated reindex requests on every `cd` into a repo.
### Agent context and search availability
`RemoteCodebaseIndexModel::codebases_for_agent_context` projects ready searchable remote repos into stable entries with a display name and path. The model only includes statuses that resolve to `RemoteCodebaseSearchAvailability::Ready`, so unindexed, indexing, failed, or otherwise unavailable repos do not appear as searchable codebase context.
For active-session search, `active_repo_availability` resolves an explicit repo path first when it matches known status, otherwise falls back to the active repo for the host or current working directory. Ready availability carries the remote path, root hash, and embedding config needed by downstream search plumbing.
## Protocol shape and local resync contrast
Local resync has two distinct product shapes but no wire protocol. First-time indexing goes through the same manager that owns the local file watcher, persisted metadata, and snapshot storage. Manual resync is an in-process settings action: the settings page dispatches `ManualResync(PathBuf)` and directly calls `CodebaseIndexManager::try_manual_resync_codebase`. Drop/delete similarly calls the manager directly through settings UI actions. Local code can distinguish “index this repo for the first time,” “retry or manually resync this already-indexed repo,” and “drop this repo” by calling different Rust methods because the UI and index manager share memory.
The local pattern also keeps resync conservative. A manual resync only applies when a codebase is already known to the manager. Navigation or repo discovery does not imply a full resync if a ready or stale index already exists; local indexing keeps search available against the last ready root while watcher-driven or manual sync work catches up.
Remote resync crosses the client/daemon boundary. This PR currently models that distinction explicitly with two proto messages:
- `IndexCodebase { repo_path, auth_token }` requests indexing for a repo that may not have an index yet.
- `ResyncCodebase { repo_path, auth_token }` requests a manual full resync of a repo that is already indexed.
That explicit remote protocol mirrors the local product distinction between initial indexing and retry/resync affordances, while making the daemon-side behavior readable in logs, telemetry operation names, and request dispatch. It also lets the daemon return a clear unavailable status when asked to resync a repo it does not know about.
There is a reasonable simplification to consider later: make remote `IndexCodebase` mean “ensure indexed, and if already indexed, perform the manual full resync requested by the client.” That would remove `ResyncCodebase` from the proto and make remote indexing idempotent through one request type. This PR does not implement that simplification so the persistence/status bootstrap work remains isolated from protocol churn.
If a follow-up removes `ResyncCodebase`, it should update all of these seams together:
- proto oneof and message definition,
- `RemoteServerClient::resync_codebase`,
- `RemoteServerOperation::ResyncCodebase`,
- `RemoteCodebaseIndexMutation::Resync`,
- `RemoteServerModel::handle_resync_codebase`,
- client round-trip tests.
The daemon `handle_index_codebase` behavior would then need to explicitly call `try_manual_resync_codebase` for already-indexed repos.
## Mirroring local patterns in remote indexing
Remote indexing should feel like the local feature even though the implementation is split across processes. The strongest local patterns to preserve are:
- A single indexing manager owns per-repo index lifecycle, watcher integration, snapshot validation, sync state, retrieval state, and drop/resync behavior.
- Startup restores persisted codebase metadata first, then queues valid persisted indices through the same manager path as fresh indices.
- Snapshot corruption or incompatibility invalidates the snapshot and falls back to rebuild rather than leaving a repo permanently broken.
- Ready and stale states keep search available through the last ready root; queued and indexing states suppress duplicate indexing requests.
- Settings and speedbump UI decide when to index, retry, resync, or drop; the index manager executes those decisions.
- Manual retry/resync is separate from passive navigation. Navigating into a known repo should update active context, not force a rebuild.
This PR mirrors those patterns by reusing `CodebaseIndexManager`, injecting daemon snapshot storage instead of adding a remote-only manager, feeding daemon-restored metadata into the same persisted build queue, and using `RemoteCodebaseIndexModel` as the client-side analog of local availability state. The remote model records active repo context, applies daemon status snapshots, filters agent context to ready searchable repos, and avoids duplicate auto-index requests when a repo is already ready, stale, queued, or indexing.
The places where remote intentionally differs from local are the process boundary and persistence scope:
- Local can call manager methods directly; remote must encode user actions as proto messages.
- Local can consume full app state because it is the app; the daemon reads the same persisted shape but consumes only codebase-index data because it is a headless worker.
- Local snapshot storage uses the app default; the daemon injects a remote-server data root.
- Local UI state and daemon indexing state synchronize through explicit status snapshots and deltas rather than shared memory.
Future remote improvements should continue to ask “what is the closest local pattern?” before adding remote-specific behavior. Examples:
- If local treats retry as a manual resync of a known repo, remote should either keep a clearly named `ResyncCodebase` message or make `IndexCodebase` explicitly perform that same known-repo resync behavior.
- If local keeps stale search available, remote status pushes should include the last ready root hash for stale states so agent search can continue.
- If local snapshot validation removes invalid persisted metadata, daemon startup should remove invalid daemon metadata and push disabled/failed status deltas rather than silently dropping client state.
- If local settings are the source of truth for user intent, remote daemon handlers should execute explicit client decisions and avoid inventing new daemon-only enablement policy.
## Error handling and security
The daemon should report persistence startup failures through the existing SQLite error reporting and telemetry path, then degrade by starting without restored metadata rather than crashing the app path.
Remote indexing requests that can cause daemon-to-Warp-service calls carry `auth_token` in the protocol payload. The token must never be logged or persisted. The daemon uses it for request-scoped outbound auth and should reject missing or invalid credentials rather than treating the daemon's cached token as ambient authority for proxy-socket writers.
Status values sent to the client should avoid exposing implementation details beyond what the client needs: repo path, lifecycle state, progress, failure message, root hash when search is ready/stale, and embedding config when root hash is present.
## Testing strategy
Unit coverage should focus on the client model and protocol seams:
- snapshot application replaces host-scoped statuses and leaves other hosts untouched,
- incremental status updates update one repo,
- ready/stale statuses with usable root hashes are searchable,
- queued/indexing statuses suppress duplicate automatic indexing,
- failed/unavailable/missing-root statuses allow recovery indexing,
- agent context includes only ready searchable remote repos,
- remote client round-trip tests cover the current indexing/resync/drop request messages.
Persistence coverage should verify that the SQLite reader restores both app state and codebase index metadata through the full `PersistedData` payload. Startup-level coverage should verify that daemon launch consumes only codebase index metadata from that payload and passes daemon-scoped snapshot storage into `CodebaseIndexManager::new_with_snapshot_storage`.
Targeted validation for this PR should include the remote-server client tests, `RemoteCodebaseIndexModel` tests, and a compile check for the app/remote-server crates touched by the persistence and proto changes. Full PR validation should still use the repository presubmit expectations before pushing.
## Risks and mitigations
### Daemon accidentally consumes app state
Consuming full app data in the daemon would couple daemon startup to UI/session state and app-only singleton assumptions. Mitigate by reading the shared `PersistedData` shape, then making the `LaunchMode::RemoteServerDaemon` branch in `initialize_app` preserve only `persisted_workspaces` and default app-only restored fields before model initialization.
### Duplicate indexing on navigation
Remote navigation events can fire often. Without the status guard, entering a known repo could repeatedly enqueue indexing. Mitigate with `should_request_auto_index_for_navigated_git_repo` and tests for ready, stale, queued, indexing, failed, and missing states.
### Client and daemon status drift
If the client misses daemon state during reconnect, tool availability and settings rows can be stale. Mitigate by pushing `CodebaseIndexStatusesSnapshot` after initialize/reconnect and using incremental updates only after that bootstrap.
### Protocol complexity
Keeping both `IndexCodebase` and `ResyncCodebase` makes the remote protocol larger than the minimal idempotent shape. Mitigate by documenting the local-resync parity rationale now and treating removal/folding as a follow-up protocol cleanup rather than mixing it into this persistence PR.
## Definition of done
Remote daemon startup reads the shared persisted data shape but restores only known codebase index metadata into daemon startup state.
The daemon uses daemon-scoped snapshot storage through `CodebaseIndexManager::new_with_snapshot_storage`.
The client receives a bootstrap status snapshot and applies subsequent status deltas.
Remote auto-indexing does not re-request indexing for ready/stale/queued/indexing repos on navigation.
Ready remote repos can appear in agent codebase context.
The PR tech spec documents the persistence architecture and the `IndexCodebase` versus `ResyncCodebase` protocol tradeoff without removing `ResyncCodebase`.