galaxy/specs/APP-3788/TECH-file-tree-protocol.md

Remote Server File Tree Protocol — Tech Spec

Linear: APP-3788

Problem

The remote server binary (crates/remote_server) currently only handles Initialize/InitializeResponse. We need to:

1. Boot repo metadata models on the server so it can index directories and keep file trees up to date
2. Let the client tell the server which directories to index (via NavigatedToDirectory)
3. Let the client fetch the initial tree and receive subsequent incremental updates as push messages

Current State

Remote server (crates/remote_server)

• ServerModel singleton handles stdin/stdout protobuf I/O
• run() boots a headless warpui app with only ServerModel
• Proto schema has only Initialize/InitializeResponse

repo_metadata crate

• LocalRepoMetadataModel — indexes repos, subscribes to DetectedRepositories for auto-indexing, has emit_incremental_updates: bool field and emits IncrementalUpdateReady when enabled
• DetectedRepositories singleton — runs async git detection via detect_possible_git_repo(), emits DetectedGitRepo events. Uses DirectoryWatcher to register watch directories.
• DirectoryWatcher singleton — manages filesystem watchers and routes changes to Repository subscribers via a TaskQueue
• LocalRepoMetadataModel also supports lazy-loaded non-git directories via index_lazy_loaded_path() (first-level-only tree, loaded: false on subdirectories)
• The incremental update types (RepoMetadataUpdate, FileTreeEntryUpdate, etc.) already exist in file_tree_update.rs

Key insight on two separate watchers

DirectoryWatcher and LocalRepoMetadataModel each own their own BulkFilesystemWatcher. DirectoryWatcher's watcher feeds the Repository model (git status, etc.), while LocalRepoMetadataModel's watcher feeds the file tree. Both need to be running on the server.

Proposed Changes

1. Proto schema additions (remote_server.proto)

Names and fields mirror the Rust types in repo_metadata/src/file_tree_update.rs 1:1 for trivial conversion.

// ── Shared file tree sub-messages ─────────────────────────────────
// Mirror the Rust types in repo_metadata/src/file_tree_update.rs.

message RepoNodeMetadata {
  oneof node {
    DirectoryNodeMetadata directory = 1;
    FileNodeMetadata file = 2;
  }
}

message DirectoryNodeMetadata {
  string path = 1;
  bool ignored = 2;
  bool loaded = 3;
}

message FileNodeMetadata {
  string path = 1;
  optional string extension = 2;
  bool ignored = 3;
}

// Mirrors FileTreeEntryUpdate in Rust.
message FileTreeEntryUpdate {
  string parent_path_to_replace = 1;
  repeated RepoNodeMetadata subtree_metadata = 2;
}

// ── Client → server ───────────────────────────────────────────────

// "I navigated to this directory, please index it."
message NavigatedToDirectory {
  string path = 1;
}

// Response after the server has run git detection on the requested path.
//
// - is_git = true:  A git repo was found. indexed_path is the repo root.
//                   Full indexing runs in the background; the client should
//                   wait for RepositoryIndexedPush before calling FetchFileTree.
// - is_git = false: No git repo. The directory was lazily indexed at first
//                   level. indexed_path is the standardized input path.
//                   The client can call FetchFileTree immediately.
message NavigatedToDirectoryResponse {
  string indexed_path = 1;
  bool is_git = 2;
}

// "Give me the current tree for this repo."
message FetchFileTree {
  string repo_path = 1;
}

// Sent as one or more responses for the same request_id.
// Client accumulates entries until sync_complete = true.
message FetchFileTreeResponse {
  string repo_path = 1;
  repeated FileTreeEntryUpdate entries = 2;
  bool sync_complete = 3;
}

// ── Server → client push (empty request_id) ───────────────────────

// Mirrors RepoMetadataUpdate in Rust.
message FileTreeUpdatePush {
  string repo_path = 1;
  repeated string remove_entries = 2;
  repeated FileTreeEntryUpdate update_entries = 3;
}

// A repository finished full indexing and is ready for FetchFileTree.
message RepositoryIndexedPush {
  string repo_path = 1;
}

Updated envelopes:

message ClientMessage {
  string request_id = 1;
  oneof message {
    Initialize initialize = 2;
    NavigatedToDirectory navigated_to_directory = 3;
    FetchFileTree fetch_file_tree = 4;
  }
}

message ServerMessage {
  string request_id = 1;
  oneof message {
    InitializeResponse initialize_response = 2;
    ErrorResponse error = 3;
    NavigatedToDirectoryResponse navigated_to_directory_response = 4;
    FetchFileTreeResponse fetch_file_tree_response = 5;
    FileTreeUpdatePush file_tree_update = 6;
    RepositoryIndexedPush repository_indexed = 7;
  }
}

Push messages use an empty request_id to distinguish them from request/response pairs.

2. Server-side model bootstrap

Update remote_server::run() to register the repo metadata singletons:

AppBuilder::new_headless(...).run(|ctx| {
    ctx.add_singleton_model(DirectoryWatcher::new);
    ctx.add_singleton_model(DetectedRepositories::default_entity);
    ctx.add_singleton_model(|ctx| {
        let mut model = LocalRepoMetadataModel::new(ctx);
        model.set_emit_incremental_updates(true);
        model
    });
    ctx.add_singleton_model(ServerModel::new);
});

This automatically wires up the existing DetectedRepositories → LocalRepoMetadataModel subscription and the watcher → LocalRepoMetadataModel update pipeline.

New API needed: LocalRepoMetadataModel::set_emit_incremental_updates(&mut self, enabled: bool) (or a builder-style constructor parameter).

3. Server-side message handling

NavigatedToDirectory

When the server receives NavigatedToDirectory { path }:

1. Await detect_possible_git_repo(path) — this checks the in-memory cache first (instant if already known), otherwise walks up the directory tree checking for .git (fast filesystem metadata, not full indexing)
2. If a git repo was found (Some(git_root)):
   • Full indexing was already triggered by the DetectedGitRepo → LocalRepoMetadataModel subscription inside detect_possible_git_repo
   • Respond with { indexed_path: git_root, is_git: true }
   • Client waits for RepositoryIndexedPush before calling FetchFileTree
3. If no git repo (None):
   • Call index_lazy_loaded_path(path) for first-level-only data
   • Respond with { indexed_path: standardized_path, is_git: false }
   • Client can call FetchFileTree immediately

FetchFileTree

When the server receives FetchFileTree { repo_path }:

1. Look up the repository in LocalRepoMetadataModel via get_repository(&repo_path)
2. If Indexed: serialize the full FileTreeEntry as one or more FetchFileTreeResponse chunks (see section on streaming pagination below)
3. If Pending: return ErrorResponse — the client retries after receiving RepositoryIndexedPush
4. If Failed or not found: return ErrorResponse

Serialization: Walk the FileTreeEntry's state_map and parent_to_child_map to produce FileTreeEntryUpdate entries. This is the same shape as RepoMetadataUpdate but for the full tree.

Incremental update push

The ServerModel subscribes to LocalRepoMetadataModel's IncrementalUpdateReady events. On receiving the event:

1. Convert the RepoMetadataUpdate to FileTreeUpdatePush proto
2. Send as a ServerMessage with empty request_id

4. Conversion layer: Rust types ↔ Proto

Add a new module crates/remote_server/src/file_tree_proto.rs with:

• RepoMetadataUpdate → FileTreeUpdatePush proto
• FileTreeEntry → FetchFileTreeResponse proto (full tree serialization with chunking)
• Proto FetchFileTreeResponse / FileTreeUpdatePush → RepoMetadataUpdate for client-side application

These conversions are straightforward because the Rust types in file_tree_update.rs were designed to mirror the proto schema 1:1.

5. Client-side changes

RemoteServerClient additions

Add methods to the client:

• navigate_to_directory(&self, path: String) -> Result<NavigatedToDirectoryResponse>
• fetch_file_tree(&self, repo_path: String) -> Result<FetchFileTreeResponse> (accumulates chunked responses)
• Handle push messages (FileTreeUpdatePush, RepositoryIndexedPush) in the client's reader loop and emit them as client events

RemoteServerClient event handling

The client's reader task receives ServerMessages. For push messages (empty request_id), route to event emission instead of completing a pending request:

RemoteServerClientEvent::FileTreeUpdated { update: RepoMetadataUpdate }
RemoteServerClientEvent::RepositoryIndexed { repo_path: String }

The downstream consumer (future file tree view integration) subscribes to these events and calls RemoteRepoMetadataModel::apply_incremental_update() for FileTreeUpdated, and RemoteRepoMetadataModel::insert_repository() for the initial tree after calling fetch_file_tree.

Design Decisions

1. NavigatedToDirectory: await git detection, then branch

The local FileTreeView::update_directory_contents (view.rs:703) uses a two-pronged approach: check for a git repo first, fall back to lazy-loading if none is found. The remote server mirrors this but runs git detection synchronously within the request handling so the client gets a definitive answer in one round trip:

1. Server awaits detect_possible_git_repo(path) — checks in-memory cache first (instant for known repos), otherwise walks up the directory tree (fast filesystem metadata checks, not full indexing)
2. If git repo found: respond with { indexed_path: git_root, is_git: true }. Full indexing was already triggered by DetectedGitRepo → LocalRepoMetadataModel. Client waits for RepositoryIndexedPush before FetchFileTree.
3. If no git repo: server calls index_lazy_loaded_path(path) for first-level data, responds with { indexed_path: path, is_git: false }. Client calls FetchFileTree immediately.

This avoids the unnecessary eager lazy-load for git repos (which would be thrown away when full indexing completes) and gives the client clear instructions in a single response.

2. Initial tree fetch: server-controlled streaming pagination

Pagination is controlled by the server based on actual response size, not tree depth (a flat repo could have huge amounts of data at each level). The protocol:

1. Server serializes the tree top-to-bottom (breadth-first or depth-first pre-order)
2. Each FetchFileTreeResponse chunk contains a batch of entries plus a bool sync_complete flag
3. The server segments by a target byte budget per chunk (e.g. 256KB)
4. The client renders progressively as chunks arrive, and knows the full tree is loaded when sync_complete = true

Multiple FetchFileTreeResponse messages are sent for the same request_id. The client accumulates them and applies each chunk to the RemoteRepoMetadataModel as it arrives.

End-to-End Flow

Case A: Directory is inside a git repo

                        Client                                      Server
                          │                                           │
  User navigates to       │  NavigatedToDirectory { path }            │
  /home/user/project/src  │ ────────────────────────────────────────> │
                          │                                           │── await detect_possible_git_repo(path)
                          │                                           │   → found git root /home/user/project
                          │                                           │   (full indexing triggered in bg)
                          │  Response { indexed_path: .../project,     │
                          │             is_git: true }                 │
                          │ <──────────────────────────────────────── │
                          │                                           │
  Client waits...         │   ... full repo indexing completes ...     │
                          │                                           │
                          │  RepositoryIndexedPush { repo_path }      │
                          │ <──────────────────────────────────────── │
                          │                                           │
  Now fetch full tree     │  FetchFileTree { repo_path }              │
                          │ ────────────────────────────────────────> │
                          │                                           │
                          │  FetchFileTreeResponse { ... true }       │ ← full tree (chunked)
                          │ <──────────────────────────────────────── │
                          │                                           │
                          │     ... file watcher detects changes ...   │
                          │                                           │
                          │  FileTreeUpdatePush { incremental }       │ ← push, empty request_id
                          │ <──────────────────────────────────────── │

Case B: Directory is NOT a git repo

                        Client                                      Server
                          │                                           │
  User navigates to       │  NavigatedToDirectory { path }            │
  /tmp/some-dir           │ ────────────────────────────────────────> │
                          │                                           │── await detect_possible_git_repo → None
                          │                                           │── index_lazy_loaded_path(path)
                          │  Response { indexed_path: /tmp/some-dir,   │
                          │             is_git: false }                │
                          │ <──────────────────────────────────────── │
                          │                                           │
  Fetch immediately       │  FetchFileTree { repo_path }              │
                          │ ────────────────────────────────────────> │
                          │                                           │
                          │  FetchFileTreeResponse { ... true }       │ ← first-level tree
                          │ <──────────────────────────────────────── │

Follow-ups (out of scope)

• Wire the client events to RemoteRepoMetadataModel and FileTreeView
• LoadDirectory request for expanding collapsed directories over the network
• Subscription management (unsubscribe from updates when file tree is closed)