306 lines
16 KiB
Markdown
306 lines
16 KiB
Markdown
# Remote Server File Tree Protocol — Tech Spec
|
|
|
|
Linear: [APP-3788](https://linear.app/warpdotdev/issue/APP-3788)
|
|
|
|
## Problem
|
|
|
|
The remote server binary (`crates/remote_server`) currently only handles `Initialize`/`InitializeResponse`. We need to:
|
|
1. Boot repo metadata models on the server so it can index directories and keep file trees up to date
|
|
2. Let the client tell the server which directories to index (via `NavigatedToDirectory`)
|
|
3. Let the client fetch the initial tree and receive subsequent incremental updates as push messages
|
|
|
|
## Current State
|
|
|
|
### Remote server (`crates/remote_server`)
|
|
- `ServerModel` singleton handles stdin/stdout protobuf I/O
|
|
- `run()` boots a headless warpui app with only `ServerModel`
|
|
- Proto schema has only `Initialize`/`InitializeResponse`
|
|
|
|
### repo_metadata crate
|
|
- `LocalRepoMetadataModel` — indexes repos, subscribes to `DetectedRepositories` for auto-indexing, has `emit_incremental_updates: bool` field and emits `IncrementalUpdateReady` when enabled
|
|
- `DetectedRepositories` singleton — runs async git detection via `detect_possible_git_repo()`, emits `DetectedGitRepo` events. Uses `DirectoryWatcher` to register watch directories.
|
|
- `DirectoryWatcher` singleton — manages filesystem watchers and routes changes to `Repository` subscribers via a `TaskQueue`
|
|
- `LocalRepoMetadataModel` also supports lazy-loaded non-git directories via `index_lazy_loaded_path()` (first-level-only tree, `loaded: false` on subdirectories)
|
|
- The incremental update types (`RepoMetadataUpdate`, `FileTreeEntryUpdate`, etc.) already exist in `file_tree_update.rs`
|
|
|
|
### Key insight on two separate watchers
|
|
`DirectoryWatcher` and `LocalRepoMetadataModel` each own their own `BulkFilesystemWatcher`. `DirectoryWatcher`'s watcher feeds the `Repository` model (git status, etc.), while `LocalRepoMetadataModel`'s watcher feeds the file tree. Both need to be running on the server.
|
|
|
|
## Proposed Changes
|
|
|
|
### 1. Proto schema additions (`remote_server.proto`)
|
|
|
|
Names and fields mirror the Rust types in `repo_metadata/src/file_tree_update.rs` 1:1 for trivial conversion.
|
|
|
|
```proto
|
|
// ── Shared file tree sub-messages ─────────────────────────────────
|
|
// Mirror the Rust types in repo_metadata/src/file_tree_update.rs.
|
|
|
|
message RepoNodeMetadata {
|
|
oneof node {
|
|
DirectoryNodeMetadata directory = 1;
|
|
FileNodeMetadata file = 2;
|
|
}
|
|
}
|
|
|
|
message DirectoryNodeMetadata {
|
|
string path = 1;
|
|
bool ignored = 2;
|
|
bool loaded = 3;
|
|
}
|
|
|
|
message FileNodeMetadata {
|
|
string path = 1;
|
|
optional string extension = 2;
|
|
bool ignored = 3;
|
|
}
|
|
|
|
// Mirrors FileTreeEntryUpdate in Rust.
|
|
message FileTreeEntryUpdate {
|
|
string parent_path_to_replace = 1;
|
|
repeated RepoNodeMetadata subtree_metadata = 2;
|
|
}
|
|
|
|
// ── Client → server ───────────────────────────────────────────────
|
|
|
|
// "I navigated to this directory, please index it."
|
|
message NavigatedToDirectory {
|
|
string path = 1;
|
|
}
|
|
|
|
// Response after the server has run git detection on the requested path.
|
|
//
|
|
// - is_git = true: A git repo was found. indexed_path is the repo root.
|
|
// Full indexing runs in the background; the client should
|
|
// wait for RepositoryIndexedPush before calling FetchFileTree.
|
|
// - is_git = false: No git repo. The directory was lazily indexed at first
|
|
// level. indexed_path is the standardized input path.
|
|
// The client can call FetchFileTree immediately.
|
|
message NavigatedToDirectoryResponse {
|
|
string indexed_path = 1;
|
|
bool is_git = 2;
|
|
}
|
|
|
|
// "Give me the current tree for this repo."
|
|
message FetchFileTree {
|
|
string repo_path = 1;
|
|
}
|
|
|
|
// Sent as one or more responses for the same request_id.
|
|
// Client accumulates entries until sync_complete = true.
|
|
message FetchFileTreeResponse {
|
|
string repo_path = 1;
|
|
repeated FileTreeEntryUpdate entries = 2;
|
|
bool sync_complete = 3;
|
|
}
|
|
|
|
// ── Server → client push (empty request_id) ───────────────────────
|
|
|
|
// Mirrors RepoMetadataUpdate in Rust.
|
|
message FileTreeUpdatePush {
|
|
string repo_path = 1;
|
|
repeated string remove_entries = 2;
|
|
repeated FileTreeEntryUpdate update_entries = 3;
|
|
}
|
|
|
|
// A repository finished full indexing and is ready for FetchFileTree.
|
|
message RepositoryIndexedPush {
|
|
string repo_path = 1;
|
|
}
|
|
```
|
|
|
|
Updated envelopes:
|
|
|
|
```proto
|
|
message ClientMessage {
|
|
string request_id = 1;
|
|
oneof message {
|
|
Initialize initialize = 2;
|
|
NavigatedToDirectory navigated_to_directory = 3;
|
|
FetchFileTree fetch_file_tree = 4;
|
|
}
|
|
}
|
|
|
|
message ServerMessage {
|
|
string request_id = 1;
|
|
oneof message {
|
|
InitializeResponse initialize_response = 2;
|
|
ErrorResponse error = 3;
|
|
NavigatedToDirectoryResponse navigated_to_directory_response = 4;
|
|
FetchFileTreeResponse fetch_file_tree_response = 5;
|
|
FileTreeUpdatePush file_tree_update = 6;
|
|
RepositoryIndexedPush repository_indexed = 7;
|
|
}
|
|
}
|
|
```
|
|
|
|
Push messages use an empty `request_id` to distinguish them from request/response pairs.
|
|
|
|
### 2. Server-side model bootstrap
|
|
|
|
Update `remote_server::run()` to register the repo metadata singletons:
|
|
|
|
```rust
|
|
AppBuilder::new_headless(...).run(|ctx| {
|
|
ctx.add_singleton_model(DirectoryWatcher::new);
|
|
ctx.add_singleton_model(DetectedRepositories::default_entity);
|
|
ctx.add_singleton_model(|ctx| {
|
|
let mut model = LocalRepoMetadataModel::new(ctx);
|
|
model.set_emit_incremental_updates(true);
|
|
model
|
|
});
|
|
ctx.add_singleton_model(ServerModel::new);
|
|
});
|
|
```
|
|
|
|
This automatically wires up the existing `DetectedRepositories` → `LocalRepoMetadataModel` subscription and the watcher → `LocalRepoMetadataModel` update pipeline.
|
|
|
|
New API needed: `LocalRepoMetadataModel::set_emit_incremental_updates(&mut self, enabled: bool)` (or a builder-style constructor parameter).
|
|
|
|
### 3. Server-side message handling
|
|
|
|
#### `NavigatedToDirectory`
|
|
|
|
When the server receives `NavigatedToDirectory { path }`:
|
|
|
|
1. Await `detect_possible_git_repo(path)` — this checks the in-memory cache first (instant if already known), otherwise walks up the directory tree checking for `.git` (fast filesystem metadata, not full indexing)
|
|
2. If a git repo was found (`Some(git_root)`):
|
|
- Full indexing was already triggered by the `DetectedGitRepo` → `LocalRepoMetadataModel` subscription inside `detect_possible_git_repo`
|
|
- Respond with `{ indexed_path: git_root, is_git: true }`
|
|
- Client waits for `RepositoryIndexedPush` before calling `FetchFileTree`
|
|
3. If no git repo (`None`):
|
|
- Call `index_lazy_loaded_path(path)` for first-level-only data
|
|
- Respond with `{ indexed_path: standardized_path, is_git: false }`
|
|
- Client can call `FetchFileTree` immediately
|
|
|
|
#### `FetchFileTree`
|
|
|
|
When the server receives `FetchFileTree { repo_path }`:
|
|
|
|
1. Look up the repository in `LocalRepoMetadataModel` via `get_repository(&repo_path)`
|
|
2. If `Indexed`: serialize the full `FileTreeEntry` as one or more `FetchFileTreeResponse` chunks (see section on streaming pagination below)
|
|
3. If `Pending`: return `ErrorResponse` — the client retries after receiving `RepositoryIndexedPush`
|
|
4. If `Failed` or not found: return `ErrorResponse`
|
|
|
|
Serialization: Walk the `FileTreeEntry`'s `state_map` and `parent_to_child_map` to produce `FileTreeEntryUpdate` entries. This is the same shape as `RepoMetadataUpdate` but for the full tree.
|
|
|
|
#### Incremental update push
|
|
|
|
The `ServerModel` subscribes to `LocalRepoMetadataModel`'s `IncrementalUpdateReady` events. On receiving the event:
|
|
|
|
1. Convert the `RepoMetadataUpdate` to `FileTreeUpdatePush` proto
|
|
2. Send as a `ServerMessage` with empty `request_id`
|
|
|
|
### 4. Conversion layer: Rust types ↔ Proto
|
|
|
|
Add a new module `crates/remote_server/src/file_tree_proto.rs` with:
|
|
|
|
- `RepoMetadataUpdate` → `FileTreeUpdatePush` proto
|
|
- `FileTreeEntry` → `FetchFileTreeResponse` proto (full tree serialization with chunking)
|
|
- Proto `FetchFileTreeResponse` / `FileTreeUpdatePush` → `RepoMetadataUpdate` for client-side application
|
|
|
|
These conversions are straightforward because the Rust types in `file_tree_update.rs` were designed to mirror the proto schema 1:1.
|
|
|
|
### 5. Client-side changes
|
|
|
|
#### `RemoteServerClient` additions
|
|
|
|
Add methods to the client:
|
|
|
|
- `navigate_to_directory(&self, path: String) -> Result<NavigatedToDirectoryResponse>`
|
|
- `fetch_file_tree(&self, repo_path: String) -> Result<FetchFileTreeResponse>` (accumulates chunked responses)
|
|
- Handle push messages (`FileTreeUpdatePush`, `RepositoryIndexedPush`) in the client's reader loop and emit them as client events
|
|
|
|
#### `RemoteServerClient` event handling
|
|
|
|
The client's reader task receives `ServerMessage`s. For push messages (empty `request_id`), route to event emission instead of completing a pending request:
|
|
|
|
```rust
|
|
RemoteServerClientEvent::FileTreeUpdated { update: RepoMetadataUpdate }
|
|
RemoteServerClientEvent::RepositoryIndexed { repo_path: String }
|
|
```
|
|
|
|
The downstream consumer (future file tree view integration) subscribes to these events and calls `RemoteRepoMetadataModel::apply_incremental_update()` for `FileTreeUpdated`, and `RemoteRepoMetadataModel::insert_repository()` for the initial tree after calling `fetch_file_tree`.
|
|
|
|
## Design Decisions
|
|
|
|
### 1. NavigatedToDirectory: await git detection, then branch
|
|
|
|
The local `FileTreeView::update_directory_contents` (`view.rs:703`) uses a two-pronged approach: check for a git repo first, fall back to lazy-loading if none is found. The remote server mirrors this but runs git detection synchronously within the request handling so the client gets a definitive answer in one round trip:
|
|
|
|
1. Server awaits `detect_possible_git_repo(path)` — checks in-memory cache first (instant for known repos), otherwise walks up the directory tree (fast filesystem metadata checks, not full indexing)
|
|
2. If git repo found: respond with `{ indexed_path: git_root, is_git: true }`. Full indexing was already triggered by `DetectedGitRepo` → `LocalRepoMetadataModel`. Client waits for `RepositoryIndexedPush` before `FetchFileTree`.
|
|
3. If no git repo: server calls `index_lazy_loaded_path(path)` for first-level data, responds with `{ indexed_path: path, is_git: false }`. Client calls `FetchFileTree` immediately.
|
|
|
|
This avoids the unnecessary eager lazy-load for git repos (which would be thrown away when full indexing completes) and gives the client clear instructions in a single response.
|
|
|
|
### 2. Initial tree fetch: server-controlled streaming pagination
|
|
|
|
Pagination is controlled by the server based on actual response size, not tree depth (a flat repo could have huge amounts of data at each level). The protocol:
|
|
|
|
1. Server serializes the tree top-to-bottom (breadth-first or depth-first pre-order)
|
|
2. Each `FetchFileTreeResponse` chunk contains a batch of entries plus a `bool sync_complete` flag
|
|
3. The server segments by a target byte budget per chunk (e.g. 256KB)
|
|
4. The client renders progressively as chunks arrive, and knows the full tree is loaded when `sync_complete = true`
|
|
|
|
Multiple `FetchFileTreeResponse` messages are sent for the same `request_id`. The client accumulates them and applies each chunk to the `RemoteRepoMetadataModel` as it arrives.
|
|
|
|
## End-to-End Flow
|
|
|
|
### Case A: Directory is inside a git repo
|
|
|
|
```
|
|
Client Server
|
|
│ │
|
|
User navigates to │ NavigatedToDirectory { path } │
|
|
/home/user/project/src │ ────────────────────────────────────────> │
|
|
│ │── await detect_possible_git_repo(path)
|
|
│ │ → found git root /home/user/project
|
|
│ │ (full indexing triggered in bg)
|
|
│ Response { indexed_path: .../project, │
|
|
│ is_git: true } │
|
|
│ <──────────────────────────────────────── │
|
|
│ │
|
|
Client waits... │ ... full repo indexing completes ... │
|
|
│ │
|
|
│ RepositoryIndexedPush { repo_path } │
|
|
│ <──────────────────────────────────────── │
|
|
│ │
|
|
Now fetch full tree │ FetchFileTree { repo_path } │
|
|
│ ────────────────────────────────────────> │
|
|
│ │
|
|
│ FetchFileTreeResponse { ... true } │ ← full tree (chunked)
|
|
│ <──────────────────────────────────────── │
|
|
│ │
|
|
│ ... file watcher detects changes ... │
|
|
│ │
|
|
│ FileTreeUpdatePush { incremental } │ ← push, empty request_id
|
|
│ <──────────────────────────────────────── │
|
|
```
|
|
|
|
### Case B: Directory is NOT a git repo
|
|
|
|
```
|
|
Client Server
|
|
│ │
|
|
User navigates to │ NavigatedToDirectory { path } │
|
|
/tmp/some-dir │ ────────────────────────────────────────> │
|
|
│ │── await detect_possible_git_repo → None
|
|
│ │── index_lazy_loaded_path(path)
|
|
│ Response { indexed_path: /tmp/some-dir, │
|
|
│ is_git: false } │
|
|
│ <──────────────────────────────────────── │
|
|
│ │
|
|
Fetch immediately │ FetchFileTree { repo_path } │
|
|
│ ────────────────────────────────────────> │
|
|
│ │
|
|
│ FetchFileTreeResponse { ... true } │ ← first-level tree
|
|
│ <──────────────────────────────────────── │
|
|
```
|
|
|
|
## Follow-ups (out of scope)
|
|
|
|
- Wire the client events to `RemoteRepoMetadataModel` and `FileTreeView`
|
|
- `LoadDirectory` request for expanding collapsed directories over the network
|
|
- Subscription management (unsubscribe from updates when file tree is closed)
|