first pass of merging in warp (doesn't build)
This commit is contained in:
Ryan Ward
@@ -0,0 +1,123 @@
|
||||
# gh-10207 — File tree falls back to lazy loading on `ExceededMaxFileLimit`
|
||||
|
||||
## Context
|
||||
|
||||
[Issue #10207](https://github.com/warpdotdev/warp/issues/10207): Project Explorer
|
||||
silently shows populated folders (e.g. `.agents`) as empty when a repo exceeds
|
||||
`MAX_FILES_PER_REPO` (100,000) during initial indexing. The reporter has
|
||||
~153k tracked files in `~/code` and sees `Failed to build file tree for
|
||||
repository: ExceededMaxFileLimit` in the log.
|
||||
|
||||
Root cause:
|
||||
|
||||
- `crates/repo_metadata/src/local_model.rs:892` runs `Entry::build_tree` with
|
||||
`MAX_TREE_DEPTH=200` and a 100k-file global quota
|
||||
(`crates/repo_metadata/src/local_model.rs:60`).
|
||||
- On `ExceededMaxFileLimit` (`crates/repo_metadata/src/entry.rs:14-28`,
|
||||
raised in `entry.rs:157-220`), the spawn callback marks the repo
|
||||
`IndexedRepoState::Failed`. The view (`app/src/code/file_tree/view.rs:1584-1586`)
|
||||
swaps a `Failed` state for an empty `FileTreeEntry`.
|
||||
- Lazy per-directory expansion already exists (`Entry::load`,
|
||||
`entry.rs:257-284`, with `LAZY_LOAD_FILE_LIMIT=5000`) but is unreachable
|
||||
when the root is `Failed`.
|
||||
|
||||
Scope agreed with @moirahuang and @alokedesai
|
||||
([comment](https://github.com/warpdotdev/warp/pull/10490#issuecomment-4423492563)):
|
||||
**make the file tree use lazy loading when indexing hits the maximum, and
|
||||
otherwise leave the user experience unchanged.** No toast, no indicator —
|
||||
the visible result should be "the folder expands" rather than "the folder
|
||||
expands plus a banner." A broader "always-lazy file tree" exploration is
|
||||
tracked internally as a follow-up.
|
||||
|
||||
Out of scope: repo-local skill discovery
|
||||
(`app/src/ai/skills/file_watchers/`) silently drops skills in degraded
|
||||
mode because it queries the metadata tree, and `ai::project_context::model`
|
||||
(`crates/ai/src/project_context/model.rs:298`), `warp::ai::outline::native`,
|
||||
and `ai::index::full_source_code_embedding` each call `Entry::build_tree`
|
||||
independently and hit the same hard limit. Per @moirahuang
|
||||
([comment](https://github.com/warpdotdev/warp/pull/10490#issuecomment-4427103160)),
|
||||
those surfaces will be addressed holistically in follow-up work; this
|
||||
spec is strictly scoped to the file tree.
|
||||
|
||||
## Proposed changes
|
||||
|
||||
### File tree fallback (load-bearing)
|
||||
|
||||
In the `index_repository` spawn callback (`local_model.rs:892-1030`), on
|
||||
`Err(BuildTreeError::ExceededMaxFileLimit)`, retry `Entry::build_tree` once
|
||||
with:
|
||||
|
||||
- `max_depth = 1` so the root is loaded with unloaded subdirectory entries
|
||||
(`entry.rs:142-150`).
|
||||
- `remaining_file_quota = None`. Direct-child files at depth=1 consume the
|
||||
quota (`entry.rs:214-220`), so reusing `MAX_FILES_PER_REPO` would
|
||||
re-trigger `ExceededMaxFileLimit` on the rare repo with >100k files
|
||||
*directly* under the root and reproduce the empty-tree bug. Cost is
|
||||
bounded by root-entry count since subdirectories return as unloaded
|
||||
immediately at depth=1; only top-level files allocate `FileMetadata`.
|
||||
|
||||
On retry success, install the repo as `IndexedRepoState::Indexed` via the
|
||||
existing `add_repository_internal`, which emits the usual
|
||||
`RepositoryUpdated` event. The view's existing handlers refresh the tree;
|
||||
the existing per-directory lazy-load path (`view.rs:1410-1431`,
|
||||
`LAZY_LOAD_FILE_LIMIT=5000`) handles expansion from there.
|
||||
|
||||
On retry failure (e.g. `IOError` reading the root directory), keep the
|
||||
existing `mark_repository_failed` path. This is the same outcome users
|
||||
already get for unreadable repo roots and is outside the scope of #10207.
|
||||
|
||||
No new event, no new state on `FileTreeState`, no UI plumbing in the view.
|
||||
|
||||
## Testing and validation
|
||||
|
||||
- Unit (regression for the original empty-tree bug): drive
|
||||
`Entry::build_tree` past `MAX_FILES_PER_REPO` (parameterized via a
|
||||
test-only constructor on the model or by lowering the constant under
|
||||
`#[cfg(test)]`) and assert the resulting `IndexedRepoState::Indexed`
|
||||
with the root containing depth-1 unloaded subdirectory entries. Assert
|
||||
`RepositoryUpdated` fires and `UpdatingRepositoryFailed` does not.
|
||||
- Unit (regression for the depth-1 quota gap, requires the `None` quota
|
||||
fix): fixture with `MAX_FILES_PER_REPO + 1` files placed *directly under
|
||||
the repo root*. Without the unquoted retry the fallback reproduces the
|
||||
original bug. With `remaining_file_quota = None` the test asserts
|
||||
`Indexed` with all top-level files present.
|
||||
- Manual: use the existing fixture at
|
||||
`~/code-fixtures/warp-10207-large-repo` (150,001 files). Build with
|
||||
`./script/run --dont-open` and `open -a target/debug/bundle/osx/WarpOss.app
|
||||
~/code-fixtures/warp-10207-large-repo`. Verify:
|
||||
1. `~/Library/Logs/warp-oss.log` contains the "indexed in degraded
|
||||
mode" warn line (or whatever the implementation logs — non-empty,
|
||||
non-error).
|
||||
2. `.agents`, `src`, etc. expand and show their contents.
|
||||
3. **No toast, no banner.** UI should look like any other repo.
|
||||
|
||||
## Risks and mitigations
|
||||
|
||||
- **Watcher pressure on 153k-file repos.** The recursive `register_path`
|
||||
call already happens today for any indexed repo; this PR does not
|
||||
change the watcher footprint. If users report watcher CPU/memory
|
||||
issues, a follow-up could prune the watch to actually-loaded subtrees.
|
||||
- **Silent degradation for non-file-tree surfaces.** The team's explicit
|
||||
position is that the file tree experience should not change, so there
|
||||
is no UI signal here that the repo is in degraded mode. Repo-local
|
||||
skill discovery, project rules, outline, and codebase indexing each
|
||||
hit the same limit on their own and will go silent or partially
|
||||
silent in degraded mode. They are explicitly out of scope for this
|
||||
spec and will be addressed holistically in follow-up work.
|
||||
|
||||
## Follow-ups
|
||||
|
||||
- Always-lazy file tree (tracked internally by @moirahuang /
|
||||
@alokedesai): if the file tree is moved to lazy loading by default,
|
||||
the special-case fallback in this spec collapses into the default
|
||||
path and `MAX_FILES_PER_REPO` can be dropped from the file-tree
|
||||
pipeline.
|
||||
- Telemetry: a counter on the depth-1 fallback so we can see how often
|
||||
it fires in the wild. The existing
|
||||
`RepoMetadataTelemetryEvent::BuildTreeFailed { error: "ExceededMaxFileLimit" }`
|
||||
at `local_model.rs:1001` already covers the trigger side; pair it with
|
||||
a success counter on the fallback retry.
|
||||
- Holistic handling of degraded-mode behavior for repo-local skill
|
||||
discovery, `ai::project_context::model`, `ai::outline::native`, and
|
||||
`ai::index::full_source_code_embedding`. Tracked separately by
|
||||
@moirahuang.
|
||||
Reference in New Issue
Block a user