public/galaxy
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Initial public release of Warp.

Browse Source 
Repo-Sync-Origin: warpdotdev/warp-internal@12af1d983b
This commit is contained in:
David Stern
2026-04-28 08:43:33 -05:00
commit 0dbd3d567a
4982 changed files with 1431549 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
specs/REMOTE-1393/PRODUCT.md
+82
View File
@@ -0,0 +1,82 @@
# Product Spec: `--jq` filter flag for `oz run get` and `oz run list`
Linear: [REMOTE-1393](https://linear.app/warpdotdev/issue/REMOTE-1393)
Figma: none (CLI-only change)
## Summary
Add a `--jq "<filter>"` flag to `oz run get` and `oz run list`. When set, the command runs the server's JSON response through the given jq filter (using the `jaq` Rust implementation) and writes the filter's result to stdout instead of printing the raw response.
## Problem
[REMOTE-1374](https://linear.app/warpdotdev/issue/REMOTE-1374) landed `--output-format json` on both commands and deliberately deferred first-class JQ-style filtering as a follow-up. In practice, the scripting pattern it unlocks still requires an external `jq` binary on the user's `PATH`:
```
oz run list --state failed --output-format json | jq -r '.runs[].task_id'
```
That's fine for interactive use but awkward in CI, sandboxes, or remote environments where `jq` is not guaranteed to be installed, and it adds a process boundary that complicates error handling and quoting. Bringing filtering into the CLI itself makes one-off inspection commands easier and keeps scripts portable.
## Behavior
1. Both `oz run list` and `oz run get` accept an optional `--jq <FILTER>` flag. `<FILTER>` is a jq filter expression in the dialect implemented by [`jaq`](https://github.com/01mf02/jaq) with the full standard library enabled (via `jaq-all`). When not set, both commands behave exactly as they do today.
2. When `--jq` is set, the input to the filter is the same JSON document the command would emit under `--output-format json`:
- For `oz run list`, the full list response envelope, e.g. `{ "runs": [...], "page_info": { ... } }`.
- For `oz run get <run_id>`, the single run object returned by `GET /api/v1/agent/runs/:runId`.
3. `--jq` and `--output-format` interact as follows:
- `--jq` implies JSON mode: whatever `--output-format` is set to (including the default `pretty`), the JSON fetch path is used to obtain the response, and the filter's output is what is printed. The `pretty`/`text` table renderers are skipped entirely.
- `--output-format json` together with `--jq` is accepted and equivalent.
- `--output-format text` together with `--jq` is accepted and equivalent (table output is skipped; only the filter output is printed).
4. Output format of the filter's result:
- Each filter output value is printed on its own line, terminated by `\n`.
- **Scalar values are printed as raw text**, not JSON-encoded. Specifically: strings are printed without surrounding double quotes and without JSON escapes; numbers are printed in their natural form; `true`, `false`, and `null` are printed as the literal words `true`, `false`, and `null`. This matches `gh --jq`'s behavior ([cli/cli#3012](https://github.com/cli/cli/pull/3012)) and means `oz run list --jq '.runs[].task_id'` prints one bare task ID per line, not one JSON-quoted string per line.
- Non-scalar values (arrays and objects) are pretty-printed as JSON across multiple lines, consistent with the non-filtered `--output-format json` behavior.
- `--jq .` on either command therefore produces byte-identical output to the same command with `--output-format json` (the top-level response is always an object; objects go through the JSON path).
- If the filter produces zero output values (for example, `empty`), the command prints nothing and exits `0`.
5. Filter errors are surfaced clearly:
- A filter that fails to parse (syntax error, unknown function, etc.) causes the command to exit with a non-zero status and prints a human-readable error to stderr that includes the filter text and, where `jaq` provides it, a location within the filter. Parse errors fire _before_ any HTTP request is issued, so a typo in `--jq` never triggers a network call or consumes credits.
- A runtime error during filter execution (for example, `.foo | .bar` on a number) causes the command to exit with a non-zero status and prints the `jaq` runtime error message to stderr. Any output values already produced before the error are still written to stdout.
6. `--jq` does not change which HTTP requests the CLI makes: on success, exactly one `GET /api/v1/agent/runs` (or `.../runs/:runId`) request, identical to the request issued by `--output-format json` without `--jq`. On a filter parse error, zero requests are made (see invariant 5). The filter is applied entirely client-side on the parsed response.
7. `--jq` does not change server-side filtering behavior. Existing `oz run list` filter flags (`--state`, `--source`, etc.) compose with `--jq`: the server narrows the response first, and the jq filter runs on the narrowed response.
8. `--jq` is documented in `oz run list --help` and `oz run get --help`. The help text:
- Names the flag (`--jq <FILTER>`).
- Explains that it runs the command's JSON response through a jq filter.
- Notes that `--jq` implies JSON output and that the filter dialect is jq-compatible via `jaq`.
- Gives at least one example invocation.
9. Invariants that must not regress:
- `oz run list` and `oz run get` with no `--jq` produce byte-identical output to today for each `--output-format`.
- The `--output-format json` payload shape is unchanged — the filter reads the existing response, not a new one.
- `--jq ""` (empty string) is rejected as an invalid filter via the same error path as any other parse error; it does not silently pass input through. Users wanting pass-through should omit `--jq` or use `--jq .`.
- Scalar unwrapping applies only to the _top-level_ filter output. Scalars that appear _inside_ an array or object are JSON-encoded as usual (so `--jq '.runs'` quotes string fields like `title` inside the returned array, but `--jq '.runs[].title'` does not).
## Non-goals
- No explicit `-r`/`--raw-output` flag. Scalar unwrapping (invariant 4) covers the common `jq -r` use case; users who want string values to be JSON-quoted at the top level can wrap the filter, e.g. `--jq '. | tojson'`.
- No `--jq-from-file`, variables (`--arg`), or multi-input streaming. The scope is a single inline filter applied to the single JSON response.
- No `--template` / Go-template alternative ([gh formatting docs](https://cli.github.com/manual/gh_help_formatting)). Separate ask if we ever want it.
- No TTY-aware colorization or compact-when-piped output ([cli/cli#7236](https://github.com/cli/cli/pull/7236)). Output is always pretty-printed for non-scalars, same as `--output-format json` today.
- No `--jq` on other commands (e.g. `oz run conversation get`, `oz environment list`). The implementation is structured so that adding `--jq` to any command that already uses `print_raw_json` is a small, mechanical follow-up, but those rollouts are out of scope here.
- No first-class `jq`-style error codes (jq reserves specific exit codes for "no output" / "false/null output"). The CLI uses its normal success/failure exit codes.
## Example invocations
```
# Pull just the task IDs of failed runs from the last day.
oz run list --state failed --updated-after 2026-04-20T00:00:00Z \
--jq '.runs[].task_id'
# Get a single field from a run.
oz run get 01HX... --jq '.state'
# Compose with other flags; --jq runs after server-side filters.
oz run list --source CLI --limit 100 \
--jq '[.runs[] | select(.harness == "claude")] | length'
```
specs/REMOTE-1393/TECH.md
+253
View File
@@ -0,0 +1,253 @@
# Tech Spec: `--jq` filter flag for `oz run get` and `oz run list`
See `specs/REMOTE-1393/PRODUCT.md` for the product spec.
This builds on top of REMOTE-1374. The `--output-format json` wiring and the `print_raw_json` helper already exist; this change adds a filter-processing layer in front of `print_raw_json` and a pair of clap flags.
Two guiding constraints beyond the product spec:
- `--jq` is a pattern we want on any command whose default JSON path goes through `print_raw_json`. The clap wiring lives in a single reusable struct (`JsonFilter`) embedded via `#[command(flatten)]`, so adding `--jq` to a new command is a one-line change.
- Filter compilation happens at clap parse time, not after the API response lands. Typos fail fast, before auth or HTTP activity. This is achieved with a custom `value_parser` that runs `jaq_core::Compiler::compile` during argument parsing.
## Context
The JSON output path for `oz run list` and `oz run get` is already in place. The relevant code after REMOTE-1374:
- `crates/warp_cli/src/task.rs:101` — `ListTasksArgs` (all existing filter flags).
- `crates/warp_cli/src/task.rs:281` — `TaskGetArgs`.
- `app/src/ai/agent_sdk/ambient.rs:566` — `AmbientAgentRunner::list_tasks`, which branches on `OutputFormat` and calls `print_raw_json` for `Json`.
- `app/src/ai/agent_sdk/ambient.rs:593` — `AmbientAgentRunner::get_task_status`, same pattern for the single-run response.
- `app/src/ai/agent_sdk/output.rs:80` — `print_raw_json(value: &serde_json::Value) -> anyhow::Result<()>`. The only shared JSON emitter; extending it is the user's stated preference and the lowest-footprint place to plug in a filter.
- `app/Cargo.toml:171` — `serde_json.workspace = true`. The workspace already enables `serde_json` with `features = ["raw_value"]`; we'll continue to use `serde_json::Value` for the filter input.
The `jaq` crates (MIT licensed, pure-Rust) provide a jq-compatible implementation. We depend on the three relevant crates directly and skip `jaq-all` / `jaq-fmts`, since `jaq-core` 3.0 provides everything we need:
- `jaq-core` 3.0 — parser, compiler, and interpreter. Exposes `Filter`, `Compiler`, `Ctx`, `defs()`, `funs()`, and the `data::JustLut<V>` `DataT` helper that lets us skip writing a custom `Data` / `DataKind`. The crate-level example in the `jaq-core` docs is exactly the compile+run path we use.
- `jaq-std` 3.0 — the jq standard library (`length`, `map`, `select`, etc.) as additional `defs()` and `funs()` to chain into the compiler.
- `jaq-json` 2.0 — the `Val` type used across the jaq ecosystem. With `features = ["serde"]` enabled, `Val` implements `serde::Deserialize`/`Serialize`, which is the path the user called out for `serde_json` compatibility (see `https://github.com/01mf02/jaq/blob/main/jaq-json/tests/common/mod.rs`, where `from_value::<Val>` converts a `serde_json::Value` to `Val`).
All three are `no_std`-friendly so they compile cleanly in the same configurations `output.rs` already compiles in.
## Proposed changes
Five small edits: add the dependency, introduce a reusable `JsonFilter` clap component that compiles the filter at parse time, embed it in both `Args` structs, extend `print_raw_json`, and thread the filter through the two call sites.
### 1. Dependencies
In `Cargo.toml` (workspace), add under `[workspace.dependencies]`:
```toml path=null start=null
jaq-core = "3.0"
jaq-std = "3.0"
jaq-json = { version = "2.0", features = ["serde"] }
```
In both `app/Cargo.toml` and `crates/warp_cli/Cargo.toml`, add:
```toml path=null start=null
jaq-core.workspace = true
jaq-std.workspace = true
jaq-json.workspace = true
```
`warp_cli` compiles the filter at clap parse time; `app` runs it. `warp_cli` technically only uses `jaq-core` + `jaq-std` items in its own code, but depends on `jaq-json` too because the stored `Filter` type is parameterized by `jaq_json::Val` — without `Val` in scope, the `Filter` field would not resolve. The alternative (store `Option<String>` in `warp_cli` and recompile in `app`) doubles the compile work and prevents sharing a single compile source of truth; keep the direct `jaq-json` dep and forgo that split unless the binary-size hit shows up as a problem.
### 2. Reusable `JsonFilter` clap component (`crates/warp_cli/src/json_filter.rs`, new)
Introduce a small module with a `JsonFilter` struct that can be flattened into any command's `Args`. The stored value is the compiled `Filter` itself — no wrapper struct, no `Arc`, no `source()` accessor. `jaq_core::Filter` is already `Clone`, so cloning `JsonFilter` is cheap and lets it cross async boundaries.
```rust path=null start=null
use clap::Args;
use jaq_core::{
data::JustLut,
load::{Arena, File, Loader},
Compiler, Filter, Native,
};
use jaq_json::Val;
/// A compiled jaq filter parameterized by `jaq_json::Val`. This is the type
/// produced by `parse_jq_filter` and stored on `JsonFilter`.
pub type JqFilter = Filter<Native<JustLut<Val>>>;
/// CLI argument bundle that parses a `--jq <FILTER>` flag into a pre-compiled
/// jaq filter. Embed with `#[command(flatten)]` on any command whose default
/// JSON output path goes through `print_raw_json`.
#[derive(Debug, Clone, Default, Args)]
pub struct JsonFilter {
/// jq filter applied to the command's JSON response. When set, implies
/// JSON output and prints the filter result instead of the raw response.
/// Uses the jaq implementation of jq (see https://github.com/01mf02/jaq).
/// Top-level scalar outputs are printed as raw text (no JSON quoting);
/// arrays and objects are pretty-printed as JSON.
#[arg(long = "jq", value_parser = parse_jq_filter, value_name = "FILTER")]
pub filter: Option<JqFilter>,
}
fn parse_jq_filter(src: &str) -> Result<JqFilter, String> {
let arena = Arena::default();
let defs = jaq_core::defs()
.chain(jaq_std::defs())
.chain(jaq_json::defs());
let funs = jaq_core::funs::<Val>()
.chain(jaq_std::funs())
.chain(jaq_json::funs());
let loader = Loader::new(defs);
let modules = loader
.load(&arena, File { path: (), code: src })
.map_err(|errs| format!("invalid jq filter {src:?}: {errs:?}"))?;
Compiler::default()
.with_funs(funs)
.compile(modules)
.map_err(|errs| format!("invalid jq filter {src:?}: {errs:?}"))
}
```
Expose it from `crates/warp_cli/src/lib.rs`:
```rust path=null start=null
pub mod json_filter;
```
The exact spellings of the type alias and `funs::<Val>()` call may need small adjustments once we compile against `jaq-core` 3.0 locally (the crate-level example in the `jaq-core` docs is the canonical reference); the overall shape is stable.
### 3. Flatten `JsonFilter` into both commands (`crates/warp_cli/src/task.rs`)
Replace the per-command flag with a single flattened field:
```rust path=null start=null
use crate::json_filter::JsonFilter;
#[derive(Debug, Clone, Args)]
pub struct ListTasksArgs {
// ... existing fields ...
#[command(flatten)]
pub json_filter: JsonFilter,
}
#[derive(Debug, Clone, Args)]
pub struct TaskGetArgs {
// ... existing fields ...
#[command(flatten)]
pub json_filter: JsonFilter,
}
```
Because `JsonFilter` is a single-field `Args` struct with `long = "jq"`, both commands expose `--jq <FILTER>` in help output without needing per-command duplication of the doc comment. Adding `--jq` to a third command later (e.g. `oz run conversation get`) is a single `#[command(flatten)] pub json_filter: JsonFilter,` line.
### 4. Filter plumbing in `print_raw_json` (`app/src/ai/agent_sdk/output.rs`)
Extend `print_raw_json` to accept an optional pre-compiled filter. When `None`, behavior is identical to today. When `Some(filter)`, run the filter against the input value and print each output on its own line.
```rust path=null start=null
use warp_cli::json_filter::JqFilter;
pub fn print_raw_json(
value: &serde_json::Value,
jq_filter: Option<&JqFilter>,
) -> anyhow::Result<()> {
ensure_stdout_blocking();
let stdout = std::io::stdout();
let mut out = stdout.lock();
match jq_filter {
None => write_json_pretty(value, &mut out)?,
Some(filter) => run_jq_filter(value, filter, &mut out)?,
}
out.flush()?;
Ok(())
}
```
The `run_jq_filter` helper lives next to `print_raw_json` in the same file. Because the filter is already compiled, it:
1. Converts the input `serde_json::Value` to `jaq_json::Val` with `serde_json::from_value::<Val>(value.clone())`. The `.clone()` is acceptable — responses are bounded by the server's page size and this keeps the API ergonomic.
2. Constructs a `Ctx` over the filter's `lut` (`Ctx::<JustLut<Val>>::new(&filter.lut, Vars::new([]))`, matching the `jaq_core` crate-level example) and calls `filter.id.run((ctx, input)).map(unwrap_valr)`. For each yielded value:
- Round-trip the `jaq_json::Val` to `serde_json::Value` via `serde_json::to_value(val)?`. The `serde` feature on `jaq-json` provides `Val: Serialize`, so this is a direct conversion.
- If the resulting `serde_json::Value` is a scalar (`Null`, `Bool`, `Number`, or `String`), write its raw text form followed by `\n`:
- `Null` → `null`
- `Bool` → `true` or `false`
- `Number` → the number's `to_string()`
- `String` → the unescaped string content (matches `gh --jq` and `jq -r` semantics for scalars)
- Otherwise (`Array` or `Object`), write via `serde_json::to_writer_pretty(&mut out, &value)?` followed by `\n`. This keeps the existing pretty-printed, `serde_json`-driven formatting for structured output so that `--jq .` on either command's response stays byte-identical to `--output-format json` without `--jq`.
3. On runtime errors, flushes already-emitted output, writes the error to stderr via `eprintln!` (so the CLI's standard log-hint suffix isn't appended to user-authored filter errors), and returns a non-zero `anyhow::Error`.
Note that `run_jq_filter` never recompiles the filter — compilation only happens inside `warp_cli::json_filter::parse_jq_filter` at clap parse time. This is what makes the CLI fail fast on bad filters: an invalid `--jq` exits during `Args::from_env()` and never reaches the spawn path, auth refresh, or HTTP client.
### 5. Call-site wiring (`app/src/ai/agent_sdk/ambient.rs`)
Both call sites currently match on `OutputFormat` and only call `print_raw_json` in the `Json` arm. With `--jq`, the JSON path should be taken regardless of `--output-format`. Treat "JSON-or-jq" as a single branch:
```rust path=null start=null
let jq = args.json_filter.filter.as_ref();
if matches!(output_format, OutputFormat::Json) || jq.is_some() {
let response = ai_client.list_agent_runs_raw(limit, filter).await?;
super::output::print_raw_json(&response, jq)?;
} else {
let tasks = ai_client.list_ambient_agent_tasks(limit, filter).await?;
Self::print_tasks_table(&tasks);
}
```
The same shape applies to `get_task_status`. `args` must be threaded into both runner methods so `args.json_filter` is reachable — today `list_tasks` only takes `limit` and `filter`. Pass the full `ListTasksArgs`/`TaskGetArgs` (or a small borrowed struct) to the runner.
### 6. Error mapping
Parse failures surface via clap's standard error-reporting path: `parse_jq_filter` returns `Err(String)`, and clap prints `error: invalid value 'xxx' for '--jq <FILTER>': <our message>` to stderr and exits with clap's usage-error code. This happens inside `Args::from_env()` before the command is even dispatched — no auth, no HTTP, no spawn.
Runtime failures propagate as `anyhow::Error` through the existing `spawn_command` error-reporting path; the CLI prints them to stderr and exits non-zero. Their `Display` is the jaq runtime error message.
## Testing and validation
Unit tests on `JsonFilter` and `parse_jq_filter` in `crates/warp_cli/src/json_filter_tests.rs` (new):
- **Invariants 1, 5, 6 (fail-fast):** `parse_jq_filter(".foo")` returns `Ok`; `parse_jq_filter("@")` and `parse_jq_filter("")` return `Err` whose `Display` contains the filter source. This is the regression guard for fail-fast: if this test passes at the `parse_jq_filter` layer, clap's own invocation guarantees failure happens in `Args::from_env()`.
- **Invariants 1, 5 (end-to-end clap):** calling `Args::try_parse_from(["oz", "run", "list", "--jq", "@"])` returns `Err` with `clap::error::ErrorKind::ValueValidation`. Repeat for `oz run get ID --jq @`.
Unit tests on `print_raw_json` in `app/src/ai/agent_sdk/output_tests.rs` (existing file). These use `parse_jq_filter` to produce a `JqFilter` and then call a crate-private variant of `run_jq_filter` that writes to a `Vec<u8>` instead of stdout. Each test maps back to a numbered invariant in `PRODUCT.md`:
- **Invariants 1 and 9 (no regression):** `print_raw_json(value, None)` produces byte-identical output to the pre-change implementation for a representative response payload.
- **Invariants 2, 6, and 9 (identity filter):** `run_jq_filter(value, parse_jq_filter(".").unwrap(), out)` on a top-level object (the shape both endpoints return) produces the same bytes as `print_raw_json(value, None)`.
- **Invariant 4 (scalar unwrapping):** a filter that emits a string (`.runs[0].task_id`) prints the bare string (no surrounding quotes, no JSON escapes); a filter that emits a number (`.runs | length`) prints the bare number; `true`/`false`/`null` print as the literal words.
- **Invariant 4 (non-scalar output):** a filter that emits an object or array (`.runs[0]`, `.page_info`) prints pretty-printed JSON.
- **Invariant 4 (multiple outputs):** a filter that yields multiple values (`.runs[] | .task_id`) produces one raw line per value.
- **Invariant 9 (inner scalars stay JSON-encoded):** `.runs` on a payload with a string-valued `title` produces JSON output where `title` remains quoted. Scalar unwrapping applies only at the top level.
- **Invariant 4 (empty output):** `empty` produces zero bytes of stdout and returns `Ok(())`.
- **Invariant 5 (runtime error, partial output):** a filter that emits a valid value and then errors writes the valid value to `out` before returning `Err`.
Clap parsing tests in `crates/warp_cli/src/task_tests.rs`:
- Parsing `oz run list --jq ".foo"` populates `ListTasksArgs.json_filter.filter` with `Some(_)` (we don't assert deep structural equality on the compiled filter; the `parse_jq_filter`-level tests above already cover the compile happy path).
- Parsing `oz run get ID --jq ".foo"` populates `TaskGetArgs.json_filter.filter` with `Some(_)`.
Runner-level tests in `app/src/ai/agent_sdk/ambient_tests.rs` (existing file):
- With `--jq` set and `OutputFormat::Pretty`, the runner still calls `list_agent_runs_raw`/`get_agent_run_raw` (not the typed variants) and routes through `print_raw_json`. Assert with the existing mockall-generated `AIClient` mock — no new trait methods needed.
Manual validation against staging covering invariants 2, 3, 4, 5, 6, 7, and 8:
- `oz run list --jq '.runs | length'` and `oz run get <id> --jq '.state'` on a run that exists.
- `oz run list --source CLI --jq '.runs[].task_id'` (composition with existing filters, invariant 7).
- `oz run list --jq '.runs[].bogus | .x'` (runtime error, invariant 5).
- `oz run list --jq '@'` (parse error, invariants 5 and 6 — confirm via `-v`/traffic inspection that no HTTP request is made).
- `oz run list --jq empty` (empty output, invariant 4).
- `oz run list --help` and `oz run get --help` to confirm flag documentation per invariant 8.
Presubmit: `./script/presubmit` (fmt, clippy `-D warnings`, test suite). The touched crates are `warp_cli` (minor) and `warp` (app).
## Risks and mitigations
- **Dependency weight:** `jaq-core`, `jaq-std`, and `jaq-json` together pull in `num-bigint`, `indexmap`, `hifijson`, and a few smaller transitives. The binary-size delta is expected to be modest and similar to other CLI features. `warp_cli` gains all three deps since it compiles filters; `app` already transitively carries serde_json and adds the same set. We deliberately skip `jaq-all` and `jaq-fmts` — the only thing `jaq-all` adds on top of these three is multi-format conveniences we don't use.
- **Filter dialect drift:** `jaq` aims to be jq-compatible but is not byte-identical to BSD `jq` in every edge case. Mitigation: the product spec explicitly names `jaq` as the dialect, and the help text says so.
- **Scalar-unwrapping divergence from pure `jq`:** our top-level scalar unwrapping differs from `jq` (which always emits JSON-encoded values) but matches `gh --jq`. Users coming from `jq` may be surprised that top-level strings are unquoted. Mitigation: document this in the help text, and point to `| tojson` as the opt-out.
- **Partial output on runtime error:** we intentionally flush already-produced outputs before surfacing the error (invariant 5). This matches jq's behavior but means a failing filter can still leave valid JSON on stdout. Downstream scripts that parse stdout should continue to check the CLI exit code, which is unchanged. Documented in the help text is not necessary; the product spec captures it.
## Follow-ups
- Apply `--jq` to `oz run conversation get` (and `oz run get --conversation`). Mechanical once this lands — the conversation emitter also goes through JSON.
- Add `--jq` to any other command that already uses `print_raw_json` (currently only the two in this PR).
- TTY-aware colorization of non-scalar output ([cli/cli#7236](https://github.com/cli/cli/pull/7236)), ideally shared with the existing `--output-format json` path.
- `--template` as a Go-template (or handlebars) alternative to `--jq` ([gh formatting docs](https://cli.github.com/manual/gh_help_formatting)), if there's user demand.