Initial public release of Warp.
Repo-Sync-Origin: warpdotdev/warp-internal@12af1d983b
This commit is contained in:
David Stern
@@ -0,0 +1,83 @@
|
||||
# Integration Test Recording Overlays
|
||||
|
||||
## Summary
|
||||
Add an API to the integration test video recording framework that, when enabled, renders visual event metadata in an overlay layer within the recorded video. The goal is to make recorded test videos self-explanatory so a viewer can understand which mouse and keyboard actions were fired during the test without reading the test source.
|
||||
|
||||
## Problem
|
||||
Current integration test recordings show the UI state changing over time, but they do not clearly communicate which input events caused those changes. This makes recordings harder to use for debugging, regression review, demos, and collaboration because a viewer has to infer whether a state transition came from a click, a drag, a keypress, or some combination of inputs.
|
||||
|
||||
## Goals
|
||||
- Make integration test recordings easier to interpret without needing external narration or source-code context.
|
||||
- Show the user inputs that occurred at the moment they occurred in the recording.
|
||||
- Follow familiar conventions from other screen recording and presentation tools that use animated annotations for clicks, drags, and keyboard shortcuts.
|
||||
- Keep the feature opt-in through an API on the video recording framework.
|
||||
|
||||
## Non-Goals
|
||||
- Changing how tests dispatch events.
|
||||
- Building a full analytics or event-inspection UI outside the recorded video.
|
||||
- Capturing every possible event type in the first version beyond clicks, click-and-drag, and keyboard events.
|
||||
|
||||
## Primary Use Cases
|
||||
- A developer watches a failed integration test video and can immediately see which click triggered the incorrect UI response.
|
||||
- A reviewer watches a video attached to a change and can understand a drag gesture or keyboard shortcut without pausing to inspect test code.
|
||||
- A developer records a repro video for a UI bug and wants the visible annotations to explain the interaction sequence clearly.
|
||||
|
||||
## Required Recorded Events
|
||||
|
||||
### Clicks
|
||||
When a click occurs, the video should show a transient animated annotation at the pointer location. This should follow the common pattern used by screen recording apps, such as a pulse, ring, or highlight that makes the click easy to notice without obscuring the UI underneath.
|
||||
|
||||
### Click and Drag
|
||||
When a click-and-drag gesture occurs, the video should show:
|
||||
- The drag start location
|
||||
- The drag path or motion between points
|
||||
- The drag end state
|
||||
|
||||
The visualization should make it obvious that the pointer movement was part of a drag gesture rather than ordinary cursor motion.
|
||||
|
||||
### Keyboard Events
|
||||
When keyboard input occurs, the video should show the keys being fired in a compact overlay. The annotation must include:
|
||||
- Modifier keys, if any
|
||||
- The primary key being fired
|
||||
|
||||
Examples include combinations such as `Cmd+K`, `Shift+Enter`, or `Ctrl+C`, as well as non-modified keys when they are relevant.
|
||||
|
||||
## Product Requirements
|
||||
- The overlay feature must be enabled through an explicit API in the integration test video recording framework.
|
||||
- When disabled, recording behavior should remain unchanged.
|
||||
- Overlay rendering should be synchronized with the recorded event timeline so annotations appear at the correct moment in the video.
|
||||
- Overlays should be visually understandable at normal playback speed without requiring frame-by-frame inspection.
|
||||
- Animations should be clear but lightweight, avoiding excessive distraction or covering important parts of the UI for too long.
|
||||
- The system should support multiple keyboard events in sequence and render them in a way that remains readable.
|
||||
- Mouse and keyboard overlays should be visually consistent so the recording feels like a single coherent annotation system.
|
||||
|
||||
## Proposed Experience
|
||||
|
||||
### Mouse Annotations
|
||||
- Clicks use a brief animated pulse or ring centered on the pointer.
|
||||
- Drag gestures use a visible start indicator and a short-lived path or trail that communicates direction and distance.
|
||||
- The styling should feel similar to established screen recording tools that visualize cursor interactions for demos and tutorials.
|
||||
|
||||
### Keyboard Annotations
|
||||
- Key events appear as a compact on-screen overlay, likely near the bottom of the video or another consistent location that does not interfere with important UI.
|
||||
- Modifier keys and the main key should be shown together as a single composed event.
|
||||
- The overlay should be readable, transient, and consistent across repeated actions.
|
||||
|
||||
## API Expectations
|
||||
The recording framework should expose an API that enables these annotations for a given test recording session. The API should be simple enough that a test author can opt into annotated videos without changing how the test itself expresses input events.
|
||||
|
||||
At a minimum, the API should:
|
||||
- Turn overlay capture on for a recording
|
||||
- Observe or receive input events already emitted by the test framework
|
||||
- Render those events into a visual overlay layer in the final video output
|
||||
|
||||
## Design Principles
|
||||
- Prefer familiar annotation patterns over novel visuals.
|
||||
- Optimize for clarity to someone watching the video for the first time.
|
||||
- Keep the overlay declarative and easy to enable in tests.
|
||||
- Ensure the feature improves debugging value without materially changing test authoring ergonomics.
|
||||
|
||||
## Success Criteria
|
||||
- A viewer can correctly identify when a click, drag, or keyboard action occurred by watching the recording alone.
|
||||
- Keyboard shortcuts are understandable from the video, including modifier keys.
|
||||
- The overlay feels polished and familiar, matching expectations set by other screen recording applications with animated interaction annotations.
|
||||
@@ -0,0 +1,277 @@
|
||||
# Integration Test Recording Overlays Technical Spec
|
||||
|
||||
## Summary
|
||||
This spec describes the implementation of the integration test recording overlay feature described in `specs/APP-3872/PRODUCT.md`.
|
||||
|
||||
The current branch already contains the core implementation for annotated integration test recordings. This document treats that implementation as the starting point, but makes the product spec the source of truth. Where the branch behavior diverged from the intended product behavior, the implementation has been adjusted and this document reflects the corrected design.
|
||||
|
||||
## Goals
|
||||
- Provide an opt-in API for annotated integration test video recordings.
|
||||
- Capture mouse and keyboard event metadata during a recording session.
|
||||
- Composite those annotations into the final recorded video rather than requiring a separate viewer.
|
||||
- Keep the feature understandable and ergonomic for test authors.
|
||||
|
||||
## Non-Goals
|
||||
- Building a generic event-inspection UI outside the generated recording artifacts.
|
||||
- Changing how integration tests express events today.
|
||||
- Supporting every input event type in the first version beyond click, click-and-drag, and keyboard events.
|
||||
|
||||
## Current Branch Foundation
|
||||
The branch already implements the main pieces needed for this feature:
|
||||
- A `VideoRecorder` abstraction for start/stop recording, frame capture, screenshot export, and MP4 finalization
|
||||
- An `OverlayLog` that stores timestamped visual annotation events
|
||||
- Integration-step hooks that observe mouse and keyboard input as tests dispatch events
|
||||
- Final compositing that draws overlays onto recorded frames at encode time
|
||||
- A manual end-to-end exercise test in `integration/src/test/video_recording.rs`
|
||||
|
||||
This tech spec keeps that architecture and refines behavior where needed to better match the product spec.
|
||||
|
||||
## API Surface
|
||||
|
||||
### Test Author API
|
||||
The recording API is exposed through `TestStep` helpers:
|
||||
- `with_start_recording()`
|
||||
- `with_stop_recording()`
|
||||
- `with_take_screenshot(...)`
|
||||
|
||||
This means annotated recording is opt-in at the test level. A test author does not need to change how clicks, drags, or keystrokes are authored once recording has started.
|
||||
|
||||
### Environment-Based Debugging API
|
||||
The existing environment variable `WARP_INTEGRATION_TEST_VIDEO` remains available as a convenience override for automatic recording during test runs. This is useful for debugging, but the primary product-facing API remains the explicit step-based recording controls.
|
||||
|
||||
## High-Level Architecture
|
||||
|
||||
### 1. Event capture in the integration test runner
|
||||
Integration events are intercepted in the step runner before they are dispatched into the app. Relevant mouse and keyboard events are translated into overlay events and appended to an in-memory `OverlayLog`.
|
||||
|
||||
The recorded overlay event types are:
|
||||
- `MouseDown`
|
||||
- `MouseMove`
|
||||
- `MouseUp`
|
||||
- `KeyPress`
|
||||
|
||||
Keyboard events are derived from parsed `Keystroke` values and rendered using a display formatter that includes modifier keys and the main key in a single visual token.
|
||||
|
||||
Important behavior:
|
||||
- Overlay events are recorded only while the `VideoRecorder` is actively recording.
|
||||
- This prevents pre-recording or post-recording interactions from leaking into the visual annotation timeline.
|
||||
|
||||
This active-recording gating is an intentional correction to the earlier branch behavior.
|
||||
|
||||
### 2. Frame capture loop
|
||||
When integration tests are running with the `integration_tests` feature enabled, the driver starts a foreground capture loop backed by the existing window frame-capture path. The loop:
|
||||
- Sleeps when recording is off
|
||||
- Requests a frame capture while recording is on
|
||||
- Stores captured frames together with wall-clock timestamps
|
||||
|
||||
This keeps recording overhead effectively limited to periods where recording is enabled.
|
||||
|
||||
### 3. Final video compositing
|
||||
At finalization time, captured frames are encoded into MP4. Before each frame is handed to the encoder, the overlay renderer advances an `OverlayState` to the frame timestamp and draws any active annotations onto the frame buffer.
|
||||
|
||||
This approach has a few benefits:
|
||||
- It avoids mutating the live app UI just to show recording-only annotations.
|
||||
- It keeps the overlay logic isolated from the product UI.
|
||||
- It allows overlay rendering to be deterministic from the captured timeline.
|
||||
|
||||
If MP4 encoding fails, the implementation falls back to writing PNG frames.
|
||||
|
||||
### 4. How the overlay pixels are actually produced
|
||||
The implementation does not ask WarpUI to render a second overlay scene and it does not add overlays after MP4 encoding. Instead, the compositor mutates each captured frame's RGBA pixel buffer in memory before the frame is converted to RGB/YUV and handed to OpenH264.
|
||||
|
||||
The concrete flow is:
|
||||
- `VideoRecorder::finalize(...)` drains the captured `TimestampedFrame` list
|
||||
- `encode_to_mp4(...)` clones the frame's RGBA byte buffer when overlay events are present
|
||||
- `OverlayState::advance_to(...)` applies all overlay events whose timestamps are at or before the current frame time
|
||||
- `OverlayState::render_onto(...)` rasterizes the active click, drag, and keyboard visuals directly into that RGBA buffer
|
||||
- the composited RGBA data is converted to RGB, then YUV, and then encoded into the output MP4
|
||||
|
||||
This means the overlays are effectively burned into the frame pixels before encoding. The original capture remains unchanged except for the pixels touched by the overlay primitives.
|
||||
|
||||
### 5. Software rasterization model
|
||||
Overlay rendering is implemented as a small software rasterizer in `ui/src/integration/overlay.rs`.
|
||||
|
||||
The renderer works on a flat RGBA byte slice and uses alpha blending per destination pixel:
|
||||
- `blend_pixel(...)` mixes overlay color into the destination image
|
||||
- `draw_filled_circle(...)` is used for the held-mouse indicator and drag anchor
|
||||
- `draw_click_ring(...)` paints the animated post-click ring by filling only the annulus
|
||||
- `draw_thick_line(...)` stamps circular samples along each drag segment
|
||||
- `draw_key_overlay(...)` paints the rounded pill background for keyboard events
|
||||
- `draw_text(...)` writes glyph pixels directly into the same RGBA buffer
|
||||
|
||||
There is no separate retained overlay surface. We simply replace some destination pixels with alpha-blended overlay colors.
|
||||
|
||||
### 6. Coordinate scaling
|
||||
Overlay events are recorded in logical coordinates, while captured frames are stored in device pixels. The driver records the window backing scale factor into `OverlayLog`, and the compositor multiplies event coordinates by that scale factor before drawing circles, lines, and key pills. This is how the overlay positions line up with the captured image content.
|
||||
|
||||
## Rendering Model
|
||||
|
||||
### Clicks
|
||||
Clicks are represented by:
|
||||
- A filled pointer indicator while the mouse button is held
|
||||
- An animated expanding ring after mouse up
|
||||
|
||||
This matches the familiar click pulse pattern used by screen recording and demo tools.
|
||||
|
||||
### Click and drag
|
||||
Drag gestures are represented by:
|
||||
- A visible drag start anchor
|
||||
- A trail connecting the recorded drag points
|
||||
- A visible end-state ring on mouse up
|
||||
- A live pointer indicator while the drag is in progress
|
||||
|
||||
The earlier branch implementation already had a drag trail and end-state ring, but it did not emphasize the drag start strongly enough. The implementation has been adjusted to render an explicit drag-start anchor so the gesture is easier to read in playback.
|
||||
|
||||
### Keyboard events
|
||||
Keyboard events are rendered as transient pill overlays near the bottom of the frame. Each pill contains:
|
||||
- Any modifier keys involved in the event
|
||||
- The main key being fired
|
||||
|
||||
Examples:
|
||||
- `⌘C`
|
||||
- `⌃A`
|
||||
- `⇧Enter`
|
||||
|
||||
To better match the product requirement that rapid keyboard sequences remain readable, the implementation now renders multiple recent key events as a small stack instead of showing only the most recent keypress. This is another intentional correction relative to the earlier branch behavior.
|
||||
|
||||
### Keyboard glyph rendering details
|
||||
The current implementation does not use system fonts or vector glyph rasterization for keyboard overlays. Instead, it uses:
|
||||
- an embedded 8x16 bitmap font table for printable ASCII characters
|
||||
- handwritten 8x16 bitmap glyphs for modifier and special-key symbols such as `⌘`, `⌃`, `⌥`, `⇧`, `↩`, `⇥`, and the arrow keys
|
||||
- integer upscaling (`FONT_SCALE`) to enlarge those bitmaps inside the pill
|
||||
|
||||
`keystroke_display_text(...)` still formats the keyboard shortcut text as the expected Unicode symbols, but `draw_text(...)` does not shape or rasterize them with a font engine. `get_glyph(...)` maps each character to either the ASCII bitmap table or one of the custom symbol bitmaps and then writes the glyph pixels directly into the frame buffer.
|
||||
|
||||
This is why the modifier keys appear correctly today, but also why the resulting text can look pixelated in videos.
|
||||
|
||||
To improve readability, the current implementation now prefers plain-text labels for ambiguous non-modifier special keys such as `Enter`, `Tab`, `Backspace`, and `Delete` instead of rendering those as symbolic glyphs. Modifier keys still use symbolic forms such as `⌘`, `⌃`, `⌥`, and `⇧`.
|
||||
|
||||
### Improving glyph quality
|
||||
Yes, we can improve the key and modifier glyph quality by switching to higher-resolution source bitmaps. That is the lowest-risk follow-up if the current 8x16 glyphs scaled by `FONT_SCALE` look too blocky in the final MP4.
|
||||
|
||||
The simplest version of that change would be:
|
||||
- replace the 8x16 ASCII and modifier-symbol bitmaps with a larger source set such as 16x32 or 24x48
|
||||
- either reduce the amount of integer upscaling or keep a smaller scale factor
|
||||
- keep the same direct-to-buffer rendering model and alpha blending pipeline
|
||||
|
||||
This would preserve the current deterministic, cross-platform rendering behavior while making the keyboard overlays look cleaner. It is a much smaller change than introducing a full font rasterization stack.
|
||||
|
||||
## Event-to-Overlay Mapping
|
||||
|
||||
### Mouse mapping
|
||||
- `LeftMouseDown` and the other supported mouse-down variants create `MouseDown`
|
||||
- `LeftMouseDragged` creates `MouseMove` in the overlay timeline
|
||||
- `LeftMouseUp` creates `MouseUp`
|
||||
- Saved-position click helpers also emit matching synthetic overlay events so helper-driven interactions look the same as hand-authored events in recordings
|
||||
|
||||
### Keyboard mapping
|
||||
- `KeyDown` events are converted to `KeyPress`
|
||||
- The display text is derived from the parsed `Keystroke`, not the raw typed character stream
|
||||
- This ensures modifier state is preserved in the overlay output
|
||||
|
||||
## Timing Model
|
||||
All overlay events and captured frames use wall-clock timestamps from the test process.
|
||||
|
||||
During encoding:
|
||||
- Each captured frame advances overlay state up to that frame timestamp
|
||||
- Expiring visual effects fade out according to fixed durations
|
||||
- Gaps between captured frames are normalized into repeated output frames at the target FPS
|
||||
|
||||
This means annotations are synchronized to the recorded interaction timeline rather than to test-step boundaries.
|
||||
|
||||
## Data Flow
|
||||
1. The driver creates a `VideoRecorder`, `ActionLog`, `OverlayLog`, and artifacts directory for the test run.
|
||||
2. A test step enables recording.
|
||||
3. The capture loop begins accumulating timestamped frames while the recorder is active.
|
||||
4. Integration events are dispatched as normal.
|
||||
5. Relevant events are mirrored into the `OverlayLog`.
|
||||
6. Finalization encodes the captured frames and composites overlay annotations frame-by-frame.
|
||||
7. Artifacts are written, including:
|
||||
- `recording.mp4`
|
||||
- screenshots requested by the test
|
||||
- `recording.log`
|
||||
|
||||
## Artifacts
|
||||
The implementation produces:
|
||||
- Annotated video output
|
||||
- Requested screenshots
|
||||
- A plain-text action log for timestamped event auditing
|
||||
|
||||
The action log is useful for debugging and complements the overlay video, but it is not the primary user-facing product surface.
|
||||
|
||||
## Implementation Details by Module
|
||||
|
||||
### `ui/src/integration/driver.rs`
|
||||
Responsible for:
|
||||
- Creating per-test recording state
|
||||
- Starting the frame capture loop
|
||||
- Initializing the overlay scale factor from the active window backing scale
|
||||
- Handling screenshot capture after steps
|
||||
- Finalizing the recording and writing artifacts
|
||||
|
||||
### `ui/src/integration/step.rs`
|
||||
Responsible for:
|
||||
- Exposing the step-level recording API
|
||||
- Intercepting integration events
|
||||
- Translating supported input events into overlay events
|
||||
- Ensuring overlay events are only recorded while recording is active
|
||||
|
||||
### `ui/src/integration/overlay.rs`
|
||||
Responsible for:
|
||||
- Defining overlay event types
|
||||
- Converting keystrokes into display strings
|
||||
- Tracking transient overlay state across the event timeline
|
||||
- Rendering click, drag, and keyboard overlays into RGBA frame buffers via direct pixel mutation and alpha blending
|
||||
|
||||
### `ui/src/integration/video_recorder.rs`
|
||||
Responsible for:
|
||||
- Managing recording lifecycle
|
||||
- Running the timestamped frame capture loop
|
||||
- Encoding MP4 output
|
||||
- Invoking overlay compositing during finalization
|
||||
- Falling back to PNG frames if video encoding fails
|
||||
|
||||
### `integration/src/test/video_recording.rs`
|
||||
Acts as the manual end-to-end validation flow for:
|
||||
- screenshots
|
||||
- start/stop recording
|
||||
- click overlays
|
||||
- drag overlays
|
||||
- keyboard overlays
|
||||
|
||||
## Decisions and Corrections
|
||||
|
||||
### Keep overlays composited at encode time
|
||||
We should continue rendering overlays into captured frames during finalization instead of drawing them live in the app. This keeps the production UI untouched and keeps the feature scoped to integration recording.
|
||||
|
||||
### Keep recording opt-in
|
||||
The explicit start/stop recording API matches the product spec and keeps the feature ergonomic for tests that only want overlays for a narrow slice of execution.
|
||||
|
||||
### Correct overlay collection to honor recording state
|
||||
Overlay collection must be tied to active recording. This avoids stale annotations from appearing in the output video and better matches the product requirement that overlays appear when the feature is enabled.
|
||||
|
||||
### Render short keyboard history, not just the latest key
|
||||
Showing a short stack of recent key events better matches the product goal of making recordings understandable at normal playback speed, especially for shortcuts or rapid key sequences.
|
||||
|
||||
### Make drag start visible
|
||||
The drag path alone is not always enough to communicate where a gesture began. A dedicated start anchor makes the recording easier to interpret.
|
||||
|
||||
### Prefer higher-resolution bitmaps before introducing real font rendering
|
||||
If the current keyboard overlays are too pixelated, the best next step is to increase the source bitmap resolution rather than immediately moving to system fonts. Higher-resolution bitmaps would improve visual quality without adding font discovery, shaping, rasterization libraries, or platform-specific output differences to the integration test pipeline.
|
||||
|
||||
## Validation
|
||||
Implementation was validated with:
|
||||
- `cargo check -p warpui_core --features integration_tests --manifest-path /Users/zach/Projects/warp_5/Cargo.toml`
|
||||
- `cargo check -p integration --manifest-path /Users/zach/Projects/warp_5/Cargo.toml`
|
||||
|
||||
For manual validation, the existing `integration/src/test/video_recording.rs` flow should be used to inspect:
|
||||
- click visibility
|
||||
- drag readability
|
||||
- keyboard modifier rendering
|
||||
- stacked key annotations during rapid sequences
|
||||
|
||||
## Future Extensions
|
||||
- Add richer styling configuration if test authors need different overlay density or durations
|
||||
- Add support for more pointer event types if product requirements expand
|
||||
- Replace the current 8x16 keyboard bitmaps with a higher-resolution glyph set to reduce visible pixelation in encoded videos
|
||||
- Optionally export structured overlay metadata alongside the rendered video if downstream tooling needs it
|
||||
Reference in New Issue
Block a user