public/galaxy
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

Initial public release of Warp.

Browse Source 
Repo-Sync-Origin: warpdotdev/warp-internal@12af1d983b
This commit is contained in:
David Stern
2026-04-28 08:43:33 -05:00
commit 0dbd3d567a
4982 changed files with 1431549 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
specs/zachlloyd/inline-markdown-images-in-blocklist/PRODUCT.md
+258
View File
@@ -0,0 +1,258 @@
# Inline Markdown Images in AI Block List — Product Spec
Linear: none provided
Figma: none provided
## Summary
Render supported Markdown images inline inside AI block list responses instead of always showing raw Markdown image syntax. The first release should support local file-backed images referenced by relative or absolute filesystem paths, plus Mermaid diagrams rendered on the fly in the same response flow.
This feature should make AI responses easier to read when they include screenshots, diagrams, or visual outputs, while preserving reliable copy behavior without requiring a broader block-list selection refactor in the first release. The presentation should now follow three distinct treatments from the approved mock:
- horizontal inline image rows for multiple image references that appear in the same Markdown line
- a stacked “Images N” card for consecutive block-level image references
- a framed Mermaid block with a titled header and white diagram canvas
Successfully rendered file-backed images and Mermaid diagrams should both reuse Warp’s existing fullscreen lightbox treatment for click-to-expand viewing.
When an image cannot be resolved or the format is unsupported, the block list should continue to show the original Markdown text rather than rendering a broken or partial image UI.
## Problem
AI responses increasingly include Markdown image references and Mermaid diagrams. In the AI block list today, that content is more useful as source text than as rendered output, but it is much harder to read and scan when the user actually wants to view the visual result inline.
Warp already supports Markdown image blocks elsewhere in the app and already has Mermaid-to-SVG rendering for notebook Markdown. The AI block list currently lacks the equivalent presentation and interaction model. That creates three problems:
- visual content in AI responses is harder to understand because users see raw Markdown instead of the image or diagram
- rendered output is inconsistent with other Markdown-capable surfaces in Warp
- copy and selection behavior for mixed text-and-image content is undefined unless we specify it explicitly
## Goals
- Render supported local Markdown images inline inside AI block list responses.
- Support both absolute file paths and relative file paths.
- Resolve relative file paths relative to the current working directory of the session associated with the AI block.
- Support Mermaid diagrams in the same overall rollout, rendering them inline from their Markdown source.
- Ensure the feature works across Warp’s supported platforms, including WASM.
- Use the same supported image format set that Warp’s existing Markdown/image rendering stack already supports today: JPEG/JPG, PNG, GIF, WebP, and SVG.
- Show the referenced file link below each successfully rendered file-backed image in the AI block list.
- Preserve stable copy behavior for rendered images and Mermaid diagrams without introducing a new cross-renderer selection model in this first release.
- Allow right-click copy on a rendered image or Mermaid diagram, copying the underlying Markdown source rather than image bytes.
- Fall back cleanly to the raw Markdown source when the image cannot be found, cannot be resolved, or is not a supported format.
- Gate block list image rendering behind a dedicated block-list-specific feature flag.
- Respect the existing Mermaid rendering flag in addition to the new block list image flag, so Mermaid diagrams render in the AI block list only when both flags are enabled.
- Reuse Warp’s existing fullscreen lightbox treatment for both rendered file-backed images and rendered Mermaid diagrams.
## Non-Goals
- Supporting remote image URLs in this first release.
- Supporting Markdown data URLs or other inline-encoded image sources in this first release.
- Adding image editing, resizing controls, zoom controls, or image-specific toolbars.
- Copying image bytes to the clipboard.
- Adding drag-and-drop, save-image, or open-image actions.
- Supporting additional image formats beyond the set Warp already renders today.
- Shipping file-backed images first and Mermaid later; both are part of the same implementation sequence.
## User Experience
### Scope
This feature applies to AI block list responses that render Markdown content.
It should work for:
- newly streamed AI responses
- previously rendered responses reopened from history
- restored conversation views that display the same AI block content
- all supported client platforms, including WASM
### Supported sources
The first release should support two kinds of inline visual Markdown content:
1. Standard Markdown image references that point to local files.
2. Mermaid diagrams authored in Markdown that Warp renders into an image representation at runtime.
For file-backed images, the initial supported path types are:
- absolute filesystem paths
- relative filesystem paths
Relative paths resolve against the current working directory of the session associated with the AI block, not against a notebook file path or any repository root heuristic.
That working directory should be the one recorded with the AI block when the response was rendered. Relative file paths should be re-resolved against that recorded working directory when the block is later restored or re-rendered.
### Supported formats
For file-backed images, the AI block list should support the same image formats Warp’s existing shared image renderer already supports today:
- JPEG / JPG
- PNG
- GIF
- WebP
- SVG
If the referenced file exists but is not one of those supported formats, Warp should not attempt a degraded or best-effort render. It should show the original Markdown text exactly as it does today.
### File-backed image rendering
When AI output contains a valid Markdown image whose source resolves to a supported local file, the image should render inline in the normal block list flow where the Markdown image appeared.
Rendered images should behave like block content in the response:
- they appear between the surrounding text in source order
- multiple images in one response all render in place
- images can appear alongside normal paragraphs, lists, code blocks, and tables without changing the rendering behavior of those surrounding elements
There are two distinct file-backed image layouts:
1. Inline image runs
When multiple parser-recognized Markdown image references appear on the same line, separated only by whitespace, Warp should render them as a horizontal image row:
- each image appears above its own visible source-path label
- each item preserves aspect ratio
- the row stays left-aligned in the response flow
- the path label under each image keeps the existing hover highlight and click-to-open behavior for file links
- if that path points to a binary file such as PNG, JPG, GIF, or WebP, activating the path should use the platform-default opener instead of Warp's editor or the user's configured `$EDITOR`
2. Block-level image groups
When Markdown images appear as standalone block items on separate lines, consecutive successfully rendered images should be grouped into a bordered card:
- the card header should read `Images N`, where `N` is the number of rendered images in that grouped block
- each row in the card should show a square thumbnail on the left and the source path on the right
- the source path keeps the same hover highlight and click-to-open behavior as other file links in AI output
- binary file paths in that row should open with the platform-default application rather than any editor target
- unsupported or unresolved images should fall back to raw Markdown rather than appearing inside the grouped card
For file-backed images that render successfully, Warp should also show the referenced file link/path in the layout appropriate for that treatment:
- the displayed link text should use the file path/source text from the Markdown image reference
- it should participate in the normal link-highlighting and click behavior users already expect for file links in AI block content
- binary file links should open with the operating system's default file opener; on macOS this means using `open` rather than routing through Warp's editor or `$EDITOR`
- it should remain visible in restored conversations the same way it does for newly rendered output
If a Markdown image reference cannot be resolved to an existing local file, the block list should show the original Markdown image text instead of a rendered image.
Rendered file-backed images should also support click-to-expand behavior:
- clicking a rendered image opens it in Warp’s existing fullscreen lightbox treatment
- the lightbox should reuse the same fullscreen overlay treatment already used elsewhere in Warp rather than introducing a new one-off viewer
- when the current AI message contains multiple successfully rendered visual items, the lightbox should allow previous/next navigation across those visuals in source order
- for file-backed images, the fullscreen viewer should keep showing the image source/path as descriptive text
### Mermaid rendering
Mermaid diagrams should be supported as part of the same feature and should render inline in the AI block list as visual content rather than raw Mermaid source.
Mermaid rendering in the AI block list requires two independent conditions to be true:
- the existing global Mermaid rendering flag is enabled
- the new AI block list image-rendering flag is enabled
If either flag is disabled, Mermaid content in the AI block list should continue to fall back to raw Markdown/source rendering.
From the user’s perspective:
- a Mermaid diagram in AI Markdown should appear as a rendered diagram in a titled, bordered block within the response flow
- the block should use a dark header treatment with a Mermaid-specific title and a white inner canvas for the rendered diagram
- rendering may happen asynchronously
- the UI must not block while the diagram is being generated
- clicking a rendered Mermaid diagram should open that diagram in the same fullscreen lightbox treatment used for rendered block-list images
- when there are multiple rendered visuals in the same AI message, the lightbox should support previous/next navigation between them in source order
If Mermaid rendering fails for any reason, the block list should fall back to showing the original Mermaid Markdown source rather than a broken image state.
### Layout and sizing
Rendered file-backed images and rendered Mermaid diagrams should preserve aspect ratio and remain fully visible without cropping or distortion, but the exact sizing now depends on the treatment:
- inline image rows use medium-height thumbnails with per-image widths derived from aspect ratio and capped so the row remains visually balanced
- block-level image groups use square thumbnail slots for each image row
- Mermaid uses a framed block with a padded white canvas that contains the rendered diagram
The inline/block treatments remain the default in-flow presentation. Clicking a rendered image or Mermaid diagram should open the shared fullscreen lightbox overlay as a secondary viewing mode, without changing the inline layout itself.
### Fallback behavior
Fallback behavior is important and should be predictable.
For file-backed images, if any of the following are true:
- the path cannot be resolved
- the file does not exist
- the format is unsupported
- the asset cannot be loaded successfully
then Warp should show the original Markdown image syntax inline, matching current behavior, instead of rendering an error-specific image container.
For Mermaid diagrams, if rendering fails, Warp should show the original Mermaid Markdown source instead of a rendered diagram.
### Selection and copy behavior
This first release should keep the existing block-list selection behavior for text, code blocks, tables, and other already-supported content. It should not introduce a broader refactor to make rendered images and Mermaid diagrams participate in mixed drag-selection across multiple renderer types.
Instead, rendered visual content should support copy through explicit copy surfaces:
- block-level copy actions should include the underlying Markdown source for rendered images and Mermaid diagrams
- right-click copy on a rendered image or Mermaid diagram should copy the underlying Markdown source for that visual element
For this first release, it is acceptable for copy to serialize visual content back into raw Markdown rather than attempting rich HTML or image clipboard output.
### Right-click copy behavior
Right-clicking a rendered image or Mermaid diagram should provide a copy action.
That copy action should place the underlying Markdown source on the clipboard:
- for file-backed images, the original Markdown image reference
- for Mermaid diagrams, the original Mermaid Markdown source
This copy action should not place image bytes on the clipboard in this release.
### Mixed content behavior
The AI block list must support responses that contain:
- only text
- only one image
- only one Mermaid diagram
- multiple file-backed images
- multiple Mermaid diagrams
- mixed text, images, and Mermaid diagrams in one response
Each supported visual element should render independently in the correct source position. Unsupported or unresolved items should remain raw Markdown text without preventing supported neighbors from rendering.
### Streaming and restored behavior
The feature should behave consistently for both streamed and restored responses.
For streamed responses:
- file-backed images should render once enough Markdown is present to identify the image reference
- Mermaid diagrams should render once enough source is present to identify and generate the diagram
- rendering updates should not corrupt surrounding content or unexpectedly clear an active selection
For restored responses:
- supported images and Mermaid diagrams should render the same way they do in the initial response view
- the same fallback rules should apply if the asset is unavailable at restore time
## Success Criteria
- A Markdown image in an AI block list response renders inline when it references a supported local file.
- Relative file paths resolve against the session’s working directory rather than a notebook/document location.
- Relative file paths are re-resolved against the working directory recorded with the AI block at render time when the block is restored or re-rendered.
- Absolute file paths render correctly when they point to a supported local image.
- The initial supported file format set is JPEG/JPG, PNG, GIF, WebP, and SVG.
- Successfully rendered file-backed images show their source path in the mock-aligned position for their layout, with normal link highlighting behavior.
- Activating the source path for a rendered binary asset opens it with the operating system's default app instead of Warp's editor or the user's configured `$EDITOR`.
- Mermaid diagrams in AI Markdown render inline in the AI block list only when both the existing Mermaid flag and the new block list image-rendering flag are enabled.
- Multiple image references on the same line render as a horizontal inline image row with one path label per image.
- Consecutive standalone image references render as a grouped `Images N` card with thumbnail-plus-path rows.
- Rendered images and Mermaid diagrams preserve aspect ratio and use the new mock-aligned sizing treatments.
- If a file cannot be found or the file format is unsupported, Warp shows the original Markdown text rather than a broken rendered state.
- If Mermaid rendering fails, Warp shows the original Mermaid Markdown source.
- Existing block-list text/code/table selection behavior continues to work as it does today after image rendering is added.
- Block-level copy actions write the underlying raw Markdown for rendered images and Mermaid diagrams into the clipboard.
- Right-click copy on a rendered image or Mermaid diagram copies its underlying Markdown source.
- Clicking a rendered file-backed image or Mermaid diagram opens the shared fullscreen lightbox treatment.
- The fullscreen lightbox supports previous/next navigation across the renderable visual items in the same AI message, in source order.
- Multiple supported images in one AI response render independently and do not interfere with surrounding text.
- Restored AI blocks and newly streamed AI blocks follow the same rendering and fallback rules.
- The implementation behaves correctly on all supported platforms, including WASM.
- The feature has automated verification coverage in addition to manual validation.
## Validation
- Unit tests for markdown/image section parsing and flag-gating behavior, including Mermaid requiring both flags.
- Unit tests for relative-path resolution against block metadata that stores the original working directory.
- Unit tests for fallback behavior when a file is missing, unsupported, or Mermaid rendering is disabled or fails.
- Automated tests covering block-level copy serialization for mixed content that includes text, images, Mermaid diagrams, code blocks, and other special Markdown-rendered content.
- Integration tests that exercise AI block list rendering end to end, including restored-block behavior and mixed-content copy behavior.
- Manual validation with an AI response containing a relative-path PNG image.
- Manual validation with an AI response containing an absolute-path JPEG image.
- Manual validation with one example each for SVG, GIF, and WebP image references.
- Manual validation that an inline image run renders as a horizontal row with the path label under each image and the expected link highlighting.
- Manual validation that consecutive standalone image lines render as an `Images N` card with thumbnail-plus-path rows.
- Manual validation that clicking the source path for a rendered PNG or other binary image opens it with the platform-default app rather than Warp's editor.
- Manual validation that a missing local file falls back to the original Markdown image syntax.
- Manual validation that an unsupported format falls back to the original Markdown image syntax.
- Manual validation that a Mermaid diagram renders in the new framed block treatment with a white inner canvas.
- Manual validation that Mermaid in the AI block list does not render unless both required flags are enabled.
- Manual validation that a Mermaid render failure falls back to raw Mermaid Markdown.
- Manual validation of a response containing text, one image, more text, and a Mermaid diagram.
- Manual validation of a response containing multiple images in sequence.
- Manual validation that clicking a rendered inline image opens the shared fullscreen lightbox with the expected image selected.
- Manual validation that clicking a rendered Mermaid diagram opens the same fullscreen lightbox treatment.
- Manual validation that previous/next navigation in the lightbox walks the rendered visuals from the same AI message in source order.
- Manual validation that Escape and scrim click dismiss the lightbox and return the user to the block list.
- Manual validation that existing text/code/table selection behavior is unchanged after image rendering is added.
- Manual validation that block-level copy produces the expected text plus raw Markdown image or Mermaid source in document order.
- Manual validation that right-click copy on a rendered image or Mermaid diagram copies the underlying Markdown source.
- Manual validation that the same response renders correctly after the conversation is restored from history.
- Regression validation that disabling the feature flag restores the current raw-Markdown behavior.
specs/zachlloyd/inline-markdown-images-in-blocklist/TECH.md
+388
View File
@@ -0,0 +1,388 @@
# Inline Markdown Images in AI Block List — Tech Spec
Product spec: `specs/zachlloyd/inline-markdown-images-in-blocklist/PRODUCT.md`
## Problem
The AI block list does not currently support the product behavior described in `PRODUCT.md`: rendering supported Markdown images and Mermaid diagrams inline inside AI responses with the approved inline-row, grouped-image, and Mermaid-card treatments.
Warp already has the underlying primitives needed to build this behavior:
- local image asset loading
- responsive image sizing
- Mermaid-to-SVG rendering
- block-list rendering of other markdown section types like code and tables
Today, block-list markdown is split into plain text, code, and table sections. Plain text is rendered with `FormattedTextElement`, which intentionally treats `FormattedTextLine::Image` as a line break rather than a visual block. As a result, Markdown image syntax degrades to raw text in the AI block list, and Mermaid fences continue to render through the existing code-path behavior rather than the product's visual treatment.
The implementation should add support for the required file-backed image and Mermaid behaviors while keeping the change narrowly scoped:
- no new cross-renderer selection model
- no broad markdown-renderer refactor
- no new dependency edge from the block list into a rendering stack it does not already use
This should be a surgical extension of the existing section-based block-list renderer, with explicit copy behavior for rendered visuals, a visible file-link label below successfully rendered file-backed images, raw-Markdown fallback when rendering is unavailable, and reuse of Warp’s existing fullscreen lightbox treatment when the user clicks a rendered visual.
## Relevant Code
- `specs/zachlloyd/inline-markdown-images-in-blocklist/PRODUCT.md` — approved product behavior
- `app/src/ai/agent/util.rs (24-186)` — `parse_markdown_into_text_and_code_sections`; current block-list markdown section splitter
- `app/src/ai/agent/mod.rs (1281-1361)` — `AgentOutputText`, `AgentOutputTable`, and `AIAgentTextSection`
- `app/src/ai/agent/mod.rs (405-505)` — `AIAgentOutput::all_text`; current message traversal used by rendering and link detection
- `app/src/ai/agent/mod.rs (1547-1628)` — `Display for AIAgentOutputMessage`; block/conversation copy formatting for markdown sections
- `app/src/ai/agent/mod.rs (2722-2779)` — `AIAgentExchange::format_output_for_copy` and `format_for_copy`
- `app/src/ai/blocklist/block/view_impl/output.rs (246-345)` — message-level block-list output rendering loop
- `app/src/ai/blocklist/block/view_impl/common.rs (869-952)` — `render_text_sections`; current section-to-UI mapping
- `app/src/ai/blocklist/block/view_impl/common.rs (986-1084)` — `render_rich_text_output_text_section`; current plain-text markdown renderer
- `app/src/ai/blocklist/block/view_impl/common.rs (1461-1646)` — `render_code_output_section`; current code-block renderer
- `app/src/workspace/lightbox_view.rs` — shared fullscreen lightbox view with Escape and left/right navigation
- `crates/ui_components/src/lightbox.rs` — reusable lightbox component and navigation controls
- `app/src/workspace/action.rs` — workspace actions for opening and updating the shared lightbox
- `app/src/workspace/view.rs` — workspace-level lightbox lifecycle and focus handling
- `app/src/ai/blocklist/block.rs (2230-2246)` — per-section handle allocation for code and tables
- `app/src/ai/blocklist/block.rs (4313-4476)` — `selected_text`, `clear_all_selections`, and `clear_other_selections`; current selection model
- `app/src/ai/blocklist/block.rs (4946-5002)` — block-level copy scope
- `app/src/ai/blocklist/controller.rs (62-86)` — `SessionContext`; current working-directory capture
- `app/src/ai/blocklist/persistence.rs (24-31)` — persisted exchange `working_directory`
- `app/src/ai/blocklist/history_model.rs (2102-2154)` — restored query history carries working directory
- `app/src/ai/blocklist/block.rs (906-919)` — `AIBlock::new` accepts `current_working_directory` and `shell_launch_data`
- `crates/markdown_parser/src/markdown_parser.rs (286-326)` — `parse_image`; current image recognition behavior
- `crates/markdown_parser/src/markdown_parser.rs (466-475)` — `parse_inline_markdown`; current inline parser surface
- `crates/markdown_parser/src/markdown_parser_test.rs (2171-2320)` — current parser coverage for images
- `crates/warpui_core/src/elements/formatted_text_element.rs (1580-1591, 1661-1671)` — images are currently treated as line breaks in the block-list rich-text path
- `crates/editor/src/content/text.rs (280-362)` — image markdown round-trip behavior in the editor stack
- `crates/editor/src/content/text.rs (526-579)` — `CodeBlockType::Mermaid` gating
- `crates/editor/src/content/edit.rs (56-91)` — native/WASM asset-source resolution rules
- `crates/editor/src/content/mermaid_diagram.rs (20-67)` — in-memory Mermaid SVG asset generation and sizing
- `crates/warpui_core/src/image_cache.rs` — supported shared image types (JPEG, PNG, GIF, WebP, SVG)
- `crates/warp_features/src/lib.rs (525-540, 817-858)` — markdown-related feature flags
- `app/src/lib.rs (2462-2471)` — app-side feature-flag wiring for markdown tables and Mermaid
- `app/Cargo.toml (667-746)` — compile-time feature declarations
- `app/src/ai/agent/util_tests.rs` — block-list markdown parser/unit test pattern
- `crates/editor/src/content/edit_tests.rs (293-404)` — Mermaid layout test pattern
- `crates/editor/src/content/text_tests.rs (101-127)` — Mermaid block-type gating tests
- `crates/integration/tests/INTEGRATION_TESTING.md` — integration test registration flow
## Current State
### Markdown parsing in the AI block list
The block list currently parses AI markdown with a line-oriented splitter in `app/src/ai/agent/util.rs (24-186)`. It recognizes:
- plain markdown text
- fenced code blocks, including linked-code metadata
- GFM tables
It does not emit explicit image or Mermaid section types today. Standard Markdown image syntax therefore remains embedded inside `AIAgentTextSection::PlainText`, and Mermaid code fences remain ordinary `Code` sections.
### Current Markdown semantics vs current parser behavior
Standard Markdown/CommonMark uses `![alt](source)` as image syntax. There is not a separate standard syntax that means “this should be shown as a literal file path instead of rendered as an image.” If the author wants literal text, they escape the Markdown or place it in code spans/blocks.
Our current `markdown_parser` does not fully implement that inline image semantic yet. `crates/markdown_parser/src/markdown_parser.rs (286-326)` only recognizes images when they occupy a standalone line; inline paragraph images like `text ![img](foo.png) more text` are currently treated as plain text by this parser. For this redesign, the block list still should not invent a generic “render some arbitrary inline image-looking text” rule. The only extension should be a narrow line parser for runs of parser-compatible image references separated only by whitespace so we can distinguish:
- consecutive standalone image lines, which stay block-level
- multiple image references on the same line, which become an inline image row
### Why images disappear today
Plain-text markdown sections are rendered with `render_rich_text_output_text_section` in `app/src/ai/blocklist/block/view_impl/common.rs (986-1084)`, which delegates to `FormattedTextElement`. In `crates/warpui_core/src/elements/formatted_text_element.rs (1580-1591, 1661-1671)`, `FormattedTextLine::Image(_)`, `FormattedTextLine::Embedded(_)`, and `FormattedTextLine::HorizontalRule` are all treated as line-break-like layout items rather than renderable content. That is the immediate reason that block-list Markdown images never show up.
### Asset and Mermaid support already exists elsewhere
Warp already has the low-level capabilities this feature needs:
- shared image format support in `crates/warpui_core/src/image_cache.rs`
- asset-source resolution, including WASM-safe behavior, in `crates/editor/src/content/edit.rs (56-91)`
- Mermaid SVG generation and sizing in `crates/editor/src/content/mermaid_diagram.rs (20-67)`
- Mermaid code-block identification in `crates/editor/src/content/text.rs (526-579)`
The app crate already depends on `warp_editor`, and the block list already embeds editor-backed code blocks. Reusing editor Mermaid/image helpers therefore does not introduce a new crate dependency edge.
Warp also already has a reusable fullscreen lightbox path at the workspace layer. `WorkspaceAction::OpenLightbox` / `UpdateLightboxImage` drive `LightboxView`, which already supports Escape dismissal, left/right keyboard navigation, and previous/next buttons. The block-list visual renderer should reuse that path rather than inventing a new fullscreen viewer.
### Working-directory metadata already exists
The product requirement for resolving relative paths against the working directory captured when the AI block rendered is already compatible with existing data flow:
- session context captures `current_working_directory` in `app/src/ai/blocklist/controller.rs (62-86)`
- persisted exchanges store `working_directory` in `app/src/ai/blocklist/persistence.rs (24-31)`
- restored history loads it back in `app/src/ai/blocklist/history_model.rs (2102-2154)`
- `AIBlock::new` already receives both `current_working_directory` and `shell_launch_data`
We do not need a new persistence field for this feature; we need to thread existing metadata into the visual-section renderers.
### Selection is intentionally out of scope
The current selection model in `app/src/ai/blocklist/block.rs (4313-4476)` is fragmented across:
- block-level text selection
- child-view-native selection for code editors and other special renderers
That architecture does not support a continuous mixed-content selection model across text, code, tables, images, and Mermaid. This feature should not attempt to fix that. Instead, it should:
- preserve existing text/code/table selection behavior
- add explicit copy affordances for rendered images and Mermaid diagrams
- ensure block-level copy/export includes the correct raw Markdown for visual sections
## Proposed Changes
### 1. Add a dedicated block-list image feature flag
Add a new feature flag dedicated to AI block-list markdown visuals, e.g. `BlocklistMarkdownImages`.
Implementation points:
- add `BlocklistMarkdownImages` to `crates/warp_features/src/lib.rs`
- add `blocklist_markdown_images = []` to `app/Cargo.toml`
- wire it in `app/src/lib.rs` alongside the existing markdown table/Mermaid flags
- enable it in `DOGFOOD_FLAGS`, but leave it out of `PREVIEW_FLAGS` until the surface is stable
Mermaid rendering in the block list should require:
- `FeatureFlag::BlocklistMarkdownImages`
- `FeatureFlag::MarkdownMermaid`
File-backed images in the block list should require only `BlocklistMarkdownImages`.
I do not recommend reusing the dormant generic `MarkdownImages` flag for this launch. It is not currently wired through `app/src/lib.rs` or `app/Cargo.toml`, and this feature needs rollout control specific to the AI block list.
### 2. Extend the AI markdown section model with visual sections
Extend `AIAgentTextSection` in `app/src/ai/agent/mod.rs` with explicit visual section variants:
- `Image { image: AgentOutputImage }`
- `MermaidDiagram { diagram: AgentOutputMermaidDiagram }`
Proposed payloads:
- `AgentOutputImage`
- `alt_text: String`
- `source: String`
- `markdown_source: String`
- `layout: AgentOutputImageLayout`
- `AgentOutputMermaidDiagram`
- `source: String`
- `markdown_source: String`
The key design choice is to preserve Markdown source on the section payload rather than reconstructing it later from rendered state. That keeps:
- block-level copy/export exact
- right-click copy trivial
- fallback rendering simple
For images, `markdown_source` can initially be canonicalized to `![alt](source)` because the parser only returns `alt_text` and `source`, not byte-accurate source spans. That is consistent with the editor stack’s current image markdown round-trip behavior.
### 3. Teach the section splitter to extract images and Mermaid explicitly
Update `parse_markdown_into_text_and_code_sections` in `app/src/ai/agent/util.rs` so that it can distinguish inline image runs from block-level image lines instead of treating every extracted image section identically.
The change should remain incremental:
- keep current code-fence boundary detection so linked-code metadata parsing stays unchanged
- keep current table extraction behavior
- for plain-text regions, continue scanning line-by-line
- keep using `markdown_parser` for standalone image-line extraction
- add a narrow helper that recognizes lines containing two or more parser-compatible image references separated only by whitespace and emits them as `AIAgentTextSection::Image` sections tagged with `AgentOutputImageLayout::Inline`
- emit standalone image lines as `AIAgentTextSection::Image` tagged with `AgentOutputImageLayout::Block`
- when a fenced code block language resolves to Mermaid, emit `AIAgentTextSection::MermaidDiagram` instead of `Code`
- leave everything else in `PlainText`
Two important constraints:
- image extraction should still stay narrow and markdown-driven rather than becoming a generic loose regex over arbitrary prose
- Mermaid section extraction should not be gated at parse time; the section payload should still preserve Mermaid source even when the runtime flags are off, so the renderer can fall back cleanly to raw Markdown without reparsing restored conversations
### 4. Keep the existing section renderer and add visual sections surgically
Do not replace the current block-list markdown rendering architecture for this feature. Instead, extend `render_text_sections` in `app/src/ai/blocklist/block/view_impl/common.rs (869-952)` with grouped visual renderers:
- inline image-row rendering for consecutive `AgentOutputImageLayout::Inline` sections
- grouped block-image row rendering for consecutive `AgentOutputImageLayout::Block` sections
- a Mermaid card renderer that uses the new framed treatment
These should follow the same overall section-rendering pattern already used for:
- code sections
- table sections
This keeps the implementation targeted:
- no unified markdown content view
- no mixed-content selection refactor
- no change to `AIBlock::selected_text` semantics beyond whatever is needed for explicit image/Mermaid copy affordances
As part of that renderer pass, build a source-ordered collection of the successfully renderable visual sections in the current AI message. That shared collection should include both successfully rendered file-backed images and successfully rendered Mermaid diagrams. Clicking any rendered image or Mermaid diagram should dispatch `WorkspaceAction::OpenLightbox` with that shared collection and the clicked section’s initial index, so previous/next navigation works across the other renderable visuals from the same message.
### 5. Reuse existing editor helpers without a new dependency edge
The new visual-section renderers should reuse existing editor helpers rather than inventing a separate rendering stack.
For file-backed images:
- extract or adapt the existing asset-resolution helper in `crates/editor/src/content/edit.rs (56-91)` so the block list can resolve relative paths against a base directory rather than a document path
- pass the AI block’s stored working directory as that base directory
- on native platforms, canonicalize when possible
- on WASM, preserve the existing no-canonicalization behavior
For Mermaid:
- use `mermaid_asset_source` from `crates/editor/src/content/mermaid_diagram.rs (20-67)`
- derive the rendered max width from the loaded asset's intrinsic size rather than calling a separate Mermaid layout helper from the block-list renderer
- honor `FeatureFlag::MarkdownMermaid` in addition to the new block-list image flag
For actual rendering:
- use a block-list-local renderer in the existing section flow
- use WarpUI image elements and existing block-list spacing/styling conventions
- preserve the existing right-click-to-copy Markdown behavior on every rendered visual
- for inline image runs, render medium-height image tiles with per-image labels below
- for grouped block images, render stacked thumbnail-plus-path rows using the full source path text
- for Mermaid, render a bordered card shell with a titled header and a white inner canvas that contains the diagram
- on left click, open the shared workspace lightbox instead of a block-list-specific fullscreen modal for both rendered file-backed images and rendered Mermaid diagrams
- keep the path labels on the shared link-detection pipeline so hover/click behavior remains unchanged
- allow the implementation to harden shared image/text primitives where needed for this surface, including guarding non-finite image rects, exposing intrinsic SVG dimensions through shared image metadata, and constraining single-line text hit testing to valid Y bounds
This follows the same broad pattern as the block list’s existing special markdown sections: section-specific render helpers living inside the current block-list message renderer, with reusable lower-level helpers coming from crates the app already depends on.
### 6. Render fallback content instead of broken visuals
The renderer should decide per visual section whether it can render as a visual block or must fall back to raw Markdown.
For file-backed images:
- if `BlocklistMarkdownImages` is disabled, render `markdown_source`
- if the path cannot be resolved, render `markdown_source`
- if the asset fails to load or is an unsupported image type, render `markdown_source`
- only include images in an inline row or grouped block row set when that individual image succeeds; unsupported or missing images fall back to raw Markdown in source order
For Mermaid:
- if either `BlocklistMarkdownImages` or `MarkdownMermaid` is disabled, render `markdown_source`
- if Mermaid SVG generation fails, render `markdown_source`
This keeps fallback behavior local to rendering and avoids mutating stored section payloads based on transient failures.
### 7. Make block-level copy and export section-aware
Update the markdown copy/export paths in `app/src/ai/agent/mod.rs (1547-1628, 2722-2779)` so that `Image` and `MermaidDiagram` sections serialize from their preserved Markdown source.
Rules:
- block-level output copy uses `markdown_source` for image and Mermaid sections
- conversation export continues to be markdown, not image bytes or HTML
- right-click copy on a rendered image or Mermaid diagram writes that section’s `markdown_source`
This gives us correct image/Mermaid copy behavior without changing the existing selection architecture.
### 8. Render the file-link label with local link detection
The new file-link label under rendered file-backed images should reuse the existing text/link rendering helpers rather than introducing a special-case link widget.
Implementation shape:
- keep the current global output link-detection pipeline unchanged for the underlying markdown section data
- in the successful image-render path, render the link text in the layout-specific position required by the product behavior
- keep using the shared `DetectedLinksState` keyed by the section index so the path labels get normal file-link highlighting and click behavior without a second link-detection system
- keep binary-file link opening on the standard file-open path so those labels resolve to the platform default app rather than `$EDITOR`
- keep find/highlight behavior scoped to the existing global text pipeline rather than refactoring search indexing for this label in the same change
This keeps the change narrowly targeted while still giving users a visible, interactive file link under rendered file-backed images.
### 9. Preserve existing selection behavior
Do not add a new drag-selection model for rendered visuals in this change.
The expected behavior for this implementation is:
- existing text/code/table selection behavior remains unchanged
- rendered images and Mermaid diagrams expose explicit copy behavior through right-click/context-menu affordances
- block-level copy/export includes the right raw Markdown for visual sections
That is the intended “minimal change” scope for this feature.
### 10. Keep restored conversations deterministic
Restored conversations should not depend on reparsing with different metadata than the original render.
The renderer should use:
- the section payload stored on the conversation output
- the `working_directory` already persisted with the exchange/block
- current runtime flags to decide whether to render a visual or raw fallback
That means restored blocks will:
- resolve relative file images against the original captured working directory
- render Mermaid only when both flags are currently enabled
- still show raw Markdown if the file is gone or the platform cannot load it
## End-to-End Flow
1. The AI backend streams markdown text into an `AIAgentOutputMessageType::Text`.
2. `parse_markdown_into_text_and_code_sections` in `app/src/ai/agent/util.rs` identifies plain text, code, tables, parser-recognized images, and Mermaid sections.
3. The parsed output stores explicit `Image` and `MermaidDiagram` sections alongside their raw Markdown source.
4. `app/src/ai/blocklist/block/view_impl/output.rs` renders the new visual sections through the existing section-based block-list renderer alongside plain text, code, and tables.
5. The visual-section renderers receive the AI block’s stored `current_working_directory` and `shell_launch_data`.
6. File-backed image sections resolve their source path relative to that working directory.
7. Mermaid sections build an in-memory SVG asset source with `mermaid_asset_source`.
8. The block-list renderer derives a source-ordered lightbox collection from the renderable visual sections in that message.
9. Each visual section either renders as an image block or falls back to a plain markdown block depending on feature flags and asset/render success.
10. When a file-backed image renders successfully, the renderer also shows a clickable/highlighted file link beneath or beside it:
- inline image rows show the basename / final path segment
- block image rows show the full source path
11. Clicking a rendered image or Mermaid diagram dispatches `WorkspaceAction::OpenLightbox` with that shared visual collection and the clicked section’s initial index.
12. Right-click copy on a rendered image or Mermaid diagram writes the section’s `markdown_source`.
13. Clicking a rendered file link continues to use the shared file-open flow, which means binary file labels open in the platform default app instead of `$EDITOR`.
14. Block-level copy and conversation export continue to emit markdown, using `markdown_source` for visual sections.
15. When the conversation is restored later, the same section payload renders again using the stored working-directory metadata and current flags.
## Risks and Mitigations
### Risk: selection expectations drift beyond the implementation
The current AI block architecture does not support unified mixed-content selection across renderer types. If the specs or implementation drift back toward that requirement, this feature would expand substantially.
Mitigation:
- keep the product and tech specs explicit that this change does not add a new cross-renderer selection model
- keep the scope focused on rendering, explicit copy affordances, and block-level copy/export correctness
### Risk: parser scope does not match full Markdown image semantics
Standard Markdown treats images as inline syntax, but our current parser only recognizes standalone-image lines, and this redesign needs a narrow same-line image-row case.
Mitigation:
- keep the new same-line parser limited to lines that consist entirely of parser-compatible image references separated only by whitespace
- do not treat arbitrary prose containing `![...]` as inline-renderable image content
- treat fuller CommonMark-style inline image support as follow-up parser work
### Risk: relative path resolution diverges between notebooks and AI blocks
The existing editor helper resolves relative image paths against a document path, while AI blocks need to resolve against the captured session working directory.
Mitigation:
- factor path resolution into a base-directory-oriented helper
- update notebook and AI-block callers to use it explicitly
- add native and WASM coverage
### Risk: regressions to existing section rendering
Adding two more section types to the current renderer could affect section indexing, link-detection offsets, or restored-block bookkeeping.
Mitigation:
- keep section ordering explicit in `AIAgentOutput::all_text` and render traversal
- follow existing per-section bookkeeping patterns for code/table sections
- add mixed-content coverage that includes ordinary code blocks and tables, not just images
## Testing and Validation
### Unit tests
- extend `app/src/ai/agent/util_tests.rs` with:
- standalone image extraction
- same-line inline image-run extraction
- standalone multi-line block-image extraction
- image + table + text ordering
- Mermaid section extraction
- add block-list markdown copy tests covering:
- image sections serialize to `![alt](source)`
- Mermaid sections serialize to fenced mermaid markdown
- add native/WASM unit tests around relative path resolution against a base directory
- add focused rendering/helper coverage for:
- section-index preservation across skipped empty text sections
- lightbox index mapping in source order
- inline-image basename labeling
- visual width guard rails for invalid sizes
- add shared primitive regression tests for:
- invalid image rect rejection
- SVG intrinsic image sizing
- single-line text hit testing respecting Y bounds
### Integration tests
- A reasonable first integration test is a single manual-observation test, `test_restored_ai_block_renders_mermaid_and_local_images`, in `crates/integration/src/test/agent_mode.rs`.
- That test would restore a synthetic `ConversationData` through `load_conversation_from_tasks`, set `InputContext.directory` to `crates/warpui_core/test_data`, and capture a real-display screenshot of one AI response that contains same-line local image markdown plus a Mermaid fence.
- It would be intentionally narrow: it would prove the restored AI block list can render both surfaces through the real conversation-hydration path without introducing protobuf fixtures or test-only UI hooks.
- Recommended follow-up integration coverage remains:
- an AI response containing plain text, a linked code block, a markdown table, a relative-path image, and a Mermaid diagram
- block-level copy serialization for mixed content
- restored-conversation rendering and fallback behavior
- disabled-flag fallback to raw Markdown
- shared-lightbox initial selection and previous/next navigation in message source order
### Manual validation
- relative and absolute local image paths for PNG, JPEG, GIF, WebP, and SVG
- inline same-line image rows with path labels below each image
- grouped standalone block-image rows with thumbnail-plus-path content
- missing-file fallback
- unsupported-format fallback
- Mermaid render success and Mermaid render failure fallback in the new framed treatment
- click-to-expand from both rendered images and Mermaid diagrams into the shared lightbox
- previous/next keyboard and button navigation inside the lightbox
- right-click copy on both file-backed images and Mermaid diagrams
- block-level copy on a response containing text, code, table, image, and Mermaid content
- restored conversation rendering after reopening history
- WASM sanity pass to verify path handling and fallback behavior do not panic
## Follow-ups
- full CommonMark-style inline image support if we decide to expand `markdown_parser` beyond its current standalone-image behavior
- unified mixed-content selection across block-list renderer types
- consolidating `MarkdownImages` and `BlocklistMarkdownImages` if Warp later ships a broader app-wide markdown-image rollout
- richer image interactions such as open/save/zoom, if product wants them later