galaxy/specs/zachlloyd/inline-markdown-images-in-blocklist/TECH.md

# Inline Markdown Images in AI Block List — Tech Spec
Product spec: `specs/zachlloyd/inline-markdown-images-in-blocklist/PRODUCT.md`

## Problem
The AI block list does not currently support the product behavior described in `PRODUCT.md`: rendering supported Markdown images and Mermaid diagrams inline inside AI responses with the approved inline-row, grouped-image, and Mermaid-card treatments.

Warp already has the underlying primitives needed to build this behavior:

- local image asset loading
- responsive image sizing
- Mermaid-to-SVG rendering
- block-list rendering of other markdown section types like code and tables

Today, block-list markdown is split into plain text, code, and table sections. Plain text is rendered with `FormattedTextElement`, which intentionally treats `FormattedTextLine::Image` as a line break rather than a visual block. As a result, Markdown image syntax degrades to raw text in the AI block list, and Mermaid fences continue to render through the existing code-path behavior rather than the product's visual treatment.

The implementation should add support for the required file-backed image and Mermaid behaviors while keeping the change narrowly scoped:

- no new cross-renderer selection model
- no broad markdown-renderer refactor
- no new dependency edge from the block list into a rendering stack it does not already use

This should be a surgical extension of the existing section-based block-list renderer, with explicit copy behavior for rendered visuals, a visible file-link label below successfully rendered file-backed images, raw-Markdown fallback when rendering is unavailable, and reuse of Warp’s existing fullscreen lightbox treatment when the user clicks a rendered visual.

## Relevant Code
- `specs/zachlloyd/inline-markdown-images-in-blocklist/PRODUCT.md` — approved product behavior
- `app/src/ai/agent/util.rs (24-186)` — `parse_markdown_into_text_and_code_sections`; current block-list markdown section splitter
- `app/src/ai/agent/mod.rs (1281-1361)` — `AgentOutputText`, `AgentOutputTable`, and `AIAgentTextSection`
- `app/src/ai/agent/mod.rs (405-505)` — `AIAgentOutput::all_text`; current message traversal used by rendering and link detection
- `app/src/ai/agent/mod.rs (1547-1628)` — `Display for AIAgentOutputMessage`; block/conversation copy formatting for markdown sections
- `app/src/ai/agent/mod.rs (2722-2779)` — `AIAgentExchange::format_output_for_copy` and `format_for_copy`
- `app/src/ai/blocklist/block/view_impl/output.rs (246-345)` — message-level block-list output rendering loop
- `app/src/ai/blocklist/block/view_impl/common.rs (869-952)` — `render_text_sections`; current section-to-UI mapping
- `app/src/ai/blocklist/block/view_impl/common.rs (986-1084)` — `render_rich_text_output_text_section`; current plain-text markdown renderer
- `app/src/ai/blocklist/block/view_impl/common.rs (1461-1646)` — `render_code_output_section`; current code-block renderer
- `app/src/workspace/lightbox_view.rs` — shared fullscreen lightbox view with Escape and left/right navigation
- `crates/ui_components/src/lightbox.rs` — reusable lightbox component and navigation controls
- `app/src/workspace/action.rs` — workspace actions for opening and updating the shared lightbox
- `app/src/workspace/view.rs` — workspace-level lightbox lifecycle and focus handling
- `app/src/ai/blocklist/block.rs (2230-2246)` — per-section handle allocation for code and tables
- `app/src/ai/blocklist/block.rs (4313-4476)` — `selected_text`, `clear_all_selections`, and `clear_other_selections`; current selection model
- `app/src/ai/blocklist/block.rs (4946-5002)` — block-level copy scope
- `app/src/ai/blocklist/controller.rs (62-86)` — `SessionContext`; current working-directory capture
- `app/src/ai/blocklist/persistence.rs (24-31)` — persisted exchange `working_directory`
- `app/src/ai/blocklist/history_model.rs (2102-2154)` — restored query history carries working directory
- `app/src/ai/blocklist/block.rs (906-919)` — `AIBlock::new` accepts `current_working_directory` and `shell_launch_data`
- `crates/markdown_parser/src/markdown_parser.rs (286-326)` — `parse_image`; current image recognition behavior
- `crates/markdown_parser/src/markdown_parser.rs (466-475)` — `parse_inline_markdown`; current inline parser surface
- `crates/markdown_parser/src/markdown_parser_test.rs (2171-2320)` — current parser coverage for images
- `crates/warpui_core/src/elements/formatted_text_element.rs (1580-1591, 1661-1671)` — images are currently treated as line breaks in the block-list rich-text path
- `crates/editor/src/content/text.rs (280-362)` — image markdown round-trip behavior in the editor stack
- `crates/editor/src/content/text.rs (526-579)` — `CodeBlockType::Mermaid` gating
- `crates/editor/src/content/edit.rs (56-91)` — native/WASM asset-source resolution rules
- `crates/editor/src/content/mermaid_diagram.rs (20-67)` — in-memory Mermaid SVG asset generation and sizing
- `crates/warpui_core/src/image_cache.rs` — supported shared image types (JPEG, PNG, GIF, WebP, SVG)
- `crates/warp_features/src/lib.rs (525-540, 817-858)` — markdown-related feature flags
- `app/src/lib.rs (2462-2471)` — app-side feature-flag wiring for markdown tables and Mermaid
- `app/Cargo.toml (667-746)` — compile-time feature declarations
- `app/src/ai/agent/util_tests.rs` — block-list markdown parser/unit test pattern
- `crates/editor/src/content/edit_tests.rs (293-404)` — Mermaid layout test pattern
- `crates/editor/src/content/text_tests.rs (101-127)` — Mermaid block-type gating tests
- `crates/integration/tests/INTEGRATION_TESTING.md` — integration test registration flow

## Current State

### Markdown parsing in the AI block list
The block list currently parses AI markdown with a line-oriented splitter in `app/src/ai/agent/util.rs (24-186)`. It recognizes:

- plain markdown text
- fenced code blocks, including linked-code metadata
- GFM tables

It does not emit explicit image or Mermaid section types today. Standard Markdown image syntax therefore remains embedded inside `AIAgentTextSection::PlainText`, and Mermaid code fences remain ordinary `Code` sections.

### Current Markdown semantics vs current parser behavior
Standard Markdown/CommonMark uses `![alt](source)` as image syntax. There is not a separate standard syntax that means “this should be shown as a literal file path instead of rendered as an image.” If the author wants literal text, they escape the Markdown or place it in code spans/blocks.

Our current `markdown_parser` does not fully implement that inline image semantic yet. `crates/markdown_parser/src/markdown_parser.rs (286-326)` only recognizes images when they occupy a standalone line; inline paragraph images like `text ![img](foo.png) more text` are currently treated as plain text by this parser. For this redesign, the block list still should not invent a generic “render some arbitrary inline image-looking text” rule. The only extension should be a narrow line parser for runs of parser-compatible image references separated only by whitespace so we can distinguish:

- consecutive standalone image lines, which stay block-level
- multiple image references on the same line, which become an inline image row

### Why images disappear today
Plain-text markdown sections are rendered with `render_rich_text_output_text_section` in `app/src/ai/blocklist/block/view_impl/common.rs (986-1084)`, which delegates to `FormattedTextElement`. In `crates/warpui_core/src/elements/formatted_text_element.rs (1580-1591, 1661-1671)`, `FormattedTextLine::Image(_)`, `FormattedTextLine::Embedded(_)`, and `FormattedTextLine::HorizontalRule` are all treated as line-break-like layout items rather than renderable content. That is the immediate reason that block-list Markdown images never show up.

### Asset and Mermaid support already exists elsewhere
Warp already has the low-level capabilities this feature needs:

- shared image format support in `crates/warpui_core/src/image_cache.rs`
- asset-source resolution, including WASM-safe behavior, in `crates/editor/src/content/edit.rs (56-91)`
- Mermaid SVG generation and sizing in `crates/editor/src/content/mermaid_diagram.rs (20-67)`
- Mermaid code-block identification in `crates/editor/src/content/text.rs (526-579)`

The app crate already depends on `warp_editor`, and the block list already embeds editor-backed code blocks. Reusing editor Mermaid/image helpers therefore does not introduce a new crate dependency edge.

Warp also already has a reusable fullscreen lightbox path at the workspace layer. `WorkspaceAction::OpenLightbox` / `UpdateLightboxImage` drive `LightboxView`, which already supports Escape dismissal, left/right keyboard navigation, and previous/next buttons. The block-list visual renderer should reuse that path rather than inventing a new fullscreen viewer.

### Working-directory metadata already exists
The product requirement for resolving relative paths against the working directory captured when the AI block rendered is already compatible with existing data flow:

- session context captures `current_working_directory` in `app/src/ai/blocklist/controller.rs (62-86)`
- persisted exchanges store `working_directory` in `app/src/ai/blocklist/persistence.rs (24-31)`
- restored history loads it back in `app/src/ai/blocklist/history_model.rs (2102-2154)`
- `AIBlock::new` already receives both `current_working_directory` and `shell_launch_data`

We do not need a new persistence field for this feature; we need to thread existing metadata into the visual-section renderers.

### Selection is intentionally out of scope
The current selection model in `app/src/ai/blocklist/block.rs (4313-4476)` is fragmented across:

- block-level text selection
- child-view-native selection for code editors and other special renderers

That architecture does not support a continuous mixed-content selection model across text, code, tables, images, and Mermaid. This feature should not attempt to fix that. Instead, it should:

- preserve existing text/code/table selection behavior
- add explicit copy affordances for rendered images and Mermaid diagrams
- ensure block-level copy/export includes the correct raw Markdown for visual sections

## Proposed Changes

### 1. Add a dedicated block-list image feature flag
Add a new feature flag dedicated to AI block-list markdown visuals, e.g. `BlocklistMarkdownImages`.

Implementation points:
- add `BlocklistMarkdownImages` to `crates/warp_features/src/lib.rs`
- add `blocklist_markdown_images = []` to `app/Cargo.toml`
- wire it in `app/src/lib.rs` alongside the existing markdown table/Mermaid flags
- enable it in `DOGFOOD_FLAGS`, but leave it out of `PREVIEW_FLAGS` until the surface is stable

Mermaid rendering in the block list should require:
- `FeatureFlag::BlocklistMarkdownImages`
- `FeatureFlag::MarkdownMermaid`

File-backed images in the block list should require only `BlocklistMarkdownImages`.

I do not recommend reusing the dormant generic `MarkdownImages` flag for this launch. It is not currently wired through `app/src/lib.rs` or `app/Cargo.toml`, and this feature needs rollout control specific to the AI block list.

### 2. Extend the AI markdown section model with visual sections
Extend `AIAgentTextSection` in `app/src/ai/agent/mod.rs` with explicit visual section variants:

- `Image { image: AgentOutputImage }`
- `MermaidDiagram { diagram: AgentOutputMermaidDiagram }`

Proposed payloads:
- `AgentOutputImage`
  - `alt_text: String`
  - `source: String`
  - `markdown_source: String`
  - `layout: AgentOutputImageLayout`
- `AgentOutputMermaidDiagram`
  - `source: String`
  - `markdown_source: String`

The key design choice is to preserve Markdown source on the section payload rather than reconstructing it later from rendered state. That keeps:

- block-level copy/export exact
- right-click copy trivial
- fallback rendering simple

For images, `markdown_source` can initially be canonicalized to `![alt](source)` because the parser only returns `alt_text` and `source`, not byte-accurate source spans. That is consistent with the editor stack’s current image markdown round-trip behavior.

### 3. Teach the section splitter to extract images and Mermaid explicitly
Update `parse_markdown_into_text_and_code_sections` in `app/src/ai/agent/util.rs` so that it can distinguish inline image runs from block-level image lines instead of treating every extracted image section identically.

The change should remain incremental:
- keep current code-fence boundary detection so linked-code metadata parsing stays unchanged
- keep current table extraction behavior
- for plain-text regions, continue scanning line-by-line
- keep using `markdown_parser` for standalone image-line extraction
- add a narrow helper that recognizes lines containing two or more parser-compatible image references separated only by whitespace and emits them as `AIAgentTextSection::Image` sections tagged with `AgentOutputImageLayout::Inline`
- emit standalone image lines as `AIAgentTextSection::Image` tagged with `AgentOutputImageLayout::Block`
- when a fenced code block language resolves to Mermaid, emit `AIAgentTextSection::MermaidDiagram` instead of `Code`
- leave everything else in `PlainText`

Two important constraints:
- image extraction should still stay narrow and markdown-driven rather than becoming a generic loose regex over arbitrary prose
- Mermaid section extraction should not be gated at parse time; the section payload should still preserve Mermaid source even when the runtime flags are off, so the renderer can fall back cleanly to raw Markdown without reparsing restored conversations

### 4. Keep the existing section renderer and add visual sections surgically
Do not replace the current block-list markdown rendering architecture for this feature. Instead, extend `render_text_sections` in `app/src/ai/blocklist/block/view_impl/common.rs (869-952)` with grouped visual renderers:

- inline image-row rendering for consecutive `AgentOutputImageLayout::Inline` sections
- grouped block-image row rendering for consecutive `AgentOutputImageLayout::Block` sections
- a Mermaid card renderer that uses the new framed treatment

These should follow the same overall section-rendering pattern already used for:
- code sections
- table sections

This keeps the implementation targeted:
- no unified markdown content view
- no mixed-content selection refactor
- no change to `AIBlock::selected_text` semantics beyond whatever is needed for explicit image/Mermaid copy affordances

As part of that renderer pass, build a source-ordered collection of the successfully renderable visual sections in the current AI message. That shared collection should include both successfully rendered file-backed images and successfully rendered Mermaid diagrams. Clicking any rendered image or Mermaid diagram should dispatch `WorkspaceAction::OpenLightbox` with that shared collection and the clicked section’s initial index, so previous/next navigation works across the other renderable visuals from the same message.

### 5. Reuse existing editor helpers without a new dependency edge
The new visual-section renderers should reuse existing editor helpers rather than inventing a separate rendering stack.

For file-backed images:
- extract or adapt the existing asset-resolution helper in `crates/editor/src/content/edit.rs (56-91)` so the block list can resolve relative paths against a base directory rather than a document path
- pass the AI block’s stored working directory as that base directory
- on native platforms, canonicalize when possible
- on WASM, preserve the existing no-canonicalization behavior

For Mermaid:
- use `mermaid_asset_source` from `crates/editor/src/content/mermaid_diagram.rs (20-67)`
- derive the rendered max width from the loaded asset's intrinsic size rather than calling a separate Mermaid layout helper from the block-list renderer
- honor `FeatureFlag::MarkdownMermaid` in addition to the new block-list image flag

For actual rendering:
- use a block-list-local renderer in the existing section flow
- use WarpUI image elements and existing block-list spacing/styling conventions
- preserve the existing right-click-to-copy Markdown behavior on every rendered visual
- for inline image runs, render medium-height image tiles with per-image labels below
- for grouped block images, render stacked thumbnail-plus-path rows using the full source path text
- for Mermaid, render a bordered card shell with a titled header and a white inner canvas that contains the diagram
- on left click, open the shared workspace lightbox instead of a block-list-specific fullscreen modal for both rendered file-backed images and rendered Mermaid diagrams
- keep the path labels on the shared link-detection pipeline so hover/click behavior remains unchanged
- allow the implementation to harden shared image/text primitives where needed for this surface, including guarding non-finite image rects, exposing intrinsic SVG dimensions through shared image metadata, and constraining single-line text hit testing to valid Y bounds

This follows the same broad pattern as the block list’s existing special markdown sections: section-specific render helpers living inside the current block-list message renderer, with reusable lower-level helpers coming from crates the app already depends on.

### 6. Render fallback content instead of broken visuals
The renderer should decide per visual section whether it can render as a visual block or must fall back to raw Markdown.

For file-backed images:
- if `BlocklistMarkdownImages` is disabled, render `markdown_source`
- if the path cannot be resolved, render `markdown_source`
- if the asset fails to load or is an unsupported image type, render `markdown_source`
- only include images in an inline row or grouped block row set when that individual image succeeds; unsupported or missing images fall back to raw Markdown in source order

For Mermaid:
- if either `BlocklistMarkdownImages` or `MarkdownMermaid` is disabled, render `markdown_source`
- if Mermaid SVG generation fails, render `markdown_source`

This keeps fallback behavior local to rendering and avoids mutating stored section payloads based on transient failures.

### 7. Make block-level copy and export section-aware
Update the markdown copy/export paths in `app/src/ai/agent/mod.rs (1547-1628, 2722-2779)` so that `Image` and `MermaidDiagram` sections serialize from their preserved Markdown source.

Rules:
- block-level output copy uses `markdown_source` for image and Mermaid sections
- conversation export continues to be markdown, not image bytes or HTML
- right-click copy on a rendered image or Mermaid diagram writes that section’s `markdown_source`

This gives us correct image/Mermaid copy behavior without changing the existing selection architecture.

### 8. Render the file-link label with local link detection
The new file-link label under rendered file-backed images should reuse the existing text/link rendering helpers rather than introducing a special-case link widget.

Implementation shape:
- keep the current global output link-detection pipeline unchanged for the underlying markdown section data
- in the successful image-render path, render the link text in the layout-specific position required by the product behavior
- keep using the shared `DetectedLinksState` keyed by the section index so the path labels get normal file-link highlighting and click behavior without a second link-detection system
- keep binary-file link opening on the standard file-open path so those labels resolve to the platform default app rather than `$EDITOR`
- keep find/highlight behavior scoped to the existing global text pipeline rather than refactoring search indexing for this label in the same change

This keeps the change narrowly targeted while still giving users a visible, interactive file link under rendered file-backed images.

### 9. Preserve existing selection behavior
Do not add a new drag-selection model for rendered visuals in this change.

The expected behavior for this implementation is:
- existing text/code/table selection behavior remains unchanged
- rendered images and Mermaid diagrams expose explicit copy behavior through right-click/context-menu affordances
- block-level copy/export includes the right raw Markdown for visual sections

That is the intended “minimal change” scope for this feature.

### 10. Keep restored conversations deterministic
Restored conversations should not depend on reparsing with different metadata than the original render.

The renderer should use:
- the section payload stored on the conversation output
- the `working_directory` already persisted with the exchange/block
- current runtime flags to decide whether to render a visual or raw fallback

That means restored blocks will:
- resolve relative file images against the original captured working directory
- render Mermaid only when both flags are currently enabled
- still show raw Markdown if the file is gone or the platform cannot load it

## End-to-End Flow
1. The AI backend streams markdown text into an `AIAgentOutputMessageType::Text`.
2. `parse_markdown_into_text_and_code_sections` in `app/src/ai/agent/util.rs` identifies plain text, code, tables, parser-recognized images, and Mermaid sections.
3. The parsed output stores explicit `Image` and `MermaidDiagram` sections alongside their raw Markdown source.
4. `app/src/ai/blocklist/block/view_impl/output.rs` renders the new visual sections through the existing section-based block-list renderer alongside plain text, code, and tables.
5. The visual-section renderers receive the AI block’s stored `current_working_directory` and `shell_launch_data`.
6. File-backed image sections resolve their source path relative to that working directory.
7. Mermaid sections build an in-memory SVG asset source with `mermaid_asset_source`.
8. The block-list renderer derives a source-ordered lightbox collection from the renderable visual sections in that message.
9. Each visual section either renders as an image block or falls back to a plain markdown block depending on feature flags and asset/render success.
10. When a file-backed image renders successfully, the renderer also shows a clickable/highlighted file link beneath or beside it:
   - inline image rows show the basename / final path segment
   - block image rows show the full source path
11. Clicking a rendered image or Mermaid diagram dispatches `WorkspaceAction::OpenLightbox` with that shared visual collection and the clicked section’s initial index.
12. Right-click copy on a rendered image or Mermaid diagram writes the section’s `markdown_source`.
13. Clicking a rendered file link continues to use the shared file-open flow, which means binary file labels open in the platform default app instead of `$EDITOR`.
14. Block-level copy and conversation export continue to emit markdown, using `markdown_source` for visual sections.
15. When the conversation is restored later, the same section payload renders again using the stored working-directory metadata and current flags.

## Risks and Mitigations

### Risk: selection expectations drift beyond the implementation
The current AI block architecture does not support unified mixed-content selection across renderer types. If the specs or implementation drift back toward that requirement, this feature would expand substantially.

Mitigation:
- keep the product and tech specs explicit that this change does not add a new cross-renderer selection model
- keep the scope focused on rendering, explicit copy affordances, and block-level copy/export correctness

### Risk: parser scope does not match full Markdown image semantics
Standard Markdown treats images as inline syntax, but our current parser only recognizes standalone-image lines, and this redesign needs a narrow same-line image-row case.

Mitigation:
- keep the new same-line parser limited to lines that consist entirely of parser-compatible image references separated only by whitespace
- do not treat arbitrary prose containing `![...]` as inline-renderable image content
- treat fuller CommonMark-style inline image support as follow-up parser work

### Risk: relative path resolution diverges between notebooks and AI blocks
The existing editor helper resolves relative image paths against a document path, while AI blocks need to resolve against the captured session working directory.

Mitigation:
- factor path resolution into a base-directory-oriented helper
- update notebook and AI-block callers to use it explicitly
- add native and WASM coverage

### Risk: regressions to existing section rendering
Adding two more section types to the current renderer could affect section indexing, link-detection offsets, or restored-block bookkeeping.

Mitigation:
- keep section ordering explicit in `AIAgentOutput::all_text` and render traversal
- follow existing per-section bookkeeping patterns for code/table sections
- add mixed-content coverage that includes ordinary code blocks and tables, not just images

## Testing and Validation

### Unit tests
- extend `app/src/ai/agent/util_tests.rs` with:
  - standalone image extraction
  - same-line inline image-run extraction
  - standalone multi-line block-image extraction
  - image + table + text ordering
  - Mermaid section extraction
- add block-list markdown copy tests covering:
  - image sections serialize to `![alt](source)`
  - Mermaid sections serialize to fenced mermaid markdown
- add native/WASM unit tests around relative path resolution against a base directory
- add focused rendering/helper coverage for:
  - section-index preservation across skipped empty text sections
  - lightbox index mapping in source order
  - inline-image basename labeling
  - visual width guard rails for invalid sizes
- add shared primitive regression tests for:
  - invalid image rect rejection
  - SVG intrinsic image sizing
  - single-line text hit testing respecting Y bounds

### Integration tests
- A reasonable first integration test is a single manual-observation test, `test_restored_ai_block_renders_mermaid_and_local_images`, in `crates/integration/src/test/agent_mode.rs`.
- That test would restore a synthetic `ConversationData` through `load_conversation_from_tasks`, set `InputContext.directory` to `crates/warpui_core/test_data`, and capture a real-display screenshot of one AI response that contains same-line local image markdown plus a Mermaid fence.
- It would be intentionally narrow: it would prove the restored AI block list can render both surfaces through the real conversation-hydration path without introducing protobuf fixtures or test-only UI hooks.
- Recommended follow-up integration coverage remains:
  - an AI response containing plain text, a linked code block, a markdown table, a relative-path image, and a Mermaid diagram
  - block-level copy serialization for mixed content
  - restored-conversation rendering and fallback behavior
  - disabled-flag fallback to raw Markdown
  - shared-lightbox initial selection and previous/next navigation in message source order

### Manual validation
- relative and absolute local image paths for PNG, JPEG, GIF, WebP, and SVG
- inline same-line image rows with path labels below each image
- grouped standalone block-image rows with thumbnail-plus-path content
- missing-file fallback
- unsupported-format fallback
- Mermaid render success and Mermaid render failure fallback in the new framed treatment
- click-to-expand from both rendered images and Mermaid diagrams into the shared lightbox
- previous/next keyboard and button navigation inside the lightbox
- right-click copy on both file-backed images and Mermaid diagrams
- block-level copy on a response containing text, code, table, image, and Mermaid content
- restored conversation rendering after reopening history
- WASM sanity pass to verify path handling and fallback behavior do not panic

## Follow-ups
- full CommonMark-style inline image support if we decide to expand `markdown_parser` beyond its current standalone-image behavior
- unified mixed-content selection across block-list renderer types
- consolidating `MarkdownImages` and `BlocklistMarkdownImages` if Warp later ships a broader app-wide markdown-image rollout
- richer image interactions such as open/save/zoom, if product wants them later