Initial public release of Warp.

Repo-Sync-Origin: warpdotdev/warp-internal@12af1d983b

crates/warp_completer/src/parsers/README.md

@@ -0,0 +1,276 @@
+# Basic parser
+
+This parser is a type driven, recursive descent parser. This work in progress guide goes over a friendly overview of the workings of it with examples. In some places `Did you know?` notes are included, they are good to know tips because the parser relies on them.
+
+# Steps
+1. Lex. - [What happens when we start the parse](#what-happens-when-we-start-the-parse)
+2. Lite Parse. - [Working with the tokens](#working-with-the-tokens)
+3. Type driven full parse. - [Full parse](#)
+
+## What happens when we start the parse?
+
+Let's say we are interested in parsing the input `warp --disable-telemetry`. Command calls generally have the form `cmd <arg1> <arg2> <argN>` where `<arg>`s are positional parameters. Let's use the primary functions of the parser and inspect. The first step we do is calling the tokenizer (the function `lex` here):
+
+```rust
+let input = "warp --disable_telemetry";
+let start_offset = 0;
+
+let (tokens, _) = lex(input, start_offset);
+println!("{:#?}", tokens);
+```
+
+The output is:
+
+```rust
+(
+    [
+        Token {
+            contents: Baseline(
+                "warp",
+            ),
+            span: Span {
+                start: 0,
+                end: 4,
+            },
+        },
+        Token {
+            contents: Space,
+            span: Span {
+                start: 4,
+                end: 5,
+            },
+        },
+        Token {
+            contents: Baseline(
+                "--disable-telemetry",
+            ),
+            span: Span {
+                start: 5,
+                end: 24,
+            },
+        },
+    ],
+    None,
+)
+```
+
+In effect, we receive here the input source *tokenized*. The `start_offset` is used by the parser to calculate the spans of each bare word. Inspecting a little more we can notice the tokens with a `span` field on them. The `Span` [struct type](../meta.rs) has the fields `start` and `end` number that are helpful for knowing where something is.
+
+#### Did you know? The `Span` struct type.
+
+Let's use the span numbers from the output above, and use the `Span` struct associated `slice` function. This function takes a string as input and will return a slice of it using the `Span`'s `start` and `end` values. Let's create three `Span`s with the numbers from the output above and slice each from the input string fed to the lexer.
+
+```rust
+let input = "warp --disable-telemetry";
+
+let word1 = Span::new(0,4);
+let word2 = Span::new(4,5);
+let word3 = Span::new(5,24);
+
+assert_eq!(word1.slice(input), "warp");
+assert_eq!(word2.slice(input), " ");
+assert_eq!(word3.slice(input), "--disable-telemetry");
+```
+
+## Working with the tokens
+
+The next step of the basic parser isn't all that different from lexing/parsing from traditional parsing. The task right now is to understand the boundaries of the tokens and get the general forms ready for a full parse. We are not interested here in doing anything more just yet since each of these tokens can be passed to commands that have no signature registered (*more on this later*). We call this step the `Lite` parse step.
+
+A very simplistic view of the grammar rules below:
+
+```
+LiteRootNode    := LiteGroup
+LiteGroup       := LitePipeline (';' LitePipeline)*
+LitePipeline    := LiteCommand ('|' LiteCommand)*
+LiteCommand     := argument+
+// (*more grammar later*)
+```
+
+These are represented as structs that basic parser generates, they are:
+
+```rust
+pub struct LiteRootNode {
+    pub groups: Vec<LiteGroup>,
+}
+
+pub struct LiteGroup {
+    pub pipelines: Vec<LitePipeline>,
+}
+
+pub struct LitePipeline {
+    pub commands: Vec<LiteCommand>,
+}
+
+pub struct LiteCommand {
+    // this is important!
+    pub parts: Vec<Spanned<String>>,
+    pub post_whitespace: Option<Span>,
+}
+```
+
+#### Did you know? The `Spanned<T>` generic struct.
+
+The `LiteCommand` has a `parts` field that holds a vector of `Spanned<String>`. We mentioned about the `Span` type before. Here, we talk about a generic `Spanned<T>` that allows wrapping any `T` with a `Span` value along with it. Here is the type along with a few examples using it's helper functions.
+
+```rust
+pub struct Spanned<T> {
+    pub span: Span,
+    pub item: T,
+}
+
+let example = Spanned { item: String::from("warp"), span: Span::new(0,4) };
+assert_eq!(example.item, "warp".to_string());
+assert_eq!(example.span, Span::new(0,4));
+
+let example = String::from("warp").spanned(Span::new(0,4));
+assert_eq!(example.item, "warp".to_string());
+assert_eq!(example.span, Span::new(0,4));
+
+let example = "warp -p --disable-telemetry";
+
+let full_span = Span::new(0, example.len());
+let first_flag_span = Span::new(5,7);
+
+assert_eq!(first_flag_span.slice(example), "-p");
+assert_eq!(first_flag_span.until(full_span), Span::new(5,27));
+assert_eq!(first_flag_span.until(full_span).slice(example), "-p --disable-telemetry");
+
+```
+
+This is useful because as soon as the `lite` parse step happens. We get as output everything we need with `spans` correctly calculated. Let's go ahead and do a lite parse (the function `parse_tokens`) to our original example using as input the tokens generated by the lexer when processing the input `warp --disable-telemetry`, like so:
+
+```rust
+let input = "warp --disable-telemetry";
+let start_offset = 0;
+
+let (tokens, _) = lex(input, start_offset);
+let (lite_node, _) = parse_tokens(tokens);
+
+let expected_word1 = String::from("warp").spanned(Span::new(0,4));
+let expected_word2 = String::from("--disable-telemetry").spanned(Span::new(5,24));
+
+assert_eq!(lite_node.groups[0].pipelines[0].commands[0].parts, vec![expected_word1, expected_word2]);
+assert_eq!(lite_node.groups[0].pipelines[0].commands.len(), 1);
+
+println!("{:#?}", lite_node);
+```
+
+We get a nice lite node:
+```rust
+LiteRootNode {
+    groups: [
+        LiteGroup {
+            pipelines: [
+                LitePipeline {
+                    commands: [
+                        LiteCommand {
+                            parts: [
+                                Spanned {
+                                    span: Span {
+                                        start: 0,
+                                        end: 4,
+                                    },
+                                    item: "warp",
+                                },
+                                Spanned {
+                                    span: Span {
+                                        start: 5,
+                                        end: 24,
+                                    },
+                                    item: "--disable-telemetry",
+                                },
+                            ],
+                            post_whitespace: None,
+                        },
+                    ],
+                },
+            ],
+        },
+    ],
+}
+```
+
+For more involved inputs (say commands separated by `|` and/or having `;`) the lite parser will effectively create the necessary `LitePipeline`s to express it, let's explore what happens if we lite parse the input `warp config-set --extension-path="/path/to/dir" ; echo $WARP_VAR"` (*two pipelines here due to the ; character*), like so:
+
+```rust
+let input = "warp config-set --extension-path=\"/path/to/dir\" ; echo $WARP_VAR";
+let start_offset = 0;
+
+let (tokens, _) = lex(input, start_offset);
+let (lite_node, _) = parse_tokens(tokens);
+
+println!("{:#?}", lite_node);
+```
+
+```rust
+LiteRootNode {
+    groups: [
+        LiteGroup {
+            pipelines: [
+                LitePipeline {
+                    commands: [
+                        LiteCommand {
+                            parts: [
+                                Spanned {
+                                    span: Span {
+                                        start: 0,
+                                        end: 4,
+                                    },
+                                    item: "warp",
+                                },
+                                Spanned {
+                                    span: Span {
+                                        start: 5,
+                                        end: 15,
+                                    },
+                                    item: "config-set",
+                                },
+                                Spanned {
+                                    span: Span {
+                                        start: 16,
+                                        end: 47,
+                                    },
+                                    item: "--extension-path=\"/path/to/dir\"",
+                                },
+                            ],
+                            post_whitespace: Some(
+                                Span {
+                                    start: 47,
+                                    end: 48,
+                                },
+                            ),
+                        },
+                    ],
+                },
+                LitePipeline {
+                    commands: [
+                        LiteCommand {
+                            parts: [
+                                Spanned {
+                                    span: Span {
+                                        start: 50,
+                                        end: 54,
+                                    },
+                                    item: "echo",
+                                },
+                                Spanned {
+                                    span: Span {
+                                        start: 55,
+                                        end: 64,
+                                    },
+                                    item: "$WARP_VAR",
+                                },
+                            ],
+                            post_whitespace: None,
+                        },
+                    ],
+                },
+            ],
+        },
+    ],
+}
+ ```
+
+ ## Type driven full parse
+
+TODO

crates/warp_completer/src/parsers/errors.rs

@@ -0,0 +1,108 @@
+#![allow(dead_code)]
+use crate::meta::{Span, Spanned, SpannedItem};
+use getset::Getters;
+
+#[derive(Getters, Debug, Clone, PartialEq, PartialOrd, Eq, Ord, Hash)]
+pub struct ParseError {
+    #[get = "pub"]
+    pub reason: ParseErrorReason,
+}
+
+impl ParseError {
+    pub fn unexpected_eof(expected: impl Into<String>, span: Span) -> ParseError {
+        ParseError {
+            reason: ParseErrorReason::Eof {
+                expected: expected.into(),
+                span,
+            },
+        }
+    }
+
+    pub fn extra_tokens(actual: Spanned<impl Into<String>>) -> ParseError {
+        let Spanned { span, item } = actual;
+
+        ParseError {
+            reason: ParseErrorReason::ExtraTokens {
+                actual: item.into().spanned(span),
+            },
+        }
+    }
+
+    pub fn mismatch(expected: impl Into<String>, actual: Spanned<impl Into<String>>) -> ParseError {
+        let Spanned { span, item } = actual;
+
+        ParseError {
+            reason: ParseErrorReason::Mismatch {
+                expected: expected.into(),
+                actual: item.into().spanned(span),
+            },
+        }
+    }
+
+    pub fn internal_error(message: Spanned<impl Into<String>>) -> ParseError {
+        ParseError {
+            reason: ParseErrorReason::InternalError {
+                message: message.item.into().spanned(message.span),
+            },
+        }
+    }
+
+    pub fn argument_error(command: Spanned<impl Into<String>>, kind: ArgumentError) -> ParseError {
+        ParseError {
+            reason: ParseErrorReason::ArgumentError {
+                command: command.item.into().spanned(command.span),
+                error: kind,
+            },
+        }
+    }
+}
+
+#[derive(Debug, Clone, PartialEq, PartialOrd, Eq, Ord, Hash)]
+pub enum ParseErrorReason {
+    Eof {
+        expected: String,
+        span: Span,
+    },
+    ExtraTokens {
+        actual: Spanned<String>,
+    },
+    Mismatch {
+        expected: String,
+        actual: Spanned<String>,
+    },
+    InternalError {
+        message: Spanned<String>,
+    },
+    ArgumentError {
+        command: Spanned<String>,
+        error: ArgumentError,
+    },
+}
+
+/// ArgumentError describes various ways that the parser could fail because of unexpected arguments.
+/// These errors correspond to problems that could be identified during expansion based on the
+/// signature of a command.
+#[derive(Debug, Eq, PartialEq, Clone, Ord, Hash, PartialOrd)]
+pub enum ArgumentError {
+    /// The command specified a mandatory flag, but it was missing.
+    MissingMandatoryFlag(String),
+    /// The command specified a mandatory positional argument, but it was missing.
+    MissingMandatoryPositional {
+        name: Option<String>,
+        positional_index: usize,
+    },
+    /// A flag was found, and it should have been followed by a value, but no value was found.
+    MissingValueForName {
+        /// The name of the flag/option
+        name: Spanned<String>,
+        /// The index of the missing argument for the option,
+        /// e.g. if the flag was -D arg1 arg2, then missing_arg_index=0 corresponds to arg1
+        missing_arg_index: usize,
+    },
+    /// An argument was found, but the command does not recognize it.
+    UnexpectedArgument(Spanned<String>),
+    /// An flag was found, but the command does not recognize it.
+    UnexpectedFlag(Spanned<String>),
+    /// A recognized flag had an invalid value.
+    InvalidValueForFlag(Spanned<String>),
+}

crates/warp_completer/src/parsers/hir/mod.rs

@@ -0,0 +1,297 @@
+use warp_command_signatures::{Argument, ArgumentType, GeneratorName, Template, TemplateType};
+
+use crate::meta::{HasSpan, Span, Spanned, SpannedItem};
+use crate::parsers::{ParsedExpression, ParsedToken};
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct ShellCommand {
+    /// The name goes up to the last subcommand.
+    /// For example, it would be "git checkout" if there are any args after "checkout", not "git".
+    pub name: String,
+    pub name_span: Span,
+    pub args: CommandCallInfo,
+}
+
+impl ShellCommand {
+    pub fn new(name: ParsedToken, name_span: Span, full_span: Span) -> Self {
+        let command_name = name.as_str().to_owned();
+        let spanned_command = ParsedExpression::new(Expression::Command, name).spanned(name_span);
+        Self {
+            name: command_name,
+            name_span,
+            args: CommandCallInfo::new(spanned_command, full_span),
+        }
+    }
+
+    /// Returns the last positional, if any, in the command.
+    pub fn last_positional(&self) -> Option<Spanned<&ParsedToken>> {
+        self.args
+            .positionals
+            .as_ref()
+            .and_then(|positionals| positionals.last())
+            .map(|expression| expression.item.value().spanned(expression.span))
+    }
+
+    /// Returns the name and span of the last named argument (if any) in the command.
+    pub fn last_named_argument(&self) -> Option<Spanned<NamedArgument<'_>>> {
+        self.args.flags.as_ref().and_then(|named_args| {
+            named_args
+                .iter()
+                .filter_map(|flag| match &flag.flag_type {
+                    FlagType::Argument {
+                        value: parsed_expression,
+                        ..
+                    } => Some(
+                        NamedArgument {
+                            name: flag.name.as_str(),
+                            parsed_token: parsed_expression.value(),
+                        }
+                        .spanned(parsed_expression.span),
+                    ),
+                    _ => None,
+                })
+                .fold(None, |acc, named_arg| match acc {
+                    None => Some(named_arg),
+                    Some(acc_named_arg) => {
+                        if named_arg.span.end() > acc_named_arg.span.end() {
+                            Some(named_arg)
+                        } else {
+                            Some(acc_named_arg)
+                        }
+                    }
+                })
+        })
+    }
+}
+
+#[derive(Debug)]
+pub struct NamedArgument<'a> {
+    pub name: &'a str,
+    pub parsed_token: &'a ParsedToken,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct ExternalCommand {
+    pub name: ParsedToken,
+    pub name_span: Span,
+    pub args: CommandCallInfo,
+}
+
+impl ExternalCommand {
+    pub fn new(name: ParsedToken, name_span: Span, full_span: Span) -> Self {
+        let command_name =
+            ParsedExpression::new(Expression::Literal, name.clone()).spanned(full_span);
+        Self {
+            name,
+            name_span,
+            args: CommandCallInfo::new(command_name, full_span),
+        }
+    }
+}
+
+impl HasSpan for ExternalCommand {
+    fn span(&self) -> Span {
+        self.name_span.until(self.args.span)
+    }
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum Command {
+    /// A command that was successfully parsed via a signature in the `CommandRegistry`.
+    Classified(ShellCommand),
+
+    /// A command that is not in the `CommandRegistry`.
+    Unclassified(ExternalCommand),
+}
+
+impl Command {
+    pub fn command_name_span(&self) -> Spanned<&str> {
+        match self {
+            Command::Classified(shell_command) => {
+                shell_command.name.as_str().spanned(shell_command.name_span)
+            }
+            Command::Unclassified(external_command) => external_command
+                .name
+                .as_str()
+                .spanned(external_command.name_span),
+        }
+    }
+
+    pub fn name_span(&self) -> Span {
+        match self {
+            Self::Classified(command) => command.name_span,
+            Self::Unclassified(command) => command.name_span,
+        }
+    }
+
+    pub fn last_token(&self) -> ParsedToken {
+        let command_call_info = match self {
+            Self::Classified(command) => &command.args,
+            Self::Unclassified(command) => &command.args,
+        };
+        if command_call_info.ending_whitespace.is_some() {
+            return ParsedToken::empty();
+        }
+        if let Some(positionals) = &command_call_info.positionals {
+            if let Some(last) = positionals.last() {
+                return last.parsed_token.clone();
+            }
+        }
+        command_call_info.command_name.parsed_token.clone()
+    }
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CommandCallInfo {
+    pub command_name: Spanned<ParsedExpression>,
+    pub positionals: Option<Vec<Spanned<ParsedExpression>>>,
+    pub flags: Option<Flags>,
+    /// Any additional whitespace at the end of the command.
+    pub ending_whitespace: Option<Span>,
+    pub span: Span,
+}
+
+impl CommandCallInfo {
+    pub fn new(head: Spanned<ParsedExpression>, span: Span) -> CommandCallInfo {
+        CommandCallInfo {
+            command_name: head,
+            positionals: None,
+            flags: None,
+            ending_whitespace: None,
+            span,
+        }
+    }
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ArgType {
+    File,
+    Folder,
+    Generator(GeneratorName),
+}
+
+impl ArgType {
+    /// Returns a list of argument types we should validate.
+    /// If there are any argument types that we haven't implemented validation for, returns empty vec
+    /// because we should assume the argument is valid.
+    pub fn get_arg_types_to_validate_from_arg_signature(arg_signature: &Argument) -> Vec<Self> {
+        let mut arg_types = vec![];
+        if arg_signature.skip_generator_validation {
+            return arg_types;
+        }
+        for argument_type in arg_signature.argument_types.iter() {
+            match argument_type {
+                ArgumentType::Template(Template {
+                    type_name: TemplateType::FilesAndFolders,
+                    ..
+                }) => arg_types.extend(vec![ArgType::File, ArgType::Folder]),
+                ArgumentType::Template(Template {
+                    type_name: TemplateType::Files { must_exist: true },
+                    ..
+                }) => arg_types.push(ArgType::File),
+                ArgumentType::Template(Template {
+                    type_name: TemplateType::Folders { must_exist: true },
+                    ..
+                }) => arg_types.push(ArgType::Folder),
+                ArgumentType::Generator(generator_name) => {
+                    arg_types.push(ArgType::Generator(generator_name.clone()))
+                }
+                // We don't want to validate against hard coded suggestions.
+                ArgumentType::Suggestion(_) => (),
+                // By the time we do arg validation, the command already has aliases expanded.
+                // We don't need to do anything here.
+                ArgumentType::Alias(_) => (),
+                _ => {
+                    // If we encounter an argument type that we haven't implemented validation for,
+                    // we should assume the argument is valid.
+                    // Return empty vec because there's no need to validate known argument types.
+                    return vec![];
+                }
+            }
+        }
+        arg_types
+    }
+}
+
+/// Identifies the expression type of a parsed token.
+#[derive(Debug, Clone, Eq, PartialEq)]
+pub enum Expression {
+    /// The expression is a literal value that will be interpreted by the command. This could be
+    /// a subcommand, an argument, or a flag that will be assigned meaning once we tie it to a
+    /// specific completion spec.
+    Literal,
+    /// The expression is an environment variable.
+    Variable,
+    /// The expression is a top-level command (such as `git`).
+    Command,
+    /// The expression is an argument that has been identified by the command signature that we have an
+    /// exhaustive list of arg types to validate against.
+    /// Contains the exhaustive list of arg types we should validate against, where at least one must pass validation.
+    /// Note that if the argument can be a type that we don't have a validation for, it will be parsed as Expression::Literal
+    /// instead.
+    /// For example, in `git checkout {arg}` the arg can be a file/folder/git branch. If we don't have a git branch validation,
+    /// the arg would be parsed as a Literal instead of ValidatableArgument([file, folder]) because the list is not
+    /// exhaustive.
+    ValidatableArgument(Vec<ArgType>),
+    /// We were not able to identify what part of the command is.
+    Unknown,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum FlagType {
+    /// The flag requires no argument, such as `git --help`
+    NoArgument,
+    /// The flag has an argument, such as  `git checkout -b {BRANCH_NAME}`.
+    Argument { value: Spanned<ParsedExpression> },
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct Flag {
+    /// The name of the flag.
+    pub name: String,
+    pub name_span: Span,
+    pub flag_type: FlagType,
+}
+
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct Flags {
+    pub(crate) flags: Vec<Flag>,
+}
+
+impl Flags {
+    pub fn new() -> Flags {
+        Default::default()
+    }
+
+    pub fn iter(&self) -> impl Iterator<Item = &Flag> {
+        self.flags.iter()
+    }
+
+    pub fn is_empty(&self) -> bool {
+        self.flags.is_empty()
+    }
+}
+
+impl Flags {
+    pub fn insert_flag_with_no_argument(&mut self, name: impl Into<String>, span: Span) {
+        let name = name.into();
+        self.flags.push(Flag {
+            name,
+            name_span: span,
+            flag_type: FlagType::NoArgument,
+        });
+    }
+
+    pub fn insert_flag_with_argument(
+        &mut self,
+        name: impl Into<String>,
+        flag_span: Span,
+        expr: Spanned<ParsedExpression>,
+    ) {
+        self.flags.push(Flag {
+            name: name.into(),
+            name_span: flag_span,
+            flag_type: FlagType::Argument { value: expr },
+        })
+    }
+}

crates/warp_completer/src/parsers/legacy.rs

@@ -0,0 +1,441 @@
+//! The "legacy" implementation of internal command parsing logic that depends on the legacy:
+//! command signature struct (`warp_command_signatures::Signature`).
+use itertools::Itertools;
+use warp_command_signatures::{DynamicCompletionData, IsArgumentOptional, Opt, Signature};
+
+use crate::{
+    completer::TopLevelCommandCaseSensitivity,
+    meta::{HasSpan, Span, Spanned, SpannedItem},
+    parsers::{
+        hir::Flags, parse_arg, parse_dollar_expr, ArgumentError, FlagArgumentsCardinality,
+        FlagSignature, ParsedExpression, ParsedToken,
+    },
+    signatures::CommandRegistry,
+};
+
+use super::{
+    hir::{Command, Expression, ShellCommand},
+    parse_unclassified_command, LiteCommand, ParseError,
+};
+
+#[derive(Clone, Copy)]
+/// A `Signature` (and its corresponding generator) contained at a given index.
+pub struct SignatureAtTokenIndex<'a> {
+    pub signature: &'a Signature,
+    pub dynamic_completion_data: Option<&'a DynamicCompletionData>,
+    pub token_index: usize,
+}
+
+impl<'a> SignatureAtTokenIndex<'a> {
+    pub fn new(
+        signature: &'a Signature,
+        dynamic_completion_data: Option<&'a DynamicCompletionData>,
+        index: usize,
+    ) -> Self {
+        SignatureAtTokenIndex {
+            signature,
+            dynamic_completion_data,
+            token_index: index,
+        }
+    }
+}
+
+/// Checks whether a `LiteCommand` is in the registry.
+pub(super) fn parse_command(
+    lite_cmd: &LiteCommand,
+    tokens: &[&str],
+    parser_scope: &CommandRegistry,
+    command_case_sensitivity: TopLevelCommandCaseSensitivity,
+) -> (Option<Command>, Option<ParseError>) {
+    let mut error: Option<ParseError> = None;
+
+    if lite_cmd.parts.is_empty() {
+        (None, None)
+    } else if let Some(found_signature) = parser_scope.signature_from_tokens(
+        tokens,
+        lite_cmd.post_whitespace.is_some(),
+        command_case_sensitivity,
+    ) {
+        let (internal_command, err) = parse_internal_command(
+            lite_cmd,
+            found_signature.signature,
+            found_signature.token_index,
+        );
+
+        error = error.or(err);
+        (Some(Command::Classified(internal_command)), error)
+    } else {
+        let (command, error) = parse_unclassified_command(lite_cmd);
+        (Some(command), error)
+    }
+}
+
+/// Does a full parse of an internal command using the lite-ly parse command as a starting point
+/// This main focus at this level is to understand what flags were passed in, what positional
+/// arguments were passed in, what rest arguments were passed in and to ensure that the basic
+/// requirements in terms of number of each were met.
+fn parse_internal_command(
+    lite_cmd: &LiteCommand,
+    signature: &Signature,
+    mut idx: usize,
+) -> (ShellCommand, Option<ParseError>) {
+    log::debug!("parsing internal command {lite_cmd:?}");
+
+    // This is a known internal command, so we need to work with the arguments and parse them according to the expected types
+    let (name, name_span) = (
+        lite_cmd.parts[0..(idx + 1)]
+            .iter()
+            .map(|x| x.item.clone())
+            .collect::<Vec<String>>()
+            .join(" "),
+        Span::new(
+            lite_cmd.parts[0].span.start(),
+            lite_cmd.parts[idx].span.end(),
+        ),
+    );
+
+    let mut internal_command = ShellCommand::new(ParsedToken(name), name_span, lite_cmd.span());
+    internal_command.args.flags = Some(Flags::new());
+    internal_command.args.ending_whitespace = lite_cmd.post_whitespace;
+
+    let mut current_positional = 0;
+    let mut named = Flags::new();
+    let mut positional = vec![];
+    let mut error = None;
+    idx += 1; // Start where the arguments begin
+
+    while idx < lite_cmd.parts.len() {
+        if lite_cmd.parts[idx].item.starts_with('-') && lite_cmd.parts[idx].item.len() > 1 {
+            let (named_types, err) =
+                get_flag_signature_spec(signature, &internal_command, &lite_cmd.parts[idx]);
+
+            if err.is_none() {
+                for FlagSignature {
+                    name: full_name,
+                    is_switch,
+                    arguments_cardinality,
+                    arguments,
+                } in named_types
+                {
+                    if is_switch {
+                        // Switch flag (without arguments)
+                        named.insert_flag_with_no_argument(full_name, lite_cmd.parts[idx].span);
+                    } else if let Some(eq_pos) = lite_cmd.parts[idx].item.find('=') {
+                        // Self-contained option (--key=value)
+                        let token = &lite_cmd.parts[idx];
+                        let value_offset = eq_pos + 1;
+
+                        let value_span =
+                            Span::new(token.span.start() + value_offset, token.span.end());
+                        let value = token.item[value_offset..].to_string().spanned(value_span);
+                        // We expect there to be exactly one arg in the case of a --key=value flag.
+                        let arg_signature = arguments.first();
+                        let (arg, err) = parse_arg(&value, arg_signature);
+                        // Flag name span covers only the flag name (before '=').
+                        let flag_name_span =
+                            Span::new(token.span.start(), token.span.start() + eq_pos);
+                        named.insert_flag_with_argument(full_name, flag_name_span, arg);
+
+                        error = error.or(err);
+                    } else if idx == lite_cmd.parts.len() - 1 {
+                        // Named argument with missing value
+                        error = error.or_else(|| {
+                            Some(ParseError::argument_error(
+                                lite_cmd.parts[0].clone(),
+                                ArgumentError::MissingValueForName {
+                                    name: full_name.spanned(lite_cmd.parts[idx].span),
+                                    missing_arg_index: 0,
+                                },
+                            ))
+                        });
+                    } else {
+                        // Named argument with following value(s).
+                        // Since an option can have multiple arguments (any of which can be variadic),
+                        // we should exhaust as many following args as possible.
+                        let flag_idx = idx;
+
+                        let end = match arguments_cardinality {
+                            FlagArgumentsCardinality::Variadic => lite_cmd.parts.len(),
+                            FlagArgumentsCardinality::Fixed(num_args) => flag_idx + num_args + 1,
+                        };
+                        let mut argument_idx = idx + 1;
+
+                        // Exhaust as many args as we expect but stop early if we see another option.
+                        // Note that since we are incrementing index again in the outer loop. Let's check
+                        // boundary on the NEXT token rather than the current token.
+                        while argument_idx < end.min(lite_cmd.parts.len())
+                            && !lite_cmd
+                                .parts
+                                .get(argument_idx)
+                                .is_some_and(|part| part.item.starts_with('-'))
+                        {
+                            let arg_signature_idx = argument_idx - flag_idx - 1;
+                            // Even though the completion spec technically allows for a variadic arg to not be the last arg,
+                            // it does not make sense and doesn't happen in practice, so we assume it's the last.
+                            let arg_signature = if arg_signature_idx >= arguments.len()
+                                && matches!(
+                                    arguments_cardinality,
+                                    FlagArgumentsCardinality::Variadic
+                                ) {
+                                arguments.last()
+                            } else {
+                                arguments.get(arg_signature_idx)
+                            };
+                            let (arg, err) =
+                                parse_arg(&lite_cmd.parts[argument_idx], arg_signature);
+                            named.insert_flag_with_argument(
+                                full_name.clone(),
+                                lite_cmd.parts[flag_idx].span,
+                                arg,
+                            );
+                            error = error.or(err);
+                            argument_idx += 1;
+                        }
+
+                        // If there were a fixed number of arguments for the option, make sure
+                        // they were all exhausted. Otherwise, we're missing an arg.
+                        if matches!(arguments_cardinality, FlagArgumentsCardinality::Fixed(_))
+                            && argument_idx != end
+                        {
+                            error = error.or_else(|| {
+                                Some(ParseError::argument_error(
+                                    lite_cmd.parts[0].clone(),
+                                    ArgumentError::MissingValueForName {
+                                        name: full_name.spanned(lite_cmd.parts[flag_idx].span),
+                                        missing_arg_index: argument_idx - flag_idx - 1,
+                                    },
+                                ))
+                            });
+                        }
+
+                        // argument_idx here is one index overshoot of the last argument
+                        // token. Set the current index to be the index of last argument.
+                        idx = argument_idx - 1;
+
+                        // We consumed the argument(s) for the option, so we should stop iterating the
+                        // possible matching flags. This case shouldn't happen as CLIs don't
+                        // generally support adding multiple flags with values in a single position
+                        break;
+                    }
+                }
+            } else {
+                positional.push(
+                    ParsedExpression::new(
+                        Expression::Unknown,
+                        ParsedToken(lite_cmd.parts[idx].item.clone()),
+                    )
+                    .spanned(lite_cmd.parts[idx].span),
+                );
+
+                error = error.or(err);
+            }
+        } else if !signature.arguments().is_empty()
+            && signature.arguments().len() > current_positional
+        {
+            let arg = {
+                let arg_signature = &signature.arguments()[current_positional];
+                let (expr, err) = parse_arg(&lite_cmd.parts[idx], Some(arg_signature));
+
+                error = error.or(err);
+                expr
+            };
+
+            positional.push(arg);
+            current_positional += 1;
+        } else if let Some(arg_signature) = signature.arguments().iter().rfind(|a| a.is_variadic())
+        {
+            let (arg, err) = parse_arg(&lite_cmd.parts[idx], Some(arg_signature));
+            error = error.or(err);
+
+            positional.push(arg);
+            current_positional += 1;
+        } else {
+            let expression = if lite_cmd.parts[idx].item.starts_with('$') {
+                parse_dollar_expr(&lite_cmd.parts[idx])
+            } else {
+                ParsedExpression::new(
+                    Expression::Unknown,
+                    ParsedToken(lite_cmd.parts[idx].item.clone()),
+                )
+                .spanned(lite_cmd.parts[idx].span)
+            };
+
+            positional.push(expression);
+
+            error = error.or_else(|| {
+                Some(ParseError::argument_error(
+                    lite_cmd.parts[0].clone(),
+                    ArgumentError::UnexpectedArgument(lite_cmd.parts[idx].clone()),
+                ))
+            });
+        }
+
+        idx += 1;
+    }
+
+    if let Some(arguments) = &signature.arguments {
+        // Count the required positional arguments and ensure these have been met
+        let mut required_arg_count = 0;
+        for positional_arg in arguments {
+            if positional_arg.optional == IsArgumentOptional::Required {
+                required_arg_count += 1;
+            }
+        }
+
+        if positional.len() < required_arg_count {
+            let arg = &arguments[positional.len()];
+            error = error.or_else(|| {
+                Some(ParseError::argument_error(
+                    lite_cmd.parts[0].clone(),
+                    ArgumentError::MissingMandatoryPositional {
+                        name: arg.display_name.clone(),
+                        positional_index: positional.len(),
+                    },
+                ))
+            });
+        }
+    }
+
+    if !named.is_empty() {
+        internal_command.args.flags = Some(named);
+    }
+
+    if !positional.is_empty() {
+        internal_command.args.positionals = Some(positional);
+    }
+
+    (internal_command, error)
+}
+
+impl From<&Opt> for FlagArgumentsCardinality {
+    fn from(option: &Opt) -> Self {
+        if option.arguments().iter().any(|arg| arg.is_variadic) {
+            FlagArgumentsCardinality::Variadic
+        } else {
+            FlagArgumentsCardinality::Fixed(
+                option
+                    .arguments()
+                    .iter()
+                    .filter(|arg| arg.is_required())
+                    .count(),
+            )
+        }
+    }
+}
+
+fn get_flag_signature_spec<'a>(
+    signature: &'a Signature,
+    cmd: &'a ShellCommand,
+    arg: &'a Spanned<String>,
+) -> (Vec<FlagSignature<'a>>, Option<ParseError>) {
+    // If it's not a flag, don't bother with it.
+    if !arg.item.starts_with('-') {
+        return (vec![], None);
+    }
+
+    let case_insensitive_flags = signature.parser_directives.always_case_insensitive;
+
+    let mut flag = arg.item.as_str();
+    let mut output = vec![];
+    let mut error = None;
+
+    if signature.parser_directives.flags_are_posix_noncompliant || flag.starts_with("--") {
+        if let Some((flag_name, _value)) = flag.split_once('=') {
+            flag = flag_name;
+        }
+
+        if signature.parser_directives.flags_match_unique_prefix {
+            let all_prefix_matches = signature
+                .options()
+                .iter()
+                .filter_map(|opt| {
+                    opt.names()
+                        .find(|name| {
+                            if case_insensitive_flags {
+                                name.to_lowercase()
+                                    .starts_with(flag.to_lowercase().as_str())
+                            } else {
+                                name.starts_with(flag)
+                            }
+                        })
+                        .map(|name| FlagSignature {
+                            name: name.to_owned(),
+                            is_switch: opt.is_switch(),
+                            arguments_cardinality: FlagArgumentsCardinality::from(opt),
+                            arguments: opt.arguments(),
+                        })
+                })
+                .exactly_one();
+            match all_prefix_matches {
+                Ok(matched_flag) => output.push(matched_flag),
+                Err(_) => {
+                    error = Some(ParseError::argument_error(
+                        cmd.name.to_string().spanned(cmd.name_span),
+                        ArgumentError::UnexpectedFlag(arg.clone()),
+                    ));
+                }
+            }
+        } else if let Some(option) = signature.options().iter().find(|option| {
+            if case_insensitive_flags {
+                option
+                    .names()
+                    .map(str::to_lowercase)
+                    .contains(&flag.to_lowercase())
+            } else {
+                option.names().contains(&flag)
+            }
+        }) {
+            output.push(FlagSignature {
+                name: flag.to_owned(),
+                is_switch: option.is_switch(),
+                arguments_cardinality: FlagArgumentsCardinality::from(option),
+                arguments: option.arguments(),
+            });
+        } else {
+            error = Some(ParseError::argument_error(
+                cmd.name.to_string().spanned(cmd.name_span),
+                ArgumentError::UnexpectedFlag(arg.clone()),
+            ));
+        }
+
+    // Short flag(s) expected. They might be grouped, e.g. -Alh
+    } else {
+        let mut starting_pos = arg.span.start() + 1;
+        // Loop over each letter as its own option, i.e. instead of "-Alh", process -A, -l, -h, etc.
+        for c in flag.trim_start_matches('-').chars() {
+            let ungrouped_flag = format!("-{c}");
+
+            if let Some(option) = signature.options().iter().find(|option| {
+                if case_insensitive_flags {
+                    option
+                        .names()
+                        .map(str::to_lowercase)
+                        .contains(&ungrouped_flag.to_lowercase())
+                } else {
+                    option.names().contains(&ungrouped_flag.as_str())
+                }
+            }) {
+                // TODO(alokedesai): Check if we should be using short or long here
+                output.push(FlagSignature {
+                    name: ungrouped_flag,
+                    is_switch: option.is_switch(),
+                    arguments_cardinality: FlagArgumentsCardinality::from(option),
+                    arguments: option.arguments(),
+                });
+            } else {
+                error = Some(ParseError::argument_error(
+                    cmd.name.to_string().spanned(cmd.name_span),
+                    ArgumentError::UnexpectedFlag(
+                        arg.item
+                            .clone()
+                            .spanned(Span::new(starting_pos, starting_pos + c.len_utf8())),
+                    ),
+                ));
+            }
+
+            starting_pos += c.len_utf8();
+        }
+    }
+
+    (output, error)
+}

crates/warp_completer/src/parsers/mod.rs

@@ -0,0 +1,334 @@
+#[cfg_attr(feature = "v2", path = "v2.rs")]
+#[cfg_attr(not(feature = "v2"), path = "legacy.rs")]
+mod imp;
+use imp::*;
+
+#[cfg(not(feature = "v2"))]
+pub use imp::SignatureAtTokenIndex;
+
+mod errors;
+pub use errors::{ArgumentError, ParseError, ParseErrorReason};
+pub mod hir;
+pub mod simple;
+
+use derive_new::new;
+use itertools::Itertools;
+use lazy_static::lazy_static;
+use regex::Regex;
+use warp_command_signatures::Argument;
+
+use crate::signatures::CommandRegistry;
+use crate::{
+    completer::TopLevelCommandCaseSensitivity,
+    meta::{HasSpan, Span, Spanned, SpannedItem},
+};
+
+use hir::{ArgType, Command, Expression, ExternalCommand};
+
+lazy_static! {
+    // Regex to test for a valid environment variable name, Environment variable names used by the
+    // utilities in the Shell and Utilities volume of IEEE Std 1003.1-2001 consist solely of
+    // upper or lowercase letters, digits, and the '_' (underscore) from the characters defined in
+    // Portable Character Set and do not begin with a digit.
+    static ref ENV_VAR_NAME_REGEX: Regex = Regex::new("^[$][a-zA-Z_][a-zA-Z0-9_]*$").unwrap();
+}
+
+type SpannedKeyValue = (Spanned<String>, Spanned<String>);
+
+/// A `LitePipeline` is a series of `LiteCommand`s separated by `|`.
+#[derive(Debug, Clone, new)]
+pub struct LitePipeline {
+    pub commands: Vec<LiteCommand>,
+}
+
+impl HasSpan for LitePipeline {
+    fn span(&self) -> Span {
+        Span::from_list(&self.commands)
+    }
+}
+
+/// A `LiteCommand` is a list of words that will get meaning when processed by
+/// the parser.
+#[derive(Debug, Default, Clone)]
+pub struct LiteCommand {
+    pub parts: Vec<Spanned<String>>,
+    /// The pieces of the command that ended in a whitespace.
+    pub post_whitespace: Option<Span>,
+}
+
+impl LiteCommand {
+    /// Returns `parts` joined together by single spaces.
+    pub fn joined_by_space(&self) -> String {
+        self.parts.iter().map(|s| s.as_str()).join(" ")
+    }
+}
+
+impl HasSpan for LiteCommand {
+    fn span(&self) -> Span {
+        let span = Span::from_list(&self.parts);
+        self.post_whitespace
+            .as_ref()
+            .map(|whitespace| span.until(whitespace))
+            .unwrap_or(span)
+    }
+}
+
+/// A `LiteGroup` is a series of `LitePipeline`s, separated by `;`.
+#[derive(Debug, Clone, new)]
+pub struct LiteGroup {
+    pub pipelines: Vec<LitePipeline>,
+}
+
+impl HasSpan for LiteGroup {
+    fn span(&self) -> Span {
+        Span::from_list(&self.pipelines)
+    }
+}
+
+/// A `LiteRootNode` is the root node of the parsed AST. Essentially a series of `LiteGroup`s,
+/// separated by newlines.
+#[derive(Debug, Clone, new)]
+pub struct LiteRootNode {
+    pub groups: Vec<LiteGroup>,
+}
+
+impl HasSpan for LiteRootNode {
+    fn span(&self) -> Span {
+        Span::from_list(&self.groups)
+    }
+}
+
+/// A command classified by its arguments, flags, etc. Optionally includes an error if there was
+/// a parse error while trying to classify the command.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct ClassifiedCommand {
+    pub command: Command,
+    /// Environment variables specified on the command line.
+    /// Each entry looks like "KEY=VALUE"
+    pub env_vars: Vec<String>,
+    pub error: Option<ParseError>,
+}
+
+/// Converts a `LiteCommand` into a `Command` that is annotated with its positional arguments,
+/// flags, etc based on corresponding completion specs.
+/// Modifies `tokens` to remove any environment variables.
+/// Returns none if the command is unable to
+/// be classified.
+pub fn classify_command(
+    lite_command: LiteCommand,
+    tokens: &mut Vec<&str>,
+    command_registry: &CommandRegistry,
+    command_case_sensitivity: TopLevelCommandCaseSensitivity,
+) -> Option<ClassifiedCommand> {
+    let mut error = None;
+
+    // TODO: potentially change to mutable reference (i.e. expand_shorthand_forms
+    // directly mutates lite_command)?
+    let (lite_command, vars, err) = expand_shorthand_forms(lite_command);
+    if error.is_none() {
+        error = err;
+    }
+    // Note that the caller of expand_shorthand_forms is responsible for updating
+    // their version of tokens, given the output of variables in environment
+    // variable assignments.
+    let env_vars = tokens
+        .drain(0..vars.len())
+        .map(|s| s.to_string())
+        .collect_vec();
+
+    let (command, err) = parse_command(
+        &lite_command,
+        tokens,
+        command_registry,
+        command_case_sensitivity,
+    );
+
+    if error.is_none() {
+        error = err;
+    }
+
+    command.map(|command| ClassifiedCommand {
+        command,
+        env_vars,
+        error,
+    })
+}
+
+fn parse_arg(
+    lite_arg: &Spanned<String>,
+    arg_signature: Option<&Argument>,
+) -> (Spanned<ParsedExpression>, Option<ParseError>) {
+    if lite_arg.item == "$" || ENV_VAR_NAME_REGEX.is_match(lite_arg.item.as_str()) {
+        return (parse_dollar_expr(lite_arg), None);
+    }
+    let arg_types_to_validate =
+        arg_signature.map(ArgType::get_arg_types_to_validate_from_arg_signature);
+    if let Some(arg_types_to_validate) = arg_types_to_validate {
+        if !arg_types_to_validate.is_empty() {
+            return (
+                ParsedExpression::new(
+                    Expression::ValidatableArgument(arg_types_to_validate),
+                    ParsedToken(lite_arg.item.clone()),
+                )
+                .spanned(lite_arg.span),
+                None,
+            );
+        }
+    }
+    (
+        ParsedExpression::new(Expression::Literal, ParsedToken(lite_arg.item.clone()))
+            .spanned(lite_arg.span),
+        None,
+    )
+}
+
+fn trim_quotes(input: &str) -> String {
+    let mut chars = input.chars();
+
+    match (chars.next(), chars.next_back()) {
+        (Some('\''), Some('\'')) => chars.collect(),
+        (Some('"'), Some('"')) => chars.collect(),
+        (Some('`'), Some('`')) => chars.collect(),
+        _ => input.to_string(),
+    }
+}
+
+/// Given a lite command and its tokens, remove the environment variable assignment token
+/// from it for completion generation. Note that the caller should mutate their tokens
+/// according to the vector of variables returned (remove them from the start)!
+///
+/// TODO: consolidate with [`Command::remove_leading_env_vars`].
+fn expand_shorthand_forms(
+    mut lite_command: LiteCommand,
+) -> (LiteCommand, Vec<SpannedKeyValue>, Option<ParseError>) {
+    let mut vars = Vec::new();
+    while !lite_command.parts.is_empty() {
+        let first_command_part = match lite_command.parts.first() {
+            Some(part) if part.item != "=" && part.contains('=') => part.clone(),
+            _ => return (lite_command, vars, None),
+        };
+
+        let assignment: Vec<_> = first_command_part.split('=').collect();
+        if assignment.len() != 2 {
+            return (
+                lite_command,
+                vars,
+                Some(ParseError::mismatch(
+                    "environment variable assignment",
+                    first_command_part.clone(),
+                )),
+            );
+        } else {
+            let original_span = first_command_part.span;
+            let (variable_name, value) = (assignment[0], trim_quotes(assignment[1]));
+
+            lite_command.parts.remove(0);
+            vars.push((
+                variable_name.to_string().spanned(original_span),
+                value.spanned(original_span),
+            ));
+        }
+    }
+    (lite_command, vars, None)
+}
+
+/// Parses a command that is not in the command registry.
+fn parse_unclassified_command(lite_cmd: &LiteCommand) -> (Command, Option<ParseError>) {
+    let mut error = None;
+
+    let external_name = lite_cmd.parts[0].clone().map(|v| trim_quotes(&v));
+
+    let mut external_command = ExternalCommand::new(
+        ParsedToken(external_name.item),
+        external_name.span,
+        lite_cmd.span(),
+    );
+    external_command.args.ending_whitespace = lite_cmd.post_whitespace;
+
+    let num_parts = lite_cmd.parts.len() - 1;
+    let mut args = Vec::with_capacity(num_parts);
+
+    if lite_cmd.parts.len() > 1 {
+        for lite_arg in &lite_cmd.parts[1..] {
+            let (expr, err) = parse_arg(lite_arg, None);
+            if error.is_none() {
+                error = err;
+            }
+            args.push(expr);
+        }
+
+        external_command.args.positionals = Some(args);
+    }
+
+    (Command::Unclassified(external_command), error)
+}
+
+/// The number of arguments for an option.
+#[derive(Clone, Debug)]
+enum FlagArgumentsCardinality {
+    /// The option has exactly k required args.
+    Fixed(usize),
+    /// The option has some arg that is variadic
+    /// (it might also have some number of non-variadic args).
+    Variadic,
+}
+
+#[derive(Clone, Debug)]
+struct FlagSignature<'a> {
+    name: String,
+    is_switch: bool,
+    arguments_cardinality: FlagArgumentsCardinality,
+    arguments: &'a [Argument],
+}
+
+fn parse_dollar_expr(lite_arg: &Spanned<String>) -> Spanned<ParsedExpression> {
+    ParsedExpression::new(Expression::Variable, ParsedToken(lite_arg.item.clone()))
+        .spanned(lite_arg.span)
+}
+
+/// Newtype denoting a token that has been parsed by the parser. For example, `foo\ bar` would have
+/// would result in a parsed token value of `foo bar`.
+#[derive(Debug, Clone, Eq, PartialEq)]
+pub struct ParsedToken(String);
+
+impl ParsedToken {
+    #[cfg(feature = "test-util")]
+    pub fn new(value: impl Into<String>) -> Self {
+        Self(value.into())
+    }
+
+    pub fn empty() -> Self {
+        Self("".into())
+    }
+
+    pub fn as_str(&self) -> &str {
+        self.0.as_str()
+    }
+}
+
+/// An expression with the parsed value that the expression corresponds to.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct ParsedExpression {
+    expression: Expression,
+    parsed_token: ParsedToken,
+}
+
+impl ParsedExpression {
+    pub fn new(expression: Expression, parsed_token: ParsedToken) -> Self {
+        Self {
+            expression,
+            parsed_token,
+        }
+    }
+
+    pub fn expression(&self) -> &Expression {
+        &self.expression
+    }
+
+    pub fn value(&self) -> &ParsedToken {
+        &self.parsed_token
+    }
+}
+
+#[cfg(test)]
+mod test;

crates/warp_completer/src/parsers/simple/convert.rs

@@ -0,0 +1,57 @@
+use super::{Command, Part};
+use crate::meta::{Span, Spanned};
+use crate::parsers::{LiteCommand, LiteGroup, LitePipeline, LiteRootNode};
+use std::fmt;
+
+impl From<Spanned<Command>> for LiteCommand {
+    fn from(command: Spanned<Command>) -> Self {
+        let post_whitespace = command.item.parts.last().and_then(|part| {
+            if part.span.end() < command.span.end() {
+                Some(Span::new(part.span.end(), command.span.end()))
+            } else {
+                None
+            }
+        });
+
+        LiteCommand {
+            parts: command.item.parts.into_iter().map(Into::into).collect(),
+            post_whitespace,
+        }
+    }
+}
+
+impl From<Spanned<Part>> for Spanned<String> {
+    fn from(spanned: Spanned<Part>) -> Spanned<String> {
+        spanned.map(|part| part.to_string())
+    }
+}
+
+impl From<Option<LiteCommand>> for LiteRootNode {
+    fn from(command: Option<LiteCommand>) -> Self {
+        LiteRootNode {
+            groups: vec![LiteGroup {
+                pipelines: vec![LitePipeline {
+                    commands: command.into_iter().collect(),
+                }],
+            }],
+        }
+    }
+}
+
+impl fmt::Display for Part {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self {
+            Part::Literal(value) => f.write_str(value.as_str()),
+            Part::OpenSubshell(_) | Part::ClosedSubshell(_) => {
+                // Since we aren't evaluating the subshell, include a placeholder value
+                f.write_str("$(...)")
+            }
+            Part::Concatenated(parts) => {
+                for part in parts {
+                    part.item.fmt(f)?;
+                }
+                Ok(())
+            }
+        }
+    }
+}

crates/warp_completer/src/parsers/simple/iter.rs

@@ -0,0 +1,102 @@
+use std::iter::Peekable;
+
+use super::token::Token;
+use crate::meta::Spanned;
+
+/// Adapter for tracking the parser's current position in the input and allowing for double peek
+pub struct ParserInput<I>
+where
+    I: IntoIterator,
+{
+    iter: Peekable<I::IntoIter>,
+    peeked: Option<I::Item>,
+    pos: usize,
+    offset: usize,
+}
+
+impl<'a, I> ParserInput<I>
+where
+    I: IntoIterator<Item = Spanned<Token<'a>>>,
+{
+    pub fn new(input: I) -> Self {
+        let iter = input.into_iter().peekable();
+        ParserInput {
+            iter,
+            peeked: None,
+            pos: 0,
+            offset: 0,
+        }
+    }
+
+    /// Get the next token from the input
+    pub fn next(&mut self) -> Option<Token<'a>> {
+        let next = self.peeked.take().or_else(|| self.iter.next());
+
+        match next {
+            Some(token) => {
+                self.pos = self.offset + token.span.end();
+                Some(token.item)
+            }
+            None => None,
+        }
+    }
+
+    /// Get the next token from the input, including the span
+    fn next_with_span(&mut self) -> Option<Spanned<Token<'a>>> {
+        let next = self.peeked.take().or_else(|| self.iter.next());
+
+        if let Some(token) = &next {
+            self.pos = self.offset + token.span.end();
+        }
+
+        next
+    }
+
+    /// Get a reference to the next token without consuming it
+    pub fn peek(&mut self) -> Option<&Token<'a>> {
+        match &self.peeked {
+            Some(value) => Some(&value.item),
+            None => self.iter.peek().map(|t| &t.item),
+        }
+    }
+
+    /// Get a reference to the token after the next, without consuming either
+    pub fn peekpeek(&mut self) -> Option<&Token<'a>> {
+        if self.peeked.is_none() {
+            self.peeked = self.iter.next();
+        }
+
+        self.iter.peek().map(|t| &t.item)
+    }
+
+    /// Get the position where we are currently, immediately after the last element that was read
+    pub fn pos(&self) -> usize {
+        self.pos
+    }
+
+    /// Create a ParserInput that will yield tokens until a backtick is found and then stop,
+    /// preserving the position.
+    ///
+    /// Note: This will buffer all of the tokens until the next backtick
+    pub fn until_backtick(&mut self) -> ParserInput<Vec<Spanned<Token<'a>>>> {
+        let mut buffer = Vec::new();
+        let start = self.pos;
+
+        while let Some(token) = self.peek() {
+            match token {
+                Token::Backtick => break,
+                _ => {
+                    // Safety: We are peeking first so the next token will always exist
+                    buffer.push(self.next_with_span().unwrap());
+                }
+            }
+        }
+
+        ParserInput {
+            iter: buffer.into_iter().peekable(),
+            peeked: None,
+            pos: start,
+            offset: self.offset,
+        }
+    }
+}

crates/warp_completer/src/parsers/simple/lexer.rs

@@ -0,0 +1,218 @@
+use super::{token::Token, EscapeChar};
+use crate::meta::{Span, Spanned, SpannedItem};
+
+/// Iterator for converting a string into a series of Tokens
+///
+/// This step is very naive, it does not attempt to understand the various contexts in which tokens
+/// can appear (e.g. within double or single quotes, within subshells nested within double quotes).
+/// The parser will track all of the necessary state in order to properly interpret each token
+/// given the context.
+pub struct Lexer<'a> {
+    /// The source string that we are tokenizing
+    source: &'a str,
+    escape_char: EscapeChar,
+    /// The current byte index in the source
+    pos: usize,
+    /// Processed character that we have seen but not yet yielded
+    queued: Option<Classified<'a>>,
+    /// Whether to consider quotes as literals in the lexer.
+    parse_quotes_as_literals: bool,
+}
+
+/// The classification of a character into a known Token, Raw character value (for Literal tokens)
+/// or an Escaped token (via `\`). This intermediate classification helps to process Literal and
+/// Escaped tokens properly without needing to handle the specific parser context.
+enum Classified<'a> {
+    Token(Spanned<Token<'a>>),
+    Raw(Spanned<char>),
+    Escaped {
+        escape_char: Spanned<Token<'a>>,
+        next_token: Option<Spanned<Token<'a>>>,
+    },
+}
+
+impl From<Spanned<char>> for Classified<'_> {
+    fn from(chr: Spanned<char>) -> Self {
+        Classified::Raw(chr)
+    }
+}
+
+impl<'a> From<Spanned<Token<'a>>> for Classified<'a> {
+    fn from(tok: Spanned<Token<'a>>) -> Self {
+        Classified::Token(tok)
+    }
+}
+
+impl<'a> Lexer<'a> {
+    pub fn new(source: &'a str, escape_char: EscapeChar, parse_quotes_as_literals: bool) -> Self {
+        Lexer {
+            source,
+            escape_char,
+            pos: 0,
+            queued: None,
+            parse_quotes_as_literals,
+        }
+    }
+
+    /// Advance the underlying iterator one step, returning the next character and its span
+    fn step(&mut self) -> Option<(Span, char)> {
+        let (span, chr) = self.peek()?;
+
+        self.pos = span.end();
+        Some((span, chr))
+    }
+
+    /// Peek at the next character without consuming it, returning the span as well
+    fn peek(&mut self) -> Option<(Span, char)> {
+        let start = self.pos;
+        let chr = self.source[start..].chars().next()?;
+        let end = start + chr.len_utf8();
+
+        let span = Span::new(start, end);
+        Some((span, chr))
+    }
+
+    /// Classify the next character as either a token, escaped token, or raw value.
+    ///
+    /// Note that some tokens will consume multiple characters.
+    fn classify_next(&mut self) -> Option<Classified<'a>> {
+        if self.queued.is_some() {
+            return self.queued.take();
+        }
+
+        let (span, chr) = self.step()?;
+
+        Some(match chr {
+            '|' => {
+                if let Some((next, '|')) = self.peek() {
+                    self.next();
+                    Token::LogicalOr.spanned(span.until(next)).into()
+                } else {
+                    Token::Pipe.spanned(span).into()
+                }
+            }
+            '&' => {
+                if let Some((next, '&')) = self.peek() {
+                    self.next();
+                    Token::LogicalAnd.spanned(span.until(next)).into()
+                } else {
+                    Token::Ampersand.spanned(span).into()
+                }
+            }
+            ';' => Token::Semicolon.spanned(span).into(),
+            '\n' => Token::Newline.spanned(span).into(),
+            '(' => Token::OpenParen.spanned(span).into(),
+            ')' => Token::CloseParen.spanned(span).into(),
+            '{' => Token::OpenCurly.spanned(span).into(),
+            '}' => Token::CloseCurly.spanned(span).into(),
+            '$' => Token::Dollar.spanned(span).into(),
+            '\'' if !self.parse_quotes_as_literals => Token::SingleQuote.spanned(span).into(),
+            '"' if !self.parse_quotes_as_literals => Token::DoubleQuote.spanned(span).into(),
+            '<' => Token::RedirectInput.spanned(span).into(),
+            '>' => Token::RedirectOutput.spanned(span).into(),
+            '\\' | '`' if self.escape_char.is_char(chr) => {
+                // Consume the following character as a token by itself
+                // Note: Since the internal Lexer uses spans relative to its slice, so we need
+                // to adjust them to match our positions
+                let pos_adjust = self.pos;
+                let next_token = self
+                    .step()
+                    .and_then(|(span, _)| {
+                        Lexer::new(
+                            span.slice(self.source),
+                            self.escape_char,
+                            self.parse_quotes_as_literals,
+                        )
+                        .next()
+                    })
+                    .map(|token| {
+                        let new_start = token.span.start() + pos_adjust;
+                        let new_end = token.span.end() + pos_adjust;
+
+                        token.item.spanned(Span::new(new_start, new_end))
+                    });
+                Classified::Escaped {
+                    escape_char: Token::EscapeChar(span.slice(self.source)).spanned(span),
+                    next_token,
+                }
+            }
+            '`' => Token::Backtick.spanned(span).into(),
+            c if c.is_whitespace() => {
+                // Note: We handle newlines earlier in the match, so this excludes newlines
+                let mut span = span;
+
+                while let Some((next_span, next_chr)) = self.peek() {
+                    if next_chr.is_whitespace() && next_chr != '\n' {
+                        span = span.until(next_span);
+                        self.step();
+                    } else {
+                        break;
+                    }
+                }
+
+                Token::Whitespace(span.slice(self.source))
+                    .spanned(span)
+                    .into()
+            }
+            c => c.spanned(span).into(),
+        })
+    }
+}
+
+impl<'a> Iterator for Lexer<'a> {
+    type Item = Spanned<Token<'a>>;
+
+    fn next(&mut self) -> Option<Self::Item> {
+        match self.classify_next() {
+            None => None,
+            Some(Classified::Token(t)) => Some(t),
+            Some(Classified::Escaped {
+                escape_char,
+                next_token,
+            }) => {
+                // Note: For escaped values, we still need to include a backslash token, because
+                // there are some contexts (e.g. within a single-quoted value) where the backslash
+                // can be interpreted as a literal value. It's left up to the parser to determine
+                // how to handle the backslash.
+                self.queued = next_token.map(Into::into);
+                Some(escape_char)
+            }
+            Some(Classified::Raw(chr)) => {
+                let mut span = chr.span;
+
+                loop {
+                    match self.classify_next() {
+                        Some(Classified::Raw(chr)) => {
+                            span = span.until(chr.span);
+                        }
+                        other => {
+                            self.queued = other;
+                            break;
+                        }
+                    }
+                }
+
+                let value = span.slice(self.source);
+                Some(Token::Literal(value).spanned(span))
+            }
+        }
+    }
+
+    fn size_hint(&self) -> (usize, Option<usize>) {
+        let lower = match &self.queued {
+            Some(_) => 1,
+            None => 0,
+        };
+        let upper = self.source.len() - self.pos;
+
+        if upper < lower {
+            (lower, None)
+        } else {
+            (lower, Some(upper))
+        }
+    }
+}
+
+#[cfg(test)]
+#[path = "lexer_test.rs"]
+mod tests;

crates/warp_completer/src/parsers/simple/lexer_test.rs

@@ -0,0 +1,218 @@
+use super::*;
+
+#[test]
+fn test_lexer() {
+    let source = r#"ls | rm -rf || touch 'hello.txt' &
+cat "Hello $(ls -la)" && echo `ps \`;  {echo Goodbye😀}"#;
+    let tokens: Vec<_> = Lexer::new(source, EscapeChar::Backslash, false)
+        .map(|t| t.item)
+        .collect();
+
+    assert_eq!(
+        tokens,
+        [
+            Token::Literal("ls"),
+            Token::Whitespace(" "),
+            Token::Pipe,
+            Token::Whitespace(" "),
+            Token::Literal("rm"),
+            Token::Whitespace(" "),
+            Token::Literal("-rf"),
+            Token::Whitespace(" "),
+            Token::LogicalOr,
+            Token::Whitespace(" "),
+            Token::Literal("touch"),
+            Token::Whitespace(" "),
+            Token::SingleQuote,
+            Token::Literal("hello.txt"),
+            Token::SingleQuote,
+            Token::Whitespace(" "),
+            Token::Ampersand,
+            Token::Newline,
+            Token::Literal("cat"),
+            Token::Whitespace(" "),
+            Token::DoubleQuote,
+            Token::Literal("Hello"),
+            Token::Whitespace(" "),
+            Token::Dollar,
+            Token::OpenParen,
+            Token::Literal("ls"),
+            Token::Whitespace(" "),
+            Token::Literal("-la"),
+            Token::CloseParen,
+            Token::DoubleQuote,
+            Token::Whitespace(" "),
+            Token::LogicalAnd,
+            Token::Whitespace(" "),
+            Token::Literal("echo"),
+            Token::Whitespace(" "),
+            Token::Backtick,
+            Token::Literal("ps"),
+            Token::Whitespace(" "),
+            Token::EscapeChar("\\"),
+            Token::Backtick,
+            Token::Semicolon,
+            Token::Whitespace("  "),
+            Token::OpenCurly,
+            Token::Literal("echo"),
+            Token::Whitespace(" "),
+            Token::Literal("Goodbye😀"),
+            Token::CloseCurly,
+        ]
+    );
+}
+
+#[test]
+fn test_spans() {
+    let source = "ls -la && echo Hello' World'$(cat  😀.txt";
+    let spans: Vec<_> = Lexer::new(source, EscapeChar::Backslash, false)
+        .map(|t| t.span)
+        .collect();
+
+    assert_eq!(
+        spans,
+        [
+            Span::new(0, 2),   // ls
+            Span::new(2, 3),   // space
+            Span::new(3, 6),   // -la
+            Span::new(6, 7),   // space
+            Span::new(7, 9),   // &&
+            Span::new(9, 10),  // space
+            Span::new(10, 14), // echo
+            Span::new(14, 15), // space
+            Span::new(15, 20), // Hello
+            Span::new(20, 21), // '
+            Span::new(21, 22), // space
+            Span::new(22, 27), // World
+            Span::new(27, 28), // '
+            Span::new(28, 29), // $
+            Span::new(29, 30), // (
+            Span::new(30, 33), // cat
+            Span::new(33, 35), // double space
+            Span::new(35, 43), // 😀.txt (😀 is 4 bytes long)
+        ]
+    );
+}
+
+#[test]
+fn test_escaped_tokens() {
+    let source = r"\\\||\&&\\&&||";
+    let tokens: Vec<_> = Lexer::new(source, EscapeChar::Backslash, false)
+        .map(|t| t.item)
+        .collect();
+
+    assert_eq!(
+        tokens,
+        [
+            Token::EscapeChar("\\"),
+            Token::EscapeChar("\\"),
+            Token::EscapeChar("\\"),
+            Token::Pipe,
+            Token::Pipe,
+            Token::EscapeChar("\\"),
+            Token::Ampersand,
+            Token::Ampersand,
+            Token::EscapeChar("\\"),
+            Token::EscapeChar("\\"),
+            Token::LogicalAnd,
+            Token::LogicalOr,
+        ]
+    )
+}
+
+#[test]
+fn test_escaped_token_spans() {
+    let source = r"\\\||\&&\\&&||";
+    let spans: Vec<_> = Lexer::new(source, EscapeChar::Backslash, false)
+        .map(|t| t.span)
+        .collect();
+
+    assert_eq!(
+        spans,
+        [
+            Span::new(0, 1),   // \
+            Span::new(1, 2),   // \
+            Span::new(2, 3),   // \
+            Span::new(3, 4),   // |
+            Span::new(4, 5),   // |
+            Span::new(5, 6),   // \
+            Span::new(6, 7),   // &
+            Span::new(7, 8),   // &
+            Span::new(8, 9),   // \
+            Span::new(9, 10),  // \
+            Span::new(10, 12), // &&
+            Span::new(12, 14), // ||
+        ]
+    );
+}
+
+#[test]
+fn test_multiple_whitespace() {
+    let source = " \t  |\t  ";
+    let tokens: Vec<_> = Lexer::new(source, EscapeChar::Backslash, false)
+        .map(|t| t.item)
+        .collect();
+
+    assert_eq!(
+        tokens,
+        [
+            Token::Whitespace(" \t  "),
+            Token::Pipe,
+            Token::Whitespace("\t  "),
+        ]
+    )
+}
+
+#[test]
+fn test_backtick_escape_char() {
+    let source = r#"& "$HOME\Downloads\Warp` Setup.exe" /SP- /SILENT `t`"#;
+    let tokens: Vec<_> = Lexer::new(source, EscapeChar::Backtick, false)
+        .map(|t| (t.item, t.span))
+        .collect();
+
+    assert_eq!(
+        tokens,
+        [
+            (Token::Ampersand, Span::new(0, 1)),
+            (Token::Whitespace(" "), Span::new(1, 2)),
+            (Token::DoubleQuote, Span::new(2, 3)),
+            (Token::Dollar, Span::new(3, 4)),
+            (Token::Literal(r"HOME\Downloads\Warp"), Span::new(4, 23)),
+            (Token::EscapeChar("`"), Span::new(23, 24)),
+            (Token::Whitespace(" "), Span::new(24, 25)),
+            (Token::Literal("Setup.exe"), Span::new(25, 34)),
+            (Token::DoubleQuote, Span::new(34, 35)),
+            (Token::Whitespace(" "), Span::new(35, 36)),
+            (Token::Literal("/SP-"), Span::new(36, 40)),
+            (Token::Whitespace(" "), Span::new(40, 41)),
+            (Token::Literal("/SILENT"), Span::new(41, 48)),
+            (Token::Whitespace(" "), Span::new(48, 49)),
+            (Token::EscapeChar("`"), Span::new(49, 50)),
+            (Token::Literal("t"), Span::new(50, 51)),
+            (Token::EscapeChar("`"), Span::new(51, 52)),
+        ]
+    )
+}
+
+#[test]
+fn test_single_quote_as_literals() {
+    let source = r#"I'd like to edit app/src"#;
+    let tokens: Vec<_> = Lexer::new(source, EscapeChar::Backslash, true)
+        .map(|t| (t.item, t.span))
+        .collect();
+
+    assert_eq!(
+        tokens,
+        [
+            (Token::Literal("I'd"), Span::new(0, 3)),
+            (Token::Whitespace(" "), Span::new(3, 4)),
+            (Token::Literal("like"), Span::new(4, 8)),
+            (Token::Whitespace(" "), Span::new(8, 9)),
+            (Token::Literal("to"), Span::new(9, 11)),
+            (Token::Whitespace(" "), Span::new(11, 12)),
+            (Token::Literal("edit"), Span::new(12, 16)),
+            (Token::Whitespace(" "), Span::new(16, 17)),
+            (Token::Literal("app/src"), Span::new(17, 24)),
+        ]
+    )
+}

crates/warp_completer/src/parsers/simple/mod.rs

@@ -0,0 +1,266 @@
+use crate::meta::{Spanned, SpannedItem};
+
+mod convert;
+mod iter;
+mod lexer;
+mod parser;
+mod token;
+
+use crate::parsers::LiteCommand;
+use lexer::Lexer;
+use parser::Parser;
+use warp_util::path::EscapeChar;
+
+use string_offset::ByteOffset;
+
+/// Parse the input and return the last unclosed command to complete on using the completions
+/// infrastructure.
+///
+/// To make sure we are completing the correct thing, this will pull off the last unclosed command
+/// from the parsed input
+pub fn parse_for_completions<S: AsRef<str>>(
+    source: S,
+    escape_char: EscapeChar,
+    parse_quotes_as_literals: bool,
+) -> Option<LiteCommand> {
+    let parser = Parser::new(Lexer::new(
+        source.as_ref(),
+        escape_char,
+        parse_quotes_as_literals,
+    ));
+    let commands = parser.parse().commands;
+    commands
+        .into_iter()
+        .next_back()
+        .map(last_unclosed_command)
+        .map(Into::into)
+}
+
+/// Returns the name of the top-level command in `source`.
+///
+/// For example, if the source is "PAGER=0 git log", the top-level command is "git".
+/// If there are multiple top-level commands (e.g. `ls && git diff`), the first
+/// one is returned.
+pub fn top_level_command<S: AsRef<str>>(source: S, escape_char: EscapeChar) -> Option<String> {
+    let parser = Parser::new(Lexer::new(source.as_ref(), escape_char, false));
+    let mut command = parser.parse().commands.into_iter().next()?;
+    command.item.remove_leading_env_vars();
+
+    match command
+        .parts
+        .first()
+        .map(|p: &Spanned<Part>| p.item.clone())
+    {
+        Some(Part::Literal(p)) => Some(p),
+        _ => None,
+    }
+}
+
+/// Parses the command to retrieve the command at the given cursor pos.
+/// This is needed for command x-ray since we need to run the completion engine on tokens
+/// that might be in the middle of a command or subcommand.
+///
+/// For example, if our command was `cd ~/Desktop $(cd ~/foo)` and our cursor
+/// was at `/foo` then we want the subcommand `cd ~/foo` instead of the parent command.
+pub fn command_at_cursor_position<S: AsRef<str>>(
+    source: S,
+    escape_char: EscapeChar,
+    pos: ByteOffset,
+) -> Option<LiteCommand> {
+    let parser = Parser::new(Lexer::new(source.as_ref(), escape_char, false));
+    let commands = parser.parse().commands;
+    command_at_cursor(commands, pos).map(Into::into)
+}
+
+/// Parses the string to retrieve all commands (separated into iterator items).
+/// This is needed for error underlining since we need to determine whether each
+/// command is valid or not.
+///
+/// For example, if our command was `git commit && git log`, we would return
+/// an iterator of 2 items - the parsed commands for "git commit" and "git log".
+pub fn all_parsed_commands<S: AsRef<str>>(
+    source: S,
+    escape_char: EscapeChar,
+) -> impl Iterator<Item = LiteCommand> {
+    let parser = Parser::new(Lexer::new(source.as_ref(), escape_char, false));
+    parser.parse().commands.into_iter().map(|mut cmd| {
+        cmd.item.remove_leading_env_vars();
+        cmd.into()
+    })
+}
+
+/// Given a `command` string, returns:
+/// 1. the subcommands that make it up, including the recomposed commands at each level of nesting.
+///    For example, given "ls $(foo | echo)", this API returns ["foo", "echo", "foo | echo", "ls $(foo | echo)"]
+/// 2. whether or not the `command` included any redirection operators (i.e. '>' or '<')
+pub fn decompose_command(command: &str, escape_char: EscapeChar) -> (Vec<String>, bool) {
+    let parser = Parser::new(Lexer::new(command, escape_char, false));
+    let res = parser.parse();
+    let (commands, contains_redirection) = (res.commands, res.contains_redirection);
+
+    (
+        commands
+            .into_iter()
+            .flat_map(|cmd| cmd.item.decompose(command))
+            .collect(),
+        contains_redirection,
+    )
+}
+
+/// Retrieves the smallest complete command at a given pos.
+///
+/// Ex: For the dummy command "git status $(git stash) && git checkout main" -
+/// pos 11 would return the subcommand corresponding to "git stash" and
+/// pos 27 would return the command corresponding to "git checkout main"
+/// Returns None if the pos is not in the range of the command.
+fn command_at_cursor(commands: Vec<Spanned<Command>>, pos: ByteOffset) -> Option<Spanned<Command>> {
+    // First search for the in range top level command
+    if let Some(in_range_command) = commands.into_iter().find(|command| {
+        command.span.start() <= pos.as_usize() && command.span.end() >= pos.as_usize()
+    }) {
+        if let Some(in_range_part) = in_range_command
+            .item
+            .parts
+            .iter()
+            .find(|part| part.span.start() <= pos.as_usize() && part.span.end() >= pos.as_usize())
+            .cloned()
+        {
+            match in_range_part.item {
+                Part::ClosedSubshell(inner) | Part::OpenSubshell(inner) if !inner.is_empty() => {
+                    // If there's a subshell, recurse to find that in range command
+                    if let Some(sub_command) = command_at_cursor(inner, pos) {
+                        return Some(sub_command);
+                    }
+                }
+                _ => {}
+            }
+        }
+        return Some(in_range_command);
+    }
+    None
+}
+
+/// Retrieve the last unclosed command contained within a given command
+///
+/// Note: This may be the command itself, if it doesn't contain an open subshell
+fn last_unclosed_command(mut command: Spanned<Command>) -> Spanned<Command> {
+    // Check if the last part of the command is an OpenSubshell with at least one subcommand
+    // If it is, we repeat the process with the final command of the subshell
+    if let Some(final_part) = command.item.parts.pop() {
+        match final_part.item {
+            Part::OpenSubshell(mut inner) => {
+                if !inner.is_empty() {
+                    return last_unclosed_command(inner.pop().unwrap());
+                } else {
+                    command
+                        .item
+                        .parts
+                        .push(Part::OpenSubshell(inner).spanned(final_part.span));
+                }
+            }
+            _ => {
+                command.item.parts.push(final_part);
+            }
+        }
+    }
+
+    command
+}
+
+/// A parsed command, made up of a number of parts.
+///
+/// Each part represents an argument, with the first part being the command itself
+#[derive(Debug, PartialEq, Eq, Clone)]
+struct Command {
+    parts: Vec<Spanned<Part>>,
+}
+
+impl Command {
+    pub fn new(parts: Vec<Spanned<Part>>) -> Self {
+        Command { parts }
+    }
+
+    pub fn decompose(self, src: &str) -> Vec<String> {
+        let this_command = self
+            .parts
+            .first()
+            .zip(self.parts.last())
+            .map(|(first, last)| src[first.span.start()..last.span.end()].trim().to_string());
+
+        let mut all_commands = vec![];
+        let mut this_command_has_literal = false;
+
+        for part in self.parts {
+            match part.item {
+                Part::Literal(_) => {
+                    this_command_has_literal = true;
+                }
+                Part::ClosedSubshell(s) | Part::OpenSubshell(s) => {
+                    // Add the total subcommand as a command.
+                    if let Some((first, last)) = s.first().zip(s.last()) {
+                        all_commands
+                            .push(src[first.span.start()..last.span.end()].trim().to_string());
+                    }
+
+                    // Recursively decompose each command in the subcommand.
+                    all_commands.extend(s.into_iter().flat_map(|c| c.item.decompose(src)));
+                }
+                Part::Concatenated(s) => {
+                    // Add the total concatenation as a command.
+                    if let Some((first, last)) = s.first().zip(s.last()) {
+                        all_commands
+                            .push(src[first.span.start()..last.span.end()].trim().to_string());
+                    }
+
+                    // Recursively decompose each part in the concatenation.
+                    let new_command = Command::new(s);
+                    all_commands.extend(new_command.decompose(src));
+                }
+            }
+        }
+
+        // If this command has a literal, add it to the list.
+        match this_command {
+            Some(c) if this_command_has_literal => all_commands.push(c.to_string()),
+            _ => {}
+        }
+
+        all_commands
+    }
+
+    /// Removes the leading env-var assignments (i.e. 'KEY=VALUE' literals) from the command.
+    pub fn remove_leading_env_vars(&mut self) {
+        while !self.parts.is_empty() {
+            let Some(first_command_part) = self.parts.first() else {
+                break;
+            };
+            if first_command_part.to_string().split('=').count() != 2 {
+                break;
+            }
+            self.parts.remove(0);
+        }
+    }
+}
+
+/// An individual part of a command (argument or command name)
+#[derive(Debug, PartialEq, Eq, Clone)]
+enum Part {
+    /// Raw string value
+    Literal(String),
+
+    /// Subshell call containing one or more commands that was properly closed
+    ///
+    /// For example, $(cat file.txt) or `ls -la`
+    ClosedSubshell(Vec<Spanned<Command>>),
+
+    /// Subshell call that was _not_ properly closed in the input
+    ///
+    /// For example, $(cat file.txt or `ls -la
+    OpenSubshell(Vec<Spanned<Command>>),
+
+    /// Concatenation of several literal and/or subshell parts
+    ///
+    /// For example, Hello"World" would be a concatenation of /Hello/ and /"World"/ as individual
+    /// parts
+    Concatenated(Vec<Spanned<Part>>),
+}

crates/warp_completer/src/parsers/simple/parser.rs

@@ -0,0 +1,460 @@
+use super::iter::ParserInput;
+use super::token::Token;
+use super::{Command, Part};
+use crate::meta::{Span, Spanned, SpannedItem};
+
+/// Parser that converts a command string into a list of commands, with arguments separated
+///
+/// Handles subshells, quotes, double quotes and escapes (e.g. `\` for paths with spaces)
+pub(super) struct Parser<I>
+where
+    I: IntoIterator,
+{
+    tokens: ParserInput<I>,
+    contains_redirection: bool,
+}
+
+pub struct ParsedResult {
+    pub commands: Vec<Spanned<Command>>,
+
+    /// Whether or not redirection operators (i.e. '>', '<') were
+    /// found when parsing.
+    pub contains_redirection: bool,
+}
+
+impl<'a, I> Parser<I>
+where
+    I: IntoIterator<Item = Spanned<Token<'a>>>,
+{
+    pub fn new(input: I) -> Self {
+        Self::with_tokens(ParserInput::new(input))
+    }
+
+    fn with_tokens(tokens: ParserInput<I>) -> Self {
+        Parser {
+            tokens,
+            contains_redirection: false,
+        }
+    }
+
+    pub fn parse(mut self) -> ParsedResult {
+        let commands = self.parse_command_list(None);
+        ParsedResult {
+            commands,
+            contains_redirection: self.contains_redirection,
+        }
+    }
+
+    /// Generate a span from the given start to the current token position
+    fn span(&self, start: usize) -> Span {
+        Span::new(start, self.tokens.pos())
+    }
+
+    /// Skip over any whitespace tokens
+    fn skip_whitespace(&mut self) {
+        while let Some(Token::Whitespace(_)) = self.tokens.peek() {
+            self.tokens.next();
+        }
+    }
+
+    /// Parse a list of commands, separated by logical operators, pipes, semicolons, etc.
+    ///
+    /// Can optionally include a delimiting token that will mark the end of the command list
+    fn parse_command_list(&mut self, delimiter: Option<Token>) -> Vec<Spanned<Command>> {
+        let mut commands = Vec::new();
+        loop {
+            self.skip_whitespace();
+            if delimiter.is_some() && delimiter.as_ref() == self.tokens.peek() {
+                break;
+            }
+
+            // TODO: Add errors if there is a command in the list and the token isn't a logical
+            // separator, as it's invalid to connect commands with a grouping operator
+            match self.tokens.peek() {
+                Some(Token::OpenParen) => {
+                    self.tokens.next();
+                    let nested = self.parse_command_list(Some(Token::CloseParen));
+                    commands.extend(nested);
+                }
+                Some(Token::OpenCurly) => {
+                    self.tokens.next();
+                    let nested = self.parse_command_list(Some(Token::CloseCurly));
+                    commands.extend(nested);
+                }
+                Some(Token::CloseParen | Token::CloseCurly) => {
+                    // If we got here, that means we are hitting a grouping close when we aren't
+                    // expecting it. This is generally a parse error, however for now we will treat
+                    // it as a separator between commands.
+                    // TODO: Add an error for an unexpected character
+                    self.tokens.next();
+                }
+                Some(Token::RedirectInput | Token::RedirectOutput) => {
+                    self.contains_redirection = true;
+                    self.tokens.next();
+                }
+                Some(t) if is_valid_command_separator(t) => {
+                    // Valid separator between commands, for now we naively consume it and don't
+                    // store semantic information about the kind of separation.
+                    self.tokens.next();
+                }
+                Some(_) => {
+                    // All other tokens are part of commands
+                    commands.push(self.parse_command());
+                }
+                None => break,
+            }
+        }
+
+        // TODO: Add an error if the delimiter wasn't found
+
+        commands
+    }
+
+    /// Parse an individual command from the input
+    fn parse_command(&mut self) -> Spanned<Command> {
+        let start = self.tokens.pos();
+        let mut parts = Vec::new();
+        loop {
+            self.skip_whitespace();
+
+            match self.tokens.peek() {
+                None => break,
+                Some(Token::RedirectInput | Token::RedirectOutput) => {
+                    self.contains_redirection = true;
+                    self.tokens.next();
+                }
+                Some(t) if is_command_terminator(t) => {
+                    break;
+                }
+                _ => {
+                    parts.push(self.parse_part(parts.is_empty()));
+                }
+            }
+        }
+
+        Command::new(parts).spanned(self.span(start))
+    }
+
+    /// Parse a single part of a command (e.g. argument) from the input
+    fn parse_part(&mut self, first_part: bool) -> Spanned<Part> {
+        let mut builder = PartBuilder::new(self.tokens.pos());
+
+        loop {
+            match self.tokens.peek() {
+                None => break,
+                Some(Token::Whitespace(_)) => break,
+                Some(Token::RedirectInput | Token::RedirectOutput) => {
+                    self.contains_redirection = true;
+                    self.tokens.next();
+                }
+                Some(t) if is_command_terminator(t) => {
+                    break;
+                }
+                Some(Token::DoubleQuote) => {
+                    builder.add_part(self.parse_double_quoted_part());
+                }
+                Some(Token::SingleQuote) => {
+                    builder.add_part(self.parse_single_quoted_part());
+                }
+                Some(Token::Backtick) => {
+                    builder.add_part(self.parse_backticked_subshell());
+                }
+                Some(Token::Dollar) => {
+                    if let Some(Token::OpenParen) = self.tokens.peekpeek() {
+                        builder.add_part(self.parse_dollar_subshell());
+                    } else {
+                        // Consume the dollar token
+                        self.tokens.next();
+                        builder.add_raw(Token::Dollar.as_str());
+                    }
+                }
+                Some(Token::EscapeChar(c)) => {
+                    let c = c.to_owned();
+                    // Consume the backslash
+                    self.tokens.next();
+                    // In non-quoted contexts, backslash escapes all characters except newlines
+                    // Newline immediately following a backslash is treated as line continuation
+                    // Regardless, the backslash is not included in the raw value (besides some
+                    // special cases)
+                    match self.tokens.next() {
+                        None => {
+                            // Include the backslash if it is the last character in the output, as
+                            // there is nothing for it to escape
+                            builder.add_raw(c);
+                        }
+                        Some(Token::Newline) => {}
+                        Some(Token::Literal("~")) => {
+                            // Include the backslash if the input is `\~` to enable us to
+                            // distinguish tildes that need to be expanded into the home directory
+                            // and the raw string.
+                            builder.add_raw(c);
+                            builder.add_raw("~");
+                        }
+                        Some(token) => {
+                            // Include the backslash if this is the first part of the command, so that we
+                            //  can support escaping aliases.  eg: with alias ls=exa, the command \ls
+                            //  should be treated as the command, and we should not remove the backslash.
+                            if first_part && builder.is_empty() {
+                                builder.add_raw(c);
+                            }
+                            builder.add_raw(token.as_str());
+                        }
+                    }
+                }
+                // All tokens not handled specially are treated as literals
+                Some(literal) => {
+                    builder.add_raw(literal.as_str());
+                    // Consume the token
+                    self.tokens.next();
+                }
+            }
+        }
+
+        builder.complete(self.tokens.pos())
+    }
+
+    /// Parse a backticked subshell section
+    ///
+    /// Note: This must only be called when the next character is a backtick
+    fn parse_backticked_subshell(&mut self) -> Spanned<Part> {
+        let start = self.tokens.pos();
+
+        // Consume the backtick token
+        let check = self.tokens.next();
+        debug_assert!(matches!(check, Some(Token::Backtick)));
+
+        // Create a parser using the buffered tokens between here and the next backtick
+        // We can't use parse_command_list directly because Backtick isn't a command terminator,
+        // and we can't make it a command terminator because it's symmetric: It also represents the
+        // _start_ of a subshell.
+        let mut sub_parser = Parser::with_tokens(self.tokens.until_backtick());
+        let command_list = sub_parser.parse_command_list(None);
+
+        // Consume the closing backtick if available and classify if the subshell was closed
+        match self.tokens.next() {
+            None => Part::OpenSubshell(command_list).spanned(self.span(start)),
+            Some(t) => {
+                debug_assert!(t == Token::Backtick);
+                Part::ClosedSubshell(command_list).spanned(self.span(start))
+            }
+        }
+    }
+
+    /// Parse a dollar subshell section, i.e. one surrounded by $()
+    ///
+    /// Note: This must only be called when the next two characters are $(
+    fn parse_dollar_subshell(&mut self) -> Spanned<Part> {
+        let start = self.tokens.pos();
+
+        // Consume the dollar and open paren tokens
+        let check = self.tokens.next();
+        debug_assert!(matches!(check, Some(Token::Dollar)));
+        let check = self.tokens.next();
+        debug_assert!(matches!(check, Some(Token::OpenParen)));
+
+        let command_list = self.parse_command_list(Some(Token::CloseParen));
+
+        // Consume the close paren if available and classify the subshell was closed
+        match self.tokens.next() {
+            None => Part::OpenSubshell(command_list).spanned(self.span(start)),
+            Some(t) => {
+                debug_assert!(t == Token::CloseParen);
+                Part::ClosedSubshell(command_list).spanned(self.span(start))
+            }
+        }
+    }
+
+    /// Parse a double-quoted section
+    ///
+    /// Within double quotes, most tokens are treated as literal. The exceptions are:
+    ///
+    /// - Backticks (`) still start subshell sections
+    /// - $() can still be used for subshell sections
+    /// - Backslash (\) escapes only some characters: $ ` " \ \n
+    ///
+    /// Note: This must only be called when the next token is a double quote
+    fn parse_double_quoted_part(&mut self) -> Spanned<Part> {
+        let mut builder = PartBuilder::new(self.tokens.pos());
+
+        // Consume the opening double quote
+        let check = self.tokens.next();
+        debug_assert!(matches!(check, Some(Token::DoubleQuote)));
+
+        loop {
+            match self.tokens.peek() {
+                None => break,
+                Some(Token::DoubleQuote) => {
+                    // Consume the closing double quote
+                    self.tokens.next();
+                    break;
+                }
+                Some(Token::EscapeChar(c)) => {
+                    let c = c.to_owned();
+                    // Consume the backslash
+                    self.tokens.next();
+                    // Check the following character for escape behavior
+                    match self.tokens.next() {
+                        Some(Token::Newline) => {
+                            // Within double quotes, newline following backslash is still treated
+                            // as line continuation, so neither is included in the raw output
+                        }
+                        Some(
+                            token @ Token::Dollar
+                            | token @ Token::Backtick
+                            | token @ Token::DoubleQuote
+                            | token @ Token::EscapeChar(_),
+                        ) => {
+                            builder.add_raw(token.as_str());
+                        }
+                        Some(token) => {
+                            // For all other characters, we include the backslash as well
+                            builder.add_raw(c);
+                            builder.add_raw(token.as_str());
+                        }
+                        None => {
+                            builder.add_raw(c);
+                        }
+                    }
+                }
+                Some(Token::Backtick) => {
+                    builder.add_part(self.parse_backticked_subshell());
+                }
+                Some(Token::Dollar) => {
+                    if let Some(Token::OpenParen) = self.tokens.peekpeek() {
+                        builder.add_part(self.parse_dollar_subshell());
+                    } else {
+                        // Consume the dollar token
+                        self.tokens.next();
+                        builder.add_raw(Token::Dollar.as_str());
+                    }
+                }
+                Some(token) => {
+                    builder.add_raw(token.as_str());
+                    // Consume the token
+                    self.tokens.next();
+                }
+            }
+        }
+
+        builder.complete(self.tokens.pos())
+    }
+
+    /// Parse a single-quoted section
+    ///
+    /// Within single quotes, all tokens are literal until the closing single quote
+    /// Note: This must only be called when the next character is a single quote
+    fn parse_single_quoted_part(&mut self) -> Spanned<Part> {
+        let start = self.tokens.pos();
+        let mut buffer = String::new();
+
+        // Consume the starting single quote
+        let check = self.tokens.next();
+        debug_assert!(matches!(check, Some(Token::SingleQuote)));
+
+        while let Some(token) = self.tokens.next() {
+            match token {
+                Token::SingleQuote => break,
+                t => buffer.push_str(t.as_str()),
+            }
+        }
+
+        // TODO: Add an error if we hit the end of the output without finding a closing quote
+
+        Part::Literal(buffer).spanned(self.span(start))
+    }
+}
+
+/// Determine if a token is a valid separator between commands
+fn is_valid_command_separator(token: &Token) -> bool {
+    use Token::*;
+
+    matches!(
+        token,
+        Pipe | LogicalOr | Ampersand | LogicalAnd | Semicolon | Newline
+    )
+}
+
+/// Determine if a token is a terminator marking the end of a command
+///
+/// This includes all of the separator tokens as well as the grouping tokens
+fn is_command_terminator(token: &Token) -> bool {
+    use Token::*;
+
+    is_valid_command_separator(token)
+        || matches!(token, OpenParen | CloseParen | OpenCurly | CloseCurly)
+}
+
+/// Builder for handling combined command parts.
+///
+/// Tracks literal values and any nested parts, flattening `Part::Concatenated` into the current
+/// list as necessary.
+struct PartBuilder {
+    start: usize,
+    buffer: String,
+    buffer_start: usize,
+    sub_parts: Vec<Spanned<Part>>,
+}
+
+impl PartBuilder {
+    fn new(start: usize) -> Self {
+        Self {
+            start,
+            buffer: String::new(),
+            buffer_start: start,
+            sub_parts: Vec::new(),
+        }
+    }
+
+    /// Add a whole known sub-Part to the current Part.
+    ///
+    /// Will complete any current Literal values into a `Part::Literal` entry.
+    fn add_part(&mut self, part: Spanned<Part>) {
+        if !self.buffer.is_empty() {
+            let span = Span::new(self.buffer_start, part.span.start());
+            let literal = Part::Literal(std::mem::take(&mut self.buffer)).spanned(span);
+            self.sub_parts.push(literal);
+        }
+
+        self.buffer_start = part.span.end();
+
+        match part.item {
+            // Flatten any concatenated inner parts to create a single list
+            Part::Concatenated(inner) => {
+                self.sub_parts.extend(inner);
+            }
+            _ => self.sub_parts.push(part),
+        }
+    }
+
+    /// Add a raw string value to the current part
+    fn add_raw(&mut self, value: &str) {
+        self.buffer.push_str(value);
+    }
+
+    /// Complete the Part, finishing any Literal entries and determining the appropriate `Part`
+    /// variant to return based on the number of entries.
+    fn complete(mut self, end: usize) -> Spanned<Part> {
+        if !self.buffer.is_empty() {
+            let span = Span::new(self.buffer_start, end);
+            let literal = Part::Literal(self.buffer).spanned(span);
+            self.sub_parts.push(literal);
+        }
+
+        let span = Span::new(self.start, end);
+        match self.sub_parts.len() {
+            0 => Part::Literal(String::new()).spanned(span),
+            // Safety: We are checking the length, so if there is one element then pop will exist
+            1 => self.sub_parts.pop().unwrap(),
+            _ => Part::Concatenated(self.sub_parts).spanned(span),
+        }
+    }
+
+    /// Check if the builder is empty.
+    fn is_empty(&self) -> bool {
+        self.buffer.is_empty() && self.sub_parts.is_empty()
+    }
+}
+
+#[cfg(test)]
+#[path = "parser_test.rs"]
+mod tests;

crates/warp_completer/src/parsers/simple/parser_test.rs

@@ -0,0 +1,202 @@
+use std::collections::HashSet;
+
+use warp_util::path::EscapeChar;
+
+use crate::parsers::simple::{decompose_command, top_level_command};
+
+use super::super::lexer::Lexer;
+use super::super::{Command, Part};
+use super::*;
+
+#[test]
+fn test_parse_open_subshell() {
+    let source = r#"cat "Hello $(ls -la"#;
+    let command = Parser::new(Lexer::new(source, EscapeChar::Backslash, false)).parse_command();
+
+    assert_eq!(
+        command,
+        Command::new(vec![
+            Part::Literal("cat".into()).spanned((0, 3)),
+            Part::Concatenated(vec![
+                Part::Literal("Hello ".into()).spanned((4, 11)),
+                Part::OpenSubshell(vec![Command::new(vec![
+                    Part::Literal("ls".into()).spanned((13, 15)),
+                    Part::Literal("-la".into()).spanned((16, 19)),
+                ])
+                .spanned((13, 19)),])
+                .spanned((11, 19)),
+            ])
+            .spanned((4, 19)),
+        ])
+        .spanned((0, 19)),
+    );
+}
+
+#[test]
+fn test_parse_nested_command() {
+    let source = r#"cat "Hello $(ls -la)""#;
+    let command = Parser::new(Lexer::new(source, EscapeChar::Backslash, false)).parse_command();
+
+    assert_eq!(
+        command,
+        Command::new(vec![
+            Part::Literal("cat".into()).spanned((0, 3)),
+            Part::Concatenated(vec![
+                Part::Literal("Hello ".into()).spanned((4, 11)),
+                Part::ClosedSubshell(vec![Command::new(vec![
+                    Part::Literal("ls".into()).spanned((13, 15)),
+                    Part::Literal("-la".into()).spanned((16, 19)),
+                ])
+                .spanned((13, 19)),])
+                .spanned((11, 20)),
+            ])
+            .spanned((4, 21)),
+        ])
+        .spanned((0, 21))
+    );
+}
+
+#[test]
+fn test_parse() {
+    let source = r#"ls | rm -rf || touch 'hello.txt\' &
+cat "Hello $(ls -la)" && echo `ps \`; {echo Goodbye😀}"#;
+
+    let commands = Parser::new(Lexer::new(source, EscapeChar::Backslash, false))
+        .parse()
+        .commands;
+
+    assert_eq!(
+        commands,
+        [
+            Command::new(vec![Part::Literal("ls".into()).spanned((0, 2))]).spanned((0, 3)),
+            Command::new(vec![
+                Part::Literal("rm".into()).spanned((5, 7)),
+                Part::Literal("-rf".into()).spanned((8, 11))
+            ])
+            .spanned((5, 12)),
+            Command::new(vec![
+                Part::Literal("touch".into()).spanned((15, 20)),
+                Part::Literal("hello.txt\\".into()).spanned((21, 33))
+            ])
+            .spanned((15, 34)),
+            Command::new(vec![
+                Part::Literal("cat".into()).spanned((36, 39)),
+                Part::Concatenated(vec![
+                    Part::Literal("Hello ".into()).spanned((40, 47)),
+                    Part::ClosedSubshell(vec![Command::new(vec![
+                        Part::Literal("ls".into()).spanned((49, 51)),
+                        Part::Literal("-la".into()).spanned((52, 55)),
+                    ])
+                    .spanned((49, 55))])
+                    .spanned((47, 56)),
+                ])
+                .spanned((40, 57))
+            ])
+            .spanned((36, 58)),
+            Command::new(vec![
+                Part::Literal("echo".into()).spanned((61, 65)),
+                Part::ClosedSubshell(vec![Command::new(vec![
+                    Part::Literal("ps".into()).spanned((67, 69)),
+                    Part::Literal("\\".into()).spanned((70, 71)),
+                ])
+                .spanned((67, 71))])
+                .spanned((66, 72)),
+            ])
+            .spanned((61, 72)),
+            Command::new(vec![
+                Part::Literal("echo".into()).spanned((75, 79)),
+                Part::Literal("Goodbye😀".into()).spanned((80, 91))
+            ])
+            .spanned((75, 91)),
+        ]
+    );
+}
+
+// Test that a backslash is retained when preceding a command.
+#[test]
+fn test_backslash_before_command() {
+    let source = r#"\ls"#;
+    let command = Parser::new(Lexer::new(source, EscapeChar::Backslash, false)).parse_command();
+
+    assert_eq!(
+        command,
+        Command::new(vec![Part::Literal(r"\ls".into()).spanned((0, 3))]).spanned((0, 3))
+    );
+}
+
+// Test that a backslash is not retained in the middle of a command.
+#[test]
+fn test_backslash_in_command() {
+    let source = r#"ls \-la"#;
+    let command = Parser::new(Lexer::new(source, EscapeChar::Backslash, false)).parse_command();
+
+    assert_eq!(
+        command,
+        Command::new(vec![
+            Part::Literal("ls".into()).spanned((0, 2)),
+            Part::Literal("-la".into()).spanned((3, 7))
+        ])
+        .spanned((0, 7))
+    );
+}
+
+#[test]
+fn test_decompose_command() {
+    let test_data = vec![
+        ("ls", vec!["ls"]),
+        ("$(ls)", vec!["ls"]),
+        ("ls -la", vec!["ls -la"]),
+        ("ls && cat", vec!["ls", "cat"]),
+        (
+            "ls $(foo | echo)",
+            vec!["foo", "echo", "foo | echo", "ls $(foo | echo)"],
+        ),
+    ];
+
+    for (input, expected_output) in test_data {
+        // Compare with hashsets bc we don't care about ordering.
+        assert_eq!(
+            HashSet::<String>::from_iter(decompose_command(input, EscapeChar::Backslash).0),
+            HashSet::from_iter(expected_output.into_iter().map(ToString::to_string)),
+        );
+    }
+}
+
+#[test]
+fn test_contains_redirection() {
+    let test_data = vec![
+        ("ls < \"file.txt < tmp\"", true),
+        ("echo $(ls > file.txt)", true),
+        ("ls >> file.txt", true),
+        ("ls < file.txt", true),
+        ("foo arg1 arg2 > file.txt", true),
+        ("foo && ls > file.txt", true),
+        ("echo \"hello world\" > output.txt", true),
+        ("echo \"5>4\"", false),
+        ("echo \"This message -> shows direction\"", false),
+        ("print(\"Value must be > 0 and < 100\")", false),
+    ];
+
+    for (cmd, should_contain_redirection) in test_data {
+        let parser = Parser::new(Lexer::new(cmd, EscapeChar::Backslash, false));
+        let contains_redirection = parser.parse().contains_redirection;
+        assert_eq!(contains_redirection, should_contain_redirection);
+    }
+}
+
+#[test]
+fn test_top_level_command() {
+    let test_data = vec![
+        ("PAGER=0 git log", Some("git")),
+        ("PAGER= git log", Some("git")),
+        ("ls && git status", Some("ls")),
+        ("$(git status)", None),
+    ];
+
+    for (input, expected_output) in test_data {
+        assert_eq!(
+            top_level_command(input, EscapeChar::Backslash),
+            expected_output.map(ToString::to_string)
+        );
+    }
+}

crates/warp_completer/src/parsers/simple/token.rs

@@ -0,0 +1,53 @@
+#[derive(Debug, PartialEq, Eq)]
+pub enum Token<'a> {
+    Literal(&'a str),
+    /// Whitespace excluding newline
+    Whitespace(&'a str),
+    /// | Operator
+    Pipe,
+    /// || operator
+    LogicalOr,
+    /// & operator
+    Ampersand,
+    /// && operator
+    LogicalAnd,
+    Semicolon,
+    Newline,
+    Backtick,
+    OpenParen,
+    CloseParen,
+    OpenCurly,
+    CloseCurly,
+    Dollar,
+    SingleQuote,
+    DoubleQuote,
+    /// \ or `
+    EscapeChar(&'a str),
+    RedirectInput,
+    RedirectOutput,
+}
+
+impl Token<'_> {
+    pub fn as_str(&self) -> &str {
+        use Token::*;
+        match self {
+            Literal(value) | Whitespace(value) | EscapeChar(value) => value,
+            Pipe => "|",
+            LogicalOr => "||",
+            Ampersand => "&",
+            LogicalAnd => "&&",
+            Semicolon => ";",
+            Newline => "\n",
+            Backtick => "`",
+            OpenParen => "(",
+            CloseParen => ")",
+            OpenCurly => "{",
+            CloseCurly => "}",
+            Dollar => "$",
+            SingleQuote => "'",
+            DoubleQuote => "\"",
+            RedirectInput => "<",
+            RedirectOutput => ">",
+        }
+    }
+}

crates/warp_completer/src/parsers/test.rs

@@ -0,0 +1,436 @@
+use itertools::Itertools;
+use warp_util::path::EscapeChar;
+
+use crate::{
+    parsers::{
+        classify_command,
+        hir::{CommandCallInfo, Flags, ShellCommand},
+        simple::parse_for_completions,
+        ClassifiedCommand,
+    },
+    signatures::testing::{create_test_command_registry, test_signature},
+};
+
+#[cfg(not(feature = "v2"))]
+use crate::parsers::hir::{Flag, FlagType};
+
+use super::*;
+
+#[test]
+pub fn test_classify_command_classifies_known_command() {
+    let registry = create_test_command_registry([test_signature()]);
+
+    let lite_command = parse_for_completions("test ", EscapeChar::Backslash, false)
+        .expect("Should be able to parse input into LiteCommand");
+    let mut tokens = lite_command.parts.iter().map(|s| s.as_str()).collect_vec();
+
+    let classified_command = classify_command(
+        lite_command.clone(),
+        &mut tokens,
+        &registry,
+        TopLevelCommandCaseSensitivity::CaseSensitive,
+    );
+    assert_eq!(
+        classified_command,
+        Some(ClassifiedCommand {
+            env_vars: vec![],
+            command: Command::Classified(ShellCommand {
+                name: "test".to_owned(),
+                name_span: Span::from((0, 4)),
+                args: CommandCallInfo {
+                    command_name: Spanned {
+                        span: Span::from((0, 4)),
+                        item: ParsedExpression::new(
+                            Expression::Command,
+                            ParsedToken("test".to_owned())
+                        ),
+                    },
+                    positionals: None,
+                    flags: Some(Flags::new()),
+                    ending_whitespace: Some(Span::from((4, 5))),
+                    span: Span::from((0, 5))
+                }
+            }),
+            error: None,
+        })
+    )
+}
+
+/// TODO(CORE-2797)
+#[cfg(not(feature = "v2"))]
+#[test]
+pub fn test_classify_command_classifies_known_command_with_flags() {
+    let registry = create_test_command_registry([test_signature()]);
+
+    let lite_command = parse_for_completions("test -r --long foo", EscapeChar::Backslash, false)
+        .expect("Should be able to parse input into LiteCommand");
+    let mut tokens = lite_command.parts.iter().map(|s| s.as_str()).collect_vec();
+
+    let classified_command = classify_command(
+        lite_command.clone(),
+        &mut tokens,
+        &registry,
+        TopLevelCommandCaseSensitivity::CaseSensitive,
+    );
+    assert_eq!(
+        classified_command,
+        Some(ClassifiedCommand {
+            env_vars: vec![],
+            command: Command::Classified(ShellCommand {
+                name: "test".to_owned(),
+                name_span: Span::from((0, 4)),
+                args: CommandCallInfo {
+                    command_name: Spanned {
+                        span: Span::from((0, 4)),
+                        item: ParsedExpression::new(
+                            Expression::Command,
+                            ParsedToken("test".to_owned())
+                        ),
+                    },
+                    positionals: None,
+                    flags: Some(Flags {
+                        flags: vec![
+                            Flag {
+                                name: "-r".to_owned(),
+                                name_span: Span::from((5, 7)),
+                                flag_type: FlagType::NoArgument
+                            },
+                            Flag {
+                                name: "--long".to_owned(),
+                                name_span: Span::from((8, 14)),
+                                flag_type: FlagType::Argument {
+                                    value: Spanned {
+                                        span: Span::from((15, 18)),
+                                        item: ParsedExpression::new(
+                                            Expression::Literal,
+                                            ParsedToken("foo".to_owned())
+                                        )
+                                    }
+                                }
+                            },
+                        ]
+                    }),
+                    ending_whitespace: None,
+                    span: Span::from((0, 18))
+                }
+            }),
+            error: None
+        })
+    )
+}
+
+/// TODO(CORE-2797)
+///
+/// With exact option matching, `-r` correctly matches the `-r` switch (no arguments),
+/// so the parser advances past it and discovers the `one` subcommand. The command path
+/// becomes `"test -r one"` (the legacy parser's convention for subcommand paths).
+#[cfg(not(feature = "v2"))]
+#[test]
+pub fn test_classify_command_classifies_known_command_with_subcommand() {
+    let registry = create_test_command_registry([test_signature()]);
+
+    let lite_command = parse_for_completions("test -r one foo bar", EscapeChar::Backslash, false)
+        .expect("Should be able to parse input into LiteCommand");
+    let mut tokens = lite_command.parts.iter().map(|s| s.as_str()).collect_vec();
+
+    let classified_command = classify_command(
+        lite_command.clone(),
+        &mut tokens,
+        &registry,
+        TopLevelCommandCaseSensitivity::CaseSensitive,
+    );
+    assert_eq!(
+        classified_command,
+        Some(ClassifiedCommand {
+            env_vars: vec![],
+            command: Command::Classified(ShellCommand {
+                name: "test -r one".to_owned(),
+                name_span: Span::from((0, 11)),
+                args: CommandCallInfo {
+                    command_name: Spanned {
+                        span: Span::from((0, 11)),
+                        item: ParsedExpression::new(
+                            Expression::Command,
+                            ParsedToken("test -r one".to_owned())
+                        ),
+                    },
+                    positionals: Some(vec![
+                        Spanned {
+                            span: Span::from((12, 15)),
+                            item: ParsedExpression::new(
+                                Expression::Literal,
+                                ParsedToken("foo".to_owned())
+                            ),
+                        },
+                        Spanned {
+                            span: Span::from((16, 19)),
+                            item: ParsedExpression::new(
+                                Expression::Literal,
+                                ParsedToken("bar".to_owned())
+                            ),
+                        },
+                    ]),
+                    flags: Some(Flags::new()),
+                    ending_whitespace: None,
+                    span: Span::from((0, 19))
+                }
+            }),
+            error: None
+        })
+    )
+}
+
+#[test]
+pub fn test_classify_command_classifies_unknown_command() {
+    let registry = create_test_command_registry([]);
+
+    let lite_command = parse_for_completions("test ", EscapeChar::Backslash, false)
+        .expect("Should be able to parse input into LiteCommand");
+    let mut tokens = lite_command.parts.iter().map(|s| s.as_str()).collect_vec();
+
+    let classified_command = classify_command(
+        lite_command.clone(),
+        &mut tokens,
+        &registry,
+        TopLevelCommandCaseSensitivity::CaseSensitive,
+    );
+    assert_eq!(
+        classified_command,
+        Some(ClassifiedCommand {
+            env_vars: vec![],
+            command: Command::Unclassified(ExternalCommand {
+                name: ParsedToken("test".to_owned()),
+                name_span: Span::from((0, 4)),
+                args: CommandCallInfo {
+                    command_name: Spanned {
+                        span: Span::from((0, 5)),
+                        item: ParsedExpression::new(
+                            Expression::Literal,
+                            ParsedToken("test".to_owned())
+                        ),
+                    },
+                    positionals: None,
+                    flags: None,
+                    ending_whitespace: Some(Span::from((4, 5))),
+                    span: Span::from((0, 5))
+                }
+            }),
+            error: None,
+        })
+    )
+}
+
+#[test]
+pub fn test_classify_command_classifies_unknown_command_with_flags() {
+    let registry = create_test_command_registry([]);
+
+    let lite_command = parse_for_completions("test -r --long foo", EscapeChar::Backslash, false)
+        .expect("Should be able to parse input into LiteCommand");
+    let mut tokens = lite_command.parts.iter().map(|s| s.as_str()).collect_vec();
+
+    let classified_command = classify_command(
+        lite_command.clone(),
+        &mut tokens,
+        &registry,
+        TopLevelCommandCaseSensitivity::CaseSensitive,
+    );
+    assert_eq!(
+        classified_command,
+        Some(ClassifiedCommand {
+            env_vars: vec![],
+            command: Command::Unclassified(ExternalCommand {
+                name: ParsedToken("test".to_owned()),
+                name_span: Span::from((0, 4)),
+                args: CommandCallInfo {
+                    command_name: Spanned {
+                        span: Span::from((0, 18)),
+                        item: ParsedExpression::new(
+                            Expression::Literal,
+                            ParsedToken("test".to_owned()),
+                        ),
+                    },
+                    positionals: Some(vec![
+                        Spanned {
+                            span: Span::from((5, 7)),
+                            item: ParsedExpression::new(
+                                Expression::Literal,
+                                ParsedToken("-r".to_owned()),
+                            ),
+                        },
+                        Spanned {
+                            span: Span::from((8, 14)),
+                            item: ParsedExpression::new(
+                                Expression::Literal,
+                                ParsedToken("--long".to_owned()),
+                            ),
+                        },
+                        Spanned {
+                            span: Span::from((15, 18)),
+                            item: ParsedExpression::new(
+                                Expression::Literal,
+                                ParsedToken("foo".to_owned()),
+                            ),
+                        },
+                    ]),
+                    flags: None,
+                    ending_whitespace: None,
+                    span: Span::from((0, 18)),
+                },
+            }),
+            error: None,
+        })
+    )
+}
+
+#[test]
+pub fn test_classify_command_classifies_unknown_command_with_subcommand() {
+    let registry = create_test_command_registry([]);
+
+    let lite_command = parse_for_completions("test -r one foo bar", EscapeChar::Backslash, false)
+        .expect("Should be able to parse input into LiteCommand");
+    let mut tokens = lite_command.parts.iter().map(|s| s.as_str()).collect_vec();
+
+    let classified_command = classify_command(
+        lite_command.clone(),
+        &mut tokens,
+        &registry,
+        TopLevelCommandCaseSensitivity::CaseSensitive,
+    );
+    assert_eq!(
+        classified_command,
+        Some(ClassifiedCommand {
+            env_vars: vec![],
+            command: Command::Unclassified(ExternalCommand {
+                name: ParsedToken("test".to_owned()),
+                name_span: Span::from((0, 4)),
+                args: CommandCallInfo {
+                    command_name: Spanned {
+                        span: Span::from((0, 19)),
+                        item: ParsedExpression::new(
+                            Expression::Literal,
+                            ParsedToken("test".to_owned()),
+                        ),
+                    },
+                    positionals: Some(vec![
+                        Spanned {
+                            span: Span::from((5, 7)),
+                            item: ParsedExpression::new(
+                                Expression::Literal,
+                                ParsedToken("-r".to_owned()),
+                            ),
+                        },
+                        Spanned {
+                            span: Span::from((8, 11)),
+                            item: ParsedExpression::new(
+                                Expression::Literal,
+                                ParsedToken("one".to_owned()),
+                            ),
+                        },
+                        Spanned {
+                            span: Span::from((12, 15)),
+                            item: ParsedExpression::new(
+                                Expression::Literal,
+                                ParsedToken("foo".to_owned()),
+                            ),
+                        },
+                        Spanned {
+                            span: Span::from((16, 19)),
+                            item: ParsedExpression::new(
+                                Expression::Literal,
+                                ParsedToken("bar".to_owned())
+                            ),
+                        },
+                    ]),
+                    flags: None,
+                    ending_whitespace: None,
+                    span: Span::from((0, 19)),
+                },
+            }),
+            error: None,
+        }),
+    )
+}
+
+#[test]
+fn test_classify_command_case_sensitive() {
+    let registry = create_test_command_registry([test_signature()]);
+
+    let lite_command = parse_for_completions("TEST ", EscapeChar::Backslash, false)
+        .expect("Should be able to parse input into LiteCommand");
+    let mut tokens = lite_command.parts.iter().map(|s| s.as_str()).collect_vec();
+
+    let classified_command = classify_command(
+        lite_command.clone(),
+        &mut tokens,
+        &registry,
+        TopLevelCommandCaseSensitivity::CaseSensitive,
+    );
+
+    assert_eq!(
+        classified_command,
+        Some(ClassifiedCommand {
+            env_vars: vec![],
+            command: Command::Unclassified(ExternalCommand {
+                name: ParsedToken("TEST".to_owned()),
+                name_span: Span::from((0, 4)),
+                args: CommandCallInfo {
+                    command_name: Spanned {
+                        span: Span::from((0, 5)),
+                        item: ParsedExpression::new(
+                            Expression::Literal,
+                            ParsedToken("TEST".to_owned())
+                        ),
+                    },
+                    positionals: None,
+                    flags: None,
+                    ending_whitespace: Some(Span::from((4, 5))),
+                    span: Span::from((0, 5))
+                }
+            }),
+            error: None,
+        })
+    )
+}
+
+/// TODO(CORE-2810)
+#[cfg(not(feature = "v2"))]
+#[test]
+fn test_classify_command_case_insensitive() {
+    let registry = create_test_command_registry([test_signature()]);
+
+    let lite_command = parse_for_completions("TEST ", EscapeChar::Backslash, false)
+        .expect("Should be able to parse input into LiteCommand");
+    let mut tokens = lite_command.parts.iter().map(|s| s.as_str()).collect_vec();
+
+    let classified_command = classify_command(
+        lite_command.clone(),
+        &mut tokens,
+        &registry,
+        TopLevelCommandCaseSensitivity::CaseInsensitive,
+    );
+
+    assert_eq!(
+        classified_command,
+        Some(ClassifiedCommand {
+            env_vars: vec![],
+            command: Command::Classified(ShellCommand {
+                name: "TEST".to_owned(),
+                name_span: Span::from((0, 4)),
+                args: CommandCallInfo {
+                    command_name: Spanned {
+                        span: Span::from((0, 4)),
+                        item: ParsedExpression::new(
+                            Expression::Command,
+                            ParsedToken("TEST".to_owned())
+                        ),
+                    },
+                    positionals: None,
+                    flags: Some(Flags::new()),
+                    ending_whitespace: Some(Span::from((4, 5))),
+                    span: Span::from((0, 5))
+                }
+            }),
+            error: None,
+        })
+    )
+}

crates/warp_completer/src/parsers/v2.rs

@@ -0,0 +1,435 @@
+//! Contains the V2 implementation of internal command parsing logic that depends on the new,
+//! JS-compatible command signature struct (`crate::signatures::CommandSignature`).
+use crate::signatures::{
+    get_matching_signature_for_tokenized_input, Command, CommandRegistry, Opt,
+};
+use crate::{
+    completer::TopLevelCommandCaseSensitivity,
+    meta::{HasSpan, Span, Spanned, SpannedItem},
+};
+
+use super::parse_unclassified_command;
+use super::{
+    hir::{self, Expression, Flags, ShellCommand},
+    parse_arg, parse_dollar_expr, ArgumentError, FlagArgumentsCardinality, FlagSignature,
+    LiteCommand, ParseError, ParsedExpression, ParsedToken,
+};
+
+pub(super) fn parse_command(
+    lite_cmd: &LiteCommand,
+    tokens: &[&str],
+    command_registry: &CommandRegistry,
+    // TODO(CORE-2810)
+    _command_case_sensitivity: TopLevelCommandCaseSensitivity,
+) -> (Option<hir::Command>, Option<ParseError>) {
+    let mut error: Option<ParseError> = None;
+
+    if lite_cmd.parts.is_empty() {
+        return (None, None);
+    }
+
+    if let Some((found_signature, token_index)) = get_matching_signature_for_tokenized_input(
+        tokens,
+        lite_cmd.post_whitespace.is_some(),
+        command_registry,
+    ) {
+        let (internal_command, err) =
+            parse_internal_command(lite_cmd, found_signature, token_index);
+
+        error = error.or(err);
+        return (Some(hir::Command::Classified(internal_command)), error);
+    }
+
+    let (command, error) = parse_unclassified_command(lite_cmd);
+    (Some(command), error)
+}
+
+// This is a forked version of `super::parse_internal_command` that works with the V2 `CommandSignature` struct.
+fn parse_internal_command(
+    lite_command: &LiteCommand,
+    command_signature: &Command,
+    mut command_token_index: usize,
+) -> (ShellCommand, Option<ParseError>) {
+    log::debug!("parsing internal command {lite_command:?}");
+
+    // This is a known internal command, so we need to work with the arguments and parse them according to the expected types
+    let (name, name_span) = (
+        lite_command.parts[0..(command_token_index + 1)]
+            .iter()
+            .map(|x| x.item.clone())
+            .collect::<Vec<String>>()
+            .join(" "),
+        Span::new(
+            lite_command.parts[0].span.start(),
+            lite_command.parts[command_token_index].span.end(),
+        ),
+    );
+
+    let mut internal_command = ShellCommand::new(ParsedToken(name), name_span, lite_command.span());
+    internal_command.args.flags = Some(Flags::new());
+    internal_command.args.ending_whitespace = lite_command.post_whitespace;
+
+    let mut current_positional = 0;
+    let mut named = Flags::new();
+    let mut positional = vec![];
+    let mut error = None;
+    command_token_index += 1; // Start where the arguments begin
+
+    while command_token_index < lite_command.parts.len() {
+        if lite_command.parts[command_token_index]
+            .item
+            .starts_with('-')
+            && lite_command.parts[command_token_index].item.len() > 1
+        {
+            let (named_types, err) = get_flag_signature_spec(
+                command_signature,
+                &internal_command,
+                &lite_command.parts[command_token_index],
+            );
+
+            if err.is_none() {
+                for FlagSignature {
+                    name: full_name,
+                    is_switch,
+                    arguments_cardinality,
+                    arguments,
+                } in named_types
+                {
+                    if is_switch {
+                        // Switch flag (without arguments)
+                        named.insert_flag_with_no_argument(
+                            full_name,
+                            lite_command.parts[command_token_index].span,
+                        );
+                    } else if lite_command.parts[command_token_index].item.contains('=') {
+                        // Self-contained option (--key=value)
+                        let mut offset = 0;
+
+                        let value = lite_command.parts[command_token_index]
+                            .item
+                            .chars()
+                            .skip_while(|prop| {
+                                offset += 1;
+                                *prop != '='
+                            })
+                            .nth(1);
+
+                        offset = if value.is_none() { offset - 1 } else { offset };
+
+                        let flag_value = Span::new(
+                            lite_command.parts[command_token_index].span.start() + offset,
+                            lite_command.parts[command_token_index].span.end(),
+                        );
+                        let value = lite_command.parts[command_token_index].item[offset..]
+                            .to_string()
+                            .spanned(flag_value);
+                        // We expect there to be exactly one arg in the case of a --key=value flag.
+                        let arg_signature = arguments.first();
+                        let (arg, err) = parse_arg(&value, arg_signature);
+                        named.insert_flag_with_argument(
+                            full_name,
+                            lite_command.parts[command_token_index].span,
+                            arg,
+                        );
+
+                        error = error.or(err);
+                    } else if command_token_index == lite_command.parts.len() - 1 {
+                        // Named argument with missing value
+                        error = error.or_else(|| {
+                            Some(ParseError::argument_error(
+                                lite_command.parts[0].clone(),
+                                ArgumentError::MissingValueForName {
+                                    name: full_name
+                                        .spanned(lite_command.parts[command_token_index].span),
+                                    missing_arg_index: 0,
+                                },
+                            ))
+                        });
+                    } else {
+                        // Named argument with following value(s).
+                        // Since an option can have multiple arguments (any of which can be variadic),
+                        // we should exhaust as many following args as possible.
+                        let flag_idx = command_token_index;
+
+                        let end = match arguments_cardinality {
+                            FlagArgumentsCardinality::Variadic => lite_command.parts.len(),
+                            FlagArgumentsCardinality::Fixed(num_args) => flag_idx + num_args + 1,
+                        };
+                        let mut argument_idx = command_token_index + 1;
+
+                        // Exhaust as many args as we expect but stop early if we see another option.
+                        // Note that since we are incrementing index again in the outer loop. Let's check
+                        // boundary on the NEXT token rather than the current token.
+                        while argument_idx < end.min(lite_command.parts.len())
+                            && !lite_command
+                                .parts
+                                .get(argument_idx)
+                                .is_some_and(|part| part.item.starts_with('-'))
+                        {
+                            let arg_signature_idx = argument_idx - flag_idx - 1;
+                            // Even though the completion spec technically allows for a variadic arg to not be the last arg,
+                            // it does not make sense and doesn't happen in practice, so we assume it's the last.
+                            let arg_signature = if arg_signature_idx >= arguments.len()
+                                && matches!(
+                                    arguments_cardinality,
+                                    FlagArgumentsCardinality::Variadic
+                                ) {
+                                arguments.last()
+                            } else {
+                                arguments.get(arg_signature_idx)
+                            };
+                            let (arg, err) =
+                                parse_arg(&lite_command.parts[argument_idx], arg_signature);
+                            named.insert_flag_with_argument(
+                                full_name.clone(),
+                                lite_command.parts[flag_idx].span,
+                                arg,
+                            );
+                            error = error.or(err);
+                            argument_idx += 1;
+                        }
+
+                        // If there were a fixed number of arguments for the option, make sure
+                        // they were all exhausted. Otherwise, we're missing an arg.
+                        if matches!(arguments_cardinality, FlagArgumentsCardinality::Fixed(_))
+                            && argument_idx != end
+                        {
+                            error = error.or_else(|| {
+                                Some(ParseError::argument_error(
+                                    lite_command.parts[0].clone(),
+                                    ArgumentError::MissingValueForName {
+                                        name: full_name
+                                            .spanned(lite_command.parts[command_token_index].span),
+                                        missing_arg_index: argument_idx - flag_idx - 1,
+                                    },
+                                ))
+                            });
+                        }
+
+                        // argument_idx here is one index overshoot of the last argument
+                        // token. Set the current index to be the index of last argument.
+                        command_token_index = argument_idx - 1;
+
+                        // We consumed the argument(s) for the option, so we should stop iterating the
+                        // possible matching flags. This case shouldn't happen as CLIs don't
+                        // generally support adding multiple flags with values in a single position
+                        break;
+                    }
+                }
+            } else {
+                positional.push(
+                    ParsedExpression::new(
+                        Expression::Unknown,
+                        ParsedToken(lite_command.parts[command_token_index].item.clone()),
+                    )
+                    .spanned(lite_command.parts[command_token_index].span),
+                );
+
+                error = error.or(err);
+            }
+        } else if !command_signature.arguments.is_empty()
+            && command_signature.arguments.len() > current_positional
+        {
+            let arg = {
+                // TODO: pass v2 Argument signature to parse_arg which only accepts v1 Argument
+                // let arg_signature = &command_signature.arguments[current_positional];
+                let (expr, err) = parse_arg(&lite_command.parts[command_token_index], None);
+
+                error = error.or(err);
+                expr
+            };
+
+            positional.push(arg);
+            current_positional += 1;
+        } else if let Some(_arg_signature) = command_signature
+            .arguments
+            .iter()
+            .rfind(|a| a.is_variadic())
+        {
+            // TODO: pass v2 Argument signature to parse_arg which only accepts v1 Argument
+            let (arg, err) = parse_arg(&lite_command.parts[command_token_index], None);
+            error = error.or(err);
+
+            positional.push(arg);
+            current_positional += 1;
+        } else {
+            let expression = if lite_command.parts[command_token_index]
+                .item
+                .starts_with('$')
+            {
+                parse_dollar_expr(&lite_command.parts[command_token_index])
+            } else {
+                ParsedExpression::new(
+                    Expression::Unknown,
+                    ParsedToken(lite_command.parts[command_token_index].item.clone()),
+                )
+                .spanned(lite_command.parts[command_token_index].span)
+            };
+
+            positional.push(expression);
+
+            error = error.or_else(|| {
+                Some(ParseError::argument_error(
+                    lite_command.parts[0].clone(),
+                    ArgumentError::UnexpectedArgument(
+                        lite_command.parts[command_token_index].clone(),
+                    ),
+                ))
+            });
+        }
+
+        command_token_index += 1;
+    }
+
+    let command_arguments = &command_signature.arguments;
+
+    // Count the required positional arguments and ensure these have been met
+    let mut required_arg_count = 0;
+    for positional_arg in command_arguments.iter() {
+        if !positional_arg.optional {
+            required_arg_count += 1;
+        }
+    }
+
+    if positional.len() < required_arg_count {
+        let arg = &command_arguments[positional.len()];
+        error = error.or_else(|| {
+            Some(ParseError::argument_error(
+                lite_command.parts[0].clone(),
+                ArgumentError::MissingMandatoryPositional {
+                    name: Some(arg.name.clone()),
+                    positional_index: positional.len(),
+                },
+            ))
+        });
+    }
+
+    if !named.is_empty() {
+        internal_command.args.flags = Some(named);
+    }
+
+    if !positional.is_empty() {
+        internal_command.args.positionals = Some(positional);
+    }
+
+    (internal_command, error)
+}
+
+fn get_flag_signature_spec<'a>(
+    command_signature: &'a Command,
+    cmd: &'a ShellCommand,
+    arg: &'a Spanned<String>,
+) -> (Vec<FlagSignature<'a>>, Option<ParseError>) {
+    if arg.item.starts_with('-') {
+        // It's a flag (or set of flags)
+        let mut output = vec![];
+        let mut error = None;
+
+        let remainder: String = arg.item.chars().skip(1).collect();
+
+        if remainder.starts_with('-') {
+            // Long flag expected
+            let mut remainder: String = remainder.chars().skip(1).collect();
+
+            if remainder.contains('=') {
+                let assignment: Vec<_> = remainder.split('=').collect();
+
+                if assignment.len() != 2 {
+                    error = Some(ParseError::argument_error(
+                        cmd.name.to_string().spanned(cmd.name_span),
+                        ArgumentError::InvalidValueForFlag(arg.clone()),
+                    ));
+                } else {
+                    remainder = assignment[0].to_string();
+                }
+            }
+
+            let mut found_remainder = false;
+            for option in command_signature.options.iter() {
+                if longhand_representations_for_opt(option).contains(&remainder) {
+                    output.push(FlagSignature {
+                        name: remainder.clone(),
+                        is_switch: option.arguments.is_empty(),
+                        // TODO: pass v2 Argument signature to parse_arg which only accepts v1 Argument
+                        arguments: &[],
+                        arguments_cardinality: FlagArgumentsCardinality::from(option),
+                    });
+                    found_remainder = true;
+                }
+            }
+
+            if !found_remainder {
+                error = Some(ParseError::argument_error(
+                    cmd.name.to_string().spanned(cmd.name_span),
+                    ArgumentError::UnexpectedFlag(arg.clone()),
+                ));
+            }
+        } else {
+            // Short flag(s) expected
+            let mut starting_pos = arg.span.start() + 1;
+            for c in remainder.chars() {
+                let mut found = false;
+
+                for option in command_signature.options.iter() {
+                    if shorthand_representations_for_opt(option).contains(&c.to_string()) {
+                        // TODO(alokedesai): Check if we should be using short or long here
+                        output.push(FlagSignature {
+                            name: c.to_string(),
+                            is_switch: option.arguments.is_empty(),
+                            // TODO: pass v2 Argument signature to parse_arg which only accepts v1 Argument
+                            arguments: &[],
+                            arguments_cardinality: FlagArgumentsCardinality::from(option),
+                        });
+                        found = true;
+                    }
+                }
+
+                if !found {
+                    error = Some(ParseError::argument_error(
+                        cmd.name.to_string().spanned(cmd.name_span),
+                        ArgumentError::UnexpectedFlag(
+                            arg.item
+                                .clone()
+                                .spanned(Span::new(starting_pos, starting_pos + c.len_utf8())),
+                        ),
+                    ));
+                }
+
+                starting_pos += c.len_utf8();
+            }
+        }
+
+        (output, error)
+    } else {
+        // It's not a flag, so don't bother with it
+        (vec![], None)
+    }
+}
+
+impl From<&Opt> for FlagArgumentsCardinality {
+    fn from(option: &Opt) -> Self {
+        if option.arguments.iter().any(|arg| arg.is_variadic()) {
+            FlagArgumentsCardinality::Variadic
+        } else {
+            FlagArgumentsCardinality::Fixed(
+                option.arguments.iter().filter(|arg| !arg.optional).count(),
+            )
+        }
+    }
+}
+
+fn shorthand_representations_for_opt(opt: &Opt) -> Vec<String> {
+    opt.name
+        .iter()
+        .filter(|s| s.starts_with('-') && !s.starts_with("--"))
+        .map(|s| s[1..].to_string())
+        .collect()
+}
+
+fn longhand_representations_for_opt(opt: &Opt) -> Vec<String> {
+    opt.name
+        .iter()
+        .filter(|s| s.starts_with("--"))
+        .map(|s| s[2..].to_string())
+        .collect()
+}