Writeup of Rust's lexer

Identifier, lifetime, and label pretokens

Recall that the IDENT nonterminal is defined as follows:

Grammar

IDENT = { IDENT_START ~ XID_CONTINUE * }
IDENT_START = { XID_START | "_" }

Note: This is following the specification in Unicode Standard Annex #31 for Unicode version 16.0, with the addition of permitting underscore as the first character.

Raw lifetime or label (Rust 2021 and 2024)

Grammar

Raw_lifetime_or_label_2021 = { "'r#" ~ IDENT ~ !"'" }

Pretoken kind

RawLifetimeOrLabel

Attributes

Reserved lifetime or label prefix (Rust 2021 and 2024)

Grammar

Reserved_lifetime_or_label_prefix_2021 = { "'" ~ IDENT ~ "#" }

Pretoken kind

Reserved

Attributes

(none)

(Non-raw) lifetime or label

Grammar

Lifetime_or_label = { "'" ~ IDENT ~ !"'" }

Pretoken kind

LifetimeOrLabel

Attributes

Note: The !"'" at the end of the expression makes sure that forms like 'aaa'bbb are not accepted.

Raw identifier

Grammar

Raw_identifier = { "r#" ~ IDENT }

Pretoken kind

RawIdentifier

Attributes

Reserved prefix

Grammar

Reserved_prefix_2015 = { "r#" | "br#" }
Reserved_prefix_2021 = { IDENT ~ "#" }

Pretoken kind

Reserved

Attributes

(none)

Note: This definition must appear here in priority order. Tokens added in future which match these reserved forms wouldn't necessarily be forms of identifier.

(Non-raw) identifier

Grammar

Identifier = { IDENT }

Pretoken kind

Identifier

Attributes

Note: The Reference adds the following when discussing identifiers: "Zero width non-joiner (ZWNJ U+200C) and zero width joiner (ZWJ U+200D) characters are not allowed in identifiers." Those characters don't have XID_Start or XID_Continue, so that's only informative text, not an additional constraint.