Writeup of Rust's lexer

Ident, lifetime, and label tokens

Table of contents

• Raw lifetime or label (Rust 2021 and 2024)
• Reserved lifetime or label prefix (Rust 2021 and 2024)
• (Non-raw) lifetime or label
• Raw ident
• Reserved prefix
• (Non-raw) ident

This writeup uses the term ident to refer to a token that lexically has the form of an identifier, including keywords and _.

Note: the procedural macros system uses the name Ident to refer to what this writeup calls Ident and Raw_ident.

The following nonterminals are common to the definitions below:

Grammar

IDENT = { IDENT_START ~ XID_CONTINUE * }
IDENT_START = { XID_START | "_" }

Note: This is following the specification in Unicode Standard Annex #31 for Unicode version 16.0, with the addition of permitting underscore as the first character.

See Special terminals for the definitions of XID_START and XID_CONTINUE.

Raw lifetime or label (Rust 2021 and 2024)

Grammar

Raw_lifetime_or_label = { "'r#" ~ IDENT }

Attributes

The token's name is IDENT.

Note that the name is not NFC-normalised. See NFC normalisation for lifetime/label.

Rejection

The match is rejected if IDENT is one of the following sequences of characters:

• _
• crate
• self
• super
• Self

Reserved lifetime or label prefix (Rust 2021 and 2024)

Grammar

Reserved_lifetime_or_label_prefix = { "'" ~ IDENT ~ "#" }

Rejection

All matches are rejected.

(Non-raw) lifetime or label

Grammar

Lifetime_or_label = { "'" ~ IDENT }

Note: The Reserved_single_quoted_literal definitions make sure that forms like 'aaa'bbb are not accepted.

See Modelling lifetimes and labels for a discussion of why this model doesn't simply treat ' as punctuation.

Attributes

The token's name is IDENT.

Note that the name is not NFC-normalised. See NFC normalisation for lifetime/label.

Rejection

No matches are rejected.

Raw ident

Grammar

Raw_ident = { "r#" ~ IDENT }

Attributes

The token's represented ident is the NFC-normalised form of IDENT.

Rejection

The match is rejected if the token's represented ident would be one of the following sequences of characters:

• _
• crate
• self
• super
• Self

Reserved prefix

Grammar

Reserved_prefix_2015 = { "r#" | "br#" }
Reserved_prefix_2021 = { IDENT ~ "#" }

Rejection

All matches are rejected.

Note: This definition must appear here in priority order. Tokens added in future which match these reserved forms wouldn't necessarily be forms of identifier.

(Non-raw) ident

Grammar

Ident = { IDENT }

Note: The Reference adds the following when discussing identifiers: "Zero width non-joiner (ZWNJ U+200C) and zero width joiner (ZWJ U+200D) characters are not allowed in identifiers." Those characters don't have XID_Start or XID_Continue, so that's only informative text, not an additional constraint.

Attributes

The token's represented ident is the NFC-normalised form of IDENT

Rejection

No matches are rejected.