Tokenisation grammar
As explained above, the grammar defines a tokens nonterminal for each Rust edition. They are presented below.
The rest of the grammar is presented in the following pages in this section. The definitions of some nonterminals are repeated on multiple pages for convenience.
The definitions of the tokenisation nonterminals are presented an order consistent with their appearances in the choice expressions below. That means they appear in priority order (higher priority earlier).
The full grammar is also available on a single page.
Grammar
TOKENS_2015 = { TOKEN_2015 * }
TOKENS_2021 = { TOKEN_2021 * }
TOKENS_2024 = { TOKEN_2024 * }
TOKEN_2015 = {
Whitespace |
Line_comment |
Block_comment |
Reserved_block_comment_start |
Character_literal |
Byte_literal |
String_literal |
Byte_string_literal |
Raw_string_literal |
Raw_byte_string_literal |
Reserved_literal_2015 |
Reserved_single_quoted_literal_2015 |
Float_literal |
Reserved_float |
Integer_literal |
Lifetime_or_label |
Raw_ident |
Reserved_prefix_2015 |
Ident |
Punctuation
}
TOKEN_2021 = {
Whitespace |
Line_comment |
Block_comment |
Reserved_block_comment_start |
Character_literal |
Byte_literal |
String_literal |
Byte_string_literal |
C_string_literal |
Raw_string_literal |
Raw_byte_string_literal |
Raw_c_string_literal |
Reserved_literal_2021 |
Reserved_single_quoted_literal_2021 |
Float_literal |
Reserved_float |
Integer_literal |
Raw_lifetime_or_label |
Reserved_lifetime_or_label_prefix |
Lifetime_or_label |
Raw_ident |
Reserved_prefix_2021 |
Ident |
Punctuation
}
TOKEN_2024 = {
Whitespace |
Line_comment |
Block_comment |
Reserved_block_comment_start |
Character_literal |
Byte_literal |
String_literal |
Byte_string_literal |
C_string_literal |
Raw_string_literal |
Raw_byte_string_literal |
Raw_c_string_literal |
Reserved_literal_2021 |
Reserved_single_quoted_literal_2021 |
Reserved_guard |
Float_literal |
Reserved_float |
Integer_literal |
Raw_lifetime_or_label |
Reserved_lifetime_or_label_prefix |
Lifetime_or_label |
Raw_ident |
Reserved_prefix_2021 |
Ident |
Punctuation
}