Pretokens
Each pretoken has an extent, which is a sequence of characters taken from the input.
Each pretoken has a kind, and possibly also some attributes, as described in the tables below.
| Kind | Attributes |
|---|---|
Reserved | |
Whitespace | |
LineComment | comment content |
BlockComment | comment content |
Punctuation | mark |
Identifier | identifier |
RawIdentifier | identifier |
LifetimeOrLabel | name |
RawLifetimeOrLabel | name |
SingleQuoteLiteral | prefix, literal content, suffix |
DoubleQuoteLiteral | prefix, literal content, suffix |
RawDoubleQuoteLiteral | prefix, literal content, suffix |
IntegerDecimalLiteral | digits, suffix |
IntegerHexadecimalLiteral | digits, suffix |
IntegerOctalLiteral | digits, suffix |
IntegerBinaryLiteral | digits, suffix |
FloatLiteral | has base, body, exponent digits, suffix |
These attributes have the following types:
| Attribute | Type |
|---|---|
| body | sequence of characters |
| digits | sequence of characters |
| exponent digits | either a sequence of characters, or none |
| has base | true or false |
| identifier | sequence of characters |
| literal content | sequence of characters |
| comment content | sequence of characters |
| mark | single character |
| name | sequence of characters |
| prefix | sequence of characters |
| suffix | sequence of characters |