Pretokens
Each pretoken has an extent, which is a sequence of characters taken from the input.
Each pretoken has a kind, and possibly also some attributes, as described in the tables below.
| Kind | Attributes |
|---|---|
Reserved | |
Whitespace | |
LineComment | comment content |
BlockComment | comment content |
Punctuation | mark |
Identifier | identifier |
RawIdentifier | identifier |
LifetimeOrLabel | name |
RawLifetimeOrLabel | name |
SingleQuotedLiteral | prefix, literal content, suffix |
DoubleQuotedLiteral | prefix, literal content, suffix |
RawDoubleQuotedLiteral | prefix, literal content, suffix |
IntegerLiteral | base, digits, suffix |
FloatLiteral | body, suffix |
These attributes have the following types:
| Attribute | Type |
|---|---|
| base | binary / octal / decimal / hexadecimal |
| body | sequence of characters |
| comment content | sequence of characters |
| digits | sequence of characters |
| identifier | sequence of characters |
| literal content | sequence of characters |
| mark | single character |
| name | sequence of characters |
| prefix | sequence of characters |
| suffix | either a sequence of characters or none |