Pretokens
Each pretoken has an extent, which is a sequence of characters taken from the input.
Each pretoken has a kind, and possibly also some attributes, as described in the tables below.
Kind | Attributes |
---|---|
Reserved | |
Whitespace | |
LineComment | comment content |
BlockComment | comment content |
Punctuation | mark |
Identifier | identifier |
RawIdentifier | identifier |
LifetimeOrLabel | name |
RawLifetimeOrLabel | name |
SingleQuoteLiteral | prefix, literal content, suffix |
DoubleQuoteLiteral | prefix, literal content, suffix |
RawDoubleQuoteLiteral | prefix, literal content, suffix |
IntegerDecimalLiteral | digits, suffix |
IntegerHexadecimalLiteral | digits, suffix |
IntegerOctalLiteral | digits, suffix |
IntegerBinaryLiteral | digits, suffix |
FloatLiteral | has base, body, exponent digits, suffix |
These attributes have the following types:
Attribute | Type |
---|---|
body | sequence of characters |
digits | sequence of characters |
exponent digits | either a sequence of characters, or none |
has base | true or false |
identifier | sequence of characters |
literal content | sequence of characters |
comment content | sequence of characters |
mark | single character |
name | sequence of characters |
prefix | sequence of characters |
suffix | sequence of characters |