Pretokens

Each pretoken has an extent, which is a sequence of characters taken from the input.

Each pretoken has a kind, and possibly also some attributes, as described in the tables below.

KindAttributes
Reserved
Whitespace
LineCommentcomment content
BlockCommentcomment content
Punctuationmark
Identifieridentifier
RawIdentifieridentifier
LifetimeOrLabelname
RawLifetimeOrLabelname
SingleQuoteLiteralprefix, literal content, suffix
DoubleQuoteLiteralprefix, literal content, suffix
RawDoubleQuoteLiteralprefix, literal content, suffix
IntegerDecimalLiteraldigits, suffix
IntegerHexadecimalLiteraldigits, suffix
IntegerOctalLiteraldigits, suffix
IntegerBinaryLiteraldigits, suffix
FloatLiteralhas base, body, exponent digits, suffix

These attributes have the following types:

AttributeType
bodysequence of characters
digitssequence of characters
exponent digitseither a sequence of characters, or none
has basetrue or false
identifiersequence of characters
literal contentsequence of characters
comment contentsequence of characters
marksingle character
namesequence of characters
prefixsequence of characters
suffixsequence of characters