Scanner
Wraps a token Iterator (or tokenizes an input String directly via regular expression rules) with current-token inspection, unlimited lookahead, and consumption helpers for recursive-descent and Pratt parsers.
Subclassing Scanner (for example class MyScanner(input: String) : Scanner<TokenType>(...)) is recommended for non-trivial parsers so parser functions can accept MyScanner instead of repeating Scanner<TokenType>, and custom helper methods can be added as needed. Direct instantiation (Scanner(input, ...)) is also supported for simple cases.
Pass a dedicated enum value (e.g. TokenType.EOF) as eofType to keep Token.type non-nullable; pass null with a nullable type parameter (Scanner<TokenType?>(input, null, ...)) to avoid adding an EOF enum entry; or use Scanner<Unit?>(input, null, ...) (mapping emitted tokens to Unit) when token types aren't needed at all.
Constructors
Convenience constructor that tokenizes input directly using Lexer with the given types rules (Regex to tokenType, or Regex to null for skipped tokens such as whitespace).
Properties
Functions
Consumes and returns the current token.
Consumes a token with the given type and returns it. If the current token has a different type, an exception is thrown.
Consumes and returns a token with the given text value. If the current token text does not match, an exception is thrown.
Wraps the given exception in a parsing exception if it's not a parsing exception already. Parsing exceptions are returned unchanged. Useful to associate other exceptions encountered during parsing with a token (including the corresponding input position).
Creates a parsing exception with position context information.
If the current token text value matches the given string, it is consumed and true is returned. Otherwise, false is returned.
If the current token text value matches the given string case-insensitively, it is consumed and true is returned. Otherwise, false is returned.