Scanner

open class Scanner<T>(val input: Iterator<Token<T>>, val eofType: T)

Wraps a token Iterator (or tokenizes an input String directly via regular expression rules) with current-token inspection, unlimited lookahead, and consumption helpers for recursive-descent and Pratt parsers.

Subclassing Scanner (for example class MyScanner(input: String) : Scanner<TokenType>(...)) is recommended for non-trivial parsers so parser functions can accept MyScanner instead of repeating Scanner<TokenType>, and custom helper methods can be added as needed. Direct instantiation (Scanner(input, ...)) is also supported for simple cases.

Pass a dedicated enum value (e.g. TokenType.EOF) as eofType to keep Token.type non-nullable; pass null with a nullable type parameter (Scanner<TokenType?>(input, null, ...)) to avoid adding an EOF enum entry; or use Scanner<Unit?>(input, null, ...) (mapping emitted tokens to Unit) when token types aren't needed at all.

Constructors

Link copied to clipboard
constructor(input: Iterator<Token<T>>, eofType: T)
constructor(input: Iterator<Token<T>>, eofType: T, eofText: String)
constructor(input: String, eofType: T, vararg types: Pair<Regex, T?>)

Convenience constructor that tokenizes input directly using Lexer with the given types rules (Regex to tokenType, or Regex to null for skipped tokens such as whitespace).

Properties

Link copied to clipboard
Link copied to clipboard
Link copied to clipboard
val eofType: T
Link copied to clipboard

Functions

Link copied to clipboard
fun consume(): Token<T>

Consumes and returns the current token.

fun consume(type: T, errorMessage: (Token<T>) -> String = { "Token type $type expected." }): Token<T>

Consumes a token with the given type and returns it. If the current token has a different type, an exception is thrown.

fun consume(text: String, errorMessage: (Token<T>) -> String = { "Expected: '$text'" }): Token<T>

Consumes and returns a token with the given text value. If the current token text does not match, an exception is thrown.

fun consume(text: String, ignoreCase: Boolean, errorMessage: (Token<T>) -> String = { "Expected: '$text'" }): Token<T>
Link copied to clipboard
fun consumeIgnoreCase(text: String, errorMessage: (Token<T>) -> String = { "Expected: '$text'" }): Token<T>

Consumes and returns a token matching text case-insensitively. If the current token text does not match, an exception is thrown.

Link copied to clipboard

Wraps the given exception in a parsing exception if it's not a parsing exception already. Parsing exceptions are returned unchanged. Useful to associate other exceptions encountered during parsing with a token (including the corresponding input position).

Link copied to clipboard

Creates a parsing exception with position context information.

Link copied to clipboard
fun lookAhead(index: Int): Token<T>
Link copied to clipboard
fun require(condition: Boolean, message: () -> String)
Link copied to clipboard
fun requireEof(message: () -> String = { "EOF expected." })
Link copied to clipboard
open override fun toString(): String
Link copied to clipboard

If the current token text value matches the given string, it is consumed and true is returned. Otherwise, false is returned.

fun tryConsume(value: String, ignoreCase: Boolean): Boolean
Link copied to clipboard

If the current token text value matches the given string case-insensitively, it is consumed and true is returned. Otherwise, false is returned.