Lexer

open class Lexer<T>(input: String, vararg types: Pair<Regex, T?>, offset: Token<T>? = null) : Iterator<Token<T>>

Splits input into tokens, exposed as a lazy Iterator.

Most parsers can pass regular expression rules directly to Scanner without instantiating Lexer explicitly; use Lexer directly when you need to inspect or filter the raw token stream before passing it to a Scanner (for example via lexer.asSequence().filter { ... }.iterator()).

The type parameter T indicates the token type (typically an enum such as IDENTIFIER, NUMBER, etc., or Unit when token types aren't needed). Custom Regex instances should be stored in constants (such as RegularExpressions or top-level private vals) rather than constructed on each parse call.

Constructors

Link copied to clipboard
constructor(input: String, vararg types: Pair<Regex, T?>, offset: Token<T>? = null)

Functions

Link copied to clipboard
open operator override fun hasNext(): Boolean
Link copied to clipboard
open operator override fun next(): Token<T>