Lexer
open class Lexer<T>(input: String, types: Pair<Regex, (String) -> T?>, textNormalization: (T, String) -> String = { _, s -> s }, offset: Token<T>? = null) : Iterator<Token<T>>
Splits the input string into tokens, exposed as token iterator. The type parameter indicates the token type; typically an enum consisting of values such as IDENTIFIER, NUMBER etc.