Parsek/org.kobjects.parsek.tokenizer

Package-level declarations

Contains the lexer, scanner and some utilities such as a filter base class and some common regular expressions.

Types

abstract class Filter<T>(val filtered: Iterator<Token<T>>) : Iterator<Token<T>>

A Filter for the lexer, useful for filtering (e.g.) insignificant newlines. The accept function is not just a parameter but needs to be overridden in a subclass as it's expected to be stateful.

Lexer

open class Lexer<T>(input: String, types: Pair<Regex, (String) -> T?>, textNormalization: (T, String) -> String = { _, s -> s }, offset: Token<T>? = null) : Iterator<Token<T>>

Splits the input string into tokens, exposed as token iterator. The type parameter indicates the token type; typically an enum consisting of values such as IDENTIFIER, NUMBER etc.

ParsingException

class ParsingException(val token: Token<*>, message: String?, cause: Throwable? = null) : Exception

Exception type for parsing errors containing a reference to the current token.

RegularExpressions

object RegularExpressions

A set of regular expressions that might be useful for parsing.

Scanner

open class Scanner<T>(val input: Iterator<Token<T>>, val eofType: T, val eofText: String = "")

Class that wraps a "plain" lexer (implemented as a token iterator) for more comfortable token analysis in a parser.

Token

data class Token<T>(val localPos: Int, val pos: Int, val line: Int, val col: Int, val type: T, val text: String)

Tokens in parsek are represented by this class.