Manual: tokenizer

Converts strings into Lever tokens

This tokenizer is the first thing that happens to a Lever program that is about to be run. The tokenizer is invoked by the parser and the compiler.

To bootstrap Lever's compiler with Python, there is a ported implementation of this module in compiler/lever_parser/reader/.

Lever's tokenizer resembles Python's tokenizer a lot. The major difference is that the keywords and operators collected by the tokenizer are determined by the 'table' -argument passed in as an argument. This allows the parsing engine to have a last word on the keywords the tokenizer parses.

This tokenizer should understand..

1. Comments starting with '#' that continue to the end of the line.
2. Python's raw string syntax, eg. r"foo\bar", r'foo\bar'
3. Symbols, eg. identifiers, regex. [a-zA-Z_][a-zA-Z0-9_]*
4. Hexadecimals starting with 0x...
5. Whole numbers, regex. [0-9]+
6. Numbers with decimal point and exponent, eg. 1.23e-10, 1.2
7. Strings with single or double quotes, eg. 'hello' or "world"
8. Custom keywords or operators.

The tokenizer treats space as separator between tokens, and does not produce Literal -objects for it.

01. Keyword smearing

The keyword table given to the tokenizer may contain symbols such as '!='. For these to be correctly recognized, the non-alphabetic keywords have to be smeared.

Here's a smearing function you can use for that purpose:

default_smear = (keyword):
    for ch in keyword
        if ch.is_alpha()
            return [keyword]
    result = []
    prefix = []
    for ch in keyword
        prefix.append(ch)
        result.append("".join(prefix))
    return result

02. API

03. Internal details

The rest of this file isn't well-documented.