Manual: grammar

Describes the language for attribute grammars used during Lever's parsing.

Table of contents ↑
Table of contents
01. Structure
01.1. How to read the attribute
01.2. Capturing elements
01.3. Index and dot annotations
01.4. Annotation expressions
02. Extensions
02.1. indentation -extension
02.2. alias -extension
03. A Simple Example
04. API
05. Internal details

There is distinct effort made to make the Lever language itself easier to develop. The grammar language described here has a big role in there.

Here's the context-free attribute grammar for the grammar language, described in itself:

use alias()
    a = [symbol]
    on = ['constructive', 'use', 'terminal', 'append', 'concat']
use indentation(indent, dedent, newline)
    can_close = [')', ']', '}']

file: sep(decl, newline)

decl:
    'use' symbol '(' sep(symbol, ',') ')' (indent join(option, newline) dedent)? / use
    'constructive'? 'terminal' join(symbol, ',') / terminal
    symbol '(' sep(symbol, ',') ')' ':' body / template
    symbol ':' body / rule

option: symbol '=' '[' sep(primitive, ',') ']'

primitive:
    symbol / symbol
    string / string

body:
    prod / [.]
    indent join(prod, newline) dedent

prod: expr* ('/' [annotation, symbol / shorthand])? / prod 

expr:
    term
    '(' arg_block(prod) ')' '+' / plus
    '(' arg_block(prod) ')' '*' / star
    '(' arg_block(prod) ')' '?' / opt
    symbol '(' arg_block(prod) ')' / expand

term:
    term '+' / plus
    term '*' / star
    term '?' / opt
    symbol / symbol
    string / string
    '[' arg_block(prod) ']' / prod_set

annotation:
    annotation_term
    annotation 'append' annotation_term / append
    annotation 'concat' annotation_term / concat

annotation_term:
    '(' annotation ')'
    int / index
    '.' / dot
    '..' / dotdot
    'null' / a_null

    symbol '(' sep(annotation, ',') ')' / label
    '[' sep(annotation, ',') ']' / a_list

arg_block(x):
    sep(x, ',', ',')
    indent sep_concat(x, newline) dedent

sep_concat(x, y):
    sep(x, ',', ',')
    seq_b(x) y sep(x, ',', ',') / (. concat .)

01. Structure

The 'lhs: rhs' part in the grammar is the production rule, the '/ thing' is an annotation that the rule maps into.

01.1. How to read the attribute

The annotation coming after the rule may consists of a single symbol or even just an expression.

The annotation is a template that is filled by deducting the details from the expression.

01.2. Capturing elements

Some pieces of the right hand side rule are ignored. These are the keyword string tokens and tokens that have been marked as 'constructive', such as indentation and dedentation tokens.

If there is not a single element that is captured, the ignored elements are retrieved in whole, otherwise only the capturing elements are captured.

When there is no annotation provided: If there is only one element captured, it is captured 'as it', otherwise the captured elements are inserted into a list.

When there is a single symbol, the elements are captured into an attribute that holds the name in that symbol.

01.3. Index and dot annotations

The index and dot annotations are read such that the elements captured by the index are marked as retrieved and not double-captured by the dot annotations.

Every item, captured or not captured, has an unique index. These indices start from 1, which is merely a convention of the grammar language. You can think of the index '0' as a symbol on the left hand side if you're a programmer, although the left-hand side cannot be captured.

The single dot '.' captures an one symbol, whereas the double dot '..' captures all remaining symbols. The double dot may only occur once, and the single dot may appear before and after the double dot.

01.4. Annotation expressions

In the annotation you may denote that the contents are placed into a list, or that something gets appended into a list or that lists are concatenated together.

You may also denote that a null is explicitly passed into an annotation.

02. Extensions

Extensions do not change how the source grammar is parsed, but they will provide additional hints about how it will be used.

The use of an extension is described with the 'use' -directive, and the ordering of the extensions determines in which order they are invoked during parsing.

The symbols passed as 'arguments' are defined as constructive terminals, which means they are not gathered during parsing. These are used

02.1. indentation -extension

The 'indentation' extension allows you to define layout-sensitive grammars. The terminals provided as input provide tokens that may be used to trigger the layout sensitivity.

The indent/dedent/newline is implemented by feeding such a token whenever it may appear and when the parsing state expects it. Therefore the current implementation of layout sensitivity may result in surprising results at ambiguous grammars.

The can_close -option lets you select tokens that allow the indentation level to decrease without reaching the end of the line first.

02.2. alias -extension

The 'alias' extension is meant for situations when you want to have keywords, but do not want them to mess up with user code.

The effect of 'alias' is that whenever the parsing cannot proceed with one of the terminals in the 'on' -list, but it could proceed with one of the terminals in the 'a' -list, the alias extension will rename the terminal to allow the parsing to pass.

Similarly to the indentation extension, the details are still a bit vague on these ones, and may be subject to change.

03. A Simple Example

Here's a complicated way to print '4' with the grammar library:

The 'calc.grammar':

addition: term '+' term / add

term: int / int

The 'calc.lc':

import grammar

main = (args):
    calc_lang = grammar.read_file(dir ++ 'calc.grammar')

    result = calc_lange.read_string("1 + 3")
    result = result.traverse(traverse)
    print(result)

traverse = (name, args, loc):
    if name == 'int'
        return parse_int(args[0])
    elif name == 'add'
        return args[0] + args[1]
    else
        assert false, [name, args, loc]

04. API

read_file(filename, extension_env = default_extensions)
filename The source path to the grammar
extension_env Environment for the extensions.
returns Initiator
The simplest way to use the grammar module.
volatile
class Initiator extends object
Can initiate the parsing.

The behavior of this initiator is similar to the chartparser.Initiator. You can use this object to initiate parsing with the grammar.

volatile
+call(self, accept = null)
self not documented
accept not documented
returns not documented
not documented
volatile
+init(self, init, extensions, keywords, symbols)
internal
create_symtab(self)
self not documented
returns Symbol table to be used with the 'tokenizer' -object.
not documented
volatile
read_file(self, filename)
self not documented
filename not documented
returns Filled-up Parser -instance.
Shorthand for running the parser on a file.
volatile
read_string(self, string, source = null)
self not documented
string not documented
source not documented
returns Filled-up Parser -instance.
Shorthand for running the parser on a string.
volatile
read_tokens(self, tokens, source = null)
self not documented
tokens not documented
source not documented
returns Filled-up Parser -instance.
Shorthand for running the parser on a sequence of tokens.
volatile
class Parser extends object
Parser instance.

Wraps the parser produced from the grammar into a bit easier-to-use packaging.

volatile
+init(self, pos, init, child, extensions)
internal
accepted : property
not documented
volatile
expect : property
not documented
volatile
expecting(self, symbol)
self not documented
symbol not documented
returns not documented
not documented
volatile
finish(self)
self not documented
returns not documented
not documented
volatile
step(self, term, token, start = null, stop = null)
self not documented
term not documented
token not documented
start not documented
stop not documented
returns not documented
not documented
volatile
traverse(self, postorder_cb, blank_cb = null, resolve_cb = null)
self not documented
postorder_cb not documented
blank_cb not documented
resolve_cb not documented
returns not documented
not documented
volatile

05. Internal details

The internals of this module are not well-documented yet.

class AliasExtension extends object
not documented
volatile
+init(self, pos, args, options)
self not documented
pos not documented
args not documented
options not documented
returns not documented
not documented
volatile
finish(self, i)
self not documented
i not documented
returns not documented
not documented
volatile
step(self, i, term, token, start, stop)
self not documented
i not documented
term not documented
token not documented
start not documented
stop not documented
returns not documented
not documented
volatile
class Append extends object
not documented
volatile
+init(self, sequence, value)
self not documented
sequence not documented
value not documented
returns not documented
not documented
volatile
+repr(self)
self not documented
returns not documented
not documented
volatile
postorder(self, fn, args, loc)
self not documented
fn not documented
args not documented
loc not documented
returns not documented
not documented
volatile
class Arg extends object
not documented
volatile
+hash(self)
self not documented
returns not documented
not documented
volatile
+init(self, value)
self not documented
value not documented
returns not documented
not documented
volatile
get_marks(self, marks)
self not documented
marks not documented
returns not documented
not documented
volatile
validate(self, key...)
self not documented
key not documented
returns not documented
not documented
volatile
class Concat extends object
not documented
volatile
+init(self, left, right)
self not documented
left not documented
right not documented
returns not documented
not documented
volatile
+repr(self)
self not documented
returns not documented
not documented
volatile
postorder(self, fn, args, loc)
self not documented
fn not documented
args not documented
loc not documented
returns not documented
not documented
volatile
class Expr extends object
not documented
volatile
+call(self, stage, xenv)
self not documented
stage not documented
xenv not documented
returns not documented
not documented
volatile
+hash(self)
self not documented
returns not documented
not documented
volatile
+init(self, fn, args)
self not documented
fn not documented
args not documented
returns not documented
not documented
volatile
class Get extends object
not documented
volatile
+init(self, index)
self not documented
index not documented
returns not documented
not documented
volatile
+repr(self)
self not documented
returns not documented
not documented
volatile
postorder(self, fn, args, loc)
self not documented
fn not documented
args not documented
loc not documented
returns not documented
not documented
volatile
class IndentationExtension extends object
not documented
volatile
+init(self, pos, args, options)
self not documented
pos not documented
args not documented
options not documented
returns not documented
not documented
volatile
finish(self, i)
self not documented
i not documented
returns not documented
not documented
volatile
step(self, i, term, token, start, stop)
self not documented
i not documented
term not documented
token not documented
start not documented
stop not documented
returns not documented
not documented
volatile
class Label extends object
not documented
volatile
+init(self, name, args)
self not documented
name not documented
args not documented
returns not documented
not documented
volatile
+repr(self)
self not documented
returns not documented
not documented
volatile
postorder(self, fn, args, loc)
self not documented
fn not documented
args not documented
loc not documented
returns not documented
not documented
volatile
class List extends object
not documented
volatile
+init(self, args = null)
self not documented
args not documented
returns not documented
not documented
volatile
+repr(self)
self not documented
returns not documented
not documented
volatile
postorder(self, fn, args, loc)
self not documented
fn not documented
args not documented
loc not documented
returns not documented
not documented
volatile
class Null extends object
not documented
volatile
+repr(self)
self not documented
returns not documented
not documented
volatile
postorder(self, fn, args, loc)
self not documented
fn not documented
args not documented
loc not documented
returns not documented
not documented
volatile
class Prod extends object
not documented
volatile
+hash(self)
self not documented
returns not documented
not documented
volatile
+init(self, value, gather, annotation)
self not documented
value not documented
gather not documented
annotation not documented
returns not documented
not documented
volatile
get_marks(self, marks)
self not documented
marks not documented
returns not documented
not documented
volatile
class Sym extends object
not documented
volatile
+hash(self)
self not documented
returns not documented
not documented
volatile
+init(self, value, gather)
self not documented
value not documented
gather not documented
returns not documented
not documented
volatile
get_marks(self, marks)
self not documented
marks not documented
returns not documented
not documented
volatile
actions : object
not documented
volatile
build_grammar_language()
returns not documented
not documented
volatile
chartparser : Module
not documented
volatile
default_extensions : dict
not documented
volatile
dir : path
not documented
volatile
expand_template(env, name, args)
env not documented
name not documented
args not documented
returns not documented
not documented
volatile
expr_append(stage, xenv, args)
stage not documented
xenv not documented
args not documented
returns not documented
not documented
volatile
expr_concat(stage, xenv, args)
stage not documented
xenv not documented
args not documented
returns not documented
not documented
volatile
expr_dot(stage, xenv, args)
stage not documented
xenv not documented
args not documented
returns not documented
not documented
volatile
expr_dotdot(stage, xenv, args)
stage not documented
xenv not documented
args not documented
returns not documented
not documented
volatile
expr_index(stage, xenv, args)
stage not documented
xenv not documented
args not documented
returns not documented
not documented
volatile
expr_label(stage, xenv, args)
stage not documented
xenv not documented
args not documented
returns not documented
not documented
volatile
expr_list(stage, xenv, args)
stage not documented
xenv not documented
args not documented
returns not documented
not documented
volatile
expr_null(stage, xenv, args)
stage not documented
xenv not documented
args not documented
returns not documented
not documented
volatile
extract_annotation(expr, args)
expr not documented
args not documented
returns not documented
not documented
volatile
import : Import
not documented
volatile
language : object
not documented
volatile
make_rule(lhs, rhs, fn)
lhs not documented
rhs not documented
fn not documented
returns not documented
not documented
volatile
name = "grammar"
not documented
volatile
read_file_bare(filename)
filename not documented
returns not documented
not documented
volatile
tokenizer : Module
not documented
volatile