Manual: technical overview
Explains the project structure. Describes how the runtime ticks.
Here we explain what is in the github repository. It is also explained what is in the runtime when it loads up a program.
Table of contents ↑01. Project hierarchy
app benchmarks compiler doc documentation_considerations.tex headers internal_documentation.txt lever lever-0.9.0.grammar lever.py lib LICENSE.md local marketing native_considerations.txt performance.text README.md runtime sample.lc samples semantics_documentation.txt setup.py test.py tests tool_ideas.text tools VERSION website_considerations.txt win32_extras www
01.1. app -directory
Contains the application script that is run when the lever executable is run without arguments.
01.2. benchmarks -directory
Some programs that have been used to optimize the runtime.
01.3. compiler -directory
Contains the bootstrapping compiler to bootstrap the another one in the lib/compiler.lc
01.4. doc -directory
Contains nearly all the documentation for the project. The documentation is intended to be readable by the runtime so it's been written in a weird language.
01.5. documentation_considerations.tex
Some old discussion about solving the documentation problems in the project. They are still not entirely solved, but the ideas in this file drove them further. Should be merged with the rest of the documentation.
01.6. headers -directory
A slightly old directory that contains C FFI files that are used by libraries. They could also sit in the lib/ -directory by now, but this directory was there before the module system had improved to provide a method to load these json files from the lib.
01.7. internal_documentation.txt
A very early form of documentation about the runtime -directory. Should be also merged with the main documentation.
01.8. lever, lever.exe
The executable binary that is generated from the runtime directory.
01.9. lever-0.9.0.grammar
The machine-readable grammar file. Used by both the compiler/ and lib/compiler.lc. A form of an attribute grammar.
01.10. lever.py
A script to run lever runtime interpreted by Python. Barely ever used these days because it's so slow method to run the code.
01.11. lib -directory
A comprehensive set of modules written in Lever meant to become a standard library. It is distributed along the runtime.
01.12. LICENSE.md
License file for the repository. It's a MIT license.
01.13. local -directory
This directory gets created by the setup.py -script. It contains third party dependencies that are required along the binary.
Doing this ensures that we have recent versions about the libraries that we are using, and makes it easier to compile the project on Windows.
01.14. marketing -directory
Supposed to contain press kits and such, but right now contains only an one image.
01.15. native_considerations.txt
Some remarks and notes about compiling for native machine code. Should be merged into the rest of the documentation.
01.16. performance.text
Stub for accessibility. The contents have been moved into: https://leverlanguage.com/doc/latest/performance.html
01.17. README.md
I'm not entirely sure what I should do with this file, but it also contains some documentation. Maybe it should be slowly merged with the rest of the documentation.
01.18. runtime -directory
The part of the Lever that has to be translated and then compiled into a executable file.
01.19. sample.lc
A kind of a scratch buffer for trying out small things. A semi-temporary file.
01.20. samples -directory
Bundle of old and new sample programs, tests and experiments.
01.21. semantics_documentation.txt
An older file describing semantics of the language in terms of the grammar declarations. Should be merged into the rest of the documentation.
01.22. setup.py
Maintenance script for the project. Will be described in detail in an another section.
01.23. test.py
An old script for running test script.
01.24. tests -directory
Outdated list of tests that was tried some time back when everything barely worked.
01.25. tool_ideas.text
Related to the performance.text discussion. Will be eventually merged in there as well.
01.26. tools -directory
Variety of tools written in Lever and Python. Mostly concerned with html documentation and machine readable specs.
01.27. VERSION
Version number of the lever release.
01.28. website_considerations.txt
Some old discussion on the website contents.
01.29. win32_extras -directory
A bundle of extra binary files needed to run the produced executable on a Windows system.
01.30. www -directory
The copy from the website directory. Mostly generated with a hint of some files that haven't been.
02. Documentation effort
The doc/ -directory is itself an experiment. Overall the structure looks like it could work, but the format is not that desirable. Although the documentation is mostly in the single place that distributes it where it is needed. It is cleanly and consistently formatted, relatively easy to write. The format itself has been lacking in providing machine-assisted tools for updating the documentation.
Eventually when the format is changed, the existing documentation won't be removed. It will be automatically translated into the new format. There are plans for a binary format of one kind. Designed to be easy to edit by an editor, have dense representation for the information and allow a good diff algorithm to be implemented on the contents.
03. Setup script functionality
When you do 'python setup.py -h' you get...
usage: setup.py [-h]
{build-local,compile,compile-lib,win32-dist,refresh-docs,stub,update-html-docs}
...
positional arguments:
{build-local,compile,compile-lib,win32-dist,refresh-docs,stub,update-html-docs}
build-local Compile the statically linked third party dependencies
for Lever
compile Compile the lever runtime
compile-lib Compile lib/ contents
win32-dist Create win32 distribution
refresh-docs Refresh the source index and then update all
documentation
stub Produce documentation stub for an item in the source
index
update-html-docs Update HTML documentation
optional arguments:
-h, --help show this help message and exit
'python setup.py build-local' downloads and builds the dependencies needed on Windows. If you don't have zlib or libuv or you're unsure whether they're new enough on your system, you may want to run this on Linux as well.
'python setup.py compile' runs little bit of dependency checking/troubleshooting and retrieves pypy into it's own path, so that it would be really easy to build this project. I've tried to make it sure that the program does one of: nags, attempts to get the dependencies, lists out the missing dependencies rather than that it would go into compiling and fail sometime during the compiling wasting your time.
Sometimes there can still be a hitch in the compiling, but it's not the default behavior here.
'python setup.py compile-lib' is run implicitly after compile. It uses the python compiler to compile bytecode for as many files in lib/ as it can. Because it is slightly out of sync it won't succeed with all of them, but it should succeed just enough that the lib/compiler.lc is able to run.
'python setup.py win32-dist' some long time ago this built the 0.8.0 win32 standalone distribution. Maybe it'll build the 0.9.0 standalone at some point!
'python setup.py refresh-docs' prints the module index into 'doc/source_index.json'. It probably doesn't work all the time, but when it's time to use it we'll fix it. It collects all the modules and functions so that we can quickly generate documentation stubs for them.
I don't like yet how this works when it comes to updating the documentation, so it'll change eventually.
'python setup.py update-html-docs' builds the HTML documentation in the www/. Everything in that directory isn't build, but the stuff that comes from the doc/ is rewritten. Lets see when it needs to be improved.
04. When Lever starts
Lever has two modes of startup. If you pass in a script, it runs that script. If no script is given the interpreter runs the 'app/main.lc' which provides a read-eval-print-loop.
When the runtime starts up, it inserts some initial tasks into a queue and then starts up an eventloop. The eventloop is provided by the library libuv. The whole system implements co-operative task switching with greenlets.
Thread spawning in Lever is not complete, but the multiple threads are supported. This means that we have both in-thread and between-threads concurrency present in our runtime. I know that this isn't potentially very ideal in every situation. Later, we will probably try to solve some of the problems with write access control or such.
After the event loop is loaded, the runtime sets up a module scope, importing routines, some basic I/O utilities and starts up the main script. Libuv handling of stdin/stdout/stderr can be a nuisance quite too often, so we'll have to change some of that eventually.
The scoping in Lever does not have an idea of a mutable global space. It can be illustrated by the following script:
scope = %"import".scope # Get the scope of this module.
while scope
print(scope.local)
scope = scope.parent
The 'import' and .scope within it are ordinary variables that can be found from the module that is given. They are also not mandatory in a module, but without them the import clauses do not work, of course.
It prints out the following kind of a hierarchy:
path("module's directory")
path("${lever_path}/lib")
path("builtin:/")
This points out where the modules are searched from. These are the directories assigned for our module scopes and they form a hierarchy.
The topmost module scope is always created when the Lever starts up. It loads modules from the same directory where the starting script is.
Currently the '/lib' scope is also created at startup. It points out where to search the system libraries from that have been written in lever.
The 'builtin://' is pointing to the modules implemented by the runtime. Most of it comes from the runtime/stdlib/ -directory. The contents of this scope are immutable.
Each scope object memoizes the modules that are already loaded and attempts to fetch the memoized module. Explicit reload command needs to be called if it is desirable to reload the module.
Solving the module system without having mutable globals was proposed by the maintainer of Python's module system. I think it is nearly just like I want it.
You can create new scopes yourself and discard them when you no longer need them. It makes it very easy to implement a plugin framework for your application.
Modules themselves can take a parent module. The variable is fetched from the parent module if it isn't present in the module itself. The 'base' module is used as a base by default, but you can change that too when you introduce a new module scope.
05. Runtime structure
Lever's runtime builds hierarchically.
On the lowest layer we have runtime/continuations.py and runtime/core.py. These parts define interactions between event loop and continuations, and those interactions sometimes spread far into the remaining parts of the runtime because they allow tasks to wait asynchronously.
The runtime/space/ -directory describes our object system. It contains most primitive objects and containers that we happen to have in the system.
The runtime/base/ describes the base module in our system. Although some of the values in that base module are filled up in all around the runtime/ -directory.
The runtime/evaluator/ contains the interpreter. It is loading the module from json-like objects that are defined by 'runtime/bon.py' -module. This is a custom format specific to Lever and it may need much more work before it is satisfying enough.
The runtime/stdlib/ contains the 'standard modules' that are defined by the runtime. They provide functionality that could not be provided otherwise. They are gathered in the runtime/module_resolution.py which defines the module scopes and import mechanisms.
The uv_ -prefixed files are all libuv related helpers and utilities. There is a still mix of two interfaces there, one that I started with and another iteration of it. It is in the 'proof of concept' -phase. Once it matters a bit more, we should move on to building good I/O streaming primitives around the features of the libuv. Lets hope that we won't depend too much on the kinks of that API.
The runtime/vectormath.py provides some vector, matrix, quaternion arithmetic along trigonometric functionality. Some of the operations there belongs into the stdlib, though I would prefer to provide 'pi', 'tau', 'sin', 'cos' so on.. in the base module because they are very useful functions and variables.
pathobj.py is provides a layer to transform windows paths to POSIX and HTML paths. This is used implicitly in lot of places of our runtime because it's nice if every file path is in a consistent format. Additionally these paths are all wrapped into path -object that makes it easy to work on them. On the retrospect the important thing is consistent path handling, not hiding of Windows-style paths. We may adjust the design to that direction later on.