TON618beta-2.0.0

Architecture — how the interpreter works

TON618 is a classic tree-walking interpreter, made of three stages that run in a straight pipeline:

source text (.ton file)
        │
        ▼
┌───────────────┐   turns characters into a flat list of Tokens
│     Lexer     │   (numbers, strings, identifiers, keywords, operators)
└───────┬───────┘
        ▼  tokens
┌───────────────┐   recursive-descent parsing: groups Tokens into an
│     Parser    │   Abstract Syntax Tree (a tree of Stmt/Expr nodes)
└───────┬───────┘   according to the language's grammar
        ▼  AST
┌───────────────┐   walks the AST directly and executes it — no bytecode,
│  Interpreter  │   no separate compilation step; each node is evaluated
└───────────────┘   the moment it's reached
ton618/
├── Makefile
├── include/                 # headers (.hpp) — declarations and design notes
│   ├── Token.hpp              — the language's vocabulary (TokenType, Token)
│   ├── Lexer.hpp                — text -> tokens
│   ├── Ast.hpp                    — the AST node shapes (Expr, Stmt)
│   ├── Parser.hpp                   — tokens -> AST
│   ├── Value.hpp                      — runtime value representation
│   ├── Environment.hpp                  — variable scopes (name -> Value, chained)
│   ├── Interpreter.hpp                    — walks the AST, runs the program
│   ├── Debugger.hpp                         — breakpoints / step mode
│   ├── LocalServer.hpp                        — raw-socket HTTP server (ton.serve/get/post)
│   ├── HttpClient.hpp                           — raw-socket HTTP client (the ton.requests module)
│   ├── JsonParser.hpp                             — JSON text -> Value (ton.json's json_parse)
│   ├── Platform.hpp                                 — OS/Termux/architecture auto-detection
│   └── Version.hpp                                    — the interpreter's own version string
├── src/                     # implementations (.cpp), one per header above,
│                              plus main.cpp (CLI entry point: reads a file, runs the
│                              Lexer -> Parser -> Interpreter pipeline, and wires up
│                              --debug / --break / --version / --update / --uninstall-ton / install / uninstall)
├── exemples/                # example .ton scripts, including one that exercises most
│                              of the language (test_new_features.ton), plus
│                              documentation.html, playground.html, and ton618-lite.js
│                              (a JS reimplementation used only by the playground)
├── scripts/                 # install.sh/.ps1/.bat, uninstall.sh/.ps1/.bat, rebuild.sh,
│                              zig-strip.sh, and doc_templates/ (see `make docs` above)
└── modules/                 # shared .ton modules (see the module guide above)

Every header starts with a short comment block explaining its role in the pipeline and, where relevant, exactly what to touch first if you want to extend that part of the language — read those before diving into the .cpp files.

Two design choices worth calling out because they shape almost every other file:

  • AST/Value nodes are "fat structs", not a class hierarchy. Expr and Stmt (Ast.hpp) and Value (Value.hpp) each hold every field any of their variants might need, with a type tag saying which ones apply. This trades a bit of memory for a codebase where evaluate()/execute() are single switch statements instead of a scattered visitor pattern — much easier to read end-to-end for a language this size. Some node kinds reuse another kind's fields (documented next to the enum) instead of adding new ones, to keep the structs from growing unbounded — SWITCH, for instance, reuses IF's condition/elseBranch fields for its subject expression and default block.
  • Control flow uses C++ exceptions. break, continue, and return are implemented as tiny exception types (BreakException, ContinueException, ReturnException, see Interpreter.hpp) thrown from execute() and caught by the nearest enclosing loop, switch, or function call. This mirrors, almost one-to-one, how those constructs unwind nested blocks in the AST — no manual "did we break?" flag needs to be threaded through every recursive call.

A third thing worth knowing if you're touching the standard library: valuesEqual() (a free function near the top of src/Interpreter.cpp) is the single source of truth for what "equal" means — the ==/!= operators, switch/case matching, and the contains()/indexOf() natives all call it, so a fix or a new value kind only needs handling in one place.

Internals since beta-2.0.0, for contributors:

  • break/continue/return are no longer C++ exceptions: Interpreter::execute() returns a Signal (NONE, BREAK, CONTINUE, RETURN) that loops and calls check. Only real errors use exceptions (LineError for interpreter errors, ThrownValue for throw).
  • Names are interned into integers (Symbols in Environment.hpp) and scopes are small flat vectors, with a hash index only for big scopes such as the globals. A block that declares nothing doesn't get its own scope at all.
  • Classes are ClassObj and objects InstanceObj (Value.hpp); method calls go through Interpreter::callMethod, which also implements uniform call syntax.
  • The standard library added in 2.0 lives in src/Stdlib.cpp (defineNativesV2()).
  • main.cpp runs scripts on a thread with a 512 MB stack, so deep recursion reaches the interpreter's own depth limit (10,000 calls) instead of crashing.
  • tests/*.ton with tests/*.expected is the regression suite: make test (also run by CI).

Together these make typical scripts about 6x faster than beta-1.x (a fib(25) plus a 1,000,000-iteration loop: 2.0 s → 0.3 s).

Edit this page on GitHub