Skip to content

dpmxl.parser

Tokenizer and recursive-descent parser. Nodes keep their source spans. See Language for what parses.

dpmxl.parser

Tokenizer and recursive-descent parser for DPM-XL expressions.

tokenize splits an expression into typed Token objects that suit syntax highlighting as well as parsing: selections are split into their table, axis and option tokens rather than captured whole. parse builds the expression tree of dpmxl.ast from those tokens.

DpmXlSyntaxError 

DpmXlSyntaxError(
    message: str,
    expression: str,
    position: int,
    end: int | None = None,
)

Bases: ValueError

Raised when an expression does not match the DPM-XL grammar.

Attributes:

  • message –

    What was expected or found, in plain words.

  • expression –

    The complete source expression.

  • position –

    Zero-based offset where the offending text starts.

  • end –

    Offset just past the offending text.

Examples:

>>> try:
...     parse("{tA} >=")
... except DpmXlSyntaxError as error:
...     print(error.message, error.position)
expression ends where an operand is expected 7

Record the failing source range.

Parameters:

  • message (str) –

    What was expected or found.

  • expression (str) –

    The complete source expression.

  • position (int) –

    Zero-based character offset where the error starts.

  • end (int | None, default: None ) –

    Offset just past the offending text; defaults to one character after position (or position at the end of the expression).

message instance-attribute 

message = message

expression instance-attribute 

expression = expression

position instance-attribute 

position = position

end instance-attribute 

end = max(end, position)

pointer 

pointer() -> str

Render the source line and mark the failing range.

Returns:

  • str ( str ) –

    Two lines suitable for terminal output: the source line and a

  • str –

    ^ under the error's start, followed by ~ under the rest of

  • str –

    the offending text on that line.

Examples:

>>> try:
...     parse("{tA, r0010} +* 1")
... except DpmXlSyntaxError as error:
...     print(error.pointer())
{tA, r0010} +* 1
             ^

Token

Bases: NamedTuple

One lexical token: its kind, source text and half-open offsets.

text is exactly expression[start:end], quotes, brackets and backticks included. kind is one of TOKEN_KINDS:

  • keyword: if, and, with, where, group by, window keywords inside over (...), and default/interval in a selection;
  • type: a parameter type such as number or set-item;
  • function: a known function name followed by (;
  • operator: + - * / & % = != <> < <= > >= := <-;
  • punctuation: ( ) , : ; [ ];
  • brace: { or } around a selection or set;
  • table, table_group, variable, operation, parameter: a selection reference with its prefix, such as tC_01.00;
  • axis: a row, column or sheet coordinate, range or wildcard, such as r0010, c0010-0050 or s*;
  • number, string, date, boolean, null: literals;
  • item: an item such as [eba_qEC:qx01];
  • period: a time_shift or window period code (A, Q, ...);
  • identifier: a key, temporary or other name, including escaped names such as x.y;
  • eof: the empty token that ends every token list.

Attributes:

  • kind (str) –

    The token kind, as above.

  • text (str) –

    The exact source text.

  • start (int) –

    Offset of the first character.

  • end (int) –

    Offset just past the last character.

kind instance-attribute 

kind: str

text instance-attribute 

text: str

start instance-attribute 

start: int

end instance-attribute 

end: int

tokenize 

tokenize(expression: str) -> list[Token]

Split an expression into typed tokens.

The tokenizer follows the lexical modes of the DPM-XL grammar: inside {...} it reads selection components (table, axis and option tokens, parameter types), elsewhere expression tokens. Whitespace is skipped; concatenating the token texts with the source between them gives the expression back. See Token for the kinds.

Returns:

  • list[Token] –

    list[Token]: Tokens ending with an eof token.

Raises:

  • DpmXlSyntaxError –

    On an unterminated literal, brace or item, or an unknown character.

Examples:

>>> [(token.kind, token.text) for token in tokenize("{tT, r0010} > 0")]
...
[('brace', '{'), ('table', 'tT'), ('punctuation', ','), ('axis', 'r0010'),
 ('brace', '}'), ('operator', '>'), ('number', '0'), ('eof', '')]

parse 

parse(expression: str) -> Node

Parse a complete DPM-XL expression.

A trailing ; is allowed. Several statements and assignments need parse_program.

Returns:

  • Node ( Node ) –

    The expression tree; every node keeps its source span.

Raises:

Examples:

>>> node = parse("{tT, r0010} > 0")
>>> type(node).__name__, node.op, node.span
('Binary', '>', (0, 15))
>>> node.left.spec.text()
'{tT, r0010}'

parse_program 

parse_program(source: str) -> Program

Parse statements separated by ;, with assignments.

A statement is an expression (with an optional with context), a temporary assignment name := expression or a persistent assignment {target} <- expression. A trailing ; is allowed.

Returns:

  • Program ( Program ) –

    The statements, also when there is only one.

Raises:

Examples:

>>> program = parse_program("x := {tT, r0010}; x > 0")
>>> [type(statement).__name__ for statement in program.statements]
['Assignment', 'Binary']
>>> program.statements[0].name
'x'

parse_selection 

parse_selection(
    body: str, *, expression: str = "", offset: int = 0
) -> SelectionSpec

Parse the inside of a {...} cell selection or with context.

Parameters:

  • body (str) –

    Text between the braces.

  • expression (str, default: '' ) –

    Complete source for error reporting.

  • offset (int, default: 0 ) –

    Offset of body inside expression.

Returns:

  • SelectionSpec ( SelectionSpec ) –

    Parsed table, axes and options.

Raises:

Examples:

>>> spec = parse_selection("tC_01.00, r0010-0050, c*")
>>> spec.table, spec.rows.parts, spec.columns.wildcard
('C_01.00', (('0010', '0050'),), True)