# dpm-xl
A Python parser and evaluator for **DPM-XL**, the expression language of EBA
validation rules and similar checks. Give it a rule and your report data; it
tells you which records fail and explains each failure in plain numbers.
dpm-xl is an independent project, not affiliated with or endorsed by the
European Banking Authority.
A failure reads like this:
```text
1,204 should equal 1,211 (1,180 + 31): difference 7, at least 5.5 given the reported precision
```
It is for teams that check supervisory reports before they are filed, build
validation into a reporting platform, or analyse and edit DPM-XL rules.
- Parses every construct of the DPM-XL grammar, including every EBA
validation rule and precondition of DPM releases 3.4–4.4 (14,000+ distinct
expressions, about 0.1 ms each).
- Evaluates them with EBA's interval arithmetic, null logic and key joins, and
points each failure at the cells involved.
- Finds problems before evaluation, for [editors](guides/editor.md) and
[CI](guides/ci.md): every syntax error, unknown functions and tables,
coordinates a table lacks, keys and operand types, each with its position
and, where one fits, a suggested fix.
- Has no dependencies. The optional `columnar` extra adds a polars evaluator
for reports with millions of open-table rows.
- Reads no files and knows no DPM release: data comes through a three-method
[provider](guides/provider.md), so the same evaluator runs against a report
upload, a database or test fixtures. [dpm-xbrl](https://luukhopman.github.io/dpm-xbrl/)
reads xBRL-CSV and XBRL-XML reports; the
[end-to-end guide](guides/end-to-end.md) connects the two.
## Installation
```bash
pip install dpm-xl # Python 3.11+, no dependencies
pip install "dpm-xl[columnar]" # adds the polars evaluator
```
## In 30 seconds
```python
from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule, explain
layout = TableLayout("C_01.00", rows=("0010", "0020", "0030"), columns=("0010", "0020"))
values = { # (table, row, column, sheet, open keys) -> value with XBRL decimals
("C_01.00", "0010", "0010", None, ()): Number.of(100, 0),
("C_01.00", "0020", "0010", None, ()): Number.of(60, 0),
("C_01.00", "0030", "0010", None, ()): Number.of(40, 0),
("C_01.00", "0010", "0020", None, ()): Number.of(1204, 0),
("C_01.00", "0020", "0020", None, ()): Number.of(1180, 0),
("C_01.00", "0030", "0020", None, ()): Number.of(31, 0),
}
provider = MemoryProvider([layout], values)
expression = (
"with {tC_01.00, c*, default: 0, interval: true}: {r0010} = {r0020} + {r0030}"
)
result = evaluate_rule(expression, provider, precondition="{v_C_01.00}")
print(result.status, result.counts)
for outcome in result.failures:
print(outcome.key, explain(result.evaluation, expression, outcome.key).summary)
```
```text
failed {'passed': 1, 'failed': 1, 'null': 0}
{'c': '0020'} 1,204 should equal 1,211 (1,180 + 31): difference 7, at least 5.5 given the reported precision
```
Column `0010` adds up; in column `0020`, 1,204 is 7 short of 1,180 + 31, more
than the rounding of three whole-unit amounts (±1.5) can explain.
## Toolkit
**Part of the dpm toolkit**, three Python packages that work together:
| Package | What it does |
| :-- | :-- |
| 
| Parse and evaluate DPM-XL validation rules **(this package)** |
| 
| Read, write and check xBRL-CSV and XBRL-XML reports |
| 
| Generate test reports that satisfy DPM-XL rules |
The [toolkit tour](https://luukhopman.github.io/dpm-xl/guides/toolkit-tour/)
takes one report through all three: generated with dpm-synth, read with
dpm-xbrl, then evaluated and explained with dpm-xl.
## Where next
- **[Getting started](getting-started.md)**: install, evaluate a first rule,
read the explanation and use the CLI.
- **[How-to guides](guides/index.md)**: connect your storage, evaluate rule
files, explain failures in a UI, evaluate large reports, run in threads.
- **[Concepts](concepts/index.md)**: records and keys, null logic, interval
arithmetic and the rest of the semantics.
- **[Reference](reference/index.md)**: the language, the CLI, the data
formats and the Python API.
- **[AI agents](agents.md)**: the essential API on one page, and
[`llms.txt`](https://luukhopman.github.io/dpm-xl/llms.txt) /
[`llms-full.txt`](https://luukhopman.github.io/dpm-xl/llms-full.txt).
- **[Project](project/changelog.md)**: changelog, stability policy, security,
contributing and licence.
# Getting started
This tutorial takes you from installation to an explained rule failure, first
in Python and then on the command line. It takes about ten minutes.
## Install
dpm-xl needs Python 3.11 or newer and has no dependencies.
```bash
pip install dpm-xl
```
or, in a project managed with uv:
```bash
uv add dpm-xl
```
Check the installation; it prints the installed version:
```bash
dpm-xl --version
```
## Parse a rule
A DPM-XL rule compares cells of report tables. This one says that in every
column of table `C_01.00`, row `0010` is the sum of rows `0020` and `0030`:
```python
from dpmxl import parse, tables
expression = (
"with {tC_01.00, c*, default: 0, interval: true}: {r0010} = {r0020} + {r0030}"
)
tree = parse(expression)
print(type(tree).__name__, tables(tree))
```
```text
With ('C_01.00',)
```
`with {...}:` sets a context that every selection in the body inherits:
`{r0010}` means `{tC_01.00, r0010, c*, default: 0, interval: true}`. `c*`
selects every column; `default: 0` fills cells that were not reported;
`interval: true` allows for the rounding of reported amounts (see
[Interval arithmetic](concepts/intervals.md)).
A rule that does not parse raises
`DpmXlSyntaxError` with a plain message and
the position of the offending text:
```python
from dpmxl import DpmXlSyntaxError
try:
parse("with {tC_01.00, c0010}: {r0010} = {r0020} +* {r0030}")
except DpmXlSyntaxError as error:
print(error.message)
print(error.pointer())
```
```text
unexpected '*' where an operand is expected
with {tC_01.00, c0010}: {r0010} = {r0020} +* {r0030}
^
```
## Provide report data
The evaluator reads no files. It asks a *provider* for the cells a selection
covers. `MemoryProvider` holds a report in
memory: table layouts plus values keyed by `(table, row, column, sheet, open
keys)`.
```python
from dpmxl import MemoryProvider, Number, TableLayout
layout = TableLayout("C_01.00", rows=("0010", "0020", "0030"), columns=("0010", "0020"))
values = {
("C_01.00", "0010", "0010", None, ()): Number.of(100, 0),
("C_01.00", "0020", "0010", None, ()): Number.of(60, 0),
("C_01.00", "0030", "0010", None, ()): Number.of(40, 0),
("C_01.00", "0010", "0020", None, ()): Number.of(1204, 0),
("C_01.00", "0020", "0020", None, ()): Number.of(1180, 0),
("C_01.00", "0030", "0020", None, ()): Number.of(31, 0),
}
provider = MemoryProvider([layout], values)
```
`Number.of(1204, 0)` is a fact reported with XBRL `decimals="0"`: 1,204
rounded to whole units, so anything from 1,203.5 to 1,204.5.
```python
amount = Number.of(1204, 0)
print(amount.low, amount.high)
```
```text
1203.5 1204.5
```
For your own storage you write a provider with three methods; see
[Connect your data](guides/provider.md).
## Evaluate
`evaluate_rule` returns a
`RuleResult`. It never raises for a bad rule or
bad data; problems become the status `error`.
```python
from dpmxl import evaluate_rule
result = evaluate_rule(expression, provider, precondition="{v_C_01.00}")
print(result.status, result.counts)
for outcome in result.outcomes:
print(outcome.key, outcome.passed)
```
```text
failed {'passed': 1, 'failed': 1, 'null': 0}
{'c': '0010'} True
{'c': '0020'} False
```
The rule has one *record* per column, because `c*` selects two columns; the
record key `{'c': '0020'}` names the failing one. The precondition
`{v_C_01.00}` makes the rule apply only when table `C_01.00` was reported;
`MemoryProvider` treats every table with a value as reported.
## Explain a failure
`explain` fills the failing record's formula with the
reported values:
```python
from dpmxl import explain
explanation = explain(result.evaluation, expression, {"c": "0020"})
print(explanation.summary)
print(explanation.filled)
print(explanation.difference, explanation.gap, explanation.direction)
print([point.ref for point in explanation.primary])
```
```text
1,204 should equal 1,211 (1,180 + 31): difference 7, at least 5.5 given the reported precision
1,204 = 1,180 + 31
7 5.5 too_low
[('C_01.00', '0010', '0020', None, ())]
```
The sides differ by 7. Each of the three amounts may be off by up to 0.5, so
1.5 of that difference could be rounding; 5.5 cannot. `primary` holds the
cells on the left of the comparison, the values that disagree; a user
interface can highlight them by their `ref`.
## Use the CLI
The same evaluation runs from the command line against a JSON data file. The
repository's [`examples/`](https://github.com/luukhopman/dpm-xl/tree/main/examples)
folder has a rule file and a data file:
```console
$ dpm-xl eval examples/rules.json examples/data.json
v0001: failed
records: 1 passed, 1 failed, 0 not evaluated
- c=0020: 1,204 should equal 1,211 (1,180 + 31): difference 7, at least 5.5 given the reported precision
v0002: failed
records: 1 passed, 1 failed, 0 not evaluated
- qCPI=LEI-B: -5 should exceed 0: difference 5
v0003: passed
records: 1 passed, 0 failed, 0 not evaluated
v0004: not_applicable
message: precondition not met
records: 0 passed, 0 failed, 0 not evaluated
4 rules: 2 failed, 1 passed, 1 not_applicable
$ echo $?
1
```
The exit status is 1 because a rule failed. `dpm-xl check` only parses, which
is what a CI job for a rule repository needs:
```console
$ dpm-xl check examples/rules.json
6/6 expressions parsed
```
## Next steps
- [Connect your data](guides/provider.md): write a provider for your own
storage.
- [Explain failures in a UI](guides/explain.md): everything an
`Explanation` holds.
- [Records and keys](concepts/records-and-keys.md): how selections become
records and how operators join them.
- [Language reference](reference/language.md): every construct with its
syntax, types, errors and tested examples; the
[quick reference](reference/quick-reference.md) has it on one page.
# How-to guides
Recipes for specific tasks. They assume you have done the
[getting started](../getting-started.md) tutorial; every example runs as
shown (the test suite executes them).
**Connect and evaluate**
- [Connect your data](provider.md): write a `DataProvider` for your storage,
including the optional `open_keys`, `table_group`, `operation` and
`earlier_report` hooks.
- [Evaluate a rule file](rule-files.md): run a rule set against a report, in
Python or with `dpm-xl eval`.
- [Explain failures in a UI](explain.md): sentences, filled formulas and the
cells to highlight.
- [From an XBRL report to explained failures](end-to-end.md): read a report
with dpm-xbrl, bind it to a provider, evaluate and explain.
- [Toolkit tour](toolkit-tour.md): generate a report with dpm-synth, read it
with dpm-xbrl, then evaluate and explain its rules.
**Scale up**
- [Evaluate large reports](columnar.md): the columnar engine for millions of
open-table rows, with the reference fallback.
- [Run rules in threads](threads.md): what is safe to share.
- [Compare with earlier reports](time-shift.md): `time_shift` and
`annualise`.
**Build tooling**
- [Check rule syntax in CI](ci.md): `dpm-xl check` and a pytest test.
- [Build editor support](editor.md): tokens, error ranges and the TextMate
grammar.
- [Analyse rules statically](analyse.md): the tree, the selections and the
tables a rule reads.
# Connect your data
The evaluator never reads report files. Everything it knows about a report
comes from a *provider*: an object with three methods, plus up to four
optional ones. This guide builds a provider over plain Python data, one
method at a time; replace the dictionaries with queries against your database,
a report loader or test fixtures.
## The contract
```python
from collections.abc import Sequence
from typing import Protocol
from dpmxl import CellQuery, DataPoint, Value
class DataProvider(Protocol):
def select(self, query: CellQuery) -> Sequence[DataPoint]: ...
def table_reported(self, table: str) -> bool | None: ...
def parameter(self, name: str) -> Value: ...
```
| Method | Answers | Contract |
| --- | --- | --- |
| `select(query)` | Every selection `{tT, r…, c…, s…}` | Return **every defined cell** of the closed axes the query covers, with `value=None` where nothing was reported, so `default` and `isnull` work. For an open table, return one point per reported key combination with the key components in `DataPoint.keys`. Raise `LookupError` for an unknown table. |
| `table_reported(table)` | `{v_TABLE}` filing-indicator tests | `True` or `False`; `None` when the provider cannot tell. |
| `parameter(name)` | `{vName}` and `{p_Name, type}` | The report parameter, such as `refPeriod`, or `None` when unknown. |
| Optional method | Answers | Without it, or when it returns `None` |
| --- | --- | --- |
| `open_keys(table) -> Sequence[str]` | The key names of an open table | An open table with no reported rows yields a selection without keys. |
| `table_group(code) -> Sequence[str] \| None` | `{g_Group, …}` table groups | The rule is `unsupported`. |
| `operation(code) -> str \| None` | `{o_Code, …}` operation references | The rule is `unsupported`. |
| `earlier_report(period, count) -> DataProvider \| None` | `time_shift` on another period's report | The rule is `unsupported`. |
The protocol is `DataProvider`. It is
`runtime_checkable`, so `isinstance(provider, DataProvider)` checks the three
required methods.
## The data
Two tables: `C_02.00` is closed (fixed rows and columns), `C_27.00` is open
(one row per counterparty, keyed by `qCPI`). Row `0030` of `C_02.00` was not
reported.
```python
from dpmxl import Number
CLOSED = {"C_02.00": {"rows": ("0010", "0020", "0030"), "columns": ("0010",)}}
FACTS = {
("C_02.00", "0010", "0010"): Number.of("250.4", 1),
("C_02.00", "0020", "0010"): Number.of(-3, 0),
}
OPEN = {"C_27.00": {"columns": ("0010", "0020"), "keys": ("qCPI",)}}
ROWS = {
"C_27.00": [
{"qCPI": "LEI-A", "0010": Number.of(120, 0), "0020": "bank"},
{"qCPI": "LEI-B", "0010": Number.of(-5, 0), "0020": "fund"},
]
}
```
Values are `Number` (use `Number.of(value, decimals)` for reported facts, with
the fact's XBRL `decimals`; `None` or `"INF"` means exact), `str`, `bool`,
`datetime.date`, `Item` (`Item.parse("eba_qEC:qx01")`),
`ValueSet` or `None`.
## Required methods
`select` receives a `CellQuery`: the table code
and the row, column and sheet selectors, each `None` when the rule does not
restrict that axis. `axis_matcher` compiles a
selector into a fast callable that handles single codes, ranges (`r0010-0050`),
lists and wildcards, and compares codes the way DPM does
(`normalize_code`: `r310` matches `0310`).
```python
from collections.abc import Sequence
from dpmxl import CellQuery, DataPoint, Value, axis_matcher
class ReportProvider:
"""A DataProvider over CLOSED, FACTS, OPEN and ROWS."""
def __init__(self, rows=ROWS, reported=("C_02.00", "C_27.00")):
self.rows = rows
self.reported = set(reported)
def select(self, query: CellQuery) -> Sequence[DataPoint]:
rows, columns = axis_matcher(query.rows), axis_matcher(query.columns)
if query.table in CLOSED:
layout = CLOSED[query.table]
return [ # every defined cell, reported or not
DataPoint(
query.table,
row=row,
column=column,
value=FACTS.get((query.table, row, column)),
ref=f"{query.table}/{row}/{column}",
)
for row in layout["rows"]
if rows(row)
for column in layout["columns"]
if columns(column)
]
if query.table in OPEN:
layout = OPEN[query.table]
return [ # one point per reported key combination and column
DataPoint(
query.table,
column=column,
keys={key: record[key] for key in layout["keys"]},
value=record.get(column),
ref=f"{query.table}/{record['qCPI']}/{column}",
)
for record in self.rows.get(query.table, [])
for column in layout["columns"]
if columns(column)
]
raise LookupError(f"unknown table {query.table}")
def table_reported(self, table: str) -> bool | None:
return table in self.reported
def parameter(self, name: str) -> Value:
return {"refPeriod": "2026-12-31"}.get(name)
```
`DataPoint.ref` is whatever identity you give the cell (a database id, a
location in the report file); it comes back in every outcome so a user
interface can highlight the cell. It must be hashable.
```python
from dpmxl import DataProvider, evaluate_rule, explain
provider = ReportProvider()
print(isinstance(provider, DataProvider))
for expression in (
"{tC_02.00, c0010} >= 0",
"{tC_27.00, c0010} > 0",
'{tC_27.00, c0020} = "bank"',
"{tC_99.00, c0010} > 0",
):
result = evaluate_rule(expression, provider)
print(expression, "->", result.status, result.counts, result.message or "")
for outcome in result.failures:
cells = [point.ref for point in outcome.points]
print(
" ",
outcome.key,
cells,
explain(result.evaluation, expression, outcome.key).summary,
)
```
```text
True
{tC_02.00, c0010} >= 0 -> failed {'passed': 1, 'failed': 1, 'null': 1}
{'r': '0020'} ['C_02.00/0020/0010'] -3 should be at least 0: difference 3
{tC_27.00, c0010} > 0 -> failed {'passed': 1, 'failed': 1, 'null': 0}
{'qCPI': 'LEI-B'} ['C_27.00/LEI-B/0010'] -5 should exceed 0: difference 5
{tC_27.00, c0020} = "bank" -> failed {'passed': 1, 'failed': 1, 'null': 0}
{'qCPI': 'LEI-B'} ['C_27.00/LEI-B/0020'] "fund" should equal "bank"
{tC_99.00, c0010} > 0 -> error {'passed': 0, 'failed': 0, 'null': 0} unknown table C_99.00
```
Row `0030` was not reported, so its record is null ("not evaluated") rather
than a failure; with `default: 0` in the selection it would be compared as 0.
That only works because `select` returned the unreported cell with
`value=None`. The unknown table became the status `error` with the
`LookupError` message.
!!! warning "Return unreported cells"
A provider that returns only reported facts for a closed table silently
turns "missing" into "absent": `default` never applies, `isnull` never
sees the cell and presence checks such as
`if not isnull({r0010}) then not isnull({r0030}) endif` pass when they
should fail.
## `open_keys`: the keys of an empty open table
When an open table has no reported rows, `select` returns nothing, and the
evaluator cannot know that the selection is keyed by `qCPI`. A rule that
groups by that key is then an error:
```python
empty = ReportProvider(rows={})
expression = "sum({tC_27.00, c0010} group by qCPI) > 0"
result = evaluate_rule(expression, empty)
print(result.status, result.message)
```
```text
error group by 'qCPI' is not a key of the operand (no keys)
```
`open_keys(table)` names the key components, so the empty selection keeps the
table's key structure, as the specification requires:
```python
class KeyedProvider(ReportProvider):
def open_keys(self, table: str) -> Sequence[str]:
return OPEN.get(table, {}).get("keys", ())
result = evaluate_rule(expression, KeyedProvider(rows={}))
print(result.status, result.counts)
```
```text
not_applicable {'passed': 0, 'failed': 0, 'null': 0}
```
## `table_group`: `{g_…}` selections
A table group selects the same cells from several tables. `table_group(code)`
returns the group's table codes, or `None` for an unknown group. With several
tables, records are keyed by the table code `t`, and the tables must share
their keys.
```python
CLOSED["C_02.01"] = {"rows": ("0010",), "columns": ("0010",)}
FACTS["C_02.01", "0010", "0010"] = Number.of(7, 0)
class GroupProvider(KeyedProvider):
def table_group(self, code: str) -> Sequence[str] | None:
return {"G_02": ("C_02.00", "C_02.01")}.get(code)
result = evaluate_rule("{g_G_02, r0010, c0010} > 0", GroupProvider())
print(result.status, [outcome.key for outcome in result.outcomes])
print(evaluate_rule("{g_G_02, r0010, c0010} > 0", KeyedProvider()).message)
```
```text
passed [{'t': 'C_02.00'}, {'t': 'C_02.01'}]
table group G_02 needs a provider that resolves table groups
```
## `operation`: `{o_…}` references
An operation reference evaluates another rule's expression. `operation(code)`
returns that expression's text, or `None` for an unknown operation. The
expression is evaluated on its own: the referring rule's context, temporaries
and assignments do not reach it. Axes on the reference keep the matching
records, `default` fills nulls and `interval: false` makes numbers exact. An
operation that refers to itself is an error.
```python
class OperationProvider(KeyedProvider):
def operation(self, code: str) -> str | None:
return {"v1000_a": "{tC_02.00, c0010} * 2"}.get(code)
expression = "{o_v1000_a, r0010} > 600"
result = evaluate_rule(expression, OperationProvider())
outcome = result.failures[0]
print(result.status, [point.ref for point in outcome.points])
print(explain(result.evaluation, expression, outcome.key).summary)
```
```text
failed ['C_02.00/0010/0010']
500.8 should exceed 600: difference 99.2
```
## `earlier_report`: `time_shift` across periods
`time_shift(x, Q, 1, refPeriod)` compares with the report one quarter
earlier. `earlier_report(period, count)` returns that report as another
provider, or `None` when you do not have it; `period` is `A`, `S`, `Q`, `M`,
`W` or `D`, and `count` is positive for earlier and negative for later
reports.
```python
from dataclasses import replace
class PreviousQuarter(KeyedProvider):
"""The same report one quarter earlier, with C_02.00 r0010 at 240."""
def select(self, query):
return [
replace(point, value=Number.of(240, 0)) if point.row == "0010" else point
for point in super().select(query)
]
class TimeProvider(KeyedProvider):
def earlier_report(self, period: str, count: int):
return PreviousQuarter() if (period, count) == ("Q", 1) else None
expression = (
"{tC_02.00, r0010, c0010} <= time_shift({tC_02.00, r0010, c0010}, Q, 1) * 1.01"
)
result = evaluate_rule(expression, TimeProvider())
outcome = result.failures[0]
print([point.ref for point in outcome.points])
print(explain(result.evaluation, expression, outcome.key).summary)
shifted_year = "{tC_02.00, r0010, c0010} >= time_shift({tC_02.00, r0010, c0010}, A, 1)"
print(evaluate_rule(shifted_year, TimeProvider()).message)
```
```text
['C_02.00/0010/0010', EarlierRef(period='Q', count=1, ref='C_02.00/0010/0010')]
250.4 should not exceed 242.4 (time_shift(240, Q, 1) × 1.01): difference 8
time_shift needs the report 1 year(s) earlier, which the provider does not have
```
Points read from the earlier report carry
`EarlierRef(period, count, ref)` as their `ref`,
so they stay distinct from the current report's cell with the same `ref`.
[Compare with earlier reports](time-shift.md) covers `time_shift` in depth.
## `MemoryProvider`
`MemoryProvider` implements all of the above
for data you can hold in memory. Its values are keyed by
`(table, row, column, sheet, keys)`, where `keys` is a tuple of
`(name, value)` pairs for open tables and `()` for closed ones; the optional
hooks take `table_groups`, `operations` and `earlier_reports` mappings:
```python
from dpmxl import MemoryProvider, TableLayout
memory = MemoryProvider(
[
TableLayout("C_02.00", rows=("0010", "0020"), columns=("0010",)),
TableLayout("C_27.00", columns=("0010",), open_keys=("qCPI",)),
],
{
("C_02.00", "0010", "0010", None, ()): Number.of(250, 0),
("C_27.00", None, "0010", None, (("qCPI", "LEI-A"),)): Number.of(120, 0),
},
reported=["C_02.00", "C_27.00"],
parameters={"refPeriod": "2026-12-31"},
table_groups={"G_02": ["C_02.00"]},
operations={"v1000_a": "{tC_02.00, c0010} * 2"},
earlier_reports={("Q", 1): MemoryProvider([], {})},
)
print(evaluate_rule("{tC_27.00, c0010} > 0", memory).status)
```
```text
passed
```
`reported` defaults to every table that has a value; a `TableLayout` with a
`template` code also answers `{v_TEMPLATE}` for that template.
## Performance and threads
- To evaluate many rules against one report, pass one
`SelectionCache` per report
(`evaluate_rule(..., cache=cache)`) so each selection is read once.
- `select` is called once per distinct selection, so a provider over a
database should fetch a table's cells in one query.
- A provider shared across threads must be safe to read concurrently;
`MemoryProvider` is. See [Run rules in threads](threads.md).
- For reports with millions of open-table rows, a columnar provider returns
polars frames instead of `DataPoint`s; see
[Evaluate large reports](columnar.md).
The repository's
[`examples/custom_provider.py`](https://github.com/luukhopman/dpm-xl/blob/main/examples/custom_provider.py)
is a complete provider in 40 lines.
# Evaluate a rule file
A validation run evaluates a few hundred to a few thousand rules, each with an
optional precondition, against one report. This guide does that in Python and
on the command line, using the files in the repository's
[`examples/`](https://github.com/luukhopman/dpm-xl/tree/main/examples) folder.
## Rule files
A rule source is one of:
- a single expression, given on the command line;
- a text file with one expression per line, where `#` starts a comment line
and blank lines are ignored
([`examples/rules.txt`](https://github.com/luukhopman/dpm-xl/blob/main/examples/rules.txt));
- a `.json` file holding a list of objects with `code`, `expression` and an
optional `precondition`
([`examples/rules.json`](https://github.com/luukhopman/dpm-xl/blob/main/examples/rules.json)).
```json
[
{
"code": "v0001",
"expression": "with {tC_01.00, c*, default: 0, interval: true}: {r0010} = {r0020} + {r0030}",
"precondition": "{v_C_01.00}"
},
{"code": "v0002", "expression": "{tC_27.00, c0010} > 0"}
]
```
[Data formats](../reference/data-formats.md#rule-files) has the details.
## In Python
Share one `SelectionCache` across the rules
of a report, so each selection is read from the provider once. Parsing is
memoized for you (`parse_cached`).
```python
import json
from pathlib import Path
from dpmxl import SelectionCache, evaluate_rule, explain
from dpmxl.cli import load_memory_provider
provider = load_memory_provider(json.loads(Path("examples/data.json").read_text()))
rules = json.loads(Path("examples/rules.json").read_text())
cache = SelectionCache()
for rule in rules:
result = evaluate_rule(
rule["expression"],
provider,
precondition=rule.get("precondition"),
cache=cache,
)
print(f"{rule['code']}: {result.status}")
for outcome in result.failures:
explanation = explain(result.evaluation, rule["expression"], outcome.key)
print(" ", explanation.summary)
```
```text
v0001: failed
1,204 should equal 1,211 (1,180 + 31): difference 7, at least 5.5 given the reported precision
v0002: failed
-5 should exceed 0: difference 5
v0003: passed
v0004: not_applicable
```
`load_memory_provider` builds the in-memory report the CLI reads from a
[data file](../reference/data-formats.md#data-files); in your application the
provider is your own ([Connect your data](provider.md)).
Each result has one of five statuses:
| Status | Meaning |
| --- | --- |
| `passed` | At least one record held and none failed. |
| `failed` | At least one record failed; `result.failures` lists them. |
| `not_applicable` | The precondition was not met, or no record could be evaluated (all null). |
| `error` | Syntax error, unknown table, type mismatch, or a division by zero that reaches the result; `result.message` says which. |
| `unsupported` | A `time_shift`, `{o_…}` or `{g_…}` the provider cannot resolve. |
[Preconditions and results](../concepts/results.md) explains each one.
`evaluate_rule` never raises for a bad rule or bad data, so one broken rule
does not stop the run.
## On the command line
`dpm-xl eval` evaluates a rule source against a data file and explains every
failure:
```console
$ dpm-xl eval examples/rules.txt examples/data.json
line 2: failed
records: 1 passed, 1 failed, 0 not evaluated
- c=0020: 1,204 should equal 1,211 (1,180 + 31): difference 7, at least 5.5 given the reported precision
line 3: failed
records: 1 passed, 1 failed, 0 not evaluated
- qCPI=LEI-B: -5 should exceed 0: difference 5
line 4: passed
records: 1 passed, 0 failed, 0 not evaluated
3 rules: 2 failed, 1 passed
```
A single expression takes its precondition from `--precondition`:
```console
$ dpm-xl eval "{tC_01.00, r0010, c0010} > 200" examples/data.json --precondition "{v_C_01.00}"
status: failed
records: 0 passed, 1 failed, 0 not evaluated
- scalar: 100 should exceed 200: difference 100
```
`--json` prints the same results as one JSON document for other programs:
```console
$ dpm-xl eval "{tC_27.00, c0010} > 0" examples/data.json --json
{
"schema_version": 1,
"results": [
{
"rule": "expression",
"expression": "{tC_27.00, c0010} > 0",
"status": "failed",
"message": null,
"counts": {
"passed": 1,
"failed": 1,
"null": 0
},
"failures": [
{
"key": {
"qCPI": "LEI-B"
},
"summary": "-5 should exceed 0: difference 5",
"cells": [
"{tC_27.00, c0010, qCPI=LEI-B}"
]
}
]
}
]
}
```
The exit status is 0 when every rule passed or did not apply, 1 when a rule
failed and 3 when a rule is an error or unsupported, or the input cannot be
read. The [CLI reference](../reference/cli.md) lists every option and the JSON
layout.
# Explain failures in a UI
A failing rule is only useful when someone can see why. For every failing
record, `explain` returns an
`Explanation`: a sentence, the formula with the
reported values filled in, both sides of the comparison with their intervals,
and the cells involved. This guide turns that into a payload for a review
screen.
## The report
```python
from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule, explain
amounts = {
("0010", "0010"): 100,
("0020", "0010"): 60,
("0030", "0010"): 40,
("0010", "0020"): 1204,
("0020", "0020"): 1180,
("0030", "0020"): 31,
}
provider = MemoryProvider(
[TableLayout("C_01.00", rows=("0010", "0020", "0030"), columns=("0010", "0020"))],
{
("C_01.00", row, column, None, ()): Number.of(amount, 0)
for (row, column), amount in amounts.items()
},
)
expression = (
"with {tC_01.00, c*, default: 0, interval: true}: {r0010} = {r0020} + {r0030}"
)
result = evaluate_rule(expression, provider)
outcome = result.failures[0]
explanation = explain(result.evaluation, expression, outcome.key)
```
`result.evaluation` holds the intermediate values of every node, which
`explain` reads. Pass the same `expression` text the rule was evaluated from.
## What an explanation holds
```python
print(explanation.kind)
print(explanation.summary)
print(explanation.formula)
print(explanation.filled)
print(explanation.relation, "|", explanation.verb)
print(
explanation.difference,
explanation.gap,
explanation.tolerance,
explanation.direction,
)
```
```text
comparison
1,204 should equal 1,211 (1,180 + 31): difference 7, at least 5.5 given the reported precision
{r0010} = {r0020} + {r0030}
1,204 = 1,180 + 31
= | should equal
7 5.5 1.5 too_low
```
| Field | Meaning |
| --- | --- |
| `kind` | `comparison`, or for other failures `missing` (a cell must be reported), `not_empty` (must be empty), `membership` (must be one of a set), `format` (must match a pattern) or `condition`. |
| `summary` | One sentence for the reviewer. |
| `formula`, `filled` | The failing statement's formula, and the same with values (`1,204 = 1,180 + 31`). Formulas are rendered from the tree, with uniform spacing, a typographic minus for negative terms and long runs of zeros collapsed to `+ 0 (x13)`. |
| `relation`, `verb` | The comparison symbol (`=`, `≠`, `≤`, …) and its wording (`should equal`). |
| `difference` | Left minus right, as text. |
| `tolerance` | How far apart the sides may be from reported precision alone (the sum of both sides' thresholds). |
| `gap` | The distance that remains after allowing for that precision, rounded down so that "at least" never overstates it. |
| `direction` | `too_high` when the left side exceeds the right, `too_low` otherwise. It says which side is larger, not which value is wrong. |
| `left`, `right` | Each side as a `Side`. |
| `operands` | Every leaf operand (cell, aggregate, parameter) as an `Operand`: its source text, value and cells. |
| `primary` | The cells on the left of the failing comparison: the values that disagree. |
Fields that do not apply are `None`; a failure that is not a comparison has
no sides, difference or gap.
## Sides and intervals
```python
for side in (explanation.left, explanation.right):
print(
side.formula,
"|",
side.filled,
"|",
side.value,
f"[{side.low}, {side.high}]",
side.terms,
)
```
```text
{r0010} | 1,204 | 1,204 [1,203.5, 1,204.5] 1
{r0020} + {r0030} | 1,180 + 31 | 1,211 [1,210, 1,212] 2
```
With `interval: true`, each reported number stands for an interval (see
[Interval arithmetic](../concepts/intervals.md)). The left side can be at
most 1,204.5 and the right side at least 1,210, so they are at least 5.5
apart: that is the `gap`.
## Highlight the cells
Every data point carries the `ref` your provider gave it.
`outcome.points` are all the cells the record used; `explanation.primary` is
the subset that disagrees. `MemoryProvider` uses the cell key as its `ref`;
your provider can use a database id or a location in the report file.
```python
def payload(rule_code, expression, result, outcome):
explanation = explain(result.evaluation, expression, outcome.key)
return {
"rule": rule_code,
"record": {name: str(value) for name, value in outcome.key.items()},
"summary": explanation.summary,
"formula": explanation.filled,
"highlight": [point.label() for point in explanation.primary],
"cells": [point.label() for point in outcome.points],
}
import json
print(json.dumps(payload("v0001", expression, result, outcome), indent=2))
```
```text
{
"rule": "v0001",
"record": {
"c": "0020"
},
"summary": "1,204 should equal 1,211 (1,180 + 31): difference 7, at least 5.5 given the reported precision",
"formula": "1,204 = 1,180 + 31",
"highlight": [
"{tC_01.00, r0010, c0020}"
],
"cells": [
"{tC_01.00, r0010, c0020}",
"{tC_01.00, r0020, c0020}",
"{tC_01.00, r0030, c0020}"
]
}
```
`DataPoint.label()` renders a cell as a
DPM-XL selection; `point.ref` is the identity to look it up by.
## Failures that are not comparisons
Presence checks, membership and pattern rules get their own sentence:
```python
from dpmxl import Item
layout = TableLayout("T", rows=("0010", "0020", "0030", "0040"), columns=("0010",))
values = {
("T", "0010", "0010", None, ()): Number.of(5, 0),
("T", "0020", "0010", None, ()): "AB-12",
("T", "0030", "0010", None, ()): Item.parse("eba_qEC:qx9"),
}
report = MemoryProvider([layout], values)
for rule in (
"if not isnull({tT, r0010, c0010}) then not isnull({tT, r0040, c0010}) endif",
"isnull({tT, r0010, c0010})",
"{tT, r0030, c0010} in {[eba_qEC:qx1], [eba_qEC:qx2]}",
'match({tT, r0020, c0010}, "[0-9]+")',
):
failed = evaluate_rule(rule, report)
explained = explain(failed.evaluation, rule, failed.failures[0].key)
print(f"{explained.kind}: {explained.summary}")
```
```text
missing: {tT, r0040, c0010} must be reported
not_empty: {tT, r0010, c0010} must be empty, but 5 is reported
membership: [qEC:qx9] is not one of [qEC:qx1], [qEC:qx2]
format: "AB-12" does not have the required format
```
## Many failures on a large report
- Explaining is cheap per record, but a rule on a large open table can fail
thousands of times. Explain the first few hundred for the screen and count
the rest with `result.counts`.
- `evaluate_rule(..., trace=False)` skips keeping intermediate values: faster
and leaner when you only need statuses. Such a result cannot be explained.
- To explain a few known failing records of a large closed selection, evaluate
again with `only_keys={"c": ["0020"]}`, which keeps only those closed-axis
codes in every selection. Do not use it for rules with aggregates, whose
totals would cover only the kept records.
- The columnar evaluator returns failing keys but no trace; explain its
failures by evaluating those rules again with `evaluate_rule`
([Evaluate large reports](columnar.md)).
Explanation and error wording may improve in minor releases; the field names
and `kind` values may not ([Stability](../project/stability.md)).
# Evaluate large reports
Some EBA reports have open tables with millions of rows. The reference
evaluator builds one Python record per row, which takes seconds per rule at
that size. [`dpmxl.columnar`](../reference/api/columnar.md) evaluates the same
language with the same semantics on polars frames, in milliseconds per rule.
It needs the `columnar` extra:
```bash
pip install "dpm-xl[columnar]"
```
The columnar evaluator covers the constructs of almost every EBA rule. For
anything else it raises
`ColumnarUnsupported`, and you evaluate
that rule with the reference evaluator instead. The two evaluators agree;
the test suite checks them case by case.
## A frame provider
A `FrameProvider` is a
`DataProvider` whose `select_frame(query)`
returns a polars frame instead of a list of `DataPoint`s. The frame has one
row per selected data point, in the
[frame schema](../reference/api/columnar.md#dpmxl.columnar--frame-schema):
| Column | Type | Content |
| --- | --- | --- |
| `cell` | any | Opaque cell reference |
| `r`, `c`, `s` | text or null | Raw closed coordinates |
| `k:` | text (may be categorical) | One per open key: text, canonical items (`category:code`) or ISO dates |
| `num` | `Decimal(38, 12)` | Numeric value |
| `thr` | `Decimal(38, 12)` | Its interval threshold, `0.5·10^-decimals` (0 when exact) |
| `txt` | text | Text, canonical item or ISO date value |
| `flag` | Boolean | Boolean value |
All four value columns are null for a cell with no fact, and a cell and
open-key combination appears at most once.
Here an open table of 20,000 counterparties, with amounts reported in whole
units:
```python
from decimal import Decimal
import polars as pl
from dpmxl.columnar import CELL, DEC, FLAG, NUM, THR, TXT
exposures = pl.DataFrame(
{
"counterparty": [f"LEI-{i:06d}" for i in range(20_000)],
"amount": [Decimal(i % 1000 - 3) for i in range(20_000)],
}
)
class ExposureFrames:
"""A frame provider over one open table, C_27.00, keyed by qCPI."""
def select_frame(self, query):
if query.table != "C_27.00":
raise LookupError(f"unknown table {query.table}")
return exposures.select(
pl.concat_str(pl.lit("C_27.00/"), pl.col("counterparty")).alias(CELL),
pl.lit(None, pl.String).alias("r"),
pl.lit("0010").alias("c"),
pl.lit(None, pl.String).alias("s"),
pl.col("counterparty").alias("k:qCPI"),
pl.col("amount").cast(DEC).alias(NUM),
pl.lit(Decimal("0.5")).cast(DEC).alias(THR), # decimals = 0
pl.lit(None, pl.String).alias(TXT),
pl.lit(None, pl.Boolean).alias(FLAG),
)
def table_reported(self, table):
return table == "C_27.00"
def parameter(self, name):
return None
```
The query's selectors are applied by the evaluator, so `select_frame` may
return the whole table; filtering by `query.columns` first saves memory on
wide tables. `value_columns` maps one Python
value to the four value columns, and
`frame_from_points` builds a frame from
`DataPoint`s.
## Evaluate
`evaluate_rule_columnar` returns a
`ColumnarResult` with the status, the record
counts and the keys of the failing records. Share one
`FrameCache` across the rules of a report.
```python
from dpmxl.columnar import FrameCache, evaluate_rule_columnar
cache = FrameCache()
result = evaluate_rule_columnar("{tC_27.00, c0010} > 0", ExposureFrames(), cache=cache)
print(result.status, result.passed, result.failed, result.null)
print(result.failing_keys[:2])
print(result.failing_frame.height)
```
```text
failed 19920 80 0
[{'qCPI': 'LEI-000000'}, {'qCPI': 'LEI-000001'}]
80
```
`failing_keys` lists the first `max_failing_keys` (1,000 by default) failing
keys as dictionaries; `failing_frame` has all of them, one string column per
key. Statuses are the same five as the reference evaluator's.
## Fall back on `ColumnarUnsupported`
Statements and assignments, set operators such as `union`, analytic
functions, the `&` operator, functions other than the aggregates, `filter`,
`abs`, `isnull`, `nvl`, `match`, `len` and `min`/`max` (so `time_shift`,
`annualise`, `substr`, the mathematical and the date functions), typed
parameter, operation and table-group references, values that do not fit
`Decimal(38, 12)`, arithmetic that would overflow it, record-level errors
(a zero divisor or a type error, which the reference evaluator confines to
its record), ordering items or text that may be an item and a `default`
standing in for a record another operand has
([Records and keys](../concepts/records-and-keys.md#defaults-fill-in-missing-records))
all raise `ColumnarUnsupported`. Catch it and evaluate the rule with
`evaluate_rule`, which needs `select`. A
provider can offer both: here `select` builds `DataPoint`s from the same
frame.
```python
from dpmxl import DataPoint, Number, SelectionCache, evaluate_rule, explain
from dpmxl.columnar import ColumnarUnsupported
class Exposures(ExposureFrames):
"""Both interfaces: frames for the columnar evaluator, points for the rest."""
def select(self, query):
if query.table != "C_27.00":
raise LookupError(f"unknown table {query.table}")
return [
DataPoint(
"C_27.00",
column="0010",
keys={"qCPI": lei},
value=Number.of(amount, 0),
ref=f"C_27.00/{lei}",
)
for lei, amount in exposures.iter_rows()
]
def open_keys(self, table):
return ("qCPI",) if table == "C_27.00" else ()
def run_rule(expression, provider, frames, points):
"""Evaluate on frames; fall back to the reference evaluator."""
try:
result = evaluate_rule_columnar(expression, provider, cache=frames)
except ColumnarUnsupported as reason:
result = evaluate_rule(expression, provider, cache=points, trace=False)
return "reference", result.status, result.counts["failed"], str(reason)
return "columnar", result.status, result.failed, None
provider, frames, points = Exposures(), FrameCache(), SelectionCache()
for expression in (
"{tC_27.00, c0010} > -5",
"total := sum({tC_27.00, c0010}); total > 0",
):
print(run_rule(expression, provider, frames, points))
```
```text
('columnar', 'passed', 0, None)
('reference', 'passed', 0, 'Program')
```
The `ColumnarUnsupported` message names what the columnar evaluator lacks.
Log it if you want to know which rules take the slower path.
## Explain the failures
A `ColumnarResult` carries no trace, so explanations come from the reference
evaluator. Evaluate only the rules that failed again with `evaluate_rule`
(on a large table this takes as long as any reference evaluation, so cap the
number of explanations you show):
```python
expression = "{tC_27.00, c0010} > 0"
traced = evaluate_rule(expression, provider, cache=points)
for outcome in traced.failures[:3]:
print(
outcome.key["qCPI"], explain(traced.evaluation, expression, outcome.key).summary
)
```
```text
LEI-000000 -3 should exceed 0: difference 3
LEI-000001 -2 should exceed 0: difference 2
LEI-000002 -1 should exceed 0: difference 1
```
## Precision
Columnar values are `Decimal(38, 12)`: sums and differences are exact,
products and quotients round to 12 decimals (thresholds of quotients round
up), where the reference evaluator keeps 34 significant digits. An exact
comparison that hinges on the 13th decimal can therefore differ; values that
do not fit `Decimal(38, 12)` raise `ColumnarUnsupported` rather than being
rounded.
## Any `DataProvider` as frames
`PointsFrameProvider` adapts any
`DataProvider`, which is the quickest way to try the columnar evaluator on
existing data. It converts each selection's `DataPoint`s, so the speed-up is
smaller than with frames built directly from your storage:
```python
from dpmxl.columnar import PointsFrameProvider
print(
evaluate_rule_columnar(
"{tC_27.00, c0010} > 0", PointsFrameProvider(provider)
).failed
)
```
```text
80
```
## Threads
`evaluate_rule_columnar` may run concurrently on one report, and polars
releases the GIL, so threads speed up a rule run on several cores. A shared
`FrameCache` builds each selection once while other threads wait for it,
within its row budget (`FrameCache(max_rows=20_000_000)`, least recently used
selections evicted first). See [Run rules in threads](threads.md).
# Run rules in threads
`evaluate_rule` and `evaluate_rule_columnar` may run concurrently on one
report. What may be shared between threads:
| Object | Safe to share | Notes |
| --- | --- | --- |
| `parse_cached` | yes | Parsed trees are immutable and serve every report. |
| `SelectionCache` | yes | Two threads that miss the same selection at once may both build it; either result is kept. Use one per report. |
| `FrameCache` | yes | Builds each selection once while other threads asking for it wait. Use one per report. |
| `MemoryProvider` | yes | Read-only after construction. |
| Your provider | if it is safe to read concurrently | A provider with a database connection per instance is not; use a connection pool or one provider per thread. |
| `Evaluator` | no | One per evaluation; `evaluate_rule` creates its own. |
The reference evaluator is pure Python, so threads help it only when your
provider waits on I/O. The columnar evaluator spends its time in polars,
which releases the GIL, so threads use several cores.
## A rule run on a thread pool
```python
from concurrent.futures import ThreadPoolExecutor
from dpmxl import MemoryProvider, Number, SelectionCache, TableLayout, evaluate_rule
provider = MemoryProvider(
[TableLayout("C_01.00", rows=("0010", "0020", "0030"), columns=("0010", "0020"))],
{
("C_01.00", row, column, None, ()): Number.of(amount, 0)
for (row, column), amount in {
("0010", "0010"): 100,
("0020", "0010"): 60,
("0030", "0010"): 40,
("0010", "0020"): 1204,
("0020", "0020"): 1180,
("0030", "0020"): 31,
}.items()
},
)
rules = {
"v0001": "with {tC_01.00, c*, default: 0, interval: true}: {r0010} = {r0020} + {r0030}",
"v0002": "with {tC_01.00, c0010}: {r0020} >= {r0030}",
"v0003": "{tC_01.00, r0010, c0020} < 1000",
}
cache = SelectionCache() # one per report, shared by every thread
def run(code):
return code, evaluate_rule(rules[code], provider, cache=cache).status
with ThreadPoolExecutor(max_workers=4) as pool:
for code, status in pool.map(run, rules):
print(code, status)
```
```text
v0001 failed
v0002 passed
v0003 failed
```
`pool.map` returns results in submission order, whatever order the threads
finish in.
## Columnar rules on threads
The same pattern with the columnar evaluator and a shared `FrameCache`,
falling back to the reference evaluator for rules it does not cover (see
[Evaluate large reports](columnar.md)):
```python
from dpmxl.columnar import (
ColumnarUnsupported,
FrameCache,
PointsFrameProvider,
evaluate_rule_columnar,
)
frames = FrameCache()
frame_provider = PointsFrameProvider(provider)
def run_columnar(code):
try:
return code, evaluate_rule_columnar(
rules[code], frame_provider, cache=frames
).status
except ColumnarUnsupported:
return code, evaluate_rule(rules[code], provider, cache=cache).status
with ThreadPoolExecutor(max_workers=4) as pool:
print(dict(pool.map(run_columnar, rules)))
```
```text
{'v0001': 'failed', 'v0002': 'passed', 'v0003': 'failed'}
```
## Processes
For many reports at once, evaluate each report in its own process (for
example with `ProcessPoolExecutor`) and use threads within a report. Parsed
trees, caches and providers are per process; `parse_cached` warms up again
in each.
# Compare with earlier reports
`time_shift(x, period, count[, key])` compares a value with the same value
some periods earlier. It works in one of two ways, depending on `key`:
- **Another report.** When `key` is not a key component of `x` (usually
`refPeriod`, or no key at all), `x` is evaluated on the report `count`
periods earlier, which the provider's `earlier_report(period, count)`
returns. This is how EBA rules compare with the previous reporting period.
- **Within the report.** When `key` is a date key component of `x`, the
values of that key move `count` periods later, so each record lines up
with the one `count` periods before it.
Periods are `A` (year), `S` (semester), `Q` (quarter), `M` (month), `W`
(week) and `D` (day). Dates move by calendar months (12, 6, 3 or 1; a
month's last day stays the last day) or by 7 or 1 days. A negative `count`
looks forward.
## Against the previous report
`MemoryProvider` takes the earlier reports as `earlier_reports`, keyed by
`(period, count)`. Your own provider implements `earlier_report`
([Connect your data](provider.md#earlier_report-time_shift-across-periods)).
```python
from datetime import date
from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule, explain
layout = [TableLayout("C_02.00", rows=("0010",), columns=("0010",))]
september = MemoryProvider(
layout,
{("C_02.00", "0010", "0010", None, ()): Number.of(240, 0)},
parameters={"refPeriod": date(2026, 9, 30)},
)
december = MemoryProvider(
layout,
{("C_02.00", "0010", "0010", None, ()): Number.of(260, 0)},
parameters={"refPeriod": date(2026, 12, 31)},
earlier_reports={("Q", 1): september},
)
growth = "{tC_02.00, r0010, c0010} <= time_shift({tC_02.00, r0010, c0010}, Q, 1, refPeriod) * 1.05"
result = evaluate_rule(growth, december)
outcome = result.failures[0]
print(explain(result.evaluation, growth, outcome.key).summary)
for point in outcome.points:
print(point.ref)
```
```text
260 should not exceed 252 (time_shift(240, Q, 1, refPeriod) × 1.05): difference 8
('C_02.00', '0010', '0010', None, ())
EarlierRef(period='Q', count=1, ref=('C_02.00', '0010', '0010', None, ()))
```
The explanation shows the earlier report's value. Points read from the earlier
report carry `EarlierRef` as their `ref`,
wrapping that report's own `ref`, so a user interface can tell the two cells
apart and link to the earlier report.
When the provider has no such report, the rule is `unsupported` rather than
failed:
```python
year_on_year = "{tC_02.00, r0010, c0010} >= time_shift({tC_02.00, r0010, c0010}, A, 1)"
result = evaluate_rule(year_on_year, december)
print(result.status, "|", result.message)
```
```text
unsupported | time_shift needs the report 1 year(s) earlier, which the provider does not have
```
Without an `earlier_report` method at all, the message says that the rule
compares with the previous reporting period, which a single report does not
include. Evaluating with a shared `SelectionCache` keeps a separate cache for
each earlier report.
## Within one report, on a date key
An open table keyed by a date can compare each date with the one before it:
```python
history = MemoryProvider(
[TableLayout("C_99.00", columns=("0010",), open_keys=("qDAT",))],
{
("C_99.00", None, "0010", None, (("qDAT", date(2026, 3, 31)),)): Number.of(
100, 0
),
("C_99.00", None, "0010", None, (("qDAT", date(2026, 6, 30)),)): Number.of(
90, 0
),
("C_99.00", None, "0010", None, (("qDAT", date(2026, 9, 30)),)): Number.of(
120, 0
),
},
)
rising = "{tC_99.00, c0010} >= time_shift({tC_99.00, c0010}, Q, 1, qDAT)"
result = evaluate_rule(rising, history)
for outcome in result.outcomes:
print(outcome.key["qDAT"], outcome.passed)
```
```text
2026-06-30 False
2026-09-30 True
```
The shifted operand has records for June and September (March plus one
quarter, June plus one quarter, and September moved past the data), so the
comparison has records where both sides do: June fails (90 < 100), September
passes (120 ≥ 90).
The explanation fills the shifted operand with the value of the record it
came from, here March's:
```python
june = result.failures[0]
print(explain(result.evaluation, rising, june.key).summary)
```
```text
90 should be at least 100 (time_shift(100, Q, 1, qDAT)): difference 10
```
## `annualise`
`annualise(x, fiscalYearEnd, key)` scales a year-to-date amount to a full
year: `x · 12 / months`, where `months` counts the months from the start of
the fiscal year that ends in month `fiscalYearEnd` up to and including the
month of the date in `key`. `key` is a date key component of `x`, or else a
report parameter such as `refPeriod`. The month is treated as complete, so
`annualise` is meant for period-end dates.
```python
result = evaluate_rule("annualise({tC_99.00, c0010}, 12, qDAT) > 300", history)
for outcome in result.outcomes:
print(outcome.key["qDAT"], outcome.passed)
```
```text
2026-03-31 True
2026-06-30 False
2026-09-30 False
```
March annualises to 100 · 12 / 3 = 400, June to 90 · 12 / 6 = 180 and
September to 120 · 12 / 9 = 160.
## Columnar evaluation
The columnar evaluator hands every `time_shift` and `annualise` rule to the
reference evaluator by raising `ColumnarUnsupported`
([Evaluate large reports](columnar.md)).
# Check rule syntax in CI
A repository of rules (exported from a DPM release, or written by hand)
should never hold an expression that does not parse. `dpm-xl check` parses
every expression and precondition of a rule source and exits with status 1
when any of them fails. With `--strict` it also fails on other errors, such
as an unknown function or a call with the wrong operands, and with
`--layouts` it checks tables and coordinates too
([CLI reference](../reference/cli.md#dpm-xl-check)).
## With the CLI
```console
$ dpm-xl check examples/rules.json
6/6 expressions parsed
$ echo $?
0
```
A syntax error names the rule and points at the offending text:
```console
$ dpm-xl check "with {tC_01.00, c0010}: {r0010} = {r0020} +* {r0030}"
argument: error: unexpected '*' where an operand is expected
with {tC_01.00, c0010}: {r0010} = {r0020} +* {r0030}
^
0/1 expressions parsed; 1 error
$ echo $?
1
```
In a rule file the source is `:expression` or `:precondition`
(JSON rule files) or `line :expression` (text files). `--json` prints the
same as a document, with the zero-based character `start` and `end` of each
problem, for annotating a pull request:
```console
$ dpm-xl check "if {tC_01.00, r0010} > 0 then true" --json
{
"schema_version": 1,
"parsed": 0,
"total": 1,
"errors": [
{
"source": "argument",
"message": "expected 'endif', found end of expression",
"position": 34,
"end": 34
}
],
"diagnostics": [
{
"source": "argument",
"severity": "error",
"code": "syntax",
"message": "expected 'endif', found end of expression",
"start": 34,
"end": 34,
"subject": null,
"suggestion": null
}
]
}
```
!!! note "A missing rule file is an error"
An argument that reads as a path (it ends in `.json` or `.txt`, or
holds a `/` without any brackets, operators or spaces) but names no
file fails with `error: no such rule file: ...` and exit status 3, so a
mistyped path cannot pass as an expression.
## A GitHub Actions job
```yaml
name: Rules
on: [push, pull_request]
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
- run: test -f rules/rules.json
- run: uvx dpm-xl check rules/rules.json --strict
```
Pin the version (`uvx dpm-xl@1.0.0 ...`) if you want a new release to be a
deliberate change.
## With pytest
A test can check more than syntax: that every rule only refers to tables your
release defines, for example. `parse_program`
accepts everything `dpm-xl check` accepts, and
`tables` lists the tables an expression selects
from.
```python
import json
from pathlib import Path
from dpmxl import DpmXlSyntaxError, parse_program, tables
KNOWN_TABLES = {"C_01.00", "C_27.00"}
def check_rules(path):
problems = []
for rule in json.loads(Path(path).read_text()):
for part in ("expression", "precondition"):
if not rule.get(part):
continue
try:
tree = parse_program(rule[part])
except DpmXlSyntaxError as error:
problems.append(f"{rule['code']} {part}: {error.message}")
continue
unknown = set(tables(tree)) - KNOWN_TABLES
if unknown:
problems.append(
f"{rule['code']} {part}: unknown tables {sorted(unknown)}"
)
return problems
print(check_rules("examples/rules.json"))
```
```text
["v0004 expression: unknown tables ['C_99.00']"]
```
In a test, `assert not check_rules("rules/rules.json")` fails the job and
lists every problem.
## What parses
dpm-xl parses every construct of the DPM-XL grammar and every EBA validation
rule and precondition of DPM releases 3.4–4.4, except two EBA expressions with
typos (`trueendif`, `ornot`). A few readings differ from the grammar on
purpose; [Language](../reference/language.md#where-dpm-xl-differs-from-the-grammar)
lists them.
# Build editor support
dpm-xl exposes what an editor needs: typed tokens with source offsets, the
language's keywords, function names and signatures, a TextMate grammar, and
positioned diagnostics with suggested fixes, from every syntax error to
unknown tables, coordinates, keys and operands of the wrong type.
## Highlight with tokens
`tokenize` splits an expression into
`Token(kind, text, start, end)` tuples. The parser
works from the same tokens, so highlighting and parsing never disagree.
```python
from dpmxl.parser import tokenize
for token in tokenize(
"with {tC_01.00, c0010}: sum({r0010-0050} group by qEBB) >= 0.5%"
):
print(f"{token.kind:12} {token.text!r:14} {token.start}-{token.end}")
```
```text
keyword 'with' 0-4
brace '{' 5-6
table 'tC_01.00' 6-14
punctuation ',' 14-15
axis 'c0010' 16-21
brace '}' 21-22
punctuation ':' 22-23
function 'sum' 24-27
punctuation '(' 27-28
brace '{' 28-29
axis 'r0010-0050' 29-39
brace '}' 39-40
keyword 'group by' 41-49
identifier 'qEBB' 50-54
punctuation ')' 54-55
operator '>=' 56-58
number '0.5' 59-62
operator '%' 62-63
eof '' 63-63
```
`text` is exactly `expression[start:end]`, quotes, brackets and backticks
included. The kinds are:
| Kind | Tokens |
| --- | --- |
| `keyword` | `if`, `and`, `with`, `where`, `group by`, window keywords inside `over (…)`, and `default` / `interval` in a selection |
| `type` | A parameter type such as `number` or `set-item` |
| `function` | A known function name followed by `(` |
| `operator` | `+ - * / & % = != <> < <= > >= := <-` |
| `punctuation` | `( ) , : ; [ ]` |
| `brace` | `{` or `}` around a selection or set |
| `table`, `table_group`, `variable`, `operation`, `parameter` | A reference with its prefix: `tC_01.00`, `gG_01`, `vX` or `v_X`, `o_v1234`, `pName` |
| `axis` | A row, column or sheet coordinate, range or wildcard: `r0010`, `c0010-0050`, `s*` |
| `number`, `string`, `date`, `boolean`, `null` | Literals |
| `item` | An item such as `[eba_qEC:qx01]` |
| `period` | A `time_shift` or window period code (`A`, `Q`, …) |
| `identifier` | A key, temporary or other name, including escaped names such as `` `x.y` `` |
| `eof` | The empty token that ends every token list |
`tokenize` raises `DpmXlSyntaxError` on an unterminated literal, brace or item,
or an unknown character; an editor can fall back to the TextMate grammar for
such lines.
## Mark syntax errors
`DpmXlSyntaxError` has a plain `message`,
the zero-based `position` where the problem starts and `end`, just past the
offending text: the range to underline.
```python
from dpmxl import DpmXlSyntaxError, parse_program
source = "sum({tC_01.00, r0010} group by qEBB qX) > 0"
try:
parse_program(source)
except DpmXlSyntaxError as error:
print(
error.message,
(error.position, error.end),
repr(source[error.position : error.end]),
)
print(error.pointer())
```
```text
expected ')', found 'qX' (36, 38) 'qX'
sum({tC_01.00, r0010} group by qEBB qX) > 0
^~
```
Parse with `parse_program` in an editor: it
accepts everything a rule may hold, including `;`-separated statements and
assignments.
## Find every problem
`diagnose` returns everything that can be
found wrong without report data, as
`Diagnostic`s: a `severity` (`error` or
`warning`), a stable `code`, a `message`, the `start` and `end` offsets to
underline and, where it applies, the `subject` (the name, code or type
concerned) and a `suggestion` to replace the underlined text with, for a quick
fix. `pointer` renders one like a
syntax error.
Unlike `parse`, `diagnose` reports every syntax error:
after one, it resumes where the enclosing construct can carry on, and leaves
out errors that follow from an earlier one. Keywords run together, such as
`ornot` or `trueendif`, are pointed out as a missing space. A construct that
holds a syntax error is not checked further, but the rest is:
```python
from dpmxl import diagnose
source = "if {tC_01.00, r0010} > 0 ornot {tC_01.00, r0020} then summ(1) = = 2 endif"
for diagnostic in diagnose(source):
print(diagnostic.code, diagnostic.message)
print(diagnostic.pointer(source))
```
```text
syntax missing space: did you mean 'or not'?
if {tC_01.00, r0010} > 0 ornot {tC_01.00, r0020} then summ(1) = = 2 endif
^~~~~
unknown-function unknown function 'summ'; did you mean 'sum'?
if {tC_01.00, r0010} > 0 ornot {tC_01.00, r0020} then summ(1) = = 2 endif
^~~~
syntax unexpected '=' where an operand is expected
if {tC_01.00, r0010} > 0 ornot {tC_01.00, r0020} then summ(1) = = 2 endif
^
```
Calls are checked against `SIGNATURES`, which
the evaluators use too, so `diagnose` reports exactly the operand counts
evaluation would reject. Pass `program=True` to accept scripts with `;`,
`:=` and `<-`, as `parse_program` does.
### Tables and coordinates
Pass a lookup from table code to `TableLayout`
(or `None` for a table that does not exist) to check tables, rows, columns and
sheets, and `tables` to suggest a table for an unknown one. A problem in a
`with` context is reported once, at the context. Severities follow
evaluation: a selection that selects nothing is an error, while a code
missing from an axis list whose other codes match is a warning. A layout's
`open_axes` say which axes have keys instead of codes. What a code on such an
axis, or on an axis the table does not have, selects is up to the provider (a
`MemoryProvider` ignores it, another may select nothing), so it is a warning
when written for that table and not reported when a shared `with` context
applies it:
```python
from dpmxl import TableLayout
layouts = {
"C_01.00": TableLayout("C_01.00", rows=("0010", "0020"), columns=("0010", "0020")),
"C_27.00": TableLayout(
"C_27.00", columns=("0010",), open_keys=("qCPI",), open_axes=("rows",)
),
}
source = (
"with {tC_01.00, c0010}: {r0030} = {r0011} + {tC_27.0, c0010}"
" + sum({tC_27.00, r0010-0050, c0010})"
)
for diagnostic in diagnose(source, layouts.get, tables=layouts):
print(
f"{diagnostic.severity:7} {source[diagnostic.start : diagnostic.end]!r:12} "
f"{diagnostic.suggestion!s:8} {diagnostic.message}"
)
```
```text
error 'r0030' None row 0030 is not in table C_01.00
error 'r0011' r0010 row 0011 is not in table C_01.00; did you mean r0010?
error 'tC_27.0' tC_27.00 unknown table C_27.0; did you mean C_27.00?
warning 'r0010-0050' r* the rows of C_27.00 are open (key qCPI), so they have no codes such as r0010-0050; select them with r* or leave them out
```
A range that covers no coordinate of its table is `empty-range`, an error
when it is all the selection selects. Table groups and operations are not
checked.
### Data types
When the layouts give their cells' `data_types`, operands of the wrong type
are errors, wherever evaluation would reject them whatever the values. A
selection covering a cell without a type, or with a `default` of another
type, has no known type and is not checked:
```python
layouts["C_01.00"] = TableLayout(
"C_01.00",
rows=("0010", "0020"),
columns=("0010", "0020"),
data_types={
(row, column, None): "number" if column == "0010" else "string"
for row in ("0010", "0020")
for column in ("0010", "0020")
},
)
source = 'with {tC_01.00}: {r0010, c0010} + {r0010, c0020} > 0 and {r0020, c0010} = "x"'
for diagnostic in diagnose(source, layouts.get):
print(diagnostic.message)
```
```text
'+' needs numbers, but {r0010, c0020} is a string
'=' cannot compare a number with a string
```
### Keys and names
A name in a clause, such as `qCPI` in `[where qCPI = "A"]`, a `group by` or an
`over (partition by …)`, must be a key of what it applies to. Pass the report
`parameters` to check names too: a name that is neither a key there, nor a
parameter, nor assigned by an earlier statement of a script is an error.
Without them, a name in a `where` or `get` clause that is not a key is a
warning, since a parameter may stand in for it:
```python
source = "{tC_27.00, c0010} [where qCPX = refDate] > 0 and refPeriod > #2026-01-01#"
for diagnostic in diagnose(source, layouts.get, parameters=["refPeriod"]):
print(diagnostic.severity, diagnostic.message, diagnostic.suggestion)
```
```text
error qCPX is not a key of table C_27.00 (keys qCPI), nor a report parameter qCPI
error refDate is not a key of table C_27.00 (keys qCPI), nor a report parameter None
```
`dpm-xl check` runs the same checks over a rule file, against the tables and
parameters of a data file with `--layouts`
([CLI reference](../reference/cli.md#dpm-xl-check)).
## Complete keywords and functions
[`dpmxl.grammar`](../reference/api/grammar.md) lists the vocabulary as
frozensets: `KEYWORDS`, `WINDOW_KEYWORDS`, `SELECTION_KEYWORDS`,
`OPERATORS`, `PUNCTUATION`, `FUNCTIONS`, `AGGREGATE_FUNCTIONS`,
`ANALYTIC_FUNCTIONS`, `PARAMETER_TYPES`, `LITERAL_WORDS`, `TIME_PERIODS`,
`AXIS_LETTERS` (`r`, `c`, `s`), `REFERENCE_PREFIXES` and `TOKEN_KINDS`.
The [language reference index](../reference/language.md#index) links each
word and symbol to its documentation, for hover help.
```python
from dpmxl import grammar
print(sorted(grammar.AGGREGATE_FUNCTIONS))
print(sorted(grammar.TIME_PERIODS))
print(grammar.canonical_function("max_aggr"), grammar.canonical_function("MAX"))
```
```text
['avg', 'count', 'max_aggr', 'median', 'min_aggr', 'sum']
['A', 'D', 'M', 'Q', 'S', 'W']
max max
```
`canonical_function` gives the
`Call.name` that a spelling parses to (`max_aggr` is `max`), and `CALL_NAMES`
holds all of those names. Each has a `Signature`
in `SIGNATURES`, for signature help:
```python
signature = grammar.SIGNATURES["time_shift"]
print(signature.operands(), signature.key_operand, signature.period_operand)
print(signature.problem("time_shift", 2))
```
```text
3 or 4 operands 3 1
time_shift takes 3 or 4 operands, not 2
```
## TextMate grammar
`textmate_grammar()` builds a TextMate
grammar (scope `source.dpm-xl`) from the same vocabulary, for VS Code,
Sublime Text, GitHub Linguist or any highlighter that reads TextMate
grammars. The repository ships it as
[`editors/dpm-xl.tmLanguage.json`](https://github.com/luukhopman/dpm-xl/blob/main/editors/dpm-xl.tmLanguage.json);
regenerate it after changing `dpmxl.grammar`:
```bash
uv run python -c "import json; from dpmxl.grammar import textmate_grammar; print(json.dumps(textmate_grammar(), indent=2))" > editors/dpm-xl.tmLanguage.json
```
## Show what a rule reads
For hover information or a "go to table" action,
`references` returns every selection with
its inherited `with` context and its source span; see
[Analyse rules statically](analyse.md).
# Analyse rules statically
Before any report is loaded you can ask a rule what it reads: which tables
and cells, which filing indicators, which functions. That is how you index a
rule set by table, find the rules affected by a template change, or decide
which tables a report must contain.
## The expression tree
`parse` returns a tree of frozen dataclasses from
[`dpmxl.ast`](../reference/api/ast.md). Every node has a half-open source
`span`, so you can always show the original text of an operand.
```python
from dpmxl import parse
source = (
"with {tC_01.00, c*, default: 0}: {r0010} = {r0020} + sum({tC_02.00, r0010-0050})"
)
tree = parse(source)
comparison = tree.body
print(type(tree).__name__, type(comparison).__name__, comparison.op)
print(source[slice(*comparison.right.span)])
```
```text
With Binary =
{r0020} + sum({tC_02.00, r0010-0050})
```
`walk` yields every node, depth first, and
`children` a node's direct children (including the
`default` of a selection, a `with` context and a typed parameter):
```python
from dpmxl.ast import Call, walk
print([node.name for node in walk(tree) if isinstance(node, Call)])
```
```text
['sum']
```
The [language reference](../reference/language.md) names the node each
construct parses to. Analytic calls such as `rank(x over (…))` are
`Analytic` nodes, not `Call`s. `parse_program`
returns a `Program` whose `statements` are expressions, `Assignment`s
(`name := e`) and `PersistentAssignment`s (`{target} <- e`).
## Selections with their context
A selection inside `with` inherits the context's table, axes and options.
`references` returns every selection completed
by its context, as `Reference` objects in source
order:
```python
from dpmxl import references
source = "with {tC_01.00, c*, default: 0}: {r0010} = {r0020} + {g_G1, r0010} + {o_v123}"
for reference in references(parse(source)):
print(
f"{reference.kind:12} {reference.code:8} {reference.spec.text():36} {source[slice(*reference.span)]}"
)
```
```text
table C_01.00 {tC_01.00, r0010, c*, default: 0} {r0010}
table C_01.00 {tC_01.00, r0020, c*, default: 0} {r0020}
table_group G1 {gG1, r0010, c*, default: 0} {g_G1, r0010}
operation v123 {ov123, c*, default: 0} {o_v123}
```
`kind` is `table`, `table_group` or `operation`; for the latter two
`spec.table` is `None` and `code` names the group or operation. Typed
parameters (`{p_Name, …}`) are not selections and are not listed; targets of
persistent assignments are.
## Tables and filing indicators
`tables` lists the distinct table codes the
cell selections read. Given a provider with a `table_group` method, a group
read counts as reading every table of the group:
```python
from dpmxl import MemoryProvider, tables
tree = parse(source)
print(tables(tree))
print(tables(tree, MemoryProvider([], {}, table_groups={"G1": ["C_03.00", "C_04.00"]})))
```
```text
('C_01.00',)
('C_01.00', 'C_03.00', 'C_04.00')
```
`reported_templates` lists the
templates a rule tests with `{v_…}`, typically in its precondition:
```python
from dpmxl import reported_templates
print(reported_templates(parse("{v_C_01.00} and not {v_C_02.00}")))
```
```text
('C_01.00', 'C_02.00')
```
## Index a rule set by table
```python
import json
from collections import defaultdict
from pathlib import Path
from dpmxl import parse_program
by_table = defaultdict(list)
for rule in json.loads(Path("examples/rules.json").read_text()):
for table in tables(parse_program(rule["expression"])):
by_table[table].append(rule["code"])
print(dict(by_table))
```
```text
{'C_01.00': ['v0001', 'v0003'], 'C_27.00': ['v0002'], 'C_99.00': ['v0004']}
```
`dpm-xl check --references` prints the same selections for a rule file on the
command line ([CLI reference](../reference/cli.md#dpm-xl-check)).
# From an XBRL report to explained failures
dpm-xl evaluates rules but reads no files;
[dpm-xbrl](https://luukhopman.github.io/dpm-xbrl/) reads xBRL-CSV and
XBRL-XML reports but evaluates no rules. This guide connects the two: it
reads an EBA-style xBRL-CSV report, binds its facts to table cells, evaluates
validation rules with the columnar engine (falling back to the reference
evaluator where needed) and explains every failure.
To start from a generated report instead, see the
[toolkit tour](toolkit-tour.md).
```bash
pip install dpm-xbrl "dpm-xl[columnar]"
```
The report is a small leverage-ratio package in the repository,
[`examples/xbrl-report/`](https://github.com/luukhopman/dpm-xl/tree/main/examples/xbrl-report):
a closed table `C_47.00` and an open table `C_48.02` keyed by counterparty
(`qRDT`). Run the code from a checkout of the repository, or point
`read_report` at your own report.
## 1. Read the report
```python
from dpmxbrl import read_report
report = read_report("examples/xbrl-report")
print(report.format, report.entity_id, report.reference_date)
print(report.filing_indicators)
print(report.decimals_parameters)
for fact in report.facts[:3]:
print(fact.template, fact.datapoint, dict(fact.keys), fact.value, fact.location)
```
```text
xbrl-csv SYNTHETICDPMXL000002.CON 2026-03-31
{'C_47.00': True, 'C_48.02': True, 'C_43.00': False}
{'decimalsMonetary': -3, 'decimalsPercentage': 4}
c_47.00 dp9001 {} 812000 reports/c_47.00.csv:2
c_47.00 dp9002 {} 190000 reports/c_47.00.csv:3
c_47.00 dp9003 {} 40000 reports/c_47.00.csv:4
```
An EBA xBRL-CSV report holds facts by *datapoint* code (`dp9001`), with the
key columns of open tables in `fact.keys`. Values are text as written; the
precision of numbers comes from the report's `decimals*` parameters by data
type. See the [dpm-xbrl documentation](https://luukhopman.github.io/dpm-xbrl/)
for the fact model, XBRL-XML and the EBA filing rules it checks.
## 2. Map datapoints to cells
DPM-XL rules address cells by table, row and column, so each datapoint needs
its position. In production that mapping comes from the DPM release (the
table layouts of the taxonomy or the DPM database). For this example it is
written out:
```python
# datapoint -> (table, row, column, data type); open tables have no row.
DATAPOINTS = {
"dp9001": ("C_47.00", "0010", "0010", "monetary"), # exposure component
"dp9002": ("C_47.00", "0020", "0010", "monetary"), # exposure component
"dp9003": ("C_47.00", "0030", "0010", "monetary"), # exposure component
"dp9004": ("C_47.00", "0040", "0010", "monetary"), # total exposure
"dp9005": ("C_47.00", "0050", "0010", "monetary"), # tier 1 capital
"dp9006": ("C_47.00", "0060", "0010", "percentage"), # leverage ratio
"dp9101": ("C_48.02", None, "0010", "monetary"), # exposure
"dp9102": ("C_48.02", None, "0020", "string"), # counterparty LEI
}
OPEN_KEYS = {"C_48.02": ("qRDT",)} # key columns of the open tables
DECIMALS = {"monetary": "decimalsMonetary", "percentage": "decimalsPercentage"}
```
## 3. Bind the facts to a provider
A [provider](provider.md) answers the evaluator's questions from the report:
the cells of a selection, the filing indicators and the report parameters.
Two things matter. A closed selection returns **every defined cell**, with
`value=None` where nothing was reported, so `default` and `isnull` work.
And numbers carry their reported precision: `Number.of(text, decimals)`.
```python
from collections import defaultdict
from dpmxl import CellQuery, DataPoint, Number, axis_matcher
class XbrlReportProvider:
"""A dpm-xl DataProvider over a dpm-xbrl Report."""
def __init__(self, report, datapoints, open_keys):
self.report = report
self.open_keys_by_table = open_keys
self.cells = defaultdict(list) # closed table -> [(row, column)]
self.columns = defaultdict(set) # open table -> {column}
for table, row, column, _ in datapoints.values():
if table in open_keys:
self.columns[table].add(column)
else:
self.cells[table].append((row, column))
self.values = {} # (table, row, column, keys) -> value
self.rows = defaultdict(dict) # open table -> {keys: None}, in report order
for fact in report.facts:
if fact.datapoint not in datapoints:
continue
table, row, column, kind = datapoints[fact.datapoint]
keys = tuple((name, fact.keys[name]) for name in open_keys.get(table, ()))
self.values[table, row, column, keys] = self._value(fact.value, kind)
if keys:
self.rowstable = None
def _value(self, text, kind):
if text is None or kind == "string":
return text
return Number.of(text, self.report.decimals_parameters.get(DECIMALS[kind]))
def select(self, query):
rows, columns = axis_matcher(query.rows), axis_matcher(query.columns)
if query.table in self.cells:
return [
DataPoint(
query.table,
row=row,
column=column,
value=self.values.get((query.table, row, column, ())),
ref=f"{query.table}/r{row}/c{column}",
)
for row, column in self.cells[query.table]
if rows(row) and columns(column)
]
if query.table in self.columns:
return [
DataPoint(
query.table,
column=column,
keys=dict(keys),
value=self.values.get((query.table, None, column, keys)),
ref=f"{query.table}/c{column}/{keys0}",
)
for keys in self.rows[query.table]
for column in sorted(self.columns[query.table])
if columns(column)
]
raise LookupError(f"unknown table {query.table}")
def open_keys(self, table):
return self.open_keys_by_table.get(table, ())
def table_reported(self, table):
return self.report.filing_indicators.get(table)
def parameter(self, name):
return self.report.parameters.get(name)
provider = XbrlReportProvider(report, DATAPOINTS, OPEN_KEYS)
print(provider.select(CellQuery("C_47.00", rows=None, columns=None))[3])
```
```text
DataPoint(table='C_47.00', row='0040', column='0010', sheet=None, keys={}, value=Number(value=Decimal('1045000'), tolerance=Decimal('5E+2')), ref='C_47.00/r0040/c0010')
```
The total exposure was reported with `decimalsMonetary = -3`, rounded to
thousands, so it stands for 1,045,000 ± 500.
## 4. Evaluate with the columnar engine and fall back
`PointsFrameProvider` presents the
same provider as polars frames. Each rule runs on the columnar engine first;
when it raises `ColumnarUnsupported`,
the reference evaluator takes over. Both share a cache per report.
```python
from dpmxl import SelectionCache, evaluate_rule
from dpmxl.columnar import (
ColumnarUnsupported,
FrameCache,
PointsFrameProvider,
evaluate_rule_columnar,
)
RULES = [
(
"e0001",
"with {tC_47.00, c0010, default: 0, interval: true}: {r0040} = {r0010} + {r0020} + {r0030}",
"{v_C_47.00}",
),
(
"e0002",
"with {tC_47.00, c0010, interval: true}: {r0060} = {r0050} / {r0040}",
"{v_C_47.00}",
),
("e0003", "{tC_48.02, c0010} >= 0", "{v_C_48.02}"),
("e0004", 'match({tC_48.02, c0020}, "[A-Z0-9]{18}[0-9]{2}")', "{v_C_48.02}"),
("e0005", "{tC_43.00, r0010, c0010} > 0", "{v_C_43.00}"),
(
"e0006",
"total := sum({tC_48.02, c0010}); total <= {tC_47.00, r0040, c0010}",
None,
),
]
frames, frame_cache, cache = (
PointsFrameProvider(provider),
FrameCache(),
SelectionCache(),
)
statuses = {}
for code, expression, precondition in RULES:
try:
result = evaluate_rule_columnar(
expression, frames, precondition=precondition, cache=frame_cache
)
engine, status = "columnar", result.status
except ColumnarUnsupported:
result = evaluate_rule(
expression, provider, precondition=precondition, cache=cache, trace=False
)
engine, status = "reference", result.status
statuses[code] = status
print(f"{code}: {status:15} ({engine})")
```
```text
e0001: failed (columnar)
e0002: passed (columnar)
e0003: failed (columnar)
e0004: failed (columnar)
e0005: not_applicable (columnar)
e0006: passed (reference)
```
`e0006` is a script with a temporary assignment (`total := …`), which the
columnar engine leaves to the reference evaluator. `e0005` does not apply:
its precondition tests `C_43.00`, which the report's filing indicators mark
as not reported.
## 5. Explain the failures
The columnar result has the failing keys but no trace, so evaluate the
failed rules again with `evaluate_rule` and
explain each failing record. Its points carry the provider's `ref`s, which
lead back to the cells, and through the mapping to the report's datapoints
and file locations.
```python
from dpmxl import explain
for code, expression, precondition in RULES:
if statuses[code] != "failed":
continue
result = evaluate_rule(expression, provider, precondition=precondition, cache=cache)
for outcome in result.failures:
explanation = explain(result.evaluation, expression, outcome.key)
print(f"{code} {dict(outcome.key)}: {explanation.summary}")
print(" cells:", ", ".join(str(point.ref) for point in explanation.primary))
```
```text
e0001 {}: 1,045,000 should equal 1,042,000 (812,000 + 190,000 + 40,000): difference 3,000, at least 1,000 given the reported precision
cells: C_47.00/r0040/c0010
e0003 {'qRDT': 'QW4KNDKKV7QH2LA40639'}: -12,000 should be at least 0: difference 12,000
cells: C_48.02/c0010/QW4KNDKKV7QH2LA40639
e0004 {'qRDT': 'QW4KNDKKV7QH2LA40639'}: "QW4KNDKKV7QH2LA4063" does not have the required format
cells: C_48.02/c0020/QW4KNDKKV7QH2LA40639
```
The total exposure is 3,000 above the sum of its components. Four amounts
rounded to thousands could explain up to 2,000 of that; at least 1,000
cannot. The leverage ratio rule `e0002` passed because 52,000 / 1,045,000 is
within the precision of the reported 4.98 %.
## In production
- **Mapping.** Build `DATAPOINTS` from the DPM release the report's entry
point names, once per release, rather than by hand.
- **Large reports.** For open tables with millions of rows, build polars
frames directly from dpm-xbrl's columnar reader
(`dpmxbrl.frames.read_xbrl_csv_frames`) in the
[frame schema](../reference/api/columnar.md#dpmxl.columnar--frame-schema) instead of going
through `DataPoint`s; see [Evaluate large reports](columnar.md).
- **Filing rules first.** dpm-xbrl records the EBA filing rules a report
breaks in `report.filing`; check them before evaluating validation rules.
- **Threads.** The provider above is read-only after construction, so rules
can run on a thread pool ([Run rules in threads](threads.md)).
# Toolkit tour
Each package of the dpm toolkit does one job:
[dpm-synth](https://luukhopman.github.io/dpm-synth/) generates test reports,
[dpm-xbrl](https://luukhopman.github.io/dpm-xbrl/) reads and writes them,
and dpm-xl evaluates their validation rules. This tour takes one report
through all three: generate it, plant an error, read it back, then evaluate
the rules and explain the failure.
```bash
pip install dpm-synth # installs dpm-xl and dpm-xbrl too
```
The module is a small template in the repository,
[`examples/toolkit-tour/template.json`](https://github.com/luukhopman/dpm-xl/blob/main/examples/toolkit-tour/template.json):
a closed table `T_01.00` whose row 0010 is the sum of rows 0020 and 0030, an
open table `T_02.00` of exposures by counterparty (`qCP`), and four DPM-XL
rules. Run the code from a checkout of the repository.
## 1. Generate a report with dpm-synth
```python
from dpmsynth import generate, load_template
template = load_template("examples/toolkit-tour/template.json")
generated = generate(template, seed=7)
print(generated.passed, "of", generated.rules, "rules pass")
for fact in generated.report.facts:
print(fact.template, fact.datapoint, dict(fact.keys), fact.value)
```
```text
4 of 4 rules pass
t_01.00 dp101 {} 1126000
t_01.00 dp102 {} 971000
t_01.00 dp103 {} 155000
t_02.00 dp201 {'qCP': 'SYNTHZDE8GXD6NCF1086'} 571000
t_02.00 dp201 {'qCP': 'SYNTHEPF91DHODZDOC37'} 555000
```
Every value is generated and the counterparties are fictional LEIs. The
same seed gives the same report.
## 2. Plant an error and write the package
A synthetic report is for testing a pipeline. Change one value, as a faulty
upstream system might, and write the report as an xBRL-CSV package with
dpm-xbrl:
```python
import io
from dataclasses import replace
from dpmxbrl import write_xbrl_csv
facts = tuple(
replace(fact, value="978000") if fact.datapoint == "dp102" else fact
for fact in generated.report.facts
)
package = io.BytesIO() # or a path such as "report.zip"
write_xbrl_csv(replace(generated.report, facts=facts), package)
```
Row 0020 now says 978,000 instead of 971,000.
## 3. Read it with dpm-xbrl
```python
from dpmxbrl import read_report
report = read_report(package.getvalue())
print(report.format, report.entity_id, report.reference_date)
print(report.decimals_parameters, report.filing_indicators)
print(report.filing)
```
```text
xbrl-csv SYNTHETICDPMXL000002.CON 2026-12-31
{'decimalsMonetary': -3} {'T_01.00': True, 'T_02.00': True}
()
```
The package breaks no EBA filing rule that the file can show; read from
bytes without a `name`, its file name is not checked. Amounts are reported
in thousands (`decimalsMonetary = -3`).
## 4. Bind the facts to a provider
dpm-xl reads no files: it asks a [provider](provider.md) for the cells of
each selection. The template places every datapoint at a table, row and
column, so it gives a `MemoryProvider` its
table layouts and places the facts read by dpm-xbrl:
```python
from dpmxl import MemoryProvider, Number, TableLayout
layouts, cells = [], {}
for table in template.tables:
layouts.append(
TableLayout(
table.code,
rows=tuple(sorted({cell.row for cell in table.cells if cell.row})),
columns=tuple(sorted({cell.column for cell in table.cells})),
open_keys=tuple(key.code for key in table.open_keys),
)
)
for cell in table.cells:
cells[cell.datapoint] = table, cell
decimals = report.decimals_parameters["decimalsMonetary"] # every cell is monetary
values = {}
for fact in report.facts:
table, cell = cells[fact.datapoint]
keys = tuple((key.code, fact.keys[key.column]) for key in table.open_keys)
position = table.code, cell.row, cell.column, None, keys
values[position] = Number.of(fact.value, decimals)
reported = [code for code, filed in report.filing_indicators.items() if filed]
provider = MemoryProvider(layouts, values, reported=reported)
```
`Number.of` keeps the reported precision: 978,000 reported in thousands
stands for 978,000 ± 500. For templates with other data types, see
dpm-synth's guide to
[validating the output](https://luukhopman.github.io/dpm-synth/guides/validate-output/).
## 5. Evaluate and explain with dpm-xl
```python
from dpmxl import evaluate_rule, explain
for rule in template.rules:
result = evaluate_rule(rule.expression, provider, precondition=rule.precondition)
print(rule.code, result.status)
for outcome in result.failures:
print(" ", explain(result.evaluation, rule.expression, outcome.key).summary)
```
```text
v1 failed
1,126,000 should equal 1,133,000 (978,000 + 155,000): difference 7,000, at least 5,500 given the reported precision
v2 passed
v3 passed
v4 passed
```
dpm-xl finds the planted error. Three amounts rounded to thousands could
explain up to 1,500 of the difference; at least 5,500 they cannot.
## Where next
- [dpm-synth](https://luukhopman.github.io/dpm-synth/): write a template
for your own module, or build one from a DPM release.
- [dpm-xbrl](https://luukhopman.github.io/dpm-xbrl/): read XBRL-XML as well,
check the EBA filing rules, and read large packages as frames.
- [From an XBRL report to explained failures](end-to-end.md): the same
provider for a real report layout, with the columnar engine.
- [Explain failures in a UI](explain.md): the cells and the filled formula
behind each summary.
# Concepts
How DPM-XL rules are evaluated. These pages explain the semantics; the
[language reference](../reference/language.md) lists every construct.
- [Records and keys](records-and-keys.md): selections yield keyed records;
operators join them.
- [Null logic](null.md): defaults, null propagation, three-valued logic and
errors confined to their record.
- [Interval arithmetic](intervals.md): reported precision as intervals, and
the gap in explanations.
- [Preconditions and results](results.md): the five statuses and what a
result holds.
- [Statements and assignments](statements.md): scripts with `:=` and `<-`.
- [Sets](sets.md): set literals, parameters, operators and `in`.
- [Aggregates and analytic functions](analytics.md): `group by`, `rank` and
windows.
- [Time](time.md): dates, `time_shift` and `annualise`.
- [Parameters, operations and table groups](references.md): everything a
rule reads besides cells.
## Sources and choices
These semantics follow the EBA *DPM-XL language specification v0.9*
(operators and data model documents) and the Eurofiling interval arithmetic
functions. Where the specification is ambiguous or contradicts itself (sum
of no records, `count` over nulls, key-name aliases, the radius formulas for
products and quotients), dpm-xl makes the choice that avoids false failures,
and says so on the page concerned.
# Records and keys
DPM-XL operates on sets of *records*. A rule's result has one record per
combination of keys, and each record passes, fails or is not evaluated. Keys
are what make "row 0010 equals rows 0020 + 0030 *in every column*" one rule.
The examples on this page use a closed table `T` (rows `0010`–`0030`,
columns `0010`–`0020`) and open tables `O` and `P` keyed by `qEBB`:
```python
from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule
provider = MemoryProvider(
[
TableLayout("T", rows=("0010", "0020", "0030"), columns=("0010", "0020")),
TableLayout("O", columns=("0010", "0020"), open_keys=("qEBB",)),
TableLayout("P", columns=("0010",), open_keys=("qEBB",)),
],
{
("T", "0010", "0010", None, ()): Number.of(10, 0),
("T", "0020", "0010", None, ()): Number.of(4, 0),
("T", "0010", "0020", None, ()): Number.of(7, 0),
("T", "0020", "0020", None, ()): Number.of(7, 0),
("O", None, "0010", None, (("qEBB", "A"),)): Number.of(5, 0),
("O", None, "0010", None, (("qEBB", "B"),)): Number.of(-1, 0),
("O", None, "0020", None, (("qEBB", "A"),)): Number.of(2, 0),
("O", None, "0020", None, (("qEBB", "B"),)): Number.of(0, 0),
("P", None, "0010", None, (("qEBB", "B"),)): Number.of(-1, 0),
("P", None, "0010", None, (("qEBB", "C"),)): Number.of(3, 0),
},
)
def show(expression):
result = evaluate_rule(expression, provider)
records = [(dict(outcome.key), outcome.passed) for outcome in result.outcomes]
print(result.status, result.message or records)
```
## Selections yield records
A selection yields one record per data point. Its key holds every closed axis
that selects more than one coordinate (`r`, `c`, `s`) plus the open key
components, such as `qEBB`. An axis that is not written counts when the
table has several coordinates on it.
```python
show("{tT, r*, c0010} > 0") # key r: three records
show("{tT, r0010, c0010} > 0") # no key: one record
show("{tO, c0010} > 0") # key qEBB
```
```text
passed [({'r': '0010'}, True), ({'r': '0020'}, True), ({'r': '0030'}, None)]
passed [({}, True)]
failed [({'qEBB': 'A'}, True), ({'qEBB': 'B'}, False)]
```
Row `0030` was not reported, so its record is null: not evaluated, never a
failure (see [Null logic](null.md)).
## Operators join on shared keys
Binary operators natural-join their operands on the keys they share. A
scalar (a record without keys) broadcasts to every record of the other side.
Operands whose keys are unrelated are an error.
```python
show("{tT, r0010, c*} > {tT, r0020, c*}") # joined on c
show("{tT, r0010, c0010} > {tT, r*, c0020}") # scalar broadcast over r
show("{tO, c0010} > {tO, c0020}") # joined on qEBB
show("{tT, r0010} > {tT, c0010}") # keys r and c: unrelated
```
```text
failed [({'c': '0010'}, True), ({'c': '0020'}, False)]
passed [({'r': '0010'}, True), ({'r': '0020'}, True), ({'r': '0030'}, None)]
failed [({'qEBB': 'A'}, True), ({'qEBB': 'B'}, False)]
error operands have incompatible keys (r) and (c)
```
A join that would produce more than 2,000,000 records is an error rather
than a memory problem (`recordset.MAX_JOIN_RECORDS`).
## Defaults fill in missing records
An open table has records only for the key combinations reported in it, so
two tables, or two parts of one template, may each lack keys the other has.
An operand with a written `default` (`default: null` included) then stands
for its default where it lacks a key, as a fact variable with a fallback
value does. One rule covers every construct that combines keyed operands:
- **Joins** (operators, and functions of several operands such as `nvl`,
`max`, `match` and `filter`): the operands without a default decide the
keys, keeping those all of them have, and the others fill in their
default. When every keyed operand has a default, the result has a record
for each key any operand with all the result's keys has. Without
defaults, a join keeps only the keys both sides have.
- **`if`**: a keyed condition with a default also covers the keys a branch
has and it lacks, provided every branch without a default has them; the
condition takes its default there. A branch lacking a key takes its
default; without one, the record is null.
- **Which results keep a default**: everything that keeps a record's value
tied to the same fact keeps it, applied to the default: unary operators,
`in`, the operators and functions above (when every keyed operand has a
default), `[rename]`, a `time_shift` that moves a date key, and the
selecting clauses `[where]`, `[sub]`, `filter` and a `with` context's
`where`. A filter restricts the operand's own records, not the keys the
other operands bring: there the operand falls back to its default. An
`if` keeps the default of the branch its condition's default takes.
- **Which results have none**: `[get]` turns key components into values,
and a default has no key components, so a key the operand lacks gives
no value (DPM-XL omits records without a fact). Aggregates and analytic
functions build each record from a group or window of records: a key
without records is no group, and its value is not the operand's default
(`sum` of nothing is null, not `0`). Their results join on shared keys
only.
- A key-less operand without its record, such as `[sub]` on a key value
nobody reported, stands for its default next to keyed operands or
reported values; on its own, or with constants only, it yields no record.
```python
show("{tO, c0010} = {tP, c0010}")
show("with {default: 0}: {tO, c0010} = {tP, c0010}")
show("{tO, c0010} >= {tP, c0010, default: 0}")
show(
"with {default: null}: if not isnull({tP, c0010}) then not isnull({tO, c0010}) endif"
)
show(
"with {default: 0}: if {tO, c0010} > 0 then {tO, c0010} < 10 else {tP, c0010} = 0 endif"
)
show('{tO, c0010, default: 0} [where qEBB = "A"] <= {tP, c0010}')
```
```text
passed [({'qEBB': 'B'}, True)]
failed [({'qEBB': 'A'}, False), ({'qEBB': 'B'}, True), ({'qEBB': 'C'}, False)]
passed [({'qEBB': 'A'}, True), ({'qEBB': 'B'}, True)]
failed [({'qEBB': 'B'}, True), ({'qEBB': 'C'}, False), ({'qEBB': 'A'}, True)]
failed [({'qEBB': 'A'}, True), ({'qEBB': 'B'}, False), ({'qEBB': 'C'}, False)]
failed [({'qEBB': 'B'}, False), ({'qEBB': 'C'}, True)]
```
## Clauses change keys and records
| Clause | Effect |
| --- | --- |
| `[where condition]` | Keeps the records whose keys satisfy the condition. |
| `[get key]` | Replaces each value by the value of a key component; records whose fact is null are omitted. |
| `[sub key = value]` | Fixes a key to a value and removes it from the key. |
| `[rename a to b]` | Renames a key, so it joins with another operand's `b`; an error if the record set already has `b`. |
| `group by k, …` | In an aggregate: one record per value of the listed keys. |
A `with` context's `where` applies to each selection in its body that has
the keys it names. Its terms (split at `and`) that name a key the selection
lacks, neither a key nor a report parameter, leave that selection alone, and
a key-less selection takes none of it. A `where` or `sub` on a selection
inside the body overrides the terms that name the same key; the other terms
still apply.
```python
show('with {tO, c0010} [where qEBB = "A"]: {tO} > {tT, r0020, c0010}')
```
```text
passed [({'qEBB': 'A'}, True)]
```
## Key names
Key names in `where`, `group by`, `sub` and `get` match with or without the
DPM 2.0 `q` prefix and ignore case, so `EBB` and `ebb` find `qEBB`:
```python
show('{tO, c0010}[where EBB = "A"] > 0')
show("sum({tO, c0010} group by qEBB) > 0")
```
```text
passed [({'qEBB': 'A'}, True)]
failed [({'qEBB': 'A'}, True), ({'qEBB': 'B'}, False)]
```
Inside a clause, a key component wins over a temporary of the same name
([Statements](statements.md)).
## Codes and items
Axis codes compare the way DPM does: numeric codes as numbers padded to four
digits, so `r310` in an older rule matches row `0310`
(`normalize_code`). Items compare by code and
category, ignoring the `eba_` prefix and a category missing on either side:
`[eba_qEC:qx01]` equals `[qEC:qx01]` and `[qx01]`.
# Null logic
Reports leave cells empty, and rules must not fail because of it. DPM-XL
handles missing values with defaults, null propagation and three-valued
logic.
```python
from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule
provider = MemoryProvider(
[
TableLayout("T", rows=("0010", "0020", "0030"), columns=("0010",)),
TableLayout("O", columns=("0010", "0020"), open_keys=("qEBB",)),
],
{
("T", "0010", "0010", None, ()): Number.of(10, 0),
("T", "0020", "0010", None, ()): Number.of(4, 0),
# T r0030 is not reported
("O", None, "0010", None, (("qEBB", "A"),)): Number.of(5, 0),
("O", None, "0010", None, (("qEBB", "B"),)): Number.of(-1, 0),
("O", None, "0020", None, (("qEBB", "A"),)): Number.of(2, 0),
("O", None, "0020", None, (("qEBB", "B"),)): Number.of(0, 0),
},
)
def show(expression):
result = evaluate_rule(expression, provider)
print(
result.status, result.message or [outcome.passed for outcome in result.outcomes]
)
```
## Missing facts and defaults
A cell with no fact takes the selection's `default`; with no default it is
null. A provider must therefore return every defined cell of a closed
selection, reported or not ([Connect your data](../guides/provider.md)).
The default also stands in where another operand has an open key the
selection lacks ([Records and keys](records-and-keys.md#defaults-fill-in-missing-records)).
```python
show("{tT, r0030, c0010} > 0")
show("{tT, r0030, c0010, default: 0} >= 0")
show("isnull({tT, r0030, c0010})")
```
```text
not_applicable [None]
passed [True]
passed [True]
```
## Null propagates
Null propagates through arithmetic, comparisons and functions, except
`isnull` and `nvl(x, replacement)`. A record whose result is null is "not
evaluated", never a failure; a rule whose records are all null is
`not_applicable`.
`and`, `or` and `not` use three-valued (Kleene) logic: `null or true` is
true, `null and false` is false, anything else with null is null.
```python
show("{tT, r0030, c0010} > 0 or true")
show("{tT, r0030, c0010} > 0 and false")
```
```text
passed [True]
failed [False]
```
## Conditionals
`if` takes the `else` branch when the condition is null. Without `else`, a
Boolean `then` (filing indicators `{v_…}` and Boolean facts included) yields
true where the condition is false or null, so a guarded check passes when its
guard does not hold.
```python
show("if {tT, r0030, c0010} > 0 then false endif")
show("if {tT, r0030, c0010} > 0 then 1 else 2 endif = 2")
```
```text
passed [True]
passed [True]
```
The result has the condition's record structure. Branch keys the condition
lacks widen it, so each condition record applies to every branch record that
shares its keys. A condition with a `default` also covers the key values only
a branch has ([Records and keys](records-and-keys.md#defaults-fill-in-missing-records)).
## Errors stay in their record
Division by zero is a run-time error, as the specification requires. Like a
type error (a non-Boolean `if` condition included), it fails only its own
record: the rule is an error only if that record reaches the result. A guard
therefore works:
```python
show("{tT, r0010, c0010} / {tT, r0030, c0010, default: 0} > 1")
show(
"if {tT, r0030, c0010, default: 0} != 0 then {tT, r0010, c0010} / {tT, r0030, c0010, default: 0} > 1 endif"
)
```
```text
error division by zero
passed [True]
```
## Aggregates
Aggregates over no records are null, except `count`, which is 0. `count`
counts non-null values; the others skip nulls.
```python
show("count({tT, r0030, c0010}) = 0")
show("sum({tT, r0030, c0010}) = 0")
```
```text
passed [True]
not_applicable [None]
```
## Preconditions
A precondition makes the rule `not_applicable` unless it yields at least one
true record; null and false records do not count. See
[Preconditions and results](results.md).
# Interval arithmetic
A fact reported as 1,204 with XBRL `decimals="0"` is a rounded amount: the
true value lies anywhere from 1,203.5 to 1,204.5. A rule that adds such
amounts must allow for the rounding, or it fails reports that are correct.
EBA validation rules therefore use interval arithmetic, following the
Eurofiling interval arithmetic functions (`iaf:*`, version 1.1.1).
## Numbers with a tolerance
A `Number` is a decimal value with a symmetric
tolerance. `Number.of(value, decimals)` derives the tolerance from XBRL
`decimals`: `0.5 · 10^-decimals`. `INF` (in any case) or `None` means exact;
constants in rules and selection defaults are exact.
```python
from dpmxl import Number
print(Number.of(1204, 0))
print(Number.of(1204000, -3).tolerance, Number.of("0.0498", 4).tolerance)
print(Number.of(5, "INF"))
```
```text
Number(value=Decimal('1204'), tolerance=Decimal('0.5'))
5E+2 0.00005
Number(value=Decimal('5'), tolerance=Decimal('0'))
```
## Arithmetic
| Operation | Tolerance of the result |
| --- | --- |
| `a + b`, `a - b` | `δa + δb` |
| `a * b` | `|a·δb| + |b·δa| + δa·δb` |
| `a / b` | The largest deviation of the four endpoint quotients from `a / b` (skipping a zero divisor) |
| `-a`, `abs(a)` | `δa` |
| `min`, `max` | The chosen value's tolerance |
```python
a, b = Number.of(10, 0), Number.of("7.0", 1)
print(a + b)
print(a * b)
```
```text
Number(value=Decimal('17.0'), tolerance=Decimal('0.55'))
Number(value=Decimal('70.0'), tolerance=Decimal('4.025'))
```
Arithmetic keeps 34 significant digits (IEEE decimal128) in any thread
decimal context; the caller's `decimal` context has no effect.
## Comparisons
Two numbers compare as intervals:
| Comparison | Holds when |
| --- | --- |
| `a = b` | `|a − b| ≤ δa + δb` (the intervals overlap) |
| `a != b` | the intervals do not overlap |
| `a <= b` | `a − b ≤ δa + δb` |
| `a >= b` | `a ≥ b − (δa + δb)` |
| `a < b`, `a > b` | likewise, strictly |
```python
from dpmxl.values import compare
print(compare("=", Number.of(10, 0), Number.of("10.4")))
print(compare("=", Number.of(10, 0), Number.of("10.6")))
```
```text
True
False
```
## Switching it on and off
Interval arithmetic applies to selections with `interval: true`:
`{tC_01.00, r0010, interval: true}`, usually set once in a `with` context.
With `interval: false`, or without the option, facts are compared exactly
as reported. EBA rules state the option explicitly. Each operator and
function in the [language reference](../reference/language.md#interval)
says what it does with a tolerance.
## The gap in explanations
When a comparison fails despite the tolerance, [`explain`](../guides/explain.md)
reports each side's interval and the *gap*: how far apart the closest values
within those intervals still are. In
```text
1,204 should equal 1,211 (1,180 + 31): difference 7, at least 5.5 given the reported precision
```
the left side is 1,204 ± 0.5 and the right side 1,211 ± 1, so the difference
of 7 shrinks to at least 5.5. The gap is rounded down, so "at least" never
overstates it.
## Where the specification is ambiguous
The specification is ambiguous about the radius formulas for products and
quotients; dpm-xl uses the ones above, which avoid false failures. The
columnar evaluator uses `Decimal(38, 12)`; see
[Evaluate large reports](../guides/columnar.md#precision).
# Preconditions and results
`evaluate_rule(expression, provider, *, precondition=None)`
never raises for a bad rule or bad data. It returns a
`RuleResult` whose status is one of five:
| Status | When |
| --- | --- |
| `passed` | At least one record is true and none is false. |
| `failed` | At least one record is false. |
| `not_applicable` | The precondition is not met, or no record could be evaluated (all null). |
| `error` | A syntax error, an unknown table, a type mismatch, a rule that yields values instead of Booleans, or a run-time error (such as division by zero) in a record that reaches the result. |
| `unsupported` | The rule needs something the provider does not offer (a `time_shift` report, a `{o_…}` operation or a `{g_…}` table group), or a function dpm-xl does not know. |
```python
from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule
provider = MemoryProvider(
[TableLayout("T", rows=("0010",), columns=("0010",))],
{("T", "0010", "0010", None, ()): Number.of(10, 0)},
)
for expression, precondition in [
("{tT, r0010, c0010} > 0", "{v_T}"),
("{tT, r0010, c0010} > 0", "{v_X}"),
("{tT, r0010, c0010} >", None),
("{tX, r0010} > 0", None),
('{tT, r0010, c0010} > "a"', None),
("{tT, r0010, c0010} + 1", None),
("{o_X} > 0", None),
("foo({tT, r0010, c0010}) > 0", None),
]:
result = evaluate_rule(expression, provider, precondition=precondition)
print(f"{result.status:15} {result.message}")
```
```text
passed None
not_applicable precondition not met
error expression ends where an operand is expected at position 20
error unknown table X
error cannot compare number with string using '>'
error rule yields number values, not booleans
unsupported operation X needs a provider that resolves operations
unsupported unknown function 'foo'
```
`message` says why the status is `error`, `unsupported` or
`not_applicable`; its wording may improve in minor releases, the statuses
may not.
## Preconditions
A precondition is an expression, usually filing-indicator tests such as
`{v_C_01.00}` or `{v_C_01.00} and {v_C_02.00}`. The rule applies only when the
precondition yields at least one true record; otherwise the result is
`not_applicable` without evaluating the rule. A precondition is evaluated on
all records, also when `only_keys` restricts the rule.
`{v_TABLE}` asks the provider's `table_reported(table)`. `MemoryProvider`
answers from its `reported` codes (by default every table with a value) and
from each layout's `template`.
## What a result holds
| Field | Content |
| --- | --- |
| `status`, `message` | As above. |
| `outcomes` | One `Outcome` per record: `key` (such as `{'c': '0020'}`), `passed` (`True`, `False`, or `None` when not evaluated) and `points`, the distinct cells it was computed from. |
| `failures` | The outcomes whose `passed` is `False`. |
| `counts` | `{'passed': n, 'failed': n, 'null': n}`. |
| `evaluation` | The trace `explain` reads; `None` for syntax and provider errors. |
| `assignments` | Results of persistent assignments (`{target} <- value`) by target text, such as `{tT, r0010, c0010}` ([Statements](statements.md)). |
| `notes` | Remarks the evaluator made along the way. |
`DataPoint.ref` in `outcome.points` is whatever identity your provider gave
the cell, so a user interface can highlight it
([Explain failures in a UI](../guides/explain.md)).
## The CLI
`dpm-xl eval` prints the same statuses and exits with 0 when every rule
passed or did not apply, 1 when a rule failed and 3 when a rule is an error
or unsupported ([CLI reference](../reference/cli.md)).
# Statements and assignments
A rule is usually one expression. DPM-XL also allows a script: statements
separated by `;`, with two kinds of assignment.
| Statement | Meaning |
| --- | --- |
| `expression` | An expression, with an optional `with` context. |
| `name := expression` | A temporary assignment: `name` holds the expression's records for later statements. |
| `{target} <- expression` | A persistent assignment: the value is computed and returned in `RuleResult.assignments`. Targets are cell selections such as `{tT, r0010, c0010}` or parameters such as `{vX}`. |
Statements run in order; the **last** one gives the rule's records and
verdict. The last statement needs no `;`, and a trailing `;` is allowed.
```python
from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule
provider = MemoryProvider(
[TableLayout("T", rows=("0010", "0020"), columns=("0010", "0020"))],
{
("T", "0010", "0010", None, ()): Number.of(10, 0),
("T", "0020", "0010", None, ()): Number.of(4, 0),
("T", "0010", "0020", None, ()): Number.of(1, 0),
("T", "0020", "0020", None, ()): Number.of(4, 0),
},
)
script = "d := with {tT, c*}: {r0010} - {r0020}; d > 0"
result = evaluate_rule(script, provider)
print(
result.status, [(dict(outcome.key), outcome.passed) for outcome in result.outcomes]
)
```
```text
failed [({'c': '0010'}, True), ({'c': '0020'}, False)]
```
A temporary keeps the records of its expression, keys included. In later
statements its name wins over a report parameter of that name and loses to a
key component inside a clause. The rule's context, temporaries and
assignments do not reach operations referenced with `{o_…}`, which are
evaluated on their own.
## Persistent assignments
```python
result = evaluate_rule(
"{tT, r0010, c0010} <- {tT, r0020, c0010} * 2; {tT, r0010, c0010} = 10", provider
)
print(result.status)
for target, recordset in result.assignments.items():
print(target, [record.value for record in recordset.records.values()])
```
```text
passed
{tT, r0010, c0010} [Number(value=Decimal('8'), tolerance=Decimal('0'))]
```
Nothing is written to the report: the last statement still reads the
reported 10. The assignment's value is for the caller, keyed by the target's
normalized DPM-XL text.
## Parsing scripts
`parse_program` parses a script into a
`Program`; `parse` accepts one expression (with an
optional trailing `;`) and says so when given a script. `evaluate_rule`,
the columnar fallback and `dpm-xl check` accept scripts.
Outside clauses `<-` is the assignment, so compare with a negated operand as
`a < -b`. Inside `[…]` clauses, as in the grammar, `<-` reads `< -`.
The columnar evaluator hands every script with assignments to the reference
evaluator. The language reference gives the syntax of
[programs and statements](../reference/language.md#statements).
# Sets
Sets hold distinct non-null values in first-seen order: numbers compared
exactly, items ignoring the `eba_` prefix. They are written as literals
(`{1, 2}`, `{[eba_CU:EUR], [eba_CU:USD]}`, the empty set `{}`), come from
set parameters (`{p_Allowed, set-item}`), or are built with `set_of(x)` and
the set operators `union`, `intersect`, `setdiff` and `symdiff`. In Python a
set is a `ValueSet`.
```python
from dpmxl import Item, MemoryProvider, Number, TableLayout, ValueSet, evaluate_rule
currencies = ValueSet.of([Item.parse("eba_CU:EUR"), Item.parse("eba_CU:USD")])
provider = MemoryProvider(
[TableLayout("O", columns=("0010", "0020"), open_keys=("qEBB",))],
{
("O", None, "0010", None, (("qEBB", "A"),)): Item.parse("eba_CU:EUR"),
("O", None, "0010", None, (("qEBB", "B"),)): Item.parse("eba_CU:JPY"),
("O", None, "0020", None, (("qEBB", "A"),)): Number.of(2, 0),
("O", None, "0020", None, (("qEBB", "B"),)): Number.of(3, 0),
},
parameters={"Allowed": currencies},
)
def show(expression):
result = evaluate_rule(expression, provider)
print(
result.status, result.message or [outcome.passed for outcome in result.outcomes]
)
print(ValueSet.of([1, 2, 2, None]))
```
```text
ValueSet(members=(1, 2))
```
## Membership with `in`
`x in s` tests each record of `x` against the set:
```python
show("{tO, c0010} in {[eba_CU:EUR], [eba_CU:USD]}")
show("{tO, c0010} in {p_Allowed, set-item}")
show("{tO, c0010} in union({p_Allowed, set-item}, {[CU:JPY]})")
```
```text
failed [True, False]
failed [True, False]
passed [True, True]
```
An operand of `in` or of a set operator that is not a set stands for the set
of its non-null values, so `{c0010} in {tT, r*, c0020}` tests against the
values of a column, and `{tO, c0020} in setdiff({1, 2, 3}, {tO, c0020})`
against what the column does not hold:
```python
show("{tO, c0020} in setdiff({1, 2, 3}, {tO, c0020})")
```
```text
failed [False, False]
```
## Missing sets
A single null, such as a set parameter the report does not have, is a null
set, which makes `in` and the set operators null. A `default: {…}` on the
parameter supplies a set instead. `set_of(null)` is `{}`.
```python
show("{tO, c0010} in {p_Missing, set-item}")
show("{tO, c0010} in {p_Missing, set-item, default: {[eba_CU:JPY]}}")
```
```text
not_applicable [None, None]
failed [False, True]
```
## Restrictions
Sets cannot be compared with `=` or the other comparison operators, and
cannot be a rule's result. A set `default` on a cell selection is an error.
The columnar evaluator handles `in` with a set literal; it hands every other
set operand (parameters, selections, set operators) to the reference
evaluator.
The language reference gives the syntax and examples of
[set literals](../reference/language.md#set-literals), [`in`](../reference/language.md#in) and the
[set functions](../reference/language.md#set-functions).
# Aggregates and analytic functions
## Aggregates
`sum`, `count`, `avg`, `min`, `max` and `median` (also spelled `min_aggr`
and `max_aggr`, as in the specification) reduce an operand's records to one
value, or to one value per group with `group by`:
```dpm-xl
sum({tC_27.00, c0010}) <= {tC_01.00, r0010, c0010}
sum({tC_27.00, c0010} group by qCPI) >= 0
```
- Over no records they are null, except `count`, which is 0.
- `count` counts non-null values; the others skip nulls.
- `max` and `min` also order dates, text and items by their text (ISO dates
in date order, items by category and code), as presence checks expect.
Booleans, and numbers mixed with other values in one group, are an error.
- With several operands, `min(a, b)` and `max(a, b)` are scalar functions
that pick per record.
## Analytic functions
An analytic function keeps the operand's records: each record gets the
aggregate of the records in its *window* (or its rank) within its
*partition*.
```dpm-xl
with {tC_08.02}: rank({c0010} over (partition by qEEA order by qEEA desc)) > 0
with {tC_08.02}: sum({c0010} over (order by qEEA data points between unbounded preceding and current data point)) > 0
```
The examples below use an open table keyed by entity `qEBB` and date `qDAT`:
```python
from datetime import date
from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule
facts = [
("A", date(2026, 1, 31), 5),
("A", date(2026, 2, 28), 7),
("A", date(2026, 3, 31), 3),
("B", date(2026, 1, 31), -1),
("B", date(2026, 2, 28), 4),
("B", date(2026, 3, 31), 4),
]
provider = MemoryProvider(
[TableLayout("O", columns=("0010",), open_keys=("qEBB", "qDAT"))],
{
("O", None, "0010", None, (("qEBB", entity), ("qDAT", day))): Number.of(
amount, 0
)
for entity, day, amount in facts
},
)
def show(expression):
result = evaluate_rule(expression, provider)
for outcome in result.outcomes:
print(outcome.key["qEBB"], outcome.key["qDAT"], outcome.passed)
```
A running total per entity, in date order:
```python
show("with {tO}: sum({c0010} over (partition by qEBB order by qDAT)) > 0")
```
```text
A 2026-01-31 True
A 2026-02-28 True
A 2026-03-31 True
B 2026-01-31 False
B 2026-02-28 True
B 2026-03-31 True
```
The average of each month and the month before:
```python
show(
"with {tO}: avg({c0010} over (partition by qEBB order by qDAT data points between 1 preceding and current data point)) >= 3"
)
```
```text
A 2026-01-31 True
A 2026-02-28 True
A 2026-03-31 True
B 2026-01-31 False
B 2026-02-28 False
B 2026-03-31 True
```
## Partitions, order and windows
- **Partition and order keys** must be keys of the operand. Without
`partition by` the whole operand is one partition.
- **`rank`** needs `order by` and ranks by those keys; ties share the lowest
rank and leave gaps (1, 1, 3).
- **Default window.** Without a window clause the window is the whole
partition, or, with `order by`, every record up to the current one and its
*peers*, the records with the same values for every order key (`range
between unbounded preceding and current data point`).
- **`data points` windows** count records: `between 2 preceding and current
data point` is the current record and the two before it.
- **`range` windows** with an offset need a single number or date order key:
`3 preceding` is 3 less (3 days earlier for a date), `3 M preceding` three
calendar months earlier. A period on a number key is an error.
- **Bounds** are `unbounded preceding`, `unbounded following`, `current data
point`, and `n [A|S|Q|M|W|D] preceding|following`. `preceding` follows the
sort direction.
Analytic calls parse to `Analytic` nodes with a
`Window`, not to `Call`s, and `rank` is not in
`evaluator.AGGREGATES`. The columnar evaluator hands analytic functions to the
reference evaluator.
The language reference gives the syntax, errors and examples of each
[aggregate](../reference/language.md#aggregates) and [analytic function](../reference/language.md#analytic-functions).
# Time
## Dates
Date literals are written `#2026-12-31#`, with an optional time
(`#2026-12-31T10:00:00#`). A date equals a date-time at midnight. Report
parameters such as `refPeriod` and date key components hold dates; in Python
they are `datetime.date`.
| Function | Result |
| --- | --- |
| `year(d)`, `month(d)`, `day(d)` | The parts of a date |
| `quarter(d)`, `semester(d)` | 1–4 and 1–2 |
| `week(d)` | The ISO 8601 week |
| `date(y, m, d)` | A date; an impossible day such as `date(2026, 2, 30)` is an error |
```dpm-xl
year({p_refPeriod, date}) = 2026
date(2026, 12, 31) = #2026-12-31#
```
`max` and `min` order dates in date order.
## `time_shift`
`time_shift(x, period, count[, key])` presents `x` as it was `count` periods
earlier (later when negative), as in the EBA introduction to DPM-XL.
`period` is `A`, `S`, `Q`, `M`, `W` or `D`.
- When `key` is not a key component of `x` (`refPeriod`, or no key), `x` is
evaluated on the report the provider's `earlier_report(period, count)`
returns. Without that method, or when it returns `None`, the rule is
`unsupported`.
- When `key` is a date key component of `x`, its values move `count` periods
later instead, within the same report.
Periods move dates by calendar months (12, 6, 3 or 1; a month's last day
stays the last day) or by 7 or 1 days. A `time_shift` that would merge two
records onto one date key is an error.
## `annualise`
`annualise(x, fiscalYearEnd, key)` scales a year-to-date amount to a year:
`x · 12 / months`, where `months` counts the months from the fiscal year
that ends in month `fiscalYearEnd` up to and including the month of the date
in `key` (a date key component, else the report parameter of that name). It
treats that month as complete, so it is meant for period-end dates.
[Compare with earlier reports](../guides/time-shift.md) shows both functions
with runnable examples, and the language reference gives the syntax and
errors of [`time_shift`](../reference/language.md#time_shift), [`annualise`](../reference/language.md#annualise) and
the [date functions](../reference/language.md#date-functions).
# Parameters, operations and table groups
Besides table cells, a rule can read report-level data and other rules'
results. Each kind of reference goes to a provider method
([Connect your data](../guides/provider.md)).
| Reference | Reads | Provider method |
| --- | --- | --- |
| `{v_C_01.00}` | Whether table or template `C_01.00` was reported (its filing indicator) | `table_reported(table)` |
| `{vName}` | Report parameter `Name`, as given | `parameter(name)` |
| `{p_Name, type[, default: v]}` | Report parameter `Name`, checked against a type | `parameter(name)` |
| `{o_Code[, r…, c…, default: v, interval: b]}` | The result of operation `Code` | `operation(code)` (optional) |
| `{g_Group, r…, c…, …}` | The same cells of every table in a group | `table_group(code)` (optional) |
## Prefixes
A `_` after the `t`, `g`, `o` and `p` prefixes is cosmetic, as the DPM-XL
grammar says: `{t_T1}` is `{tT1}`. After `v` it is not: `{v_X}` tests a
filing indicator, `{vX}` reads a parameter. A backtick escapes a code that
is not a plain name: `` {t`C 01.00`} ``.
## Typed parameters
`{p_Name, type}` reads `parameter(Name)` and checks its type. The types are
`number`, `integer`, `string`, `date`, `boolean`, `item` and the set types
`set-number`, `set-string`, `set-date`, `set-item` and so on. Python numbers
become `Number`, text becomes an item or a date for those types, and a list
or set becomes a set ([Sets](sets.md)). A missing value takes the `default`,
or is null.
```python
from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule
provider = MemoryProvider(
[TableLayout("T", rows=("0010",), columns=("0010",))],
{("T", "0010", "0010", None, ()): Number.of(10, 0)},
parameters={"refPeriod": "2026-12-31", "Limit": 5, "Ccy": "eba_CU:EUR"},
)
for expression in (
"{p_refPeriod, date} = #2026-12-31#",
"{p_Limit, number} < {tT, r0010, c0010}",
"{p_Ccy, item} = [eba_CU:EUR]",
"{p_Missing, number, default: 3} = 3",
"{p_Missing, number} = 3",
"{p_Limit, date} = #2026-01-01#",
):
result = evaluate_rule(expression, provider)
print(f"{result.status:15} {result.message or ''}")
```
```text
passed
passed
passed
passed
not_applicable
error parameter Limit is not a date
```
## Operation references
`{o_Code, …}` evaluates the expression the provider's `operation(code)`
returns, on its own: the referring rule's context, temporaries and
assignments neither reach it nor receive its own. On the result:
- axes keep the matching `r`/`c`/`s` records, dropping a key selected by one
code;
- `default` fills nulls;
- `interval: false` makes numbers exact.
An operation that refers to itself, directly or through others, is an error.
Without the provider method, the rule is `unsupported`.
## Table groups
`{g_Group, …}` selects from every table the provider's `table_group(code)`
lists. With several tables the records are keyed by the table code `t`, and
the tables must share their keys. Without the provider method, the rule is
`unsupported`.
In static analysis, `references` reports
group and operation reads with `kind` and `code`, and
`tables(node, provider)` counts a group read as
reading every table of the group ([Analyse rules statically](../guides/analyse.md)).
The language reference gives the syntax of every
[reference](../reference/language.md#references) and [parameter type](../reference/language.md#typed-parameter).
# Reference
- [Quick reference](quick-reference.md): DPM-XL on one page, with the
complete grammar.
- [Language](language.md): every DPM-XL construct with its syntax, types,
null and interval behaviour, errors, tested examples and the node it
parses to, and where dpm-xl differs from the grammar.
- [CLI](cli.md): `dpm-xl check` and `dpm-xl eval`, every option, exit status
and the JSON output schema.
- [Data formats](data-formats.md): rule files and data files.
- [Python API](api/index.md): every public module, generated from the
docstrings.
# Quick reference
DPM-XL at a glance. Each entry links to the
[language reference](language.md), which gives the exact types, null
behaviour, errors and examples.
```dpm-xl
with {tC_01.00, c*, default: 0, interval: true}: {r0010} = {r0020} + {r0030}
if {v_C_27.00} then sum({tC_27.00, c0010} group by qCPI) >= 0 endif
{tC_27.00, c0020}[where qCPI <> "LEI-A"] in {p_Allowed, set-item}
total := sum({tC_27.00, c0010}); total <= {tC_01.00, r0010, c0010}
```
## Statements
| Syntax | Meaning |
| --- | --- |
| `expression` | A rule: one expression, evaluated per record ([Programs and statements](language.md#statements)) |
| `s1; s2; s3` | A script; the last statement gives the result |
| `name := e` | [Temporary assignment](language.md#temporary-assignment) for later statements |
| `{tT, r0010, c0010} <- e`, `{vX} <- e` | [Persistent assignment](language.md#persistent-assignment), returned in `RuleResult.assignments` |
| `with {tT, c*, default: 0}: e` | [Context](language.md#with) for the selections of `e` |
| `with {tO}[where qEBB = "A"]: e` | Context with a `where` on every selection of `e` |
## Selections and references
| Syntax | Meaning |
| --- | --- |
| `{tC_01.00, r0010, c0010}` | [Cells](language.md#selections) of table `C_01.00`; `t_C_01.00` and `` t`C 01.00` `` also work |
| `r0010`, `r0010-0050`, `(r0010, r0030-0050)`, `r*` | [Rows](language.md#axes): one, a range, a list, all; likewise `c` and `s` |
| `default: 0` | [Value for missing facts](language.md#default) and for keys another operand has |
| `interval: true` | [Interval arithmetic](language.md#interval) on reported precision |
| `{v_C_01.00}` | [Filing indicator](language.md#filing-indicator): was the table reported |
| `{vrefPeriod}`, `refPeriod` | [Report parameter](language.md#report-parameter), as given |
| `{p_Limit, number, default: 5}` | [Typed parameter](language.md#typed-parameter); types `number integer string date boolean item` and `set-…` of each |
| `{o_v1234, r0010}` | [Operation](language.md#operation) result |
| `{g_G01, r0010, c0010}` | [Table group](language.md#table-group), keyed by table `t` |
## Literals
| Syntax | Type |
| --- | --- |
| `12`, `-3.5`, `.5`, `1e6`, `4.5%` | [number](language.md#numbers) (`4.5%` is 0.045) |
| `"text"`, `'text'`, `''` | [text](language.md#strings), no escapes |
| `#2026-12-31#`, `#2026-12-31T10:00:00#` | [date](language.md#dates) |
| `true`, `false`, `null` | [boolean, null](language.md#boolean-and-null) |
| `[eba_CU:EUR]`, `[qEBB]` | [item](language.md#items) |
| `{1, 2}`, `{[CU:EUR], [CU:USD]}`, `{}` | [set](language.md#set-literals) |
| `` `to` ``, `` `a b` `` | [escaped name](language.md#names) |
## Operators
Loosest first; see [Precedence and associativity](language.md#precedence).
| Operators | Meaning |
| --- | --- |
| `or`, `xor` | [Logical](language.md#logical), three-valued |
| `and` | [Logical](language.md#logical), three-valued |
| `not` | [Negation](language.md#not); `not a = b` is `not (a = b)` |
| `=`, `!=`, `<>`, `<`, `<=`, `>`, `>=` | [Comparison](language.md#comparison), not chained |
| `in`, `not in` | [Membership](language.md#in) in a set |
| `&` | [Text concatenation](language.md#concatenation) |
| `+`, `-` | [Addition, subtraction](language.md#arithmetic) |
| `*`, `/` | [Multiplication, division](language.md#arithmetic) |
| unary `-`, `+` | [Sign](language.md#unary) |
| `n%`, `[…]` | [Percentage](language.md#percent), [clauses](language.md#clauses) |
| `if c then a else b endif` | [Conditional](language.md#if); `else` optional |
## Clauses
| Syntax | Meaning |
| --- | --- |
| `x[where qEBB = "A"]` | [Keep records](language.md#where) whose keys satisfy a condition |
| `x[get qEBB]` | [Replace values](language.md#get) by a key's values |
| `x[sub qEBB = "A"]` | [Fix a key](language.md#sub) and drop it |
| `x[rename qEBB to qX, …]` | [Rename keys](language.md#rename) |
Key names ignore case and the `q` prefix ([Key names](language.md#key-names)).
## Functions
| Function | Operands | Result |
| --- | --- | --- |
| [`abs(x)`](language.md#abs) | number | number |
| [`exp(x)`, `ln(x)`, `sqrt(x)`](language.md#exp) | number | number, null if undefined |
| [`power(x, y)`](language.md#power), [`log(x, base)`](language.md#log) | numbers | number, null if undefined |
| [`max(a, b, …)`, `min(a, b, …)`](language.md#max) | numbers, or dates, text, items | the largest, smallest per record |
| [`isnull(x)`](language.md#isnull) | any | boolean |
| [`nvl(x, r)`](language.md#nvl) | any | `x`, or `r` where `x` is null |
| [`len(s)`, `length(s)`](language.md#len) | text | number |
| [`substr(s, start, length)`](language.md#substr) | text, whole numbers | text |
| [`match(s, pattern)`](language.md#match) | text, regular expression | boolean |
| [`year`, `semester`, `quarter`, `month`, `week`, `day`](language.md#date-parts) | date | number |
| [`date(y, m, d)`](language.md#date) | whole numbers | date |
| [`time_shift(x, period, n, key)`](language.md#time_shift) | any, `A` `S` `Q` `M` `W` or `D`, whole number, optional name | `x` as `n` periods earlier |
| [`annualise(x, fiscalYearEnd, key)`](language.md#annualise) | number, month, name | number |
| [`set_of(x)`](language.md#set_of) | any | set |
| [`union`, `intersect`](language.md#union) | two or more sets | set |
| [`setdiff`, `symdiff`](language.md#setdiff) | two sets | set |
| [`filter(x, condition)`](language.md#filter) | any, boolean | the records of `x` where true |
## Aggregates and analytic functions
| Syntax | Meaning |
| --- | --- |
| [`sum`](language.md#sum), [`count`](language.md#count), [`avg`](language.md#avg), [`median`](language.md#median), [`max`, `min`, `max_aggr`, `min_aggr`](language.md#aggregate-max) | [Aggregates](language.md#aggregates): `sum(x)`, `sum(x group by k1, k2)` |
| `sum(x over (partition by k order by d desc data points between 1 preceding and current data point))` | [Analytic aggregate](language.md#analytic-functions) per record |
| `range between 3 M preceding and current data point` | Window by order key value; `unbounded preceding`, `unbounded following`, `n preceding`, `n following` |
| [`rank(x over (partition by k order by d))`](language.md#rank) | Rank per partition; ties share the lowest rank |
## Grammar
The complete syntax, in the [notation](language.md#notation) of the
language reference. Keywords match in any case; `group by` and the other
two-word keywords take any whitespace between their words.
```text
program ::= statement ( ";" statement )* ";"?
statement ::= name ":=" assigned | assigned
assigned ::= target "<-" context | context
target ::= selection | "{" "v" name "}" | "{" "v_" code "}"
context ::= "with" selection ( "[" "where" expression "]" )? ":" context
| expression
expression ::= disjunction
disjunction ::= conjunction ( ( "or" | "xor" ) conjunction )*
conjunction ::= negation ( "and" negation )*
negation ::= "not" negation | comparison
comparison ::= concatenation ( ( "=" | "!=" | "<>" | "<" | "<=" | ">" | ">=" ) concatenation
| "not"? "in" concatenation )?
concatenation ::= additive ( "&" additive )*
additive ::= multiplicative ( ( "+" | "-" ) multiplicative )*
multiplicative ::= unary ( ( "*" | "/" ) unary )*
unary ::= ( "+" | "-" ) unary | postfix
postfix ::= primary ( "%" | clause )*
primary ::= literal | set | selection | reference | name | call | analytic
| "if" expression "then" expression ( "else" expression )? "endif"
| "(" expression ")"
clause ::= "[" "where" expression "]"
| "[" "get" name "]"
| "[" "sub" name "=" additive "]"
| "[" "rename" name "to" name ( "," name "to" name )* "]"
selection ::= "{" component ( "," component )* "}"
component ::= source | axis | "default" ":" default
| "interval" ":" ( "true" | "false" )
source ::= ( "t" | "g" | "o" ) "_"? code
axis ::= part | "(" part ( "," part )* ")" | ( "r" | "c" | "s" ) "*"
part ::= ( "r" | "c" | "s" ) alnum+ ( "-" ( "r" | "c" | "s" )? alnum+ )?
default ::= ( "+" | "-" )? number "%"? | "(" default ")" | text | "true" | "false"
| "null" | item | date | set
reference ::= "{" "v_" code "}" | "{" "v" name "}"
| "{" "p" "_"? code "," type ( "," "default" ":" default )? "}"
type ::= ( "set-" )? ( "number" | "integer" | "string" | "date" | "boolean" | "item" )
call ::= name "(" ( expression ( "," expression )* )? ")"
| aggregate "(" expression "group" "by" name ( "," name )* ")"
| "time_shift" "(" expression "," period "," expression ( "," name )? ")"
analytic ::= ( aggregate | "rank" ) "(" expression "over" "(" partition? order? window? ")" ")"
aggregate ::= "sum" | "count" | "avg" | "median" | "max" | "min" | "max_aggr" | "min_aggr"
partition ::= "partition" "by" name ( "," name )*
order ::= "order" "by" name ( "asc" | "desc" )? ( "," name ( "asc" | "desc" )? )*
window ::= ( "data" "points" | "range" ) "between" bound "and" bound
bound ::= "unbounded" ( "preceding" | "following" ) | "current" "data" "point"
| digit+ period? ( "preceding" | "following" )
period ::= "A" | "S" | "Q" | "M" | "W" | "D"
set ::= "{" "}" | "{" expression ( "," expression )* "}"
literal ::= number "%"? | text | date | "true" | "false" | "null" | item
```
A set's first member is a literal, an item, a signed number or a
parenthesized number; braces that start otherwise are a selection or
reference. The lexical rules for `name`, `code`, `number`, `text`, `date`
and `item` are under [Lexical structure](language.md#lexical).
# Language reference
Every construct dpm-xl accepts, with its syntax, meaning, operand and result
types, null and interval behaviour, how it treats keyed records, the errors
it raises, and examples whose results the test suite computes. The
[quick reference](quick-reference.md) lists the same constructs on one
page; the [concept pages](../concepts/index.md) explain the semantics in
depth.
## How to read this page {#reading}
### Syntax notation {#notation}
Syntax is written in an EBNF in the style of the XML specification:
- `"with"` is the text itself; keywords match in any case.
- `expression` is another construct, defined on this page.
- `a b` is `a` followed by `b`; `a | b` is `a` or `b`.
- `a?` is an optional `a`; `a*` and `a+` repeat `a` zero or more, and one or
more times.
- `( … )` groups.
`name` is a [name](#names), `number` a [number literal](#numbers), `code` a
table code, `alnum` a letter or digit, `digit` a digit, and `literal` any
[literal](#literals).
### Examples and results {#examples}
The examples run against a small report, built by
[`examples/language_report.py`](https://github.com/luukhopman/dpm-xl/blob/main/examples/language_report.py):
| Table | Layout | Values |
| --- | --- | --- |
| `T` | rows `0010`–`0030`, columns `0010`, `0020` | c0010: 100, 60, 40; c0020: 5, -5 and nothing in r0030. Reported with `decimals` 0, so ±0.5 with `interval: true`. |
| `U` | rows `0010`–`0030`, column `0010` | 1, 2, 3, exact |
| `S` | row `0010`, column `0010`, sheets `0001`, `0002` | 7, 8 |
| `X` | row `0010`, columns `0010`–`0040` | `"LEI-A"`, `[eba_CU:EUR]`, `#2026-06-30#`, `true` |
| `O` | open, keys `qKEY`, `qDAT`; columns `0010`, `0020` | a, 2026-03-31: 10, EUR; b, 2026-06-30: 20, USD; c, 2026-09-30: 30, JPY |
| `P` | open, key `qKEY`; column `0010` | b: 2; d: 4 |
Report parameters: `refPeriod` = 2026-12-31, `Level` = `"solo"`, `Limit` =
5 and the set `Allowed` = `{[CU:EUR], [CU:USD]}`. Table group `G` holds `T`
and `U`; operation `v1` is `with {tT, c0010}: {r*} * 2`. In the report one
year earlier (`time_shift(…, A, 1)`), T r0010 c0010 is 50.
A result without keys is one value; a keyed result lists its records as
`key=value, …: value`, separated by `;`. Numbers are shown in full, with
`± tolerance` when they carry an [interval](../concepts/intervals.md)
tolerance; text is quoted and items are in brackets. `error: …` is an
[evaluation error](#errors) (in a keyed result, of that record only),
`unsupported: …` a reference the provider cannot resolve, `syntax error: …`
a [syntax error](#errors), and `no records` an empty result. To try an
expression of your own:
```bash
python examples/language_report.py "sum({tT, c0010, r*})"
```
### Data types {#data-types}
| Type | Values | Literal |
| --- | --- | --- |
| number | Decimals with 34 significant digits, and a tolerance from reported precision | [`12`, `4.5%`](#numbers) |
| text | Character strings | [`"text"`](#strings) |
| boolean | `true`, `false` | [`true`](#boolean-and-null) |
| date | Dates and date-times | [`#2026-12-31#`](#dates) |
| item | A member of a category, `category:code` | [`[eba_CU:EUR]`](#items) |
| set | Distinct non-null values | [`{1, 2}`](#set-literals) |
| null | No value | [`null`](#boolean-and-null) |
`integer` exists only as a [parameter type](#typed-parameter): a number
without a fraction. Each entry below states the types it takes; another
type is an evaluation error.
### Records, keys and nulls {#records}
An operand is a set of records: one value, or one value per key such as
`r=0010` or `qKEY=a`. Unless an entry says otherwise:
- Operators and functions of several operands **join** their operands on
the keys they share, and a key-less operand applies to every record
([Operators join on shared keys](../concepts/records-and-keys.md#operators-join-on-shared-keys)).
Keyed operands must have the same keys, or one operand's keys must
include every other's; anything else is the error `operands have
incompatible keys`.
- An operand with a `default` stands in for the records it lacks
([Defaults fill in missing records](../concepts/records-and-keys.md#defaults-fill-in-missing-records)).
- A **null** operand makes the result null
([Null logic](../concepts/null.md)). A record whose result is null is not
evaluated, never a failure.
- An **evaluation error** in one record, such as a division by zero, fails
only that record; it is the rule's error only if the record reaches the
result ([Errors stay in their record](../concepts/null.md#errors-stay-in-their-record)).
- A value passed on unchanged, such as an `if` branch, the value `nvl` or
`filter` keeps, or a record that `where`, `sub` or `rename` keeps, keeps
its interval tolerance.
- `where`, `sub`, `filter`, `rename`, a `with` context's `where` and a
`time_shift` of a date key keep their operand's `default`; the results of
`get`, aggregates and analytic functions have none and join on shared
keys only
([Defaults fill in missing records](../concepts/records-and-keys.md#defaults-fill-in-missing-records)).
### Errors {#errors}
| Kind | When | Rule status |
| --- | --- | --- |
| Syntax error | The text does not parse: `DpmXlSyntaxError`, with its position | `error` |
| Evaluation error | A wrong type, an unknown table, key or parameter, incompatible keys, a division by zero | `error` |
| Unsupported | The rule needs a provider method the provider lacks, or calls an unknown function | `unsupported` |
`diagnose` finds syntax errors, unknown
functions and wrong operand counts without data, and with table layouts
also unknown tables, coordinates and keys and operands of the wrong type.
Messages may be reworded in minor releases; statuses do not change
([Stability](../project/stability.md)).
## Lexical structure {#lexical}
### Whitespace and comments {#whitespace}
Spaces, tabs and line breaks separate tokens and are otherwise ignored
outside text literals. The words of `group by`, `partition by`, `order by`,
`data points` and `current data point` may be separated by any whitespace.
Expressions have no comments. Text [rule files](data-formats.md#rule-files) skip
lines that start with `#`, but that belongs to the file format.
### Case {#case}
Keywords, function names, `true`, `false`, `null` and parameter types match
in any case. Everything else is case-sensitive: the reference prefixes
`t g v o p` and the axis letters `r c s` are lower case, the period codes
`A S Q M W D` upper case, and names of temporaries match exactly.
[Key names](#key-names) in clauses ignore case.
| Expression | Result |
| --- | --- |
| `IF TRUE THEN Abs(-1) ENDIF` | `1` |
| `{TT, r0010, c0010}` | `syntax error: unknown selection component 'TT'` |
### Keywords and word boundaries {#word-boundaries}
A word is a letter or `_` followed by letters, digits, `_` and `.`, not
ending in `.`. A word is read whole, so a keyword needs whitespace or
punctuation between it and a neighbouring word: `ornot` is one name, not
`or not`, and `trueendif` is not `true endif`. A word may directly follow a
number (`2endif` reads `2 endif`), and symbols need no spaces
(`1<2and(true)`).
| Expression | Result |
| --- | --- |
| `true ornot false` | `syntax error: expected end of expression, found 'ornot'` |
| `true or(not false)` | `true` |
| `1<2and 2>1` | `true` |
`diagnose` suggests the missing space:
`did you mean 'or not'?`.
### Names {#names}
```text
name ::= word | "`" character* "`"
```
A name is a [word](#word-boundaries) that is not a reserved word, or any
text without backticks between backticks. Names stand for
[temporaries](#temporary-assignment), [report parameters](#bare-names) and,
inside [clauses](#clauses), [key components](#key-names), and parse to
`Name`. Backticks make a reserved word or text with
spaces a name: `` `to` ``, `` `a b` ``.
The keywords `and else endif get group by if in not or over rename sub then
to where with xor` are reserved everywhere. The
[window keywords](#analytic-functions) are reserved only inside `over (…)`,
`default` and `interval` only inside selections, function names only before
`(`, and period codes only where a period is expected; elsewhere they are
ordinary names.
| Expression | Result |
| --- | --- |
| `Level` | `"solo"` |
| `` `Level` `` | `"solo"` |
| `` `a.b` := 4; `a.b` + 1 `` | `5` |
| `range := 2; sum := 3; range * sum` | `6` |
| `to` | `syntax error: unexpected keyword 'to'` |
### Literals {#literals}
```text
literal ::= number | percentage | text | date | "true" | "false" | "null" | item
```
Literals are exact values. They parse to
`Literal_` with `kind` `number`, `string`, `date`,
`boolean`, `null` or `item` (whose value is an
`Item`). [Set literals](#set-literals) are described
with sets.
#### Numbers {#numbers}
```text
number ::= digit+ ( "." digit+ )? exponent? | "." digit+ exponent?
exponent ::= ( "e" | "E" ) ( "+" | "-" )? digit+
```
A decimal, read exactly: `0.1` is one tenth. A number literal has no sign;
`-3` is [unary minus](#unary) applied to `3`. A point needs a digit after
it, so `1.` is a syntax error. A number whose exponent is outside the
decimal128 range (about ±6144) is the syntax error `number out of range`.
#### Percentages {#percent}
```text
percentage ::= ( number | "(" number ")" ) "%"
```
`%` after a number literal, possibly in parentheses, divides it by 100.
After anything else, such as a name or `(1 + 1)`, it is the syntax error
`'%' must follow a number`, and so is `5%%`.
| Expression | Result |
| --- | --- |
| `12` | `12` |
| `-3.5` | `-3.5` |
| `.5` | `0.5` |
| `1.5e3` | `1500` |
| `0.1 + 0.2 = 0.3` | `true` |
| `4.5%` | `0.045` |
| `2 * 3 %` | `0.06` |
| `1.` | `syntax error: unexpected character '.'` |
| `(1)%` | `0.01` |
| `Limit%` | `syntax error: '%' must follow a number` |
#### Text {#strings}
```text
text ::= '"' character* '"' | "'" character* "'"
```
Text between double or single quotes, possibly empty, possibly spanning
lines. There are no escape sequences: a text cannot contain its own quote,
so write `"it's"` or `'say "yes"'`.
| Expression | Result |
| --- | --- |
| `"text"` | `"text"` |
| `"it's"` | `"it's"` |
| `''` | `""` |
#### Dates {#dates}
```text
date ::= "#" YYYY "-" MM "-" DD ( "T" hh ":" mm ":" ss )? "#"
```
A calendar date, or a date-time with a time part, with every digit written.
`#2026-1-1#` and impossible dates such as `#2026-02-30#` are the syntax
error `invalid date`. A date equals the date-time at its midnight.
| Expression | Result |
| --- | --- |
| `#2026-12-31#` | `2026-12-31` |
| `#2026-12-31T10:30:00#` | `2026-12-31T10:30:00` |
| `#2026-12-31# = #2026-12-31T00:00:00#` | `true` |
| `#2026-02-30#` | `syntax error: invalid date '2026-02-30'` |
#### Booleans and null {#boolean-and-null}
`true` and `false` are the Boolean values and `null` the absent value, in
any case. Null propagates through most operations
([Null logic](../concepts/null.md)).
| Expression | Result |
| --- | --- |
| `TRUE` | `true` |
| `null` | `null` |
| `isnull(null)` | `true` |
#### Items {#items}
```text
item ::= "[" category ":" code "]" | "[" code "]" | "[" "`" code "`" "]"
```
A member of a DPM category, such as a currency or a country. The category
starts with a letter and holds letters, digits, `_` and `-`; the code holds
letters, digits, `_`, `.` and `-` and ends in a letter or digit. Spaces may
pad the inside of the brackets, but not the `:`. The second form, a code
alone such as `[qEBB]` or `[ei86]`, is a property or legacy item code.
Items support only `=` and `!=`. Two items are equal when their codes match
and their categories match, ignoring an `eba_` prefix and a category
missing on either side: `[eba_CU:EUR]`, `[CU:EUR]` and `[EUR]` are equal.
Text compared with an item is read as an item.
A `[` followed by `where`, `get`, `sub` or `rename` opens a
[clause](#clauses) instead.
| Expression | Result |
| --- | --- |
| `[eba_CU:EUR]` | `[CU:EUR]` |
| `[eba_CU:EUR] = [EUR]` | `true` |
| `[CU:EUR] = "eba_CU:EUR"` | `true` |
| `[qEBB]` | `[qEBB]` |
| `[CU:EUR] < [CU:USD]` | `error: items do not support '<'` |
| `[eba_CU : EUR]` | `syntax error: invalid item '[eba_CU : EUR]'` |
### Limits {#limits}
An expression nests at most 50 levels deep (parentheses, signs, `not`,
`if`, calls, `with` bodies and chained clauses count); deeper is the
syntax error `expression too deeply nested`. Arithmetic keeps 34
significant digits.
## Programs and statements {#statements}
```text
program ::= statement ( ";" statement )* ";"?
statement ::= name ":=" assigned | assigned
assigned ::= target "<-" context | context
context ::= "with" selection ( "[" "where" expression "]" )? ":" context | expression
```
A rule is one expression. A script is statements separated by `;`, run in
order; the last statement gives the result and needs no `;`.
`parse` reads one expression (a trailing `;` is
allowed) and `parse_program` a script, which
`evaluate_rule` also accepts
([Statements and assignments](../concepts/statements.md)). Parses to
`Program`.
| Expression | Result |
| --- | --- |
| `x := 1; y := x + 1; y * 10;` | `20` |
| `1;;` | `syntax error: unexpected ';' where an operand is expected` |
### Temporary assignment `:=` {#temporary-assignment}
```text
name ":=" assigned
```
Evaluates the right side and binds its records, keys included, to `name`
for the statements that follow; the statement's result is that value. A
temporary hides a report parameter of the same name, and inside a clause a
key component of the same name hides it. Parses to
`Assignment`.
| Expression | Result |
| --- | --- |
| `d := {tT, c0010} - {tT, c0020}; d` | `r=0010: 95; r=0020: 65; r=0030: null` |
| `Level := 1; Level` | `1` |
| `x := 1; X` | `error: unknown key component or parameter 'X'` |
### Persistent assignment `<-` {#persistent-assignment}
```text
target ::= selection | "{" "v" name "}" | "{" "v_" code "}"
```
`target <- value` evaluates the value and returns it in
`RuleResult.assignments` under the target's
normalized text, such as `{tT, r0010, c0010}`. Nothing is written to the
report: later statements still read the reported values. The statement's
result is the value. Outside clauses `<-` is always this assignment, so
write a comparison with a negative operand as `a < -1`. A target that is
not a selection or `{v…}` reference is a syntax error. Parses to
`PersistentAssignment`.
| Expression | Result |
| --- | --- |
| `{tT, r0010, c0010} <- {tU, r0010, c0010} * 2` | `2` |
| `{tT, r0010, c0010} <- 5; {tT, r0010, c0010}` | `100` |
| `{vTotal} <- sum({tU})` | `6` |
### `with` {#with}
```text
"with" selection ( "[" "where" expression "]" )? ":" context
```
Sets a context for the [selections](#selections) of its body: each takes
the context's table, axes, `default` and `interval` where it does not write
its own. A nested `with` replaces the components it names. `with` may start
a statement, an assignment's value or another `with` body; elsewhere, such
as in parentheses, it is a syntax error.
The optional `where` applies, as a [`where` clause](#where), to each
selection of the body that has the keys it names. Its `and` terms that name
a key the selection lacks (neither one of its keys nor a temporary or
report parameter) leave that selection alone, and a key-less selection
takes none of it. A `where` or `sub` written on a selection replaces the
context condition's terms on the same key
([Clauses change keys and records](../concepts/records-and-keys.md#clauses-change-keys-and-records)).
Parses to `With`.
| Expression | Result |
| --- | --- |
| `with {tT, c0010}: {r0010} + {r0020}` | `160` |
| `with {tT, c0020, default: 0}: {r0030}` | `0` |
| `with {tT, c0020, default: 0}: {r0030, default: 1}` | `1` |
| `with {tT, interval: true}: {r0010, c0010}` | `100 ± 0.5` |
| `with {tT}: with {c0010}: {r0010}` | `100` |
| `with {tO, c0010}[where qKEY = "a"]: {tO}` | `qKEY=a, qDAT=2026-03-31: 10` |
| `with {tO, c0010}[where qKEY = "a"]: {c0010}[sub qKEY = "b"]` | `qDAT=2026-06-30: 20` |
| `with {tO}[where qKEY = "a"]: {c0010} + {tT, r0010, c0010}` | `qKEY=a, qDAT=2026-03-31: 110` |
| `with {tO}[where qDAT > #2026-05-01#]: {c0010} + {tP, c0010}` | `qKEY=b, qDAT=2026-06-30: 22` |
| `with {tO}[where qKEY = "a" and r = "0010"]: {c0010}` | `qKEY=a, qDAT=2026-03-31: 10` |
| `(with {tT}: {r0010, c0010})` | `syntax error: unexpected keyword 'with'` |
## Selections {#selections}
```text
selection ::= "{" component ( "," component )* "}"
component ::= source | axis | "default" ":" default | "interval" ":" ( "true" | "false" )
source ::= ( "t" | "g" | "o" ) "_"? code
code ::= ( letter | digit | "_" | "." | "-" )+ | "`" character* "`"
```
A selection reads cells. Its components are separated by commas, come in
any order and appear at most once each (one source, one selector per axis,
one `default`, one `interval`); a repeated one is a syntax error. The
source is written or comes from an enclosing [`with`](#with); without one
the selection is the evaluation error `selection … has no table`. Braces
that start with a literal, an item or a signed number are a
[set literal](#set-literals) instead, and `{v…}` and `{p…}` braces are
[references](#references). Parses to `Selection`
holding a `SelectionSpec`.
### Table `t` {#table}
`tCode` names a table, such as `tC_01.00`. A `_` after `t` is cosmetic
(`t_C_01.00` is `tC_01.00`), and backticks escape other codes:
`` t`C 01.00` ``. An unknown table is the evaluation error `unknown table
…`. The sources `g` and `o` select from a [table group](#table-group) or
an [operation](#operation).
### Rows, columns and sheets `r c s` {#axes}
```text
axis ::= part | "(" part ( "," part )* ")" | ( "r" | "c" | "s" ) "*"
part ::= ( "r" | "c" | "s" ) alnum+ ( "-" ( "r" | "c" | "s" )? alnum+ )?
```
`r`, `c` and `s` select rows, columns and sheets: one code (`r0010`), an
inclusive range (`r0010-0050`), a list of codes and ranges on one axis in
parentheses (`(c0010, c0030-0050)`), or every coordinate (`r*`). Codes hold
letters and digits; a range may repeat its axis letter (`r0010-r0050`). An
axis that is not written selects every coordinate on it.
Codes match as DPM numbers them: numeric codes as numbers padded to four
digits (`r10` is row `0010`), others ignoring case. A range is numeric when
both ends are numeric and compares text otherwise. A list cannot mix axes
or hold `*`. A coordinate the table lacks selects nothing.
**Keys.** Each axis that selects more than one coordinate is a key, `r`,
`c` or `s`, with the coordinate codes as text; open tables add their key
components, such as `qKEY`
([Selections yield records](../concepts/records-and-keys.md#selections-yield-records)).
| Expression | Result |
| --- | --- |
| `{tT, r0010, c0010}` | `100` |
| `{t_T, r10, c10}` | `100` |
| `{tT, c0010}` | `r=0010: 100; r=0020: 60; r=0030: 40` |
| `{tT, r0010-0020, c0020}` | `r=0010: 5; r=0020: -5` |
| `{tT, (r0010, r0030), c0010}` | `r=0010: 100; r=0030: 40` |
| `{tT, r0010, c*}` | `c=0010: 100; c=0020: 5` |
| `{tS}` | `s=0001: 7; s=0002: 8` |
| `{tS, s0002}` | `8` |
| `sum({tS, r0010, c0010, s*})` | `15` |
| `{tO, c0010}` | `qKEY=a, qDAT=2026-03-31: 10; qKEY=b, qDAT=2026-06-30: 20; qKEY=c, qDAT=2026-09-30: 30` |
| `{tT, r0010, r0020}` | `syntax error: duplicate rows selector` |
| `{tT, (r0010, c0010)}` | `syntax error: axis list mixes axes` |
| `{tZ, r0010}` | `error: unknown table Z` |
### `default` {#default}
```text
default ::= ( "+" | "-" )? number "%"? | "(" default ")" | text | "true" | "false" | "null" | item | date | set
```
The value of a selected cell without a fact; without `default` it is null.
A written `default`, `default: null` included, also stands in for records
that another operand has and the selection lacks
([Defaults fill in missing records](../concepts/records-and-keys.md#defaults-fill-in-missing-records)).
The value is a literal, not an expression: `default: 1 + 1` is a syntax
error. A set is a default only of a [set parameter](#typed-parameter); on a
selection it is an evaluation error.
| Expression | Result |
| --- | --- |
| `{tT, r*, c0020}` | `r=0010: 5; r=0020: -5; r=0030: null` |
| `{tT, r*, c0020, default: 0}` | `r=0010: 5; r=0020: -5; r=0030: 0` |
| `{tT, r0030, c0020, default: (-1)}` | `-1` |
| `{tO, c0010} + {tP, c0010}` | `qKEY=b, qDAT=2026-06-30: 22` |
| `{tO, c0010} + {tP, c0010, default: 0}` | `qKEY=a, qDAT=2026-03-31: 10; qKEY=b, qDAT=2026-06-30: 22; qKEY=c, qDAT=2026-09-30: 30` |
| `{tT, c0010, default: {1}}` | `error: a set default applies only to set parameters` |
### `interval` {#interval}
`interval: true` keeps each reported number's tolerance (from its XBRL
`decimals`), so that arithmetic and comparisons use
[interval arithmetic](../concepts/intervals.md); `interval: false`, or no
`interval`, makes the numbers exact. The value is `true` or `false`.
| Expression | Result |
| --- | --- |
| `{tT, r0010, c0010, interval: true}` | `100 ± 0.5` |
| `{tT, r0010, c0010, interval: true} = 100.4` | `true` |
| `{tT, r0010, c0010} = 100.4` | `false` |
| `{tT, r0010, c0010, interval: 1}` | `syntax error: interval must be true or false` |
## References {#references}
```text
reference ::= "{" "v_" code "}" | "{" "v" name "}"
| "{" "p" "_"? code "," type ( "," "default" ":" default )? "}"
```
Braces also read report-level data and other rules' results; each kind of
reference goes to a provider method
([Parameters, operations and table groups](../concepts/references.md)).
### Filing indicator `{v_…}` {#filing-indicator}
`{v_Code}` is `true` when table or template `Code` was reported, else
`false` (the provider's `table_reported`). The `_` matters: without it the
braces read a [report parameter](#report-parameter). Usually a
[precondition](../concepts/results.md#preconditions). Parses to
`TableReported`.
| Expression | Result |
| --- | --- |
| `{v_T}` | `true` |
| `{v_Z}` | `false` |
| `{v_T} and not {v_Z}` | `true` |
### Report parameter `{v…}` {#report-parameter}
`{vName}` is report parameter `Name` as the provider gives it, of any type;
null when it is missing. `Name` starts with a letter; backticks escape
other names: ``{v`a b`}``. Parses to `Parameter`.
| Expression | Result |
| --- | --- |
| `{vrefPeriod}` | `2026-12-31` |
| `{vLevel}` | `"solo"` |
| `{vMissing}` | `null` |
#### Bare names {#bare-names}
A [name](#names) reads a [temporary](#temporary-assignment) or, failing
that, the report parameter of that name; inside a clause a key component
comes first ([Key names](#key-names)). An unknown name is the evaluation
error `unknown key component or parameter`.
| Expression | Result |
| --- | --- |
| `refPeriod` | `2026-12-31` |
| `missing` | `error: unknown key component or parameter 'missing'` |
### Typed parameter `{p_…}` {#typed-parameter}
```text
type ::= "number" | "integer" | "string" | "date" | "boolean" | "item"
| "set-number" | "set-integer" | "set-string" | "set-date" | "set-boolean" | "set-item"
```
`{p_Name, type}` reads report parameter `Name` and checks its type; a value
of another type is the evaluation error `parameter … is not …`. Text is
read as a date or an item for those types, and `integer` takes whole
numbers. A missing parameter takes the `default`, or is null. A `set-…`
type takes a set of that member type, and its default may be a
[set literal](#set-literals). The `_` after `p` is cosmetic; the type comes
before the default. Parses to `TypedParameter`.
| Expression | Result |
| --- | --- |
| `{p_refPeriod, date}` | `2026-12-31` |
| `{p_Limit, number} + 1` | `6` |
| `{pLimit, integer}` | `5` |
| `{p_Level, string}` | `"solo"` |
| `{p_Allowed, set-item}` | `{[CU:EUR], [CU:USD]}` |
| `{p_Missing, boolean, default: true}` | `true` |
| `{p_Missing, item, default: [CU:EUR]}` | `[CU:EUR]` |
| `{p_Missing, set-number, default: {1, 2}}` | `{1, 2}` |
| `{p_Missing, set-integer}` | `null` |
| `{p_Missing, set-string, default: {"a"}}` | `{"a"}` |
| `{p_Missing, set-date, default: {#2026-01-01#}}` | `{2026-01-01}` |
| `{p_Missing, set-boolean, default: {true}}` | `{true}` |
| `{p_Limit, date}` | `error: parameter Limit is not a date` |
| `{p_Limit}` | `syntax error: expected ',', found '}'` |
### Operation `{o_…}` {#operation}
```text
"{" "o" "_"? code ( "," ( axis | "default" ":" default | "interval" ":" ( "true" | "false" ) ) )* "}"
```
Evaluates operation `Code`, the expression the provider's optional
`operation(code)` method returns, on its own: the referring rule's `with`
context, temporaries and assignments do not reach it. On its result, axes
keep the matching `r`, `c` and `s` records, `default` fills nulls and
`interval: false` makes numbers exact. An operation that refers to itself
is an evaluation error; without the provider method the rule is
unsupported. Parses to a `Selection` with `spec.operation`.
| Expression | Result |
| --- | --- |
| `{o_v1}` | `r=0010: 200; r=0020: 120; r=0030: 80` |
| `{o_v1, r0010}` | `200` |
| `{o_X}` | `unsupported: operation X needs a provider that resolves operations` |
### Table group `{g_…}` {#table-group}
`{g_Group, …}` selects from every table of a group, which the provider's
optional `table_group(code)` method lists. With several tables the records
gain the key `t`, the table code, and the tables must have the same keys
for the selection. Without the provider method the rule is unsupported.
Parses to a `Selection` with `spec.table_group`.
| Expression | Result |
| --- | --- |
| `{g_G, r0010, c0010}` | `t=T: 100; t=U: 1` |
| `{g_G, r0010}` | `error: the tables of group G have different keys` |
| `{g_X, r0010}` | `unsupported: table group X needs a provider that resolves table groups` |
## Operators {#operators}
### Precedence and associativity {#precedence}
From loosest to tightest binding:
| Level | Operators | Associativity | Entry |
| --- | --- | --- | --- |
| 1 | `;`, `:=`, `<-`, `with … :` | statements only | [Statements](#statements) |
| 2 | `or`, `xor` | left | [Logical operators](#logical) |
| 3 | `and` | left | [Logical operators](#logical) |
| 4 | `not` | prefix; nests (`not not x`) | [`not`](#not) |
| 5 | `=`, `!=`, `<>`, `<`, `<=`, `>`, `>=`, `in`, `not in` | none: one per operand | [Comparison](#comparison), [`in`](#in) |
| 6 | `&` | left | [Concatenation](#concatenation) |
| 7 | `+`, `-` | left | [Arithmetic](#arithmetic) |
| 8 | `*`, `/` | left | [Arithmetic](#arithmetic) |
| 9 | unary `+`, `-` | prefix; nests (`- -x`) | [Unary plus and minus](#unary) |
| 10 | `%` after a number literal, clauses `[…]` | postfix; chain left to right | [Percentages](#percent), [Clauses](#clauses) |
| 11 | literals, selections, references, names, calls, `if … endif`, `( … )` | | |
Without parentheses `not` cannot be an operand of a comparison, arithmetic
or a sign: `1 = not true` and `- not x` are syntax errors. Comparisons do
not chain: `a < b < c` is a syntax error.
| Expression | Result |
| --- | --- |
| `1 + 2 * 3` | `7` |
| `(1 + 2) * 3` | `9` |
| `10 - 4 - 3` | `3` |
| `12 / 4 / 3` | `1` |
| `-2 * -3` | `6` |
| `"a" & "b" = "ab"` | `true` |
| `1 + 1 in {2}` | `true` |
| `not 1 = 2` | `true` |
| `not true and false` | `false` |
| `true or false and false` | `true` |
| `true xor true or true` | `true` |
| `1 < 2 < 3` | `syntax error: expected end of expression, found '<'` |
| `1 = not true` | `syntax error: unexpected keyword 'not'` |
### Arithmetic `+ - * /` {#arithmetic}
```text
additive ::= multiplicative ( ( "+" | "-" ) multiplicative )*
multiplicative ::= unary ( ( "*" | "/" ) unary )*
```
Addition, subtraction, multiplication and division of numbers.
- **Types:** numbers in, a number out; exact decimal arithmetic with 34
significant digits.
- **Null:** null if either operand is null.
- **Intervals:** tolerances propagate
([Arithmetic](../concepts/intervals.md#arithmetic)).
- **Keys:** the operands [join](#records).
- **Errors:** `+ needs numbers, found …`; `division by zero` when the
divisor's value is zero.
- **Node:** `Binary`.
| Expression | Result |
| --- | --- |
| `{tT, r0020, c0010} + {tT, r0030, c0010}` | `100` |
| `{tT, c0010} - {tT, c0020}` | `r=0010: 95; r=0020: 65; r=0030: null` |
| `{tT, c0010, interval: true} * 2` | `r=0010: 200 ± 1; r=0020: 120 ± 1; r=0030: 80 ± 1` |
| `1 / 3` | `0.3333333333333333333333333333` |
| `1 + null` | `null` |
| `1 / 0` | `error: division by zero` |
| `{tT, c0010} / {tT, c0020}` | `r=0010: 20; r=0020: -12; r=0030: null` |
| `1 + "a"` | `error: + needs numbers, found string` |
| `{tT, r0010} + {tT, c0010}` | `error: operands have incompatible keys (r) and (c)` |
### Unary plus and minus {#unary}
```text
unary ::= ( "+" | "-" ) unary | postfix
```
`-x` negates a number and `+x` returns it, for each record, keeping its
tolerance. Null stays null; another type is the error `unary - needs
numbers`. Parses to `Unary`.
| Expression | Result |
| --- | --- |
| `-{tT, r0010, c*}` | `c=0010: -100; c=0020: -5` |
| `- -1` | `1` |
| `-"a"` | `error: unary - needs numbers, found string` |
### Concatenation `&` {#concatenation}
```text
concatenation ::= additive ( "&" additive )*
```
Joins two texts; null if either is null. Another type is the error `&
needs text, found …`. Binds looser than arithmetic and tighter than
comparisons. Parses to `Binary`.
| Expression | Result |
| --- | --- |
| `"LEI" & "-" & "A"` | `"LEI-A"` |
| `{tX, c0010} & "/2026"` | `"LEI-A/2026"` |
| `"a" & null` | `null` |
| `"a" & 1` | `error: & needs text, found number` |
### Comparison `= != <> < <= > >=` {#comparison}
```text
comparison ::= concatenation ( ( "=" | "!=" | "<>" | "<" | "<=" | ">" | ">=" ) concatenation )?
```
Compares two values and yields a Boolean. `<>` is `!=`; the node's `op` is
`!=`.
| Operands | Comparison |
| --- | --- |
| number, number | By value; with tolerances as intervals ([Comparisons](../concepts/intervals.md#comparisons)) |
| text, text | By character code |
| date, date | In date order; a date is the date-time at its midnight |
| boolean, boolean | `false` before `true`; the text `"true"`, `"false"`, `"1"` or `"0"` counts as a Boolean |
| item, item | `=` and `!=` only, as under [Items](#items); text counts as an item |
| date, text | As text |
- **Null:** null if either operand is null.
- **Keys:** the operands [join](#records).
- **Errors:** `cannot compare number with string using '='` and the like
for other pairs, sets included; `items do not support '<'`.
- **Node:** `Binary`.
| Expression | Result |
| --- | --- |
| `{tT, r0010, c0010} > {tT, r0020, c0010}` | `true` |
| `{tT, c0010} >= 50` | `r=0010: true; r=0020: true; r=0030: false` |
| `"a" < "b"` | `true` |
| `#2026-01-01# < #2026-01-02#` | `true` |
| `true > false` | `true` |
| `{tX, c0040} = "true"` | `true` |
| `{tX, c0020} = [CU:EUR]` | `true` |
| `1 != 2` | `true` |
| `1 <> 2` | `true` |
| `null = null` | `null` |
| `1 = "1"` | `error: cannot compare number with string using '='` |
| `{1} = {1}` | `error: cannot compare set with set using '='` |
### `in`, `not in` {#in}
```text
membership ::= concatenation "not"? "in" concatenation
```
`x in s` is `true` when `x` equals (`=`) a member of the set `s`, so `x`'s
interval tolerance applies; `x not in s` is `not (x in s)`. `s` is a
[set literal](#set-literals), a set parameter, a
[set function](#set-functions), or any other operand, which stands for the
set of its non-null values ([Sets](../concepts/sets.md)).
- **Null:** null when `x` is null, or when `s` is a single null, such as a
missing set parameter.
- **Keys:** applies to each record of `x`; the set is not joined.
- **Errors:** incompatible types, as for `=`.
- **Node:** `In`; `not in` is `Unary("not", In(…))`.
| Expression | Result |
| --- | --- |
| `{tU, c0010} in {1, 3}` | `r=0010: true; r=0020: false; r=0030: true` |
| `{tO, c0020} in {p_Allowed, set-item}` | `qKEY=a, qDAT=2026-03-31: true; qKEY=b, qDAT=2026-06-30: true; qKEY=c, qDAT=2026-09-30: false` |
| `{tO, c0020} not in {[CU:JPY]}` | `qKEY=a, qDAT=2026-03-31: true; qKEY=b, qDAT=2026-06-30: true; qKEY=c, qDAT=2026-09-30: false` |
| `60 in {tT, c0010}` | `true` |
| `{tT, r0010, c0010, interval: true} in {100.4}` | `true` |
| `null in {1}` | `null` |
| `1 in {p_Missing, set-number}` | `null` |
| `1 in {}` | `false` |
### `not` {#not}
```text
negation ::= "not" negation | comparison
```
Negates a Boolean, for each record; null stays null. Another type is the
error `not needs a boolean`. `not` binds looser than comparisons: `not a =
b` is `not (a = b)`. Parses to `Unary`.
| Expression | Result |
| --- | --- |
| `not {tT, c0010} > 50` | `r=0010: false; r=0020: false; r=0030: true` |
| `not null` | `null` |
| `not 1` | `error: not needs a boolean, found number` |
### Logical operators `and or xor` {#logical}
```text
disjunction ::= conjunction ( ( "or" | "xor" ) conjunction )*
conjunction ::= negation ( "and" negation )*
```
Conjunction, disjunction and exclusive or with three-valued (Kleene) logic:
`false and null` is `false` and `true or null` is `true`; otherwise a null
operand makes the result null, and `xor` with a null is always null. Both
operands are always evaluated.
- **Keys:** the operands [join](#records).
- **Errors:** `and needs booleans, found …`.
- **Node:** `Binary`.
| Expression | Result |
| --- | --- |
| `{tT, c0010} > 50 and {tT, c0020} > 0` | `r=0010: true; r=0020: false; r=0030: false` |
| `{v_T} or {v_Z}` | `true` |
| `true xor false` | `true` |
| `null and false` | `false` |
| `null or false` | `null` |
| `true xor null` | `null` |
| `1 and true` | `error: and needs booleans, found number` |
### `if` {#if}
```text
"if" expression "then" expression ( "else" expression )? "endif"
```
Chooses a branch by a Boolean condition; a null condition takes the `else`
branch. Without `else`, a false or null condition yields `true` when the
`then` branch is Boolean, so that a guarded check passes when its guard
does not hold, and null otherwise
([Conditionals](../concepts/null.md#conditionals)). `if … endif` is an
operand: it nests and combines with operators.
- **Keys:** a key-less condition picks a whole branch. A keyed condition
gives one record per condition record, from the matching branch record;
branch keys the condition lacks widen it. Defaults on the condition or a
branch fill in missing keys
([Defaults fill in missing records](../concepts/records-and-keys.md#defaults-fill-in-missing-records)).
- **Errors:** `if needs a boolean condition, found …`, in the record
concerned.
- **Node:** `If`.
| Expression | Result |
| --- | --- |
| `if {v_T} then {tT, r0010, c0010} > 0 endif` | `true` |
| `if {tT, c0010} > 50 then {tT, c0010} else 0 endif` | `r=0010: 100; r=0020: 60; r=0030: 0` |
| `if {tT, c0010} > 50 then {tT, c0020} > 0 endif` | `r=0010: true; r=0020: false; r=0030: true` |
| `if {tT, c0010} > 50 then {tT, c0020} endif` | `r=0010: 5; r=0020: -5; r=0030: null` |
| `if null then 1 else 2 endif` | `2` |
| `if false then 1 else if false then 2 else 3 endif endif + 1` | `4` |
| `if 1 then 2 endif` | `error: if needs a boolean condition, found number` |
## Clauses {#clauses}
```text
postfix ::= primary ( "%" | clause )*
clause ::= "[" "where" expression "]"
| "[" "get" name "]"
| "[" "sub" name "=" additive "]"
| "[" "rename" name "to" name ( "," name "to" name )* "]"
```
A clause follows any operand in brackets, usually a selection or a name.
Clauses chain left to right. Inside a clause `<-` reads `< -`, so
`[where qX<-1]` compares with -1.
### Key names {#key-names}
Names in clauses, `group by` and `over` resolve against the operand's keys:
exactly, else ignoring case and the DPM 2.0 `q` prefix, so `KEY`, `key` and
`qkey` all find `qKEY`. A name that matches no key, or several, is read as
a temporary or report parameter in `where` and `get`, and is an error in
`sub`, `rename`, `group by` and `over`. The keys `r`, `c` and `s` hold
coordinate codes as text (`r = "0010"`, not `r = 10`); open keys hold the
values the provider reports, such as text, dates and items.
### `where` {#where}
Keeps the records whose keys satisfy a condition. The condition is
evaluated for each record, with the record's key components as names, and
must yield one Boolean; records where it is false or null are dropped. It
tests keys, not values: to test values use [`filter`](#filter). The result
keeps the operand's `default`, which stands in for the keys other operands
bring ([Defaults fill in missing records](../concepts/records-and-keys.md#defaults-fill-in-missing-records)).
A key-less operand is the error `where needs a keyed operand`. Parses to
`Where`.
| Expression | Result |
| --- | --- |
| `{tT, c0010}[where r = "0010"]` | `r=0010: 100` |
| `{tT, c0010}[where r in {"0010", "0020"}]` | `r=0010: 100; r=0020: 60` |
| `{tO, c0010}[where qDAT > #2026-05-01#]` | `qKEY=b, qDAT=2026-06-30: 20; qKEY=c, qDAT=2026-09-30: 30` |
| `{tO, c0010}[where KEY <> "a" and qDAT < #2026-09-01#]` | `qKEY=b, qDAT=2026-06-30: 20` |
| `{tO, c0010}[where qDAT<-1]` | `error: cannot compare date with number using '<'` |
| `{tT, c0010}[where r = 10]` | `error: cannot compare string with number using '='` |
| `{tT, r0010, c0010}[where true]` | `error: where needs a keyed operand` |
| `{tO, c0010}[where {tO, c0010} > 10]` | `error: expected a single value, found records keyed by qKEY, qDAT` |
### `get` {#get}
Replaces each record's value by the value of one of its key components,
keeping the keys; records whose value is null are dropped, and the result
has no `default`. A name that is not a key reads the report parameter of
that name for every record. Parses to `Get`.
| Expression | Result |
| --- | --- |
| `{tO, c0010}[get qDAT]` | `qKEY=a, qDAT=2026-03-31: 2026-03-31; qKEY=b, qDAT=2026-06-30: 2026-06-30; qKEY=c, qDAT=2026-09-30: 2026-09-30` |
| `{tT, c0020}[get r]` | `r=0010: "0010"; r=0020: "0020"` |
| `{tT, c0010}[get foo]` | `error: unknown key component or parameter 'foo'` |
### `sub` {#sub}
Keeps the records whose key component equals a value, compared as with
`=`, and removes that key. The value is an additive expression (a literal,
a parameter, an item or arithmetic; parenthesize anything else) with one
value. The result keeps the operand's `default`, as for [`where`](#where).
Parses to `Sub`.
| Expression | Result |
| --- | --- |
| `{tO, c0010}[sub qKEY = "a"]` | `qDAT=2026-03-31: 10` |
| `{tO, c0010}[sub qKEY = "a"][sub qDAT = #2026-03-31#]` | `10` |
| `{tT, c0010}[sub r = "0010"]` | `100` |
| `{tO, c0010}[sub qKEY = "z"]` | `no records` |
| `{tO, c0010, default: 0}[sub qDAT = #2026-09-30#] + {tP, c0010}` | `qKEY=b: 2; qKEY=d: 4` |
| `{tO, c0010}[sub foo = 1]` | `error: unknown key component 'foo'` |
### `rename` {#rename}
Renames key components, so that operands with different key names join.
The old names resolve as [key names](#key-names); the new ones are taken as
written. Records, values and the `default` are unchanged. A new name that
is already a key component (also ignoring case and the `q` prefix), or two
pairs with the same new name, is the error `rename: key component …
already exists`; pairs apply in order, so keys cannot be swapped in one
clause. Parses to `Rename`.
| Expression | Result |
| --- | --- |
| `{tO, c0010}[rename qKEY to K, qDAT to D]` | `K=a, D=2026-03-31: 10; K=b, D=2026-06-30: 20; K=c, D=2026-09-30: 30` |
| `{tP, c0010}[rename qKEY to K] + {tO, c0010}[sub qDAT = #2026-06-30#][rename qKEY to K]` | `K=b: 22` |
| `{tO, c0010}[rename foo to K]` | `error: unknown key component 'foo'` |
| `{tO, c0010}[rename qKEY to DAT]` | `error: rename: key component qDAT already exists` |
| `{tO, c0010}[rename qKEY to X, qDAT to X]` | `error: rename: key component X already exists` |
## Functions {#functions}
```text
call ::= name "(" ( expression ( "," expression )* )? ")"
```
A word followed by `(` is a call; function names match in any case. Calls
parse to `Call` with the canonical name
(`canonical_function`), except
[analytic calls](#analytic-functions). Unless an entry says otherwise, a
function applies to each record, [joins](#records) keyed operands, and is
null when an operand is null.
- **Operand counts** are checked against
`SIGNATURES`: a wrong count is the error
`abs takes 1 operand, not 2`, and `group by` in a function that does not
aggregate is `group by is not allowed in abs`.
- **Unknown functions** parse, so that rules with newer functions can be
read, and evaluate as unsupported: `unknown function 'foo'`.
| Expression | Result |
| --- | --- |
| `ABS(-1)` | `1` |
| `abs(1, 2)` | `error: abs takes 1 operand, not 2` |
| `abs(1 group by r)` | `error: group by is not allowed in abs` |
| `foo(1)` | `unsupported: unknown function 'foo'` |
### Numeric functions {#numeric-functions}
#### `abs` {#abs}
`abs(x)`: the absolute value of a number, keeping its tolerance. Errors:
`abs needs numbers, found …`.
| Expression | Result |
| --- | --- |
| `abs(-3)` | `3` |
| `abs({tT, r*, c0020})` | `r=0010: 5; r=0020: 5; r=0030: null` |
| `abs({tT, r0020, c0020, interval: true})` | `5 ± 0.5` |
| `abs("x")` | `error: abs needs numbers, found string` |
#### `exp`, `ln`, `sqrt` {#exp}
`exp(x)`, `ln(x)` and `sqrt(x)`: the exponential, natural logarithm and
square root of a number. They are computed in binary floating point, to
about 16 significant digits, and the result is exact, without tolerance. A
result that is undefined or out of range, such as `ln(0)` or `sqrt(-1)`, is
null. Errors: `exp needs numbers, found …`.
| Expression | Result |
| --- | --- |
| `exp(0)` | `1` |
| `exp(1)` | `2.718281828459045` |
| `ln(1)` | `0` |
| `sqrt(16)` | `4` |
| `sqrt(2)` | `1.4142135623730951` |
| `ln(0)` | `null` |
| `sqrt(-1)` | `null` |
| `sqrt("a")` | `error: sqrt needs numbers, found string` |
#### `power` {#power}
`power(base, exponent)`: `base` raised to `exponent`, in binary floating
point like [`exp`](#exp); null when undefined or out of range.
| Expression | Result |
| --- | --- |
| `power(2, 10)` | `1024` |
| `power(2, 0.5)` | `1.4142135623730951` |
| `power(0, -1)` | `null` |
| `power(-8, 1 / 3)` | `null` |
#### `log` {#log}
`log(x, base)`: the logarithm of `x` to `base`, in binary floating point
like [`ln`](#exp); null outside its domain (`x` ≤ 0, `base` ≤ 0 or 1).
| Expression | Result |
| --- | --- |
| `log(8, 2)` | `3` |
| `log(100, 10)` | `2` |
| `log(8, 1)` | `null` |
| `log(-8, 2)` | `null` |
#### `max`, `min` {#max}
With two or more operands, `max(a, b, …)` and `min(a, b, …)` pick the
largest or smallest value per record, keeping its tolerance; null if any
operand is null. Numbers compare by value; dates, text and items, which
cannot be mixed with numbers, by their text (ISO dates in date order).
Booleans are an error. With one operand they are
[aggregates](#aggregate-max).
| Expression | Result |
| --- | --- |
| `max(1, 5, 3)` | `5` |
| `min({tT, c0010}, {tT, c0020})` | `r=0010: 5; r=0020: -5; r=0030: null` |
| `max({tT, c0010}, 50)` | `r=0010: 100; r=0020: 60; r=0030: 50` |
| `max(#2026-01-01#, #2026-02-01#)` | `2026-02-01` |
| `max(1, null)` | `null` |
| `max(1, "a")` | `error: max cannot compare numbers with dates, text or items` |
### Null functions {#null-functions}
#### `isnull` {#isnull}
`isnull(x)`: `true` where `x` is null, else `false`; never null. It applies
to each record, so it finds the unreported cells of a selection.
| Expression | Result |
| --- | --- |
| `isnull({tT, c0020})` | `r=0010: false; r=0020: false; r=0030: true` |
| `isnull(null)` | `true` |
| `isnull(0)` | `false` |
#### `nvl` {#nvl}
`nvl(x, replacement)`: `x`, or `replacement` where `x` is null. The
operands [join](#records), so `nvl` replaces null values but adds no
records; a `default` on `x` covers the records `x` lacks.
| Expression | Result |
| --- | --- |
| `nvl({tT, c0020}, 0)` | `r=0010: 5; r=0020: -5; r=0030: 0` |
| `nvl(null, "none")` | `"none"` |
| `nvl({tO, c0010}, {tP, c0010})` | `qKEY=b, qDAT=2026-06-30: 20` |
### Text functions {#text-functions}
#### `len`, `length` {#len}
`len(x)`, also spelled `length(x)`: the number of characters of a text.
Other values count the characters of their text form; a number counts as
written, without an exponent.
| Expression | Result |
| --- | --- |
| `len("abc")` | `3` |
| `length({tX, c0010})` | `5` |
| `len(12.50)` | `5` |
| `len(null)` | `null` |
#### `substr` {#substr}
`substr(text)`, `substr(text, start)` and `substr(text, start, length)`:
part of a text. `start` counts from 1 (default 1) and `length`, by default
the rest, is at least 0; both are whole numbers. Past the end the result is
`""`. Errors: `substr needs text, found …`, `substr start must be at least
1`, `substr length must not be negative`, `substr needs whole numbers`.
| Expression | Result |
| --- | --- |
| `substr("abcdef", 2, 3)` | `"bcd"` |
| `substr("abcdef", 3)` | `"cdef"` |
| `substr("abcdef", 10)` | `""` |
| `substr("abcdef", 0)` | `error: substr start must be at least 1` |
| `substr(123, 1)` | `error: substr needs text, found number` |
#### `match` {#match}
`match(x, pattern)`: `true` when the whole text of `x` matches the regular
expression `pattern`, in Python
[`re`](https://docs.python.org/3/library/re.html) syntax, as if anchored at
both ends. Numbers match their text. Errors: `match needs a text pattern`,
`invalid match pattern: …`.
| Expression | Result |
| --- | --- |
| `match({tX, c0010}, "LEI-[A-Z]")` | `true` |
| `match("xLEI-A", "LEI-[A-Z]")` | `false` |
| `match(2026, "[0-9]{4}")` | `true` |
| `match("a", "(")` | `error: invalid match pattern: missing ), unterminated subpattern at position 0` |
### Date functions {#date-functions}
#### `year`, `semester`, `quarter`, `month`, `week`, `day` {#date-parts}
The part of a date or date-time, as a number: `year`, `semester` (1–2),
`quarter` (1–4), `month` (1–12), `week` (the ISO 8601 week, 1–53) and
`day` of the month. Errors: `year needs a date, found …`; text is not read
as a date.
| Expression | Result |
| --- | --- |
| `year(#2026-05-31#)` | `2026` |
| `semester(#2026-07-01#)` | `2` |
| `quarter(#2026-05-31#)` | `2` |
| `month({tX, c0030})` | `6` |
| `week(#2026-01-01#)` | `1` |
| `day(#2026-05-31T10:00:00#)` | `31` |
| `year({tO, c0010}[get qDAT])` | `qKEY=a, qDAT=2026-03-31: 2026; qKEY=b, qDAT=2026-06-30: 2026; qKEY=c, qDAT=2026-09-30: 2026` |
| `year("2026-05-31")` | `error: year needs a date, found string` |
#### `date` {#date}
`date(year, month, day)`: the date of three whole numbers. Errors: `date
needs whole numbers`, `invalid date: …` for a day the month lacks.
| Expression | Result |
| --- | --- |
| `date(2026, 2, 28)` | `2026-02-28` |
| `date(2026, 12, 31) = #2026-12-31#` | `true` |
| `date(2026, 2, 30)` | `error: invalid date: 2026-02-30` |
### Time functions {#time-functions}
#### `time_shift` {#time_shift}
```text
"time_shift" "(" expression "," period "," expression ( "," name )? ")"
period ::= "A" | "S" | "Q" | "M" | "W" | "D"
```
`time_shift(x, period, count, key)` presents `x` as it was `count` periods
earlier (later when negative): years, semesters, quarters, months, weeks or
days. `period` is an upper-case code and `count` a whole number.
- When `key` is a date key component of `x`, its values move `count`
periods later within the report, so that each record joins with those of
the later date. Months move by calendar months, and a month's last day
stays the last day.
- Otherwise, without `key` or with a name such as `refPeriod` that is not a
key of `x`, `x` is evaluated on the report `count` periods earlier, from
the provider's optional `earlier_report(period, count)` method; without
that report the rule is unsupported.
Errors: `time_shift needs a period of A, S, Q, M, W or D`, `time_shift
needs a key name as its fourth operand`, `time_shift needs dates in key …`,
and `time_shift moves several … dates to …` when two records would merge.
See [Time](../concepts/time.md#time_shift) and
[Compare with earlier reports](../guides/time-shift.md).
| Expression | Result |
| --- | --- |
| `time_shift({tT, r0010, c0010}, A, 1)` | `50` |
| `{tT, r0010, c0010} - time_shift({tT, r0010, c0010}, A, 1, refPeriod)` | `50` |
| `time_shift({tO, c0010}, Q, 1, qDAT)` | `qKEY=a, qDAT=2026-06-30: 10; qKEY=b, qDAT=2026-09-30: 20; qKEY=c, qDAT=2026-12-31: 30` |
| `time_shift({tO, c0010}, S, 1, qDAT)` | `qKEY=a, qDAT=2026-09-30: 10; qKEY=b, qDAT=2026-12-31: 20; qKEY=c, qDAT=2027-03-31: 30` |
| `time_shift({tO, c0010}, W, 1, qDAT)` | `qKEY=a, qDAT=2026-04-07: 10; qKEY=b, qDAT=2026-07-07: 20; qKEY=c, qDAT=2026-10-07: 30` |
| `time_shift({tO, c0010}, D, -1, qDAT)` | `qKEY=a, qDAT=2026-03-30: 10; qKEY=b, qDAT=2026-06-29: 20; qKEY=c, qDAT=2026-09-29: 30` |
| `time_shift({tO, c0010}, M, -1, qDAT)` | `qKEY=a, qDAT=2026-02-28: 10; qKEY=b, qDAT=2026-05-31: 20; qKEY=c, qDAT=2026-08-31: 30` |
| `time_shift({tT, r0010, c0010}, Q, 1)` | `unsupported: time_shift needs the report 1 quarter(s) earlier, which the provider does not have` |
| `time_shift({tT, r0010, c0010}, a, 1)` | `error: time_shift needs a period of A, S, Q, M, W or D` |
| `time_shift({tO, c0010}, A, 1, qKEY)` | `error: time_shift needs dates in key qKEY, found string` |
#### `annualise` {#annualise}
`annualise(x, fiscalYearEnd, key)` scales a year-to-date amount to a year:
`x · 12 / months`, where `months` counts the months after the fiscal year's
last month (`fiscalYearEnd`, 1–12) up to and including the month of the
date in `key`. `key` is a date key component of `x`, else a date report
parameter. The tolerance scales with the amount. Errors: `annualise needs
a fiscal year end month of 1 to 12`, `annualise needs a date key or report
parameter …`, `annualise needs a key name as its third operand`.
| Expression | Result |
| --- | --- |
| `annualise({tO, c0010}, 12, qDAT)` | `qKEY=a, qDAT=2026-03-31: 40; qKEY=b, qDAT=2026-06-30: 40; qKEY=c, qDAT=2026-09-30: 40` |
| `annualise(30, 6, refPeriod)` | `60` |
| `annualise(30, 12, Level)` | `error: annualise needs a date key or report parameter 'Level'` |
### Set functions {#set-functions}
Each operand of a set function is a set, or any other operand, which stands
for the set of its non-null values. A single null operand, such as a
missing set parameter, makes the result null; `set_of` gives `{}` instead.
The result has no keys, and its members keep their first-seen order.
#### `set_of` {#set_of}
`set_of(x)`: the set of the non-null values of `x`.
| Expression | Result |
| --- | --- |
| `set_of({tU})` | `{1, 2, 3}` |
| `set_of({tT, c0020})` | `{5, -5}` |
| `set_of(null)` | `{}` |
#### `union`, `intersect` {#union}
`union(a, b, …)` has the members of any operand; `intersect(a, b, …)` the
members of the first operand that every other operand has. Both take two or
more operands.
| Expression | Result |
| --- | --- |
| `union({1}, {2}, {1, 3})` | `{1, 2, 3}` |
| `intersect({1, 2}, {2, 3}, {2})` | `{2}` |
| `union({tU}, {tP})` | `{1, 2, 3, 4}` |
| `union({1}, {p_Missing, set-number})` | `null` |
#### `setdiff`, `symdiff` {#setdiff}
`setdiff(a, b)` has the members of `a` that `b` lacks; `symdiff(a, b)` the
members of exactly one of the two.
| Expression | Result |
| --- | --- |
| `setdiff({1, 2}, {2, 3})` | `{1}` |
| `symdiff({1, 2}, {2, 3})` | `{1, 3}` |
| `setdiff({1, 2, 3, 4}, {tU})` | `{4}` |
### `filter` {#filter}
`filter(data, condition)` keeps the records of `data` whose `condition`
record is `true`; a false, null or non-Boolean condition drops the record.
The two operands [join](#records), so the condition is usually a comparison
on the same selection. Unlike [`where`](#where) it tests values, not keys.
The result keeps `data`'s `default`, which stands in for the keys other
operands bring.
| Expression | Result |
| --- | --- |
| `filter({tT, c0010}, {tT, c0010} > 50)` | `r=0010: 100; r=0020: 60` |
| `filter({tO, c0010}, {tO, c0020} = [CU:USD])` | `qKEY=b, qDAT=2026-06-30: 20` |
| `count(filter({tT, c0020}, {tT, c0020} > 0))` | `1` |
## Sets {#sets}
### Set literals {#set-literals}
```text
set ::= "{" "}" | "{" expression ( "," expression )* "}"
```
Braces whose first member is a literal, an item, a signed number or a
parenthesized number are a set; other braces are a
[selection](#selections). Each member is an expression with one value;
nulls and duplicates are dropped, and a set cannot contain a set. Sets
cannot be compared with `=` or the other comparisons and cannot be a rule's
result ([Sets](../concepts/sets.md)). Parses to
`SetLiteral`.
| Expression | Result |
| --- | --- |
| `{1, 2, 2}` | `{1, 2}` |
| `{(-5), 3}` | `{-5, 3}` |
| `{-5, 1 + 1}` | `{-5, 2}` |
| `{[eba_CU:EUR], [eba_CU:USD]}` | `{[CU:EUR], [CU:USD]}` |
| `{"a", #2026-01-01#, true, null}` | `{"a", 2026-01-01, true}` |
| `{}` | `{}` |
Sets are tested with [`in`](#in), built with the
[set functions](#set-functions) and read from
[set parameters](#typed-parameter).
## Aggregates {#aggregates}
```text
aggregate ::= ( "sum" | "count" | "avg" | "median" | "max" | "min" | "max_aggr" | "min_aggr" )
"(" expression ( "group" "by" name ( "," name )* )? ")"
```
An aggregate reduces an operand's records to one value, or to one value per
combination of the `group by` keys ([Key names](#key-names)), skipping null
values. Over no records an aggregate is null, except `count`, which is 0;
so is a group whose values are all null. The result has the `group by`
keys, or none, and no `default`: a key without records is no group
([Aggregates](../concepts/analytics.md#aggregates)).
- **Errors:** `group by 'qX' is not a key of the operand (r, c)`; `sum
needs numbers, found …`; `max with group by takes 1 operand, not 2`.
- **Node:** `Call` with `group_by`; with `over`, an
[analytic function](#analytic-functions).
| Expression | Result |
| --- | --- |
| `sum({tT} group by c)` | `c=0010: 200; c=0020: 0` |
| `count({tT} group by r, c)` | `r=0010, c=0010: 1; r=0010, c=0020: 1; r=0020, c=0010: 1; r=0020, c=0020: 1; r=0030, c=0010: 1; r=0030, c=0020: 0` |
| `sum({tO, c0010} group by KEY)` | `qKEY=a: 10; qKEY=b: 20; qKEY=c: 30` |
| `sum({tT} group by qX)` | `error: group by 'qX' is not a key of the operand (r, c)` |
### `sum` {#sum}
The sum of the numbers; their tolerances add up.
| Expression | Result |
| --- | --- |
| `sum({tT, c0010})` | `200` |
| `sum({tT, c0010, interval: true})` | `200 ± 1.5` |
| `sum({tT, r0030, c0020})` | `null` |
| `sum({tX, c0010})` | `error: sum needs numbers, found string` |
### `count` {#count}
The number of non-null values, of any type.
| Expression | Result |
| --- | --- |
| `count({tT})` | `5` |
| `count({tT, r0030, c0020})` | `0` |
| `count({tO} group by c)` | `c=0010: 3; c=0020: 3` |
### `avg` {#avg}
The mean of the numbers, with the sum's tolerance divided by the count.
| Expression | Result |
| --- | --- |
| `avg({tU})` | `2` |
| `avg({tT, c0010, interval: true})` | `66.66666666666666666666666667 ± 0.5` |
### `median` {#median}
The middle number, or the mean of the two middle numbers; exact, without
tolerance.
| Expression | Result |
| --- | --- |
| `median({tU})` | `2` |
| `median({tT, c0020})` | `0` |
### `max`, `min`, `max_aggr`, `min_aggr` {#aggregate-max}
With one operand, `max` and `min` are aggregates, also spelled `max_aggr`
and `min_aggr`, which take exactly one operand. They pick the largest or
smallest value, keeping its tolerance. Besides numbers they order dates,
text and items by their text (ISO dates in date order, items by category
and code), as presence checks expect; Booleans, and numbers mixed with
other values, are errors. With several operands `max` and `min` are the
[scalar functions](#max).
| Expression | Result |
| --- | --- |
| `max({tU})` | `3` |
| `min_aggr({tT, c0020})` | `-5` |
| `max_aggr({tO, c0020})` | `[CU:USD]` |
| `max({tO, c0010} group by qKEY)` | `qKEY=a: 10; qKEY=b: 20; qKEY=c: 30` |
| `max({tX})` | `error: max needs numbers, dates, text or items` |
| `max_aggr(1, 2)` | `syntax error: max_aggr takes one operand` |
## Analytic functions {#analytic-functions}
```text
analytic ::= function "(" expression "over" "(" partition? order? window? ")" ")"
function ::= "rank" | "sum" | "count" | "avg" | "median" | "max" | "min" | "max_aggr" | "min_aggr"
partition ::= "partition" "by" name ( "," name )*
order ::= "order" "by" name ( "asc" | "desc" )? ( "," name ( "asc" | "desc" )? )*
window ::= ( "data" "points" | "range" ) "between" bound "and" bound
bound ::= "unbounded" ( "preceding" | "following" )
| "current" "data" "point"
| digit+ period? ( "preceding" | "following" )
```
An analytic function keeps its operand's records: each gets the aggregate
of the records in its window, or its rank, within its partition.
`partition by` and `order by` name keys of the operand
([Key names](#key-names)); without `partition by` the whole operand is one
partition. Without a window the window is the whole partition, or with
`order by` every record up to the current one and its peers. `data points`
windows count records; `range` windows with an offset need a single number
or date order key, and `3 M preceding` is three calendar months earlier.
The result has no `default`. The window keywords are reserved only inside
`over (…)`
([Partitions, order and windows](../concepts/analytics.md#partitions-order-and-windows)).
- **Errors:** the syntax errors `rank needs an order by clause`, `abs takes
no over clause` and `a window offset is a whole number`; `sum over 'qX',
which is not a key of the operand (r, c)`; `a range window offset needs a
number or date order key`; `a range window with offsets needs one order
key`.
- **Node:** `Analytic` with a
`Window`.
| Expression | Result |
| --- | --- |
| `sum({tU} over ())` | `r=0010: 6; r=0020: 6; r=0030: 6` |
| `sum({tU} over (order by r))` | `r=0010: 1; r=0020: 3; r=0030: 6` |
| `sum({tT} over (partition by c))` | `r=0010, c=0010: 200; r=0020, c=0010: 200; r=0030, c=0010: 200; r=0010, c=0020: 0; r=0020, c=0020: 0; r=0030, c=0020: 0` |
| `avg({tU} over (order by r data points between 1 preceding and current data point))` | `r=0010: 1; r=0020: 1.5; r=0030: 2.5` |
| `sum({tU} over (order by r data points between current data point and unbounded following))` | `r=0010: 6; r=0020: 5; r=0030: 3` |
| `min_aggr({tU} over (order by r desc data points between unbounded preceding and 1 following))` | `r=0030: 2; r=0020: 1; r=0010: 1` |
| `sum({tO, c0010} over (order by qDAT range between 3 M preceding and current data point))` | `qKEY=a, qDAT=2026-03-31: 10; qKEY=b, qDAT=2026-06-30: 30; qKEY=c, qDAT=2026-09-30: 50` |
| `count({tO, c0010} over (order by qDAT asc range between current data point and 1 Q following))` | `qKEY=a, qDAT=2026-03-31: 2; qKEY=b, qDAT=2026-06-30: 2; qKEY=c, qDAT=2026-09-30: 1` |
| `sum({tU} over (order by r range between 1 preceding and current data point))` | `error: a range window offset needs a number or date order key` |
| `abs({tU} over ())` | `syntax error: abs takes no over clause` |
### `rank` {#rank}
`rank(x over (partition by … order by …))`: the position of each record in
its partition by the order keys, from 1; ties share the lowest rank and
leave gaps (1, 1, 3). Values do not matter, only keys. `rank` needs `over`
with `order by`; a window is ignored.
| Expression | Result |
| --- | --- |
| `rank({tU} over (order by r desc))` | `r=0030: 1; r=0020: 2; r=0010: 3` |
| `rank({tT} over (partition by c order by r desc))` | `r=0030, c=0010: 1; r=0020, c=0010: 2; r=0010, c=0010: 3; r=0030, c=0020: 1; r=0020, c=0020: 2; r=0010, c=0020: 3` |
| `filter({tU}, rank({tU} over (order by r)) <= 2)` | `r=0010: 1; r=0020: 2` |
| `rank({tU})` | `syntax error: rank needs an over clause` |
| `rank({tU} over (partition by r))` | `syntax error: rank needs an order by clause` |
## Where dpm-xl differs from the grammar {#where-dpm-xl-differs-from-the-grammar}
Some readings differ from the published DPM-XL grammar on purpose, so that
EBA rules keep their meaning:
- `not` binds looser than comparisons and `in`: `not a = b` is
`not (a = b)`.
- Chained comparisons such as `a < b < c` or `a in b in c`, `- not x`, and a
selection that repeats an axis, `default` or `interval` are syntax errors.
- Outside clauses `<-` is the persistent assignment, so compare with a
negated operand as `a < -b`.
- `{v_Code}` tests a filing indicator and `{vName}` reads a report
parameter: the `_` after `v` is not cosmetic, unlike after `t`, `g`, `o`
and `p`.
```dpm-xl
{tT, r0010} < {tT, r0020} < {tT, r0030}
1 in {1} in {true}
- not true
{tT, r0010, r0020}
```
In the other direction dpm-xl accepts keywords in any case, `null` and `<>`
anywhere, a last statement without `;`, `length` for `len`, codes without
leading zeros (`r310` is row `0310`), ranges that repeat the axis letter
(`r0440-r0480`), key names without the `q` prefix or in another case, item
codes without a category (`[ei86]`), and `max` and `min` over dates, text
and items.
Every distinct EBA rule and precondition expression of DPM releases 3.4–4.4
parses, except two source expressions with typos (`trueendif`, `ornot`),
which are reported as syntax errors.
## Known limitations {#limitations}
- No comments, and no escape sequences in [text](#strings).
- `with` only starts a statement, an assignment's value or another `with`
body ([`with`](#with)).
- `exp`, `ln`, `sqrt`, `power` and `log` compute in binary floating point
and drop the tolerance ([`exp`](#exp)).
- A coordinate a table lacks selects nothing rather than raising an error;
`diagnose` with table layouts warns about
it.
- `time_shift` across reports, `{o_…}` and `{g_…}` need optional provider
methods; without them a rule is unsupported
([Connect your data](../guides/provider.md)).
- The [columnar evaluator](../guides/columnar.md) implements a subset and
raises `ColumnarUnsupported` for the rest, such as scripts, `time_shift`,
analytic functions and most sets.
- Expressions nest at most 50 deep, and numbers stay within the decimal128
range ([Limits](#limits)).
## Index {#index}
Every word and symbol of `dpmxl.grammar`, with its entry.
The test suite checks that this index and the grammar module agree.
| Name | Kind | Entry |
| --- | --- | --- |
| `and` | keyword | [Logical operators](#logical) |
| `else` | keyword | [`if`](#if) |
| `endif` | keyword | [`if`](#if) |
| `get` | keyword | [`get`](#get) |
| `group by` | keyword | [Aggregates](#aggregates) |
| `if` | keyword | [`if`](#if) |
| `in` | keyword | [`in`, `not in`](#in) |
| `not` | keyword | [`not`](#not) |
| `or` | keyword | [Logical operators](#logical) |
| `over` | keyword | [Analytic functions](#analytic-functions) |
| `rename` | keyword | [`rename`](#rename) |
| `sub` | keyword | [`sub`](#sub) |
| `then` | keyword | [`if`](#if) |
| `to` | keyword | [`rename`](#rename) |
| `where` | keyword | [`where`](#where) |
| `with` | keyword | [`with`](#with) |
| `xor` | keyword | [Logical operators](#logical) |
| `asc` | window keyword | [Analytic functions](#analytic-functions) |
| `between` | window keyword | [Analytic functions](#analytic-functions) |
| `current data point` | window keyword | [Analytic functions](#analytic-functions) |
| `data points` | window keyword | [Analytic functions](#analytic-functions) |
| `desc` | window keyword | [Analytic functions](#analytic-functions) |
| `following` | window keyword | [Analytic functions](#analytic-functions) |
| `order by` | window keyword | [Analytic functions](#analytic-functions) |
| `partition by` | window keyword | [Analytic functions](#analytic-functions) |
| `preceding` | window keyword | [Analytic functions](#analytic-functions) |
| `range` | window keyword | [Analytic functions](#analytic-functions) |
| `unbounded` | window keyword | [Analytic functions](#analytic-functions) |
| `default` | selection option | [`default`](#default) |
| `interval` | selection option | [`interval`](#interval) |
| `false` | literal | [Booleans and null](#boolean-and-null) |
| `null` | literal | [Booleans and null](#boolean-and-null) |
| `true` | literal | [Booleans and null](#boolean-and-null) |
| `boolean` | parameter type | [Typed parameter](#typed-parameter) |
| `date` | parameter type | [Typed parameter](#typed-parameter) |
| `integer` | parameter type | [Typed parameter](#typed-parameter) |
| `item` | parameter type | [Typed parameter](#typed-parameter) |
| `number` | parameter type | [Typed parameter](#typed-parameter) |
| `set-boolean` | parameter type | [Typed parameter](#typed-parameter) |
| `set-date` | parameter type | [Typed parameter](#typed-parameter) |
| `set-integer` | parameter type | [Typed parameter](#typed-parameter) |
| `set-item` | parameter type | [Typed parameter](#typed-parameter) |
| `set-number` | parameter type | [Typed parameter](#typed-parameter) |
| `set-string` | parameter type | [Typed parameter](#typed-parameter) |
| `string` | parameter type | [Typed parameter](#typed-parameter) |
| `A` | period | [`time_shift`](#time_shift) |
| `S` | period | [`time_shift`](#time_shift) |
| `Q` | period | [`time_shift`](#time_shift) |
| `M` | period | [`time_shift`](#time_shift) |
| `W` | period | [`time_shift`](#time_shift) |
| `D` | period | [`time_shift`](#time_shift) |
| `abs` | function | [`abs`](#abs) |
| `annualise` | function | [`annualise`](#annualise) |
| `avg` | function | [`avg`](#avg) |
| `count` | function | [`count`](#count) |
| `date` | function | [`date`](#date) |
| `day` | function | [Date parts](#date-parts) |
| `exp` | function | [`exp`, `ln`, `sqrt`](#exp) |
| `filter` | function | [`filter`](#filter) |
| `intersect` | function | [`union`, `intersect`](#union) |
| `isnull` | function | [`isnull`](#isnull) |
| `len` | function | [`len`, `length`](#len) |
| `length` | function | [`len`, `length`](#len) |
| `ln` | function | [`exp`, `ln`, `sqrt`](#exp) |
| `log` | function | [`log`](#log) |
| `match` | function | [`match`](#match) |
| `max` | function | [`max`, `min`](#max) |
| `max_aggr` | function | [`max_aggr`, `min_aggr`](#aggregate-max) |
| `median` | function | [`median`](#median) |
| `min` | function | [`max`, `min`](#max) |
| `min_aggr` | function | [`max_aggr`, `min_aggr`](#aggregate-max) |
| `month` | function | [Date parts](#date-parts) |
| `nvl` | function | [`nvl`](#nvl) |
| `power` | function | [`power`](#power) |
| `quarter` | function | [Date parts](#date-parts) |
| `rank` | function | [`rank`](#rank) |
| `semester` | function | [Date parts](#date-parts) |
| `set_of` | function | [`set_of`](#set_of) |
| `setdiff` | function | [`setdiff`, `symdiff`](#setdiff) |
| `sqrt` | function | [`exp`, `ln`, `sqrt`](#exp) |
| `substr` | function | [`substr`](#substr) |
| `sum` | function | [`sum`](#sum) |
| `symdiff` | function | [`setdiff`, `symdiff`](#setdiff) |
| `time_shift` | function | [`time_shift`](#time_shift) |
| `union` | function | [`union`, `intersect`](#union) |
| `week` | function | [Date parts](#date-parts) |
| `year` | function | [Date parts](#date-parts) |
| `+` | operator | [Arithmetic, unary plus](#arithmetic) |
| `-` | operator | [Arithmetic, unary minus](#arithmetic) |
| `*` | operator | [Arithmetic](#arithmetic) |
| `/` | operator | [Arithmetic](#arithmetic) |
| `&` | operator | [Concatenation](#concatenation) |
| `=` | operator | [Comparison](#comparison) |
| `!=` | operator | [Comparison](#comparison) |
| `<>` | operator | [Comparison](#comparison) |
| `<` | operator | [Comparison](#comparison) |
| `<=` | operator | [Comparison](#comparison) |
| `>` | operator | [Comparison](#comparison) |
| `>=` | operator | [Comparison](#comparison) |
| `%` | operator | [Percentages](#percent) |
| `:=` | operator | [Temporary assignment](#temporary-assignment) |
| `<-` | operator | [Persistent assignment](#persistent-assignment) |
| `(` | punctuation | [Precedence (grouping), calls, axis lists](#precedence) |
| `)` | punctuation | [Precedence (grouping), calls, axis lists](#precedence) |
| `[` | punctuation | [Clauses, items](#clauses) |
| `]` | punctuation | [Clauses, items](#clauses) |
| `{` | punctuation | [Selections, references, sets](#selections) |
| `}` | punctuation | [Selections, references, sets](#selections) |
| `,` | punctuation | [Selections, calls, sets](#selections) |
| `:` | punctuation | [`with`, selection options](#with) |
| `;` | punctuation | [Programs and statements](#statements) |
| `t` | reference prefix | [Table](#table) |
| `g` | reference prefix | [Table group](#table-group) |
| `v` | reference prefix | [Filing indicator, report parameter](#filing-indicator) |
| `o` | reference prefix | [Operation](#operation) |
| `p` | reference prefix | [Typed parameter](#typed-parameter) |
| `r` | axis | [Rows, columns and sheets](#axes) |
| `c` | axis | [Rows, columns and sheets](#axes) |
| `s` | axis | [Rows, columns and sheets](#axes) |
# CLI
```text
dpm-xl check [--layouts DATA] [--strict] [--references] [--json] SOURCE
dpm-xl eval [--precondition EXPRESSION] [--json] RULES DATA
dpm-xl --version
dpm-xl [COMMAND] --help
```
`dpm-xl` is installed with the package; `python -m dpmxl.cli` runs the same.
## Rule sources
`SOURCE` and `RULES` are one of:
- the path of a `.json` file holding a list of rules
([format](data-formats.md#rule-files));
- the path of any other existing file: one expression per line, `#` starts a
comment line, blank lines are ignored;
- otherwise, the argument itself as one expression. An argument that reads
as a path (ending in `.json` or `.txt`, or holding a `/` without brackets,
operators or spaces) but names no file is an error (exit status 3).
Each expression and precondition is labelled in the output: `argument` for
an expression given directly, `:expression` and `:precondition`
for a JSON rule file, `line :expression` for a text file (`eval` labels
the rule itself: `argument` becomes `expression`, the others `` and
`line `).
## `dpm-xl check`
Checks every expression and precondition of a rule source with
`diagnose`: every syntax error, unknown
functions, calls with the wrong operands and selections without a table.
Expressions may be scripts with `;`, `:=` and `<-`. Every problem is
printed; the exit status is 1 when an expression does not parse, or with
`--strict` when any expression has an error. Warnings never change it.
| Option | Effect |
| --- | --- |
| `--layouts DATA` | Also check tables, coordinates, open-table keys and, when the tables give `data_types`, operand types against the `tables` of a [data file](data-formats.md#data-files), and names against its `parameters`. |
| `--strict` | Exit with status 1 for any error, not only for syntax errors. |
| `--references` | Also print every selection, with its `with` context applied. |
| `--json` | Print one JSON document instead of text. |
```console
$ dpm-xl check "with {tC_01.00}: {r0010} >= 0" --references
argument: {tC_01.00, r0010}
1/1 expressions parsed
$ dpm-xl check "{tC_01.00, r0010} >="
argument: error: expression ends where an operand is expected
{tC_01.00, r0010} >=
^
0/1 expressions parsed; 1 error
$ echo $?
1
$ dpm-xl check "{tC_01.00, r0011, c0010} > 0 or {tC_27.00, r0010, c0010} > 0" --layouts examples/data.json --strict
argument: error: row 0011 is not in table C_01.00; did you mean r0010?
{tC_01.00, r0011, c0010} > 0 or {tC_27.00, r0010, c0010} > 0
^~~~~
argument: warning: the rows of C_27.00 are open (key qCPI), so they have no codes such as r0010; select them with r* or leave them out
{tC_01.00, r0011, c0010} > 0 or {tC_27.00, r0010, c0010} > 0
^~~~~
1/1 expressions parsed; 1 error and 1 warning
$ echo $?
1
```
Problems go to standard error; the summary goes to standard output.
### JSON output
```console
$ dpm-xl check "with {tC_01.00}: {r0010} >= 0" --references --json
{
"schema_version": 1,
"parsed": 1,
"total": 1,
"errors": [],
"diagnostics": [],
"selections": [
{
"source": "argument",
"selection": "{tC_01.00, r0010}"
}
]
}
```
| Field | Type | Content |
| --- | --- | --- |
| `schema_version` | integer | `1`. Changes only with a new major release. |
| `parsed` | integer | Expressions and preconditions that parsed. |
| `total` | integer | Expressions and preconditions checked. |
| `errors` | list | One object per syntax error: `source` (the label), `message` (plain words), `position` and `end` (zero-based character offsets of the problem in that expression). |
| `diagnostics` | list | One object per problem, syntax errors included: `source` (the label) and the fields of a `Diagnostic`: `severity`, `code`, `message`, `start`, `end`, `subject` and `suggestion`. |
| `selections` | list | Only with `--references`: `source` and `selection` (normalized DPM-XL text) per selection. |
## `dpm-xl eval`
Evaluates every rule of a rule source against a JSON
[data file](data-formats.md#data-files) and explains each failing record.
| Option | Effect |
| --- | --- |
| `--precondition EXPRESSION` | Precondition for a single expression given as `RULES`, such as `{v_C_01.00}`. Rule files carry their own. |
| `--json` | Print one JSON document instead of text. |
```console
$ dpm-xl eval examples/rules.json examples/data.json
v0001: failed
records: 1 passed, 1 failed, 0 not evaluated
- c=0020: 1,204 should equal 1,211 (1,180 + 31): difference 7, at least 5.5 given the reported precision
v0002: failed
records: 1 passed, 1 failed, 0 not evaluated
- qCPI=LEI-B: -5 should exceed 0: difference 5
v0003: passed
records: 1 passed, 0 failed, 0 not evaluated
v0004: not_applicable
message: precondition not met
records: 0 passed, 0 failed, 0 not evaluated
4 rules: 2 failed, 1 passed, 1 not_applicable
```
For one expression the output starts with `status:`, and a failing record
without keys is shown as `scalar`:
```console
$ dpm-xl eval "{tC_01.00, r0010, c0010} / 0 > 1" examples/data.json
status: error
message: division by zero
records: 0 passed, 0 failed, 0 not evaluated
$ echo $?
3
```
### JSON output
```console
$ dpm-xl eval "{tC_27.00, c0010} > 0" examples/data.json --json
{
"schema_version": 1,
"results": [
{
"rule": "expression",
"expression": "{tC_27.00, c0010} > 0",
"status": "failed",
"message": null,
"counts": {
"passed": 1,
"failed": 1,
"null": 0
},
"failures": [
{
"key": {
"qCPI": "LEI-B"
},
"summary": "-5 should exceed 0: difference 5",
"cells": [
"{tC_27.00, c0010, qCPI=LEI-B}"
]
}
]
}
]
}
```
| Field | Type | Content |
| --- | --- | --- |
| `schema_version` | integer | `1`. Changes only with a new major release. |
| `results` | list | One object per rule, in source order. |
| `results[].rule` | string | The rule's label: its `code`, `line ` or `expression`. |
| `results[].expression` | string | The expression evaluated. |
| `results[].status` | string | `passed`, `failed`, `not_applicable`, `error` or `unsupported`. |
| `results[].message` | string or null | Why the status is `error`, `unsupported` or `not_applicable`. |
| `results[].counts` | object | `passed`, `failed` and `null` (not evaluated) record counts. |
| `results[].failures` | list | Per failing record: `key` (key name to value, as text), `summary` (the explanation sentence) and `cells` (each cell used, as DPM-XL text). |
Summaries and messages may be reworded in minor releases; field names,
statuses and keys may not.
## Exit statuses
| Status | `check` | `eval` |
| --- | --- | --- |
| 0 | Everything parsed; with `--strict`, nothing has an error. | Every rule passed or did not apply. |
| 1 | An expression or precondition does not parse; with `--strict`, one has an error. | A rule failed. |
| 2 | Usage error (unknown command or option). | Usage error. |
| 3 | The rule file cannot be read or is malformed. | A rule is `error` or `unsupported`, or the rule or data file cannot be read or is malformed. |
With several rules, `eval` exits with the highest status of any rule.
```console
$ dpm-xl eval "{tC_01.00, r0010, c0010} > 1" missing.json
error: [Errno 2] No such file or directory: 'missing.json'
$ echo $?
3
```
# Data formats
The CLI reads two JSON formats: rule files and data files. Python callers
normally build a provider instead
([Connect your data](../guides/provider.md)); the CLI's
`dpmxl.cli.load_memory_provider(document)` turns a data document into a
`MemoryProvider` if you want the same
format in Python. The `--json` output of the CLI is described in the
[CLI reference](cli.md#json-output).
## Rule files
A `.json` rule file is a list of objects, one per rule
([`examples/rules.json`](https://github.com/luukhopman/dpm-xl/blob/main/examples/rules.json)):
| Field | Type | Content |
| --- | --- | --- |
| `code` | string | The rule's label in the output. Defaults to the rule's 1-based position. |
| `expression` | string | The rule. A rule without one is skipped. |
| `precondition` | string, optional | An expression that must yield a true record for the rule to apply. |
`expression` and `precondition` may also be objects with an `expression`
field, as some rule exports nest them. A file that does not hold a list, or
a list entry that is not an object, is an error (exit status 3).
```json
[
{
"code": "v0001",
"expression": "with {tC_01.00, c*, default: 0, interval: true}: {r0010} = {r0020} + {r0030}",
"precondition": "{v_C_01.00}"
},
{"code": "v0002", "expression": "{tC_27.00, c0010} > 0"}
]
```
Any other file is a text rule file: one expression per line, `#` starts a
comment line, blank lines are ignored
([`examples/rules.txt`](https://github.com/luukhopman/dpm-xl/blob/main/examples/rules.txt)).
## Data files
A data file is one JSON object describing a report
([`examples/data.json`](https://github.com/luukhopman/dpm-xl/blob/main/examples/data.json)).
Every field is optional.
| Field | Type | Content |
| --- | --- | --- |
| `tables` | list | Table layouts: `code`; `rows`, `columns` and `sheets` (lists of codes); `open_keys` (key names of an open table); `template` (a template code whose filing indicator `{v_…}` this table answers); `open_axes` (which of `rows`, `columns` and `sheets` are open, identified by the open keys); `data_types` (cells' data types, each `row`, `column`, `sheet` and `type`: `number`, `integer`, `string`, `date`, `boolean` or `item`). The last two are optional and used only by [`dpm-xl check --layouts`](cli.md#dpm-xl-check). |
| `facts` | list | Reported values, below. |
| `reported` | list | Table or template codes with a positive filing indicator. Defaults to every table with a fact. |
| `parameters` | object | Report parameters by name, typed like fact values; a list is a set. |
| `table_groups` | object | Group code to a list of table codes, for `{g_…}`. |
| `operations` | object | Operation code to its expression, for `{o_…}`. |
| `earlier` | list | Other periods' reports for `time_shift`: `period` (`A`, `S`, `Q`, `M`, `W` or `D`), `count` (periods earlier, negative for later) and `data`, a nested data document. |
### Facts
| Field | Type | Content |
| --- | --- | --- |
| `table` | string | Table code (required). |
| `row`, `column`, `sheet` | string, optional | Closed coordinates; omit the ones the table does not have. |
| `keys` | object, optional | Open key components by name, for open tables. |
| `value` | any | The value; `null` for a nil fact. |
| `decimals` | integer or `"INF"`, optional | XBRL decimals of a number; `"INF"` or absent means exact. |
| `type` | string, optional | `number`, `item` or `date`, to read a string value as that type. |
Values are typed as follows: JSON numbers (and strings with `"type":
"number"`) become `Number` with the given `decimals`; strings starting with
`[` (or with `"type": "item"`) become items such as `[eba_CU:EUR]`; strings
with `"type": "date"` become dates; lists become sets; Booleans and other
strings stay as they are. Open key values are typed the same way, without
`type`.
### Example
This document uses every field; the Python below loads it the way
`dpm-xl eval` does and evaluates a rule against each part.
```python
from dpmxl import evaluate_rule
from dpmxl.cli import load_memory_provider
document = {
"tables": [
{"code": "C_01.00", "rows": ["0010", "0020"], "columns": ["0010"]},
{
"code": "C_27.00",
"columns": ["0010", "0020", "0030"],
"open_keys": ["qCPI"],
"template": "C_27",
"open_axes": ["rows"],
"data_types": [{"column": "0010", "type": "number"}],
},
],
"facts": [
{
"table": "C_01.00",
"row": "0010",
"column": "0010",
"value": 1204,
"decimals": 0,
},
{
"table": "C_01.00",
"row": "0020",
"column": "0010",
"value": "1180.5",
"decimals": "INF",
"type": "number",
},
{
"table": "C_27.00",
"column": "0010",
"keys": {"qCPI": "LEI-A"},
"value": 120,
"decimals": -3,
},
{
"table": "C_27.00",
"column": "0020",
"keys": {"qCPI": "LEI-A"},
"value": "[eba_CU:EUR]",
},
{
"table": "C_27.00",
"column": "0030",
"keys": {"qCPI": "LEI-A"},
"value": "2026-06-30",
"type": "date",
},
],
"reported": ["C_01.00", "C_27.00"],
"parameters": {
"refPeriod": "2026-12-31",
"Currencies": ["[eba_CU:EUR]", "[eba_CU:USD]"],
},
"table_groups": {"G_01": ["C_01.00"]},
"operations": {"v0001_a": "{tC_01.00, r0010, c0010} * 2"},
"earlier": [
{
"period": "Q",
"count": 1,
"data": {
"tables": [{"code": "C_01.00", "rows": ["0010"], "columns": ["0010"]}],
"facts": [
{
"table": "C_01.00",
"row": "0010",
"column": "0010",
"value": 1100,
"decimals": 0,
}
],
},
},
],
}
provider = load_memory_provider(document)
for expression in (
"{tC_01.00, r0010, c0010} > {tC_01.00, r0020, c0010}",
"{tC_27.00, c0020} in {p_Currencies, set-item}",
"{tC_27.00, c0030} < {p_refPeriod, date}",
"{v_C_27}",
"{g_G_01, r0010, c0010} > 0",
"{o_v0001_a} > 2000",
"{tC_01.00, r0010, c0010} >= time_shift({tC_01.00, r0010, c0010}, Q, 1)",
):
print(evaluate_rule(expression, provider).status, expression)
```
```text
passed {tC_01.00, r0010, c0010} > {tC_01.00, r0020, c0010}
passed {tC_27.00, c0020} in {p_Currencies, set-item}
passed {tC_27.00, c0030} < {p_refPeriod, date}
passed {v_C_27}
passed {g_G_01, r0010, c0010} > 0
passed {o_v0001_a} > 2000
passed {tC_01.00, r0010, c0010} >= time_shift({tC_01.00, r0010, c0010}, Q, 1)
```
A malformed document is reported as `error: …` with exit status 3: an
invalid value (a number that does not parse, `decimals` that is neither an
integer nor `INF` or is beyond the decimal128 range) names the fact by its
1-based position, as in `error: fact 1: invalid number 'x1'`; a missing
required field such as `table` gives `malformed data document`.
# Python API
The public API is every name in the `__all__` of the modules below. Names
starting with an underscore are internal. The most used names are also
importable from the `dpmxl` package itself.
| Module | Purpose |
| --- | --- |
| [`dpmxl.evaluator`](evaluator.md) | Evaluation, preconditions, per-record outcomes, caches; thread safety. |
| [`dpmxl.explain`](explain.md) | Fills a failing record's formula with operand values. |
| [`dpmxl.provider`](provider.md) | The `DataProvider` protocol and the in-memory `MemoryProvider`. |
| [`dpmxl.columnar`](columnar.md) | The same semantics on polars frames (`columnar` extra); the frame schema. |
| [`dpmxl.parser`](parser.md) | Tokenizer and recursive-descent parser. Nodes keep source spans. |
| [`dpmxl.ast`](ast.md) | Expression tree, selections, axis selectors and item literals. |
| [`dpmxl.grammar`](grammar.md) | Keywords, operators, function names and signatures; the TextMate grammar. |
| [`dpmxl.values`](values.md) | `Number` with interval tolerance, comparison and formatting. |
| [`dpmxl.recordset`](recordset.md) | Keyed record sets and the natural join used by every operator. |
| [`dpmxl.references`](references.md) | Static selections (with `with` context applied) and `{v_…}` templates. |
| [`dpmxl.diagnostics`](diagnostics.md) | Positioned problems for editors: syntax, calls, tables and coordinates. |
The command line (`dpmxl.cli`) is documented in the [CLI reference](../cli.md).
## The `dpmxl` package
```python
import dpmxl
print(dpmxl.__all__)
```
```text
['AxisSelector', 'CellQuery', 'DataPoint', 'DataProvider', 'Diagnostic', 'DpmXlSyntaxError', 'Evaluation', 'EvaluationError', 'Evaluator', 'Explanation', 'Item', 'MemoryProvider', 'Node', 'Number', 'Operand', 'Outcome', 'Recordset', 'Reference', 'RuleResult', 'SelectionCache', 'SelectionSpec', 'TableLayout', 'UnsupportedError', 'Value', 'ValueSet', '__version__', 'axis_matcher', 'axis_matches', 'compare', 'diagnose', 'evaluate_rule', 'explain', 'format_value', 'iter_points', 'normalize_code', 'parse', 'parse_cached', 'parse_program', 'parse_selection', 'references', 'reported_templates', 'tables', 'unique_points']
```
| Name | Defined in |
| --- | --- |
| `evaluate_rule`, `RuleResult`, `Outcome`, `Evaluation`, `Evaluator`, `SelectionCache`, `parse_cached` | `dpmxl.evaluator` |
| `explain`, `Explanation`, `Operand` | `dpmxl.explain` |
| `DataProvider`, `MemoryProvider`, `TableLayout`, `CellQuery`, `DataPoint`, `axis_matcher`, `axis_matches`, `normalize_code` | `dpmxl.provider` |
| `parse`, `parse_program`, `parse_selection`, `DpmXlSyntaxError` | `dpmxl.parser` |
| `Node`, `Item`, `SelectionSpec`, `AxisSelector` | `dpmxl.ast` |
| `Number`, `Value`, `ValueSet`, `compare`, `format_value` | `dpmxl.values` |
| `Recordset`, `EvaluationError`, `UnsupportedError`, `iter_points`, `unique_points` | `dpmxl.recordset` |
| `references`, `Reference`, `tables`, `reported_templates` | `dpmxl.references` |
| `diagnose`, `Diagnostic` | `dpmxl.diagnostics` |
`dpmxl.__version__` is the installed version.
# `dpmxl.evaluator`
Evaluation of rules and preconditions, per-record outcomes and the caches shared by the rules of one report. The module docstring below includes the thread-safety guarantees.
Evaluate DPM-XL expressions against a report's data provider.
`evaluate_rule` evaluates a rule against a
`DataProvider` and returns a
`RuleResult`; it never raises for a bad rule or
bad data.
The evaluator implements every construct of the DPM-XL grammar; constructs
that need data beyond one report (``time_shift``, operation and table-group
references) use optional provider methods. It keeps every intermediate
record set so that a failing record can be explained operand by operand (see
`dpmxl.explain`).
Semantics in brief:
* A selection yields one record per selected data point, keyed by varying
closed axes and open key components. Missing facts take the selection's
``default`` (``null`` when none is written); a written default also stands
in where another operand has a key the selection lacks.
* ``interval: true`` keeps each fact's decimals tolerance, so ``=`` holds when
the intervals overlap; otherwise comparisons are exact.
* Null propagates through arithmetic and comparisons; ``and``/``or`` use
three-valued (Kleene) logic. A null rule result is "not evaluated", never a
failure.
* Aggregates over no records are null, except ``count``, which is 0.
## Threads
`evaluate_rule` may run concurrently on one
report from several threads. `parse_cached`
is thread-safe. A shared `SelectionCache` is
safe too, though two threads that miss the same selection at once may both
build it. Providers shared across threads must be safe to read concurrently;
`MemoryProvider` is. An
`Evaluator` instance is not meant to be shared:
create one per evaluation.
Examples:
>>> from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule
>>> provider = MemoryProvider(
... [TableLayout("T", rows=("0010", "0020"), columns=("0010",))],
... {
... ("T", "0010", "0010", None, ()): Number.of(5, 0),
... ("T", "0020", "0010", None, ()): Number.of(-2, 0),
... },
... )
>>> result = evaluate_rule("{tT, r*, c0010} >= 0", provider)
>>> result.status, result.counts
('failed', {'passed': 1, 'failed': 1, 'null': 0})
>>> [outcome.key for outcome in result.failures]
[{'r': '0020'}]
## `dpmxl.evaluator.AGGREGATES`
```python
AGGREGATES = frozenset({'sum', 'count', 'avg', 'max', 'min', 'median'})
```
Canonical names of the aggregate functions (``Call.name`` values).
``rank`` and aggregates with ``over (...)`` are
`Analytic` nodes and are not listed.
## `dpmxl.evaluator.Evaluation`
```python
class Evaluation(root: Node, result: Recordset, trace: dict[int, Recordset] = dict(), notes: list[str] = list(), assignments: dict[str, Recordset] = dict(), shapes: dict[tuple[str, int], Any] = dict())
```
Raw evaluation result with intermediate record sets for explanation.
Pass it to `explain` (it is
`RuleResult.evaluation`).
Attributes:
root: The evaluated tree.
result: The record set of the last statement.
trace: Each node's record set by ``id(node)``; empty when evaluated
with ``trace=False``.
notes: Remarks gathered during evaluation.
### `dpmxl.evaluator.Evaluation.assignments`
```python
assignments: dict[str, Recordset] = field(default_factory=dict)
```
Results of persistent assignments (``{target} <- value``) by target,
written as normalized DPM-XL such as ``{tT, r0010, c0010}`` or ``{vX}``.
### `dpmxl.evaluator.Evaluation.shapes`
```python
shapes: dict[tuple[str, int], Any] = field(default_factory=dict, repr=False, compare=False)
```
What explanations derive from the tree alone (plain formulas, term
counts), by kind and node id; shared by every failure explained.
### `dpmxl.evaluator.Evaluation.value_at`
```python
def value_at(node: Node, binding: Mapping[str, Hashable]) -> list[Record]
```
Return the records a node produced for a key binding.
Returns:
list[Record]: Matching records; empty when the node was not
evaluated for that binding.
## `dpmxl.evaluator.Evaluator`
```python
class Evaluator(provider: DataProvider, *, trace: bool = True, cache: SelectionCache | None = None, only_keys: Mapping[str, Collection[str]] | None = None)
```
Evaluates one expression tree against one provider.
The engine behind `evaluate_rule`, which
most callers want instead: ``Evaluator.run`` raises
`EvaluationError` where
``evaluate_rule`` returns status ``error``. Create one per evaluation;
share a `SelectionCache` between them.
### `dpmxl.evaluator.Evaluator.run`
```python
def run(node: Node) -> Evaluation
```
Evaluate a complete expression or program.
Returns:
Evaluation: The result and its trace.
Raises:
EvaluationError: For a type error, an unknown table and the like;
`UnsupportedError` when the
provider cannot resolve a reference.
### `dpmxl.evaluator.Evaluator.evaluate`
```python
def evaluate(node: Node, context: SelectionSpec, bindings: Mapping[str, Value] | None) -> Recordset
```
Evaluate a node under a ``with`` context and optional key bindings.
Returns:
Recordset: The node's records.
## `dpmxl.evaluator.Outcome`
```python
class Outcome(key: Mapping[str, Hashable], passed: bool | None, raw_points: tuple[Any, ...] = ())
```
One evaluated record of a rule.
``points`` are flattened from the evaluation on first access: most
outcomes pass, and nobody reads their points.
Attributes:
key: The record key, such as ``{'c': '0020'}`` or ``{'qCPI': 'LEI-B'}``;
empty for a rule without keys. Pass it to
`explain`.
passed: ``True``, ``False``, or ``None`` when the record's result is
null (not evaluated).
raw_points: The points as the evaluation keeps them (possibly
nested); read ``points`` instead.
### `dpmxl.evaluator.Outcome.points`
```python
points: tuple[DataPoint, ...]
```
Distinct data points the record was computed from.
## `dpmxl.evaluator.RuleResult`
```python
class RuleResult(status: Status, outcomes: list[Outcome] = list(), message: str | None = None, evaluation: Evaluation | None = None, notes: list[str] = list(), assignments: dict[str, Recordset] = dict())
```
The outcome of evaluating a rule and its optional precondition.
Attributes:
status: ``passed``, ``failed``, ``not_applicable`` (precondition not
met, or no record could be evaluated), ``error`` (syntax error,
unknown table, type mismatch, division by zero that reaches the
result) or ``unsupported`` (a ``time_shift``, ``{o_...}`` or
``{g_...}`` the provider cannot resolve, or an unknown function).
outcomes: One `Outcome` per record.
message: Why the status is ``error``, ``unsupported`` or
``not_applicable``.
evaluation: The traced `Evaluation` for
`explain`; ``None`` when evaluation did not
complete.
notes: Remarks gathered during evaluation.
### `dpmxl.evaluator.RuleResult.assignments`
```python
assignments: dict[str, Recordset] = field(default_factory=dict)
```
Results of persistent assignments by target (see
`Evaluation.assignments`); the verdict
comes from the last statement.
### `dpmxl.evaluator.RuleResult.failures`
```python
failures: list[Outcome]
```
Records whose result is false.
### `dpmxl.evaluator.RuleResult.counts`
```python
counts: dict[str, int]
```
Counts of passed, failed and not-evaluated records.
## `dpmxl.evaluator.SelectionCache`
```python
class SelectionCache()
```
Selection results shared by every rule evaluated against one report.
Rules over the same report select the same cells again and again (every
sum rule on a table reads its columns); building each record set once
makes large open tables affordable. Record sets are never mutated by the
evaluator, so sharing them is safe. Use one cache per report. Rules may
share it across threads: a selection two threads miss at once is built
by both, and either result is kept.
### `dpmxl.evaluator.SelectionCache.earlier`
```python
earlier: dict[tuple[str, int], SelectionCache] = {}
```
Caches of earlier reports for ``time_shift``, by period and count.
## `dpmxl.evaluator.evaluate_rule`
```python
def evaluate_rule(expression: str | Node, provider: DataProvider, *, precondition: str | Node | None = None, trace: bool = True, cache: SelectionCache | None = None, only_keys: Mapping[str, Collection[str]] | None = None) -> RuleResult
```
Evaluate a validation rule, honouring its precondition.
Args:
expression: Rule expression text or a parsed tree.
provider: Report data.
precondition: Optional precondition; the rule is not applicable
unless it yields at least one true record.
trace: Keep intermediate values for `explain`.
cache: Selections shared by the rules of one report.
only_keys: Keep only records with these closed-axis key codes in
every selection (see `Evaluator`); the
precondition is evaluated on all records. Not meaningful with aggregates,
whose totals would cover only the kept records.
Returns:
RuleResult: Status, per-record outcomes and the evaluation trace.
Syntax errors are reported as status ``error``, not raised.
Examples:
>>> from dpmxl import MemoryProvider, Number, TableLayout, explain
>>> provider = MemoryProvider(
... [TableLayout("T", rows=("0010", "0020"), columns=("0010",))],
... {
... ("T", "0010", "0010", None, ()): Number.of(10, 0),
... ("T", "0020", "0010", None, ()): Number.of(12, 0),
... },
... )
>>> expression = "with {tT, c0010}: {r0010} >= {r0020}"
>>> result = evaluate_rule(expression, provider, precondition="{v_T}")
>>> result.status
'failed'
>>> explain(result.evaluation, expression, result.failures[0].key).summary
'10 should be at least 12: difference 2'
>>> evaluate_rule("{tT, r0010} >", provider).status
'error'
## `dpmxl.evaluator.parse_cached`
```python
def parse_cached(expression: str) -> Node
```
Parse an expression once and reuse the tree.
Trees are immutable and traces key nodes per evaluation, so one parsed
tree can serve every report a rule is evaluated against. The cache is
thread-safe and holds up to 32,768 expressions.
Returns:
Node: The expression tree.
Raises:
DpmXlSyntaxError: If the text is not a valid expression.
Examples:
>>> parse_cached("{tT, r0010} > 0") is parse_cached("{tT, r0010} > 0")
True
# `dpmxl.explain`
Explanations of failing records: a sentence, the filled formula, both sides with their intervals and the cells involved. See [Explain failures in a UI](../../guides/explain.md).
Explain a failing record in words and with the formula's values filled in.
For ``{r0010} = {r0020} + {r0030}`` failing at column ``0010``:
* ``summary``: ``1,204 should equal 1,197 (1,180 + 17): difference 7``;
* ``filled``: ``1,204 = 1,180 + 17``;
* ``left`` and ``right``: each side's formula, value, filled text and cells.
Formulas are rendered from the expression tree rather than copied from the
source, so spacing is uniform, negative terms read ``a - 226,000`` (with a
typographic minus) instead of ``a + -226,000``, and long runs of zero terms
collapse to ``+ 0 (x13)``.
Failures that are not comparisons (a value that must be reported, must be
empty, must be one of a set, or must match a pattern) get a sentence of their
own.
Explanation and error wording may improve in minor releases; field names and
``kind`` values may not.
## `dpmxl.explain.VERBS`
```python
VERBS = {'=': 'should equal', '!=': 'should differ from', '<': 'should be less than', '<=': 'should not exceed', '>': 'should exceed', '>=': 'should be at least'}
```
The verb of a failing comparison's summary, by comparison operator.
## `dpmxl.explain.Explanation`
```python
class Explanation(formula: str, filled: str, summary: str, operands: tuple[Operand, ...], difference: str | None = None, primary: tuple[DataPoint, ...] = (), kind: Kind = 'condition', relation: str | None = None, verb: str | None = None, left: Side | None = None, right: Side | None = None, gap: str | None = None, direction: Direction | None = None, tolerance: str | None = None)
```
Human-readable account of one evaluated record.
Fields whose value is formatted text (``difference``, ``gap``,
``tolerance``, the sides' values) are strings, ready for display.
Attributes:
formula: The rule body as rendered from the tree, such as
``{r0010} = {r0020} + {r0030}``.
filled: The formula with the record's values, such as
``1,204 = 1,180 + 17``.
summary: One sentence for a person, such as
``1,204 should equal 1,197 (1,180 + 17): difference 7``.
operands: Every leaf operand with its value and cells.
difference: For a comparison, the absolute difference of the sides.
kind: ``comparison``, ``missing`` (a value must be reported),
``not_empty`` (a value must be empty), ``membership`` (a value
must be one of a set), ``format`` (a value must match a pattern)
or ``condition`` (anything else).
relation: The failing comparison operator, such as ``=``.
verb: The summary's verb from `VERBS`.
left: The left side of a failing comparison.
right: The right side of a failing comparison.
tolerance: The sum of both sides' thresholds; ``None`` when exact.
### `dpmxl.explain.Explanation.primary`
```python
primary: tuple[DataPoint, ...] = ()
```
Cells on the left of the failing comparison: the values that disagree.
### `dpmxl.explain.Explanation.gap`
```python
gap: str | None = None
```
How far apart the two sides' intervals are: the smallest difference
that remains when every value may vary within its reported precision.
### `dpmxl.explain.Explanation.direction`
```python
direction: Direction | None = None
```
Which side is larger (``too_high``: left exceeds right), not a verdict
on which value is wrong.
## `dpmxl.explain.Operand`
```python
class Operand(source: str, value: str, points: tuple[DataPoint, ...])
```
One leaf operand of a failing record with its value and cells.
Attributes:
source: The operand's text, such as ``{r0020}``.
value: Its formatted value, such as ``1,180``.
points: The cells it read.
## `dpmxl.explain.Side`
```python
class Side(formula: str, filled: str, value: str, points: tuple[DataPoint, ...], terms: int, threshold: str | None = None, low: str | None = None, high: str | None = None)
```
One side of the failing comparison.
Attributes:
formula: The side's formula, such as ``{r0020} + {r0030}``.
filled: The formula with values, such as ``1,180 + 17``.
value: The side's formatted value.
points: The cells it read.
### `dpmxl.explain.Side.terms`
```python
terms: int
```
Number of operands (cells, aggregates, parameters) on this side.
### `dpmxl.explain.Side.threshold`
```python
threshold: str | None = None
```
Half-width of the side's interval from reported precision; None if exact.
### `dpmxl.explain.Side.single_cell`
```python
single_cell: bool
```
Whether the side is exactly one reported cell.
## `dpmxl.explain.explain`
```python
def explain(evaluation: Evaluation, expression: str, key: Mapping[str, Hashable]) -> Explanation
```
Explain the record of ``evaluation`` identified by ``key``.
Args:
evaluation: A traced evaluation, such as
`RuleResult.evaluation` from
`evaluate_rule` (with the default
``trace=True``).
expression: The source text the tree was parsed from.
key: The record key, as in `Outcome.key`.
Returns:
Explanation: Summary sentence, formula, filled formula and operands.
Examples:
>>> from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule
>>> layout = TableLayout("T", rows=("0010", "0020", "0030"), columns=("0010",))
>>> provider = MemoryProvider([layout], {
... ("T", "0010", "0010", None, ()): Number.of(1204, 0),
... ("T", "0020", "0010", None, ()): Number.of(1180, 0),
... ("T", "0030", "0010", None, ()): Number.of(17, 0),
... })
>>> expression = (
... "with {tT, c0010, interval: true}: {r0010} = {r0020} + {r0030}"
... )
>>> result = evaluate_rule(expression, provider)
>>> explanation = explain(result.evaluation, expression, result.failures[0].key)
>>> print(explanation.summary) # doctest: +NORMALIZE_WHITESPACE
1,204 should equal 1,197 (1,180 + 17): difference 7, at least 5.5 given the
reported precision
>>> explanation.filled, explanation.gap, explanation.direction
('1,204 = 1,180 + 17', '5.5', 'too_high')
>>> explanation.right.low, explanation.right.high
('1,196', '1,198')
>>> [point.label() for point in explanation.primary]
['{tT, r0010, c0010}']
# `dpmxl.provider`
The data contract between the evaluator and a report. See [Connect your data](../../guides/provider.md) for a walkthrough of every method, optional hooks included.
Data access contract between the evaluator and a report.
The evaluator never reads report files. It asks a
`DataProvider` for the data points a selection covers and
for report-level facts such as filing indicators. A provider is responsible for:
* resolving table codes and axis selectors against the table definitions;
* returning **every defined cell** of a closed selection, with ``value=None``
when the report has no fact for it, so ``default`` and ``isnull`` work;
* returning one data point per reported key combination for open axes, with
the open key components in `DataPoint.keys`.
`MemoryProvider` is a complete in-memory implementation
used by the tests and by tools that build reports programmatically.
## `dpmxl.provider.CELL_TYPES`
```python
CELL_TYPES = frozenset({'boolean', 'date', 'integer', 'item', 'number', 'string'})
```
Data types a `TableLayout` gives its cells.
## `dpmxl.provider.AxisMatcher`
```python
class AxisMatcher(codes: frozenset[str], numeric_ranges: tuple[tuple[int, int], ...], text_ranges: tuple[tuple[str, str], ...], wildcard: bool = False)
```
A compiled axis selector: exact codes plus inclusive ranges.
Build one with `axis_matcher` and call it
with a coordinate code.
Attributes:
codes: Normalized single codes.
numeric_ranges: Inclusive ranges with numeric bounds.
text_ranges: Inclusive ranges with text bounds, compared as text.
wildcard: Whether every coordinate is selected.
### `dpmxl.provider.AxisMatcher.single`
```python
single: str | None
```
The only selected code, when the selector names exactly one.
## `dpmxl.provider.CellQuery`
```python
class CellQuery(table: str, rows: AxisSelector | None = None, columns: AxisSelector | None = None, sheets: AxisSelector | None = None)
```
A fully resolved selection: table plus optional axis selectors.
The evaluator passes one to
`DataProvider.select` after applying
the ``with`` context. Match coordinates with
`axis_matcher`.
Attributes:
table: The table code, such as ``C_01.00``.
rows: The row selector; ``None`` when the selection names no rows,
which selects every row.
columns: The column selector, likewise.
sheets: The sheet selector, likewise.
## `dpmxl.provider.DataPoint`
```python
class DataPoint(table: str, row: str | None = None, column: str | None = None, sheet: str | None = None, keys: Mapping[str, Value] = dict(), value: Value = None, ref: Hashable | None = None)
```
One selected cell, and for open tables one reported key combination.
``ref`` is an opaque provider identity (for example a cell ID) that is
passed back in evaluation results so callers can highlight the cell.
Attributes:
table: The table code.
row: The row code, or ``None`` for a table without rows.
column: The column code, or ``None``.
sheet: The sheet code, or ``None``.
keys: Open key components by name (such as ``{"qCPI": "LEI-A"}``);
empty for closed tables.
value: The reported value, or ``None`` when nothing was reported.
ref: Any hashable identity of the cell; points read by ``time_shift``
from another report carry an
`EarlierRef`.
### `dpmxl.provider.DataPoint.coordinate`
```python
def coordinate(axis: str) -> str | None
```
Return the closed coordinate for ``r``, ``c`` or ``s``.
Returns:
str | None: The coordinate code, or ``None`` when absent.
Raises:
KeyError: For another axis.
### `dpmxl.provider.DataPoint.label`
```python
def label() -> str
```
Render a compact DPM-XL style location.
Returns:
str: For example ``{tC_01.00, r0010, c0010}``.
Examples:
>>> DataPoint("C_01.00", row="0010", column="0010").label()
'{tC_01.00, r0010, c0010}'
>>> DataPoint("C_27.00", column="0010", keys={"qCPI": "LEI-B"}).label()
'{tC_27.00, c0010, qCPI=LEI-B}'
## `dpmxl.provider.DataProvider`
```python
class DataProvider(Protocol)
```
Source of report data for one evaluation.
Three methods are required. ``select`` resolves a
`CellQuery` against the table definitions
(`axis_matcher` and
`normalize_code` implement range and
code matching) and returns:
* for a closed table, **every defined cell** the selection covers, with
``value=None`` where the report has no fact, so ``default`` and
``isnull`` work;
* for an open table, one point per reported key combination (times the
selected closed coordinates), with the key components in
`DataPoint.keys`;
and raises ``LookupError`` for a table the release does not define (the
rule's status is then ``error``). ``table_reported`` answers ``{v_TABLE}``
and ``parameter`` answers ``{vName}`` and ``{p_Name, type}``.
A provider may also define these optional methods:
* ``open_keys(table: str) -> Sequence[str]``: the open key names of a
table. Without it, an open table with no reported rows yields a
selection without keys; with it, the selection keeps the table's key
structure and is empty, as the specification requires.
* ``table_group(code: str) -> Sequence[str] | None``: the table codes of a
table group, for ``{g_Group, ...}`` selections.
* ``operation(code: str) -> str | None``: the expression of another
operation, for ``{o_Code, ...}`` references.
* ``earlier_report(period: str, count: int) -> DataProvider | None``: the
report ``count`` periods earlier (later when negative), where
``period`` is ``A``, ``S``, ``Q``, ``M``, ``W`` or ``D``, for
``time_shift``.
Without the optional method, or when it returns ``None``, rules that need
it are ``unsupported``. ``DataProvider`` is ``runtime_checkable``:
``isinstance`` checks the three required methods only.
A provider shared by several threads must be safe to read concurrently.
### `dpmxl.provider.DataProvider.select`
```python
def select(query: CellQuery) -> Sequence[DataPoint]
```
Return the data points covered by a selection.
Raises:
LookupError: If the table does not exist in the release.
### `dpmxl.provider.DataProvider.table_reported`
```python
def table_reported(table: str) -> bool | None
```
Return whether the report's filing indicators include ``table``.
``table`` is the code written in ``{v_...}``, usually a template code
such as ``C_01.00``. Returns ``None`` when the provider cannot tell,
which makes ``{v_...}`` null.
### `dpmxl.provider.DataProvider.parameter`
```python
def parameter(name: str) -> Value
```
Return a report-level parameter such as ``refPeriod``.
Returns ``None`` when the parameter is unknown. Typed parameters
(``{p_Name, type}``) convert the value: Python numbers become
`Number`, text becomes an item or date for
those types, and a list or set becomes a set.
## `dpmxl.provider.EarlierRef`
```python
class EarlierRef(period: str, count: int, ref: Hashable)
```
The ``ref`` of a data point that ``time_shift`` read from another report.
It wraps the earlier report's own ``ref`` with the shift, so the point
stays distinct from the current report's cell with the same ``ref``.
Attributes:
period: The ``time_shift`` period, such as ``Q``.
count: The number of periods.
ref: The earlier report's own ``ref`` for the cell.
## `dpmxl.provider.EarlierReport`
```python
class EarlierReport(provider: DataProvider, period: str, count: int)
```
The report ``time_shift`` reads, marking its points' refs.
Every data point it selects carries an `EarlierRef`
instead of the provider's ``ref`` (points without a ``ref`` keep ``None``); every
other method, optional ones included, is the wrapped provider's.
### `dpmxl.provider.EarlierReport.select`
```python
def select(query: CellQuery) -> Sequence[DataPoint]
```
Return the earlier report's points with marked refs.
Returns:
Sequence[DataPoint]: The points.
### `dpmxl.provider.EarlierReport.table_reported`
```python
def table_reported(table: str) -> bool | None
```
Return the earlier report's filing indicator.
Returns:
bool | None: Filing-indicator state.
### `dpmxl.provider.EarlierReport.parameter`
```python
def parameter(name: str) -> Value
```
Return an earlier report parameter.
Returns:
Value: The parameter or ``None``.
## `dpmxl.provider.MemoryProvider`
```python
class MemoryProvider(tables: Iterable[TableLayout], values: Mapping[CellKey, Value], *, reported: Iterable[str] | None = None, parameters: Mapping[str, Value] | None = None, table_groups: Mapping[str, Sequence[str]] | None = None, operations: Mapping[str, str] | None = None, earlier_reports: Mapping[tuple[str, int], DataProvider] | None = None)
```
Provider over in-memory table layouts and values.
Values are keyed by ``(table, row, column, sheet, keys)`` where ``keys`` is
a tuple of ``(name, value)`` pairs for open tables (empty for closed ones).
Absent tables in ``reported`` are treated as not reported. Every optional
`DataProvider` method is implemented. Each
point's ``ref`` is its value key.
A ``MemoryProvider`` is safe to read from several threads at once.
Examples:
>>> from dpmxl import CellQuery, Number
>>> closed = TableLayout("C_01.00", rows=("0010", "0020"), columns=("0010",))
>>> open_ = TableLayout("C_27.00", columns=("0010",), open_keys=("qCPI",))
>>> provider = MemoryProvider(
... [closed, open_],
... {
... ("C_01.00", "0010", "0010", None, ()): Number.of(100, 0),
... ("C_27.00", None, "0010", None, (("qCPI", "A"),)): Number.of(1, 0),
... },
... parameters={"refPeriod": "2026-12-31"},
... )
>>> (p.row, p.value) for p in provider.select(CellQuery("C_01.00"))
('0020', None)
>>> [p.keys for p in provider.select(CellQuery("C_27.00"))]
[{'qCPI': 'A'}]
>>> provider.table_reported("C_27.00"), provider.parameter("refPeriod")
(True, '2026-12-31')
### `dpmxl.provider.MemoryProvider.select`
```python
def select(query: CellQuery) -> Sequence[DataPoint]
```
Return closed cells, and one point per reported open key set.
Returns:
Sequence[DataPoint]: Selected data points.
Raises:
LookupError: If the table is unknown.
### `dpmxl.provider.MemoryProvider.open_keys`
```python
def open_keys(table: str) -> tuple[str, ...]
```
Return a table's open key names.
Returns:
tuple[str, ...]: Key names; empty for closed or unknown tables.
### `dpmxl.provider.MemoryProvider.table_reported`
```python
def table_reported(table: str) -> bool | None
```
Return whether a table or its template was reported.
Returns:
bool | None: Filing-indicator state.
### `dpmxl.provider.MemoryProvider.parameter`
```python
def parameter(name: str) -> Value
```
Return a report parameter.
Returns:
Value: The parameter or ``None``.
### `dpmxl.provider.MemoryProvider.table_group`
```python
def table_group(code: str) -> Sequence[str] | None
```
Return the table codes of a table group.
Returns:
Sequence[str] | None: The tables, or ``None`` for an unknown group.
### `dpmxl.provider.MemoryProvider.earlier_report`
```python
def earlier_report(period: str, count: int) -> DataProvider | None
```
Return the report ``count`` periods earlier.
Returns:
DataProvider | None: The report, or ``None`` when not given.
### `dpmxl.provider.MemoryProvider.operation`
```python
def operation(code: str) -> str | None
```
Return the expression of an operation.
Returns:
str | None: The expression, or ``None`` for an unknown operation.
## `dpmxl.provider.TableLayout`
```python
class TableLayout(code: str, rows: tuple[str, ...] = (), columns: tuple[str, ...] = (), sheets: tuple[str, ...] = (), open_keys: tuple[str, ...] = (), template: str | None = None, open_axes: tuple[AxisName, ...] = (), data_types: Mapping[tuple[str | None, str | None, str | None], str] = dict())
```
The coordinates, open keys and cell data types of one table.
`MemoryProvider` serves cells from
layouts, and `diagnose` checks selections
against them. An axis is closed (its ``rows``, ``columns`` or ``sheets``
codes), open (named in ``open_axes``: its rows, columns or sheets are
identified by the ``open_keys``, not by codes) or absent (neither).
When a table has been split into several, as when ``C_02.00`` became
``C_02.00.a`` and ``C_02.00.b``, rules may still select the original,
abstract code; describe it with the union of its parts' coordinates.
Attributes:
code: The table code, such as ``C_01.00``.
rows: Row codes; empty for a table without closed rows.
columns: Column codes; empty for a table without closed columns.
sheets: Sheet codes; empty for a table without closed sheets.
open_keys: Open key names, such as ``("qCPI",)``; empty for a closed
table.
template: The template code when it differs from ``code``, so
``{v_TEMPLATE}`` holds when any of its tables was reported.
open_axes: The open axes among ``rows``, ``columns`` and ``sheets``.
Optional: without it, an axis without codes of a table with open
keys is taken to be open or absent, and selections of it are not
checked. What a provider selects for a code on an open or absent
axis is up to it: a ``MemoryProvider`` ignores the selector, other
providers may select nothing. ``diagnose`` reports such a code as
a warning.
data_types: The data type of each cell by its ``(row, column,
sheet)``, with ``None`` for an open or absent axis: ``number``,
``integer``, ``string``, ``date``, ``boolean`` or ``item``.
Optional; `diagnose` uses it to
find operands of the wrong type. It may be partial: a selection
that covers a cell without a type has no known type, so it is
not checked. It takes no part in equality.
Raises:
ValueError: For an unknown axis in ``open_axes``, an open axis with
codes, or an unknown data type.
Examples:
>>> layout = TableLayout(
... "C_27.00",
... columns=("0010", "0020"),
... open_keys=("qCPI",),
... open_axes=("rows",),
... data_types={(None, "0010", None): "number"},
... )
>>> layout.open_axes, layout.data_types[None, "0010", None]
(('rows',), 'number')
### `dpmxl.provider.TableLayout.coordinates`
```python
def coordinates() -> Iterable[tuple[str | None, str | None, str | None]]
```
Yield every closed (row, column, sheet) combination.
Yields:
tuple[str | None, str | None, str | None]: One cell position.
## `dpmxl.provider.axis_matcher`
```python
def axis_matcher(selector: AxisSelector | None) -> AxisMatcher
```
Compile a selector once for repeated matching.
Returns:
AxisMatcher: A callable matcher; ``None`` and wildcards select all.
Examples:
>>> from dpmxl import AxisSelector
>>> rows = axis_matcher(AxisSelector(parts=(("0010", "0050"), "0100")))
>>> rows("0030"), rows("100"), rows("0060"), rows(None)
(True, True, False, False)
## `dpmxl.provider.axis_matches`
```python
def axis_matches(code: str | None, selector: AxisSelector | None) -> bool
```
Return whether a coordinate is selected.
``None`` selectors and wildcards select every coordinate. Ranges are
inclusive and compare numerically when both bounds are numeric.
Returns:
bool: Whether ``code`` is covered by ``selector``.
Examples:
>>> from dpmxl import AxisSelector
>>> axis_matches("0020", AxisSelector(parts=("0010", "0020")))
True
>>> axis_matches("0020", None)
True
## `dpmxl.provider.normalize_code`
```python
def normalize_code(code: str) -> str
```
Normalize an axis code for comparison.
Numeric codes are compared as numbers padded to four digits, so ``r310``
in an older rule matches row ``0310``. Results are memoized: a report's
rules normalize the same few thousand codes millions of times.
Returns:
str: A canonical code.
Examples:
>>> normalize_code("310"), normalize_code("0310"), normalize_code("x01")
('0310', '0310', 'X01')
# `dpmxl.columnar`
The columnar evaluator (the `columnar` extra). The module docstring below defines the frame schema a `FrameProvider` returns and the thread-safety guarantees. See [Evaluate large reports](../../guides/columnar.md).
Vectorized DPM-XL evaluation on polars data frames.
Needs the ``columnar`` extra (``pip install "dpm-xl[columnar]"``). The
record-by-record evaluator (`dpmxl.evaluator`) is the
reference implementation; this module evaluates the same language with the
same semantics on columnar data, so open tables with millions of rows evaluate
in milliseconds per rule.
Anything this module does not implement raises
`ColumnarUnsupported`, and callers of
`evaluate_rule_columnar` then fall
back to the reference evaluator, so coverage never trades off correctness.
That covers statements and assignments, set operators such as ``union``,
analytic functions, the ``&`` operator, functions other than the aggregates,
``filter``, ``abs``, ``isnull``, ``nvl``, ``match``, ``len`` and
``min``/``max`` (so ``time_shift``, ``annualise``, ``substr``, the
mathematical and the date functions), typed parameter, operation and
table-group references, values that do not fit
``Decimal(38, 12)``, arithmetic that would overflow it, any record error (a
zero divisor or a type error, which the reference evaluator confines to its
record), ordering items and text that may be an item, and a ``default``
standing in for a record another operand has.
Values are ``Decimal(38, 12)``: sums and differences are exact, products and
quotients round to 12 decimals (thresholds of quotients round up), where the
reference evaluator keeps 34 significant digits, so an exact comparison that
hinges on the 13th decimal can differ.
## Frame schema
A `FrameProvider` returns one row per selected
data point (closed cells times reported open key combinations) with these
columns:
| Column | Type | Content |
| --- | --- | --- |
| ``cell`` (``CELL``) | any | Opaque cell reference, returned in outcomes |
| ``r``, ``c``, ``s`` | text or null | Raw closed coordinates |
| ``k:`` | text, may be categorical | One per open key (see below) |
| ``num`` (``NUM``) | ``DEC`` = ``Decimal(38, 12)`` | Numeric value |
| ``thr`` (``THR``) | ``DEC`` | Threshold, ``0.5·10^-decimals`` (0 if exact) |
| ``txt`` (``TXT``) | text | Text, canonical item or ISO date value |
| ``flag`` (``FLAG``) | Boolean | Boolean value |
Open key columns hold text, canonical items (``category:code``, see
`canonical_item`) or ISO dates.
`VALUE_COLUMNS` is ``(num, thr, txt, flag)``;
all four are null for a cell with no fact, and
`value_columns` maps one Python value to them.
A cell and open-key combination appears at most once, and a cell's
coordinates are the same on every row. String columns may be categorical,
which keeps large selections compact.
`PointsFrameProvider` builds such frames
from any `DataProvider`.
Inside the evaluator a record set is a `Frame`: key
columns ``k:`` (``k:r``, ``k:c``, ``k:s`` for varying closed axes, open
keys such as ``k:qEBB``) as canonical strings, plus the same value columns.
Operators natural-join their operands on shared keys exactly like
`combine`. Where an operand with a ``default``
lacks a record another operand has, or an ``if`` condition with a default
lacks a key a branch has, which the reference evaluator fills in with the
default, this module raises
`ColumnarUnsupported`.
## Threads
`evaluate_rule_columnar` may run
concurrently on one report from several threads (polars releases the GIL).
`FrameCache` is thread-safe and builds each
selection once while other threads asking for it wait for that build. A
provider shared across threads must be safe to read concurrently;
`PointsFrameProvider` over a
`MemoryProvider` is.
Examples:
>>> from dpmxl import MemoryProvider, Number, TableLayout
>>> from dpmxl.columnar import PointsFrameProvider, evaluate_rule_columnar
>>> layout = TableLayout("T", rows=("0010", "0020"), columns=("0010",))
>>> provider = MemoryProvider([layout], {
... ("T", "0010", "0010", None, ()): Number.of(5, 0),
... ("T", "0020", "0010", None, ()): Number.of(-2, 0),
... })
>>> result = evaluate_rule_columnar(
... "{tT, r*, c0010} >= 0", PointsFrameProvider(provider)
... )
>>> result.status, result.passed, result.failed, result.failing_keys
('failed', 1, 1, [{'r': '0020'}])
## `dpmxl.columnar.CELL`
```python
CELL = 'cell'
```
Name of the opaque cell-reference column.
## `dpmxl.columnar.DEC`
```python
DEC = pl.Decimal(38, 12)
```
The polars type of the ``num`` and ``thr`` columns: ``Decimal(38, 12)``.
## `dpmxl.columnar.FLAG`
```python
FLAG = 'flag'
```
Name of the Boolean value column.
## `dpmxl.columnar.NUM`
```python
NUM = 'num'
```
Name of the numeric value column.
## `dpmxl.columnar.THR`
```python
THR = 'thr'
```
Name of the column holding the interval threshold of ``num``.
## `dpmxl.columnar.TXT`
```python
TXT = 'txt'
```
Name of the text column: text, canonical items and ISO dates.
## `dpmxl.columnar.VALUE_COLUMNS`
```python
VALUE_COLUMNS = (NUM, THR, TXT, FLAG)
```
The four value columns, ``("num", "thr", "txt", "flag")``; all null for a
cell with no fact.
## `dpmxl.columnar.ColumnarEvaluator`
```python
class ColumnarEvaluator(provider: FrameProvider, cache: FrameCache | None = None)
```
Evaluate one expression tree on frames.
Most callers want
`evaluate_rule_columnar`, which
also applies the precondition and turns the result into counts.
``evaluate`` raises `ColumnarUnsupported`
for anything it does not implement. Use one evaluator per expression; the
`FrameCache` is what rules share.
### `dpmxl.columnar.ColumnarEvaluator.evaluate`
```python
def evaluate(node: Node, context: SelectionSpec | None = None) -> Frame
```
Evaluate a node under an inherited ``with`` context.
Returns:
Frame: The node's records.
Raises:
ColumnarUnsupported: For constructs the reference engine must run.
UnsupportedError: For constructs no engine supports.
EvaluationError: For invalid expressions or data.
## `dpmxl.columnar.ColumnarResult`
```python
class ColumnarResult(status: Status, message: str | None = None, passed: int = 0, failed: int = 0, null: int = 0, failing_keys: list[dict[str, str | None]] | None = None, failing_frame: pl.DataFrame | None = None)
```
Outcome of one rule evaluated on frames.
Unlike `RuleResult` it carries no
evaluation trace: to explain a failure, evaluate the rule with
`evaluate_rule` (``only_keys`` limits it
to the failing keys) and call `explain`.
Attributes:
status: ``passed``, ``failed``, ``not_applicable``, ``error`` or
``unsupported``, as for `RuleResult`.
message: Why the status is ``error``, ``unsupported`` or
``not_applicable``.
passed: Number of records that hold.
failed: Number of records that fail.
null: Number of records whose result is null (not evaluated).
### `dpmxl.columnar.ColumnarResult.failing_keys`
```python
failing_keys: list[dict[str, str | None]] | None = None
```
Keys of the first failing records, up to ``max_failing_keys``.
### `dpmxl.columnar.ColumnarResult.failing_frame`
```python
failing_frame: pl.DataFrame | None = None
```
Keys of every failing record, one string column per key name; ``None``
when the result has no keys.
## `dpmxl.columnar.ColumnarUnsupported`
```python
class ColumnarUnsupported(Exception)
```
The expression needs the reference evaluator.
Raised by `evaluate_rule_columnar`
and the frame helpers for anything this module does not evaluate exactly;
catch it and call `evaluate_rule`
instead.
## `dpmxl.columnar.Frame`
```python
class Frame(keys: tuple[str, ...], df: pl.DataFrame, has_default: bool = False)
```
A record set: key names plus a frame of key and value columns.
Attributes:
keys: Key names, such as ``("c",)`` or ``("qCPI",)``.
df: One row per record: a ``k:`` column per key and the value
columns ``num``, ``thr``, ``txt`` and ``flag``.
has_default: Whether the records come from a selection with a
``default`` through per-record operations, so that the reference
evaluator fills in a default for keys they lack (see
`Recordset`).
### `dpmxl.columnar.Frame.kinds`
```python
kinds: frozenset[str]
```
Value columns holding at least one non-null value (one scan).
### `dpmxl.columnar.Frame.key_columns`
```python
key_columns: list[str]
```
Frame column names of the keys.
## `dpmxl.columnar.FrameCache`
```python
class FrameCache(max_rows: int = 20000000)
```
Selection frames shared by every rule evaluated against one report.
Rules reuse the same selections, but on massive open tables a selection
can hold millions of rows, so the cache keeps at most ``max_rows`` rows
and evicts the least recently used selections beyond that.
Rules may be evaluated concurrently from several threads (polars
releases the GIL): the cache is thread-safe, and
`raw_frame` and
`selection` build each missing
entry once while other threads asking for it wait for that build.
### `dpmxl.columnar.FrameCache.get_raw`
```python
def get_raw(query: CellQuery) -> pl.DataFrame | None
```
Return a cached provider frame.
Returns:
pl.DataFrame | None: The frame, or ``None``.
### `dpmxl.columnar.FrameCache.get_frame`
```python
def get_frame(spec: SelectionSpec) -> Frame | None
```
Return a cached selection record set.
Returns:
Frame | None: The frame, or ``None``.
### `dpmxl.columnar.FrameCache.put_raw`
```python
def put_raw(query: CellQuery, frame: pl.DataFrame) -> None
```
Cache a provider frame within the row budget.
### `dpmxl.columnar.FrameCache.put_frame`
```python
def put_frame(spec: SelectionSpec, frame: Frame) -> None
```
Cache a selection record set within the row budget.
### `dpmxl.columnar.FrameCache.raw_frame`
```python
def raw_frame(query: CellQuery, build: Callable[[], pl.DataFrame]) -> pl.DataFrame
```
Return a provider frame, building and caching it once.
Returns:
pl.DataFrame: The cached or newly built frame.
### `dpmxl.columnar.FrameCache.selection`
```python
def selection(spec: SelectionSpec, build: Callable[[], Frame]) -> Frame
```
Return a selection record set, building and caching it once.
Returns:
Frame: The cached or newly built record set.
## `dpmxl.columnar.FrameProvider`
```python
class FrameProvider(Protocol)
```
Columnar report data for the columnar evaluator.
The columnar counterpart of `DataProvider`.
``select_frame`` returns one row per selected data point (closed cells
times reported open key combinations) with columns ``cell`` (an opaque
cell reference), ``r``, ``c``, ``s`` (raw coordinates or null),
``k:`` for each open key (text, items or ISO dates), and the value
columns ``num``, ``thr``, ``txt``, ``flag`` (all null when the cell was
not reported). A cell and open key combination appears at most once, and
a cell's coordinates are the same on every row. String columns may be
categorical, which keeps large selections compact. See the frame schema
in the `dpmxl.columnar` module documentation.
### `dpmxl.columnar.FrameProvider.select_frame`
```python
def select_frame(query: CellQuery) -> pl.DataFrame
```
Return the selected data points as a frame.
Raises:
LookupError: If the table does not exist in the release.
### `dpmxl.columnar.FrameProvider.table_reported`
```python
def table_reported(table: str) -> bool | None
```
Answer ``{v_TABLE}``: whether the report's filing indicators include it.
Returns ``None`` when the provider cannot tell.
### `dpmxl.columnar.FrameProvider.parameter`
```python
def parameter(name: str) -> Value
```
Return a report parameter such as ``refPeriod``, or ``None``.
## `dpmxl.columnar.PointsFrameProvider`
```python
class PointsFrameProvider(provider: Any)
```
Adapt any ``DataProvider`` to ``FrameProvider``.
Each ``select_frame`` call converts the wrapped
`DataProvider`'s points with
`frame_from_points`; the other methods
delegate. It is as thread-safe as the wrapped provider.
Examples:
>>> from dpmxl import CellQuery, MemoryProvider, Number, TableLayout
>>> layout = TableLayout("T", columns=("0010",), open_keys=("qCPI",))
>>> provider = MemoryProvider(
... [layout], {("T", None, "0010", None, (("qCPI", "A"),)): Number.of(7, 0)}
... )
>>> frame = PointsFrameProvider(provider).select_frame(CellQuery("T"))
>>> frame.columns
['cell', 'r', 'c', 's', 'k:qCPI', 'num', 'thr', 'txt', 'flag']
>>> frame.select("k:qCPI", "num", "thr").row(0)
('A', Decimal('7.000000000000'), Decimal('0.500000000000'))
### `dpmxl.columnar.PointsFrameProvider.select_frame`
```python
def select_frame(query: CellQuery) -> pl.DataFrame
```
Return the provider's points as a frame.
Returns:
pl.DataFrame: The selection.
### `dpmxl.columnar.PointsFrameProvider.table_reported`
```python
def table_reported(table: str) -> bool | None
```
Delegate to the wrapped provider.
Returns:
bool | None: Filing-indicator state.
### `dpmxl.columnar.PointsFrameProvider.parameter`
```python
def parameter(name: str) -> Value
```
Delegate to the wrapped provider.
Returns:
Value: The parameter.
## `dpmxl.columnar.canonical_item`
```python
def canonical_item(item: Item) -> str
```
Return the canonical text of an item: ``category:code`` or ``code``.
Returns:
str: The canonical form used in ``txt`` and open key columns.
Examples:
>>> from dpmxl import Item
>>> canonical_item(Item.parse("eba_qEC:qx01"))
'qEC:qx01'
## `dpmxl.columnar.canonical_key`
```python
def canonical_key(key: Mapping[str, Any]) -> dict[str, str | None]
```
Canonicalize a reference outcome key for comparison with columnar keys.
Closed coordinates are normalized with
`normalize_code`; other key values become
the text the columnar evaluator uses (items as ``category:code``).
Returns:
dict[str, str | None]: Key name to canonical string.
Examples:
>>> canonical_key({"r": "10", "qCPI": "LEI-A"})
{'r': '0010', 'qCPI': 'LEI-A'}
## `dpmxl.columnar.evaluate_rule_columnar`
```python
def evaluate_rule_columnar(expression: str | Node, provider: FrameProvider, *, precondition: str | Node | None = None, cache: FrameCache | None = None, max_failing_keys: int = 1000) -> ColumnarResult
```
Evaluate a validation rule on frames, honouring its precondition.
The columnar counterpart of `evaluate_rule`:
the same statuses and counts, and failing keys in canonical form (see
`canonical_key`). Syntax and evaluation
errors are returned as status ``error``, not raised. It may run
concurrently from several threads with a shared ``cache``.
Args:
expression: Rule expression text or a parsed tree.
provider: Columnar report data.
precondition: Optional precondition; the rule is not applicable
unless it yields at least one true record.
cache: Selections shared by the rules of one report; a new cache per
call when omitted.
max_failing_keys: How many failing keys to list in
``failing_keys``; ``failing_frame`` has them all.
Returns:
ColumnarResult: Status, record counts and the keys of failing records.
Raises:
ColumnarUnsupported: When the reference evaluator must run instead.
Examples:
Fall back to the reference evaluator when needed:
>>> from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule
>>> provider = MemoryProvider(
... [TableLayout("T", rows=("0010",), columns=("0010",))],
... {("T", "0010", "0010", None, ()): Number.of(3, 0)},
... )
>>> def run(expression):
... try:
... return evaluate_rule_columnar(
... expression, PointsFrameProvider(provider)
... ).status
... except ColumnarUnsupported:
... return evaluate_rule(expression, provider).status
>>> run("{tT, r0010, c0010} > 0")
'passed'
>>> run("year(#2026-12-31#) = 2026") # dates: reference evaluator
'passed'
## `dpmxl.columnar.frame_from_points`
```python
def frame_from_points(points: Sequence[DataPoint], open_keys: Sequence[str] = ()) -> pl.DataFrame
```
Adapt data points to the frame layout of a ``FrameProvider``.
Builds the columns described by
`FrameProvider` from
`DataPoint` objects; ``cell`` holds
``repr(point.ref)``.
Args:
points: Selected data points.
open_keys: The table's open key names, kept even without points.
Returns:
pl.DataFrame: One row per point.
Raises:
ColumnarUnsupported: For numeric or Boolean key values, items
without a category, or numbers ``Decimal(38, 12)`` does not hold.
## `dpmxl.columnar.normalize_code_expr`
```python
def normalize_code_expr(column: pl.Expr) -> pl.Expr
```
Vectorized `normalize_code`.
Returns:
pl.Expr: Numeric codes zero-padded to four digits, others upper-case.
## `dpmxl.columnar.scalar_frame`
```python
def scalar_frame(value: Value) -> Frame
```
Build a key-less one-row frame holding one value.
Returns:
Frame: The scalar, with the columns of
`value_columns`.
## `dpmxl.columnar.selected_axes`
```python
def selected_axes(query: CellQuery, row: str | None, column: str | None, sheet: str | None) -> bool
```
Return whether a cell's coordinates are selected (for providers).
Returns:
bool: Whether every axis matches.
## `dpmxl.columnar.value_columns`
```python
def value_columns(value: Value) -> dict[str, Any]
```
Split a scalar value into the four value columns.
Numbers fill ``num`` and ``thr``; Booleans ``flag``; text, items
(canonical, see `canonical_item`) and
dates (ISO) ``txt``. ``None`` leaves all four null.
Returns:
dict[str, Any]: ``num``, ``thr``, ``txt`` and ``flag`` entries.
Examples:
>>> from dpmxl import Number
>>> value_columns(Number.of("1.5", 1))
{'num': Decimal('1.5'), 'thr': Decimal('0.05'), 'txt': None, 'flag': None}
>>> value_columns(True)
{'num': None, 'thr': None, 'txt': None, 'flag': True}
Raises:
ColumnarUnsupported: For a number or threshold that ``Decimal(38, 12)``
does not hold exactly.
# `dpmxl.parser`
Tokenizer and recursive-descent parser. Nodes keep their source spans. See [Language](../language.md) for what parses.
Tokenizer and recursive-descent parser for DPM-XL expressions.
`tokenize` splits an expression into typed
`Token` objects that suit syntax highlighting as well as parsing:
selections are split into their table, axis and option tokens rather than captured
whole. `parse` builds the expression tree of
`dpmxl.ast` from those tokens.
## `dpmxl.parser.DpmXlSyntaxError`
```python
class DpmXlSyntaxError(message: str, expression: str, position: int, end: int | None = None) # bases: ValueError
```
Raised when an expression does not match the DPM-XL grammar.
Attributes:
message: What was expected or found, in plain words.
expression: The complete source expression.
position: Zero-based offset where the offending text starts.
end: Offset just past the offending text.
Examples:
>>> try:
... parse("{tA} >=")
... except DpmXlSyntaxError as error:
... print(error.message, error.position)
expression ends where an operand is expected 7
### `dpmxl.parser.DpmXlSyntaxError.pointer`
```python
def pointer() -> str
```
Render the source line and mark the failing range.
Returns:
str: Two lines suitable for terminal output: the source line and a
``^`` under the error's start, followed by ``~`` under the rest of
the offending text on that line.
Examples:
>>> try:
... parse("{tA, r0010} +* 1")
... except DpmXlSyntaxError as error:
... print(error.pointer())
{tA, r0010} +* 1
^
## `dpmxl.parser.Token`
```python
class Token(NamedTuple)
```
One lexical token: its kind, source text and half-open offsets.
``text`` is exactly ``expression[start:end]``, quotes, brackets and
backticks included. ``kind`` is one of `TOKEN_KINDS`:
* ``keyword``: ``if``, ``and``, ``with``, ``where``, ``group by``, window
keywords inside ``over (...)``, and ``default``/``interval`` in a
selection;
* ``type``: a parameter type such as ``number`` or ``set-item``;
* ``function``: a known function name followed by ``(``;
* ``operator``: ``+ - * / & % = != <> < <= > >= := <-``;
* ``punctuation``: ``( ) , : ; [ ]``;
* ``brace``: ``{`` or ``}`` around a selection or set;
* ``table``, ``table_group``, ``variable``, ``operation``, ``parameter``:
a selection reference with its prefix, such as ``tC_01.00``;
* ``axis``: a row, column or sheet coordinate, range or wildcard, such as
``r0010``, ``c0010-0050`` or ``s*``;
* ``number``, ``string``, ``date``, ``boolean``, ``null``: literals;
* ``item``: an item such as ``[eba_qEC:qx01]``;
* ``period``: a ``time_shift`` or window period code (``A``, ``Q``, ...);
* ``identifier``: a key, temporary or other name, including escaped
names such as ```x.y```;
* ``eof``: the empty token that ends every token list.
Attributes:
kind: The token kind, as above.
text: The exact source text.
start: Offset of the first character.
end: Offset just past the last character.
## `dpmxl.parser.parse`
```python
def parse(expression: str) -> Node
```
Parse a complete DPM-XL expression.
A trailing ``;`` is allowed. Several statements and assignments need
`parse_program`.
Returns:
Node: The expression tree; every node keeps its source ``span``.
Raises:
DpmXlSyntaxError: If the text is not a valid expression.
Examples:
>>> node = parse("{tT, r0010} > 0")
>>> type(node).__name__, node.op, node.span
('Binary', '>', (0, 15))
>>> node.left.spec.text()
'{tT, r0010}'
## `dpmxl.parser.parse_program`
```python
def parse_program(source: str) -> Program
```
Parse statements separated by ``;``, with assignments.
A statement is an expression (with an optional ``with`` context), a
temporary assignment ``name := expression`` or a persistent assignment
``{target} <- expression``. A trailing ``;`` is allowed.
Returns:
Program: The statements, also when there is only one.
Raises:
DpmXlSyntaxError: If the text is not a valid program.
Examples:
>>> program = parse_program("x := {tT, r0010}; x > 0")
>>> [type(statement).__name__ for statement in program.statements]
['Assignment', 'Binary']
>>> program.statements[0].name
'x'
## `dpmxl.parser.parse_selection`
```python
def parse_selection(body: str, *, expression: str = '', offset: int = 0) -> SelectionSpec
```
Parse the inside of a ``{...}`` cell selection or ``with`` context.
Args:
body: Text between the braces.
expression: Complete source for error reporting.
offset: Offset of ``body`` inside ``expression``.
Returns:
SelectionSpec: Parsed table, axes and options.
Raises:
DpmXlSyntaxError: On an unknown selection component.
Examples:
>>> spec = parse_selection("tC_01.00, r0010-0050, c*")
>>> spec.table, spec.rows.parts, spec.columns.wildcard
('C_01.00', (('0010', '0050'),), True)
## `dpmxl.parser.tokenize`
```python
def tokenize(expression: str) -> list[Token]
```
Split an expression into typed tokens.
The tokenizer follows the lexical modes of the DPM-XL grammar: inside
``{...}`` it reads selection components (table, axis and option tokens,
parameter types), elsewhere expression tokens. Whitespace is skipped;
concatenating the token texts with the source between them gives the
expression back. See `Token` for the kinds.
Returns:
list[Token]: Tokens ending with an ``eof`` token.
Raises:
DpmXlSyntaxError: On an unterminated literal, brace or item, or an
unknown character.
Examples:
>>> [(token.kind, token.text) for token in tokenize("{tT, r0010} > 0")]
... # doctest: +NORMALIZE_WHITESPACE
[('brace', '{'), ('table', 'tT'), ('punctuation', ','), ('axis', 'r0010'),
('brace', '}'), ('operator', '>'), ('number', '0'), ('eof', '')]
# `dpmxl.ast`
The expression tree: frozen node classes with source `span`s. [Language](../language.md) names the node each construct parses to.
Abstract syntax tree for DPM-XL expressions.
Every node keeps the half-open source ``span`` it was parsed from, so callers
can show the original operand text next to an evaluated value without
re-printing the expression.
## `dpmxl.ast.AXIS_KEYS`
```python
AXIS_KEYS: dict[AxisName, str] = {'rows': 'r', 'columns': 'c', 'sheets': 's'}
```
``SelectionSpec`` axis field name to the record key it produces.
## `dpmxl.ast.AXIS_PREFIXES`
```python
AXIS_PREFIXES: dict[str, AxisName] = {'r': 'rows', 'c': 'columns', 's': 'sheets'}
```
Axis letter (``r``, ``c``, ``s``) to its ``SelectionSpec`` field name.
## `dpmxl.ast.MISSING`
```python
MISSING = _Missing()
```
Marker for a ``default`` that was not written; it is falsy and differs from
a written ``default: null``.
## `dpmxl.ast.Analytic`
```python
class Analytic(name: str, operand: Node, partition_by: tuple[str, ...] = (), order_by: tuple[tuple[str, bool], ...] = (), window: Window | None = None, *, span: Span = (0, 0)) # bases: Node
```
``name(operand over (partition by … order by … window))``.
``name`` is ``rank`` or an aggregate's call name (``sum``, ``count``,
``avg``, ``max``, ``min``, ``median``). ``order_by`` pairs each key with
whether it sorts descending. The result keeps the operand's records.
Attributes:
name: ``rank`` or the aggregate's call name.
operand: The aggregated expression.
partition_by: Partition key names.
order_by: ``(key, descending)`` pairs.
window: The window clause, or ``None`` when not written.
## `dpmxl.ast.Assignment`
```python
class Assignment(name: str, value: Node, *, span: Span = (0, 0)) # bases: Node
```
``name := value``: a temporary result later statements refer to.
Attributes:
name: The temporary's name.
value: The assigned expression.
## `dpmxl.ast.AxisName`
```python
AxisName: TypeAlias = Literal['rows', 'columns', 'sheets']
```
Name of a closed axis field of `SelectionSpec`.
## `dpmxl.ast.AxisSelector`
```python
class AxisSelector(wildcard: bool = False, parts: tuple[str | tuple[str, str], ...] = (), *, span: Span = (0, 0), spans: tuple[Span, ...] = ())
```
Selected coordinates on one table axis.
``parts`` holds single codes (``"0010"``) and inclusive ranges
(``("0010", "0050")``) as written, without the axis prefix. A range may
repeat the axis letter before its upper bound: ``r0010-r0050`` is
``("0010", "0050")``. The letter is dropped only when the lower bound
does not start with it, so a code that does is read as written:
``rr1-rr10`` is ``("r1", "rr10")``; write ``rr1-r10`` for ``("r1",
"r10")``.
A parsed selector keeps its source positions; like node spans they take
no part in equality, hashing or ``repr``, and selectors built directly
have none.
Attributes:
wildcard: Whether the selector is ``*`` (every coordinate).
parts: Single codes and ``(low, high)`` ranges.
span: Source span of the whole selector, such as ``r0010-0050`` or
``(r0010, r0020)``; ``(0, 0)`` when built directly.
spans: Source span of each part, in the order of ``parts``; empty for
a wildcard or a selector built directly.
Examples:
>>> from dpmxl import parse_selection
>>> rows = parse_selection("tT, (r0010, r0020-0050)").rows
>>> rows.parts, rows.span, rows.spans
(('0010', ('0020', '0050')), (4, 23), ((5, 10), (12, 22)))
### `dpmxl.ast.AxisSelector.is_single`
```python
is_single: bool
```
Whether the selector names exactly one coordinate.
### `dpmxl.ast.AxisSelector.text`
```python
def text(prefix: str) -> str
```
Render the selector in DPM-XL syntax.
Returns:
str: For example ``r*``, ``r0010`` or ``(r0010, r0020-0050)``.
## `dpmxl.ast.Binary`
```python
class Binary(op: str, left: Node, right: Node, *, span: Span = (0, 0)) # bases: Node
```
Arithmetic, comparison or boolean operator.
Attributes:
op: The operator as written canonically: ``+ - * / & = != < <= > >=``,
``and``, ``or`` or ``xor`` (``<>`` is read as ``!=``).
left: Left operand.
right: Right operand.
## `dpmxl.ast.Call`
```python
class Call(name: str, args: tuple[Node, ...], group_by: tuple[str, ...] | None = None, *, span: Span = (0, 0)) # bases: Node
```
Function or aggregate call, optionally with ``group by`` keys.
Attributes:
name: Canonical function name (see
`canonical_function`).
args: Arguments in order.
group_by: Key names of ``group by``, or ``None`` without it.
## `dpmxl.ast.Get`
```python
class Get(operand: Node, name: str, *, span: Span = (0, 0)) # bases: Node
```
``operand [get name]`` replaces values by a key component.
Attributes:
operand: The expression.
name: Key component whose value becomes the record's value.
## `dpmxl.ast.If`
```python
class If(condition: Node, then: Node, otherwise: Node | None, *, span: Span = (0, 0)) # bases: Node
```
``if c then a [else b] endif``.
Attributes:
condition: The condition.
then: Value when the condition is true.
otherwise: Value of the ``else`` branch, or ``None`` without one.
## `dpmxl.ast.In`
```python
class In(operand: Node, values: Node, *, span: Span = (0, 0)) # bases: Node
```
``operand in set``.
``values`` is a `SetLiteral` for ``in {...}``, otherwise any
set-valued expression such as a ``set-*`` parameter or ``union(...)``.
Attributes:
operand: The tested value.
values: The set.
## `dpmxl.ast.Item`
```python
class Item(category: str | None, code: str)
```
An enumerated item literal such as ``[eba_qEC:qx01]``.
``category`` is the namespace part without the ``eba_`` prefix; it is
``None`` for a bare legacy element reference such as ``[ei86]``.
Attributes:
category: Category without ``eba_``, such as ``qEC``, or ``None``.
code: Item code, such as ``qx01``.
### `dpmxl.ast.Item.parse`
```python
def parse(text: str) -> Item
```
Parse ``prefix:code`` or a bare code.
Returns:
Item: Normalized item identity; the ``eba_`` prefix is dropped.
Examples:
>>> Item.parse("eba_qEC:qx01")
Item(category='qEC', code='qx01')
>>> Item.parse("eba_qEC:qx01").matches(Item.parse("qx01"))
True
### `dpmxl.ast.Item.matches`
```python
def matches(other: Item) -> bool
```
Compare two items, ignoring a category missing on either side.
Returns:
bool: Whether both name the same item.
## `dpmxl.ast.Literal_`
```python
class Literal_(value: Decimal | str | bool | date | Item | None, kind: Literal['number', 'string', 'boolean', 'date', 'item', 'null'], *, span: Span = (0, 0)) # bases: Node
```
A constant: number, string, boolean, date, item or null.
Attributes:
value: The constant: ``Decimal``, ``str``, ``bool``, ``date`` (or
``datetime``), `Item` or ``None``.
kind: ``number``, ``string``, ``boolean``, ``date``, ``item`` or ``null``.
## `dpmxl.ast.Name`
```python
class Name(name: str, *, span: Span = (0, 0)) # bases: Node
```
A bare identifier: a key component, period unit or parameter name.
Attributes:
name: The identifier, without backticks.
## `dpmxl.ast.Node`
```python
class Node(*, span: Span = (0, 0))
```
Base class for expression nodes.
Every node has a public ``span``: the half-open ``(start, end)`` character
offsets of its source in the parsed expression, so ``expression[start:end]``
is the node's text. A parenthesized operand's span includes the
parentheses; a clause's span runs from its operand to the closing ``]``.
Spans take no part in equality or hashing. Nodes built directly default to
``(0, 0)``.
## `dpmxl.ast.Parameter`
```python
class Parameter(name: str, *, span: Span = (0, 0)) # bases: Node
```
``{vName}``: a report-level parameter such as ``ReportingLevel``.
Attributes:
name: Parameter name without the ``v`` prefix.
## `dpmxl.ast.PersistentAssignment`
```python
class PersistentAssignment(target: Node, value: Node, *, span: Span = (0, 0)) # bases: Node
```
``{target} <- value``: a computed result for a cell or variable.
``target`` is a `Selection` (``{tT, r0010, c0010}``) or a
`Parameter` (``{vX}``, also written ``{v_X}``).
Attributes:
target: The assigned cell or variable.
value: The computed expression.
## `dpmxl.ast.Program`
```python
class Program(statements: tuple[Node, ...], *, span: Span = (0, 0)) # bases: Node
```
Statements separated by ``;``; the last one gives the result.
`parse_program` returns one;
`parse` returns a single expression instead.
Attributes:
statements: The statements in order.
## `dpmxl.ast.Rename`
```python
class Rename(operand: Node, pairs: tuple[tuple[str, str], ...], *, span: Span = (0, 0)) # bases: Node
```
``operand [rename a to b, ...]`` renames key components.
Attributes:
operand: The expression.
pairs: ``(old, new)`` key names.
## `dpmxl.ast.Selection`
```python
class Selection(spec: SelectionSpec, *, span: Span = (0, 0)) # bases: Node
```
A ``{...}`` cell selection.
Attributes:
spec: The selection's components as written (without the ``with``
context; see `references`).
## `dpmxl.ast.SelectionSpec`
```python
class SelectionSpec(table: str | None = None, rows: AxisSelector | None = None, columns: AxisSelector | None = None, sheets: AxisSelector | None = None, default: Literal_ | SetLiteral | _Missing = MISSING, interval: bool | None = None, table_group: str | None = None, operation: str | None = None, *, source_span: Span = (0, 0))
```
Source, axis and option components of a ``{...}`` selection.
The source is one of ``table`` (``tC_01.00``), ``table_group``
(``gG_01``) or ``operation`` (``o_v1234``, another operation's result);
the others are ``None``.
Attributes:
table: Table code, such as ``C_01.00``.
rows: Row selector, or ``None`` when not written.
columns: Column selector, or ``None``.
sheets: Sheet selector, or ``None``.
default: The ``default:`` literal, or ``MISSING`` when not written.
interval: The ``interval:`` option, or ``None`` when not written.
table_group: Table group code of a ``{g_...}`` selection.
operation: Operation code of an ``{o_...}`` reference.
source_span: Source span of the table, table group or operation
reference, such as ``tC_01.00``; ``(0, 0)`` when there is none or
the selection was built directly. Like the selectors' spans it
takes no part in equality, hashing or ``repr``, and
`merged_into` keeps the
span of the source it keeps.
### `dpmxl.ast.SelectionSpec.merged_into`
```python
def merged_into(context: SelectionSpec) -> SelectionSpec
```
Return this selection completed by an inherited ``with`` context.
Returns:
SelectionSpec: Components written here win over the context; a
source written here replaces the context's source.
Examples:
>>> from dpmxl import parse_selection
>>> context = parse_selection("tC_01.00, c0010, interval: true")
>>> parse_selection("r0020").merged_into(context).text()
'{tC_01.00, r0020, c0010, interval: true}'
### `dpmxl.ast.SelectionSpec.axis`
```python
def axis(name: AxisName) -> AxisSelector | None
```
Return the selector for one axis.
Returns:
AxisSelector | None: ``None`` when the axis was not written.
### `dpmxl.ast.SelectionSpec.text`
```python
def text() -> str
```
Render the selection in DPM-XL syntax.
Returns:
str: A normalized ``{...}`` selection.
Examples:
>>> from dpmxl import parse_selection
>>> parse_selection("t_C_01.00 , r10-50, default: 0").text()
'{tC_01.00, r10-50, default: 0}'
## `dpmxl.ast.SetLiteral`
```python
class SetLiteral(items: tuple[Node, ...], *, span: Span = (0, 0)) # bases: Node
```
A set literal ``{a, b, ...}`` or the empty set ``{}``.
Attributes:
items: The member expressions.
## `dpmxl.ast.Span`
```python
Span: TypeAlias = tuple[int, int]
```
Half-open ``(start, end)`` character offsets into the parsed source.
## `dpmxl.ast.Sub`
```python
class Sub(operand: Node, name: str, value: Node, *, span: Span = (0, 0)) # bases: Node
```
``operand [sub name = value]`` fixes and removes a key component.
Attributes:
operand: The expression.
name: Key component to fix.
value: The value it must have.
## `dpmxl.ast.TableReported`
```python
class TableReported(table: str, *, span: Span = (0, 0)) # bases: Node
```
``{v_TABLE}``: true when the report declares the template.
Attributes:
table: The template or table code.
## `dpmxl.ast.TypedParameter`
```python
class TypedParameter(name: str, type: str, default: Literal_ | SetLiteral | _Missing = MISSING, *, span: Span = (0, 0)) # bases: Node
```
``{p_Name, type[, default: value]}``: a typed report parameter.
``type`` is one of `PARAMETER_TYPES` in lower case.
Attributes:
name: Parameter name without the ``p_`` prefix.
type: The declared type.
default: The ``default:`` value, or ``MISSING``.
## `dpmxl.ast.Unary`
```python
class Unary(op: Literal['+', '-', 'not'], operand: Node, *, span: Span = (0, 0)) # bases: Node
```
Prefix ``+``, ``-`` or ``not``.
Attributes:
op: The operator.
operand: The operand.
## `dpmxl.ast.Where`
```python
class Where(operand: Node, condition: Node, *, span: Span = (0, 0)) # bases: Node
```
``operand [where condition]`` filters records by key components.
Attributes:
operand: The filtered expression.
condition: Condition on key components.
## `dpmxl.ast.Window`
```python
class Window(unit: Literal['data points', 'range'], start: WindowBound, end: WindowBound)
```
``data points|range between start and end``.
Attributes:
unit: ``data points`` or ``range``.
start: The lower bound.
end: The upper bound.
## `dpmxl.ast.WindowBound`
```python
class WindowBound(kind: Literal['unbounded preceding', 'preceding', 'current data point', 'following', 'unbounded following'], offset: int = 0, period: str | None = None)
```
One end of an analytic window.
``kind`` is ``unbounded preceding``, ``preceding``, ``current data
point``, ``following`` or ``unbounded following``; ``offset`` counts
data points, or for a ``range`` window units of the order key, or of
``period`` (``A``, ``S``, ``Q``, ``M``, ``W``, ``D``) for a date key.
Attributes:
kind: The bound kind, as above.
offset: Number of data points or order-key units.
period: Period code of a date offset, or ``None``.
## `dpmxl.ast.With`
```python
class With(context: SelectionSpec, body: Node, condition: Node | None = None, *, span: Span = (0, 0)) # bases: Node
```
``with {context} [where condition]: body``.
The optional condition filters every selection evaluated in the body.
Attributes:
context: Selection components inherited by every selection in the body.
body: The expression.
condition: The context's ``where`` condition, or ``None``.
## `dpmxl.ast.children`
```python
def children(node: Node) -> tuple[Node, ...]
```
Return a node's direct child expressions in source order.
A selection's or ``with`` context's ``default`` and a typed parameter's
``default`` are children too, so `walk` reaches every literal.
Returns:
tuple[Node, ...]: Child nodes; literals and other leaves have none.
Examples:
>>> from dpmxl import parse
>>> [type(child).__name__ for child in children(parse("{tT} + 1"))]
['Selection', 'Literal_']
## `dpmxl.ast.walk`
```python
def walk(node: Node) -> list[Node]
```
Return the node and all descendants in pre-order.
Returns:
list[Node]: Every node of the tree.
Examples:
>>> from dpmxl import parse
>>> expression = "with {tT, default: 0}: {r0010} > {r0020} * 1.1"
>>> [type(node).__name__ for node in walk(parse(expression))]
['With', 'Literal_', 'Binary', 'Selection', 'Binary', 'Selection', 'Literal_']
>>> [
... expression[slice(*node.span)]
... for node in walk(parse(expression))
... if type(node).__name__ == "Selection"
... ]
['{r0010}', '{r0020}']
# `dpmxl.grammar`
The vocabulary of DPM-XL as frozensets, and the TextMate grammar built from it. See [Build editor support](../../guides/editor.md).
Machine-readable vocabulary of DPM-XL for editors and other tooling.
The sets below are the words and symbols the tokenizer recognizes; the tests
check that `tokenize` agrees with them. Words are lower
case; the tokenizer matches keywords in any case.
## `dpmxl.grammar.AGGREGATE_FUNCTIONS`
```python
AGGREGATE_FUNCTIONS = frozenset({'avg', 'count', 'max_aggr', 'median', 'min_aggr', 'sum'})
```
Aggregate operators; ``max`` and ``min`` with one operand aggregate too.
## `dpmxl.grammar.ANALYTIC_FUNCTIONS`
```python
ANALYTIC_FUNCTIONS = frozenset({'rank'})
```
Operators that take only an analytic ``over (...)`` clause.
## `dpmxl.grammar.AXIS_LETTERS`
```python
AXIS_LETTERS = frozenset({'r', 'c', 's'})
```
Prefixes of row, column and sheet coordinates inside a selection.
## `dpmxl.grammar.CALL_NAMES`
```python
CALL_NAMES = frozenset(canonical_function(name) for name in FUNCTIONS - ANALYTIC_FUNCTIONS)
```
Every ``Call.name`` a known function parses to. Aggregates among them are
``{canonical_function(name) for name in AGGREGATE_FUNCTIONS}``, which equals
``dpmxl.evaluator.AGGREGATES``. Analytic functions (``rank``, and aggregates
with ``over``) parse to `Analytic` instead of a call.
## `dpmxl.grammar.FUNCTIONS`
```python
FUNCTIONS = AGGREGATE_FUNCTIONS | ANALYTIC_FUNCTIONS | frozenset({'abs', 'annualise', 'date', 'day', 'exp', 'filter', 'intersect', 'isnull', 'len', 'length', 'ln', 'log', 'match', 'max', 'min', 'month', 'nvl', 'power', 'quarter', 'semester', 'set_of', 'setdiff', 'sqrt', 'substr', 'symdiff', 'time_shift', 'union', 'week', 'year'})
```
Function names, recognized when followed by ``(``.
## `dpmxl.grammar.FUNCTION_ALIASES`
```python
FUNCTION_ALIASES = {'max_aggr': 'max', 'min_aggr': 'min'}
```
Source spellings whose ``Call.name`` differs: the aggregate forms of
``max`` and ``min``, which evaluators recognize by their single operand.
## `dpmxl.grammar.KEYWORDS`
```python
KEYWORDS = frozenset({'and', 'else', 'endif', 'get', 'group by', 'if', 'in', 'not', 'or', 'over', 'rename', 'sub', 'then', 'to', 'where', 'with', 'xor'})
```
Keywords of expressions and clauses.
## `dpmxl.grammar.LITERAL_WORDS`
```python
LITERAL_WORDS = frozenset({'false', 'null', 'true'})
```
Boolean and null literals.
## `dpmxl.grammar.OPERATORS`
```python
OPERATORS = frozenset({'!=', '%', '&', '*', '+', '-', '/', ':=', '<', '<-', '<=', '<>', '=', '>', '>='})
```
Operator symbols, including the assignments ``:=`` and ``<-``.
## `dpmxl.grammar.PARAMETER_TYPES`
```python
PARAMETER_TYPES = frozenset({'boolean', 'date', 'integer', 'item', 'number', 'set-boolean', 'set-date', 'set-integer', 'set-item', 'set-number', 'set-string', 'string'})
```
Data types of a ``{p_Name, type}`` parameter reference.
## `dpmxl.grammar.PUNCTUATION`
```python
PUNCTUATION = frozenset({'(', ')', ',', ':', ';', '[', ']'})
```
Punctuation; ``{`` and ``}`` are the separate ``brace`` token kind.
## `dpmxl.grammar.REFERENCE_PREFIXES`
```python
REFERENCE_PREFIXES = {'t': 'table', 'g': 'table_group', 'v': 'variable', 'o': 'operation', 'p': 'parameter'}
```
Selection reference prefix to token kind, such as ``tC_01.00``.
## `dpmxl.grammar.SELECTION_KEYWORDS`
```python
SELECTION_KEYWORDS = frozenset({'default', 'interval'})
```
Option keywords inside a ``{...}`` selection.
## `dpmxl.grammar.SIGNATURES`
```python
SIGNATURES: Mapping[str, Signature] = MappingProxyType(_signatures())
```
The `Signature` of every name in `CALL_NAMES`.
## `dpmxl.grammar.TIME_PERIODS`
```python
TIME_PERIODS = frozenset({'A', 'S', 'Q', 'M', 'W', 'D'})
```
Period codes of ``time_shift`` and ``range`` window bounds (upper case).
## `dpmxl.grammar.TOKEN_KINDS`
```python
TOKEN_KINDS = frozenset({'axis', 'boolean', 'brace', 'date', 'eof', 'function', 'identifier', 'item', 'keyword', 'null', 'number', 'operation', 'operator', 'parameter', 'period', 'punctuation', 'string', 'table', 'table_group', 'type', 'variable'})
```
Every ``Token.kind`` that `tokenize` produces.
## `dpmxl.grammar.WINDOW_KEYWORDS`
```python
WINDOW_KEYWORDS = frozenset({'asc', 'between', 'current data point', 'data points', 'desc', 'following', 'order by', 'partition by', 'preceding', 'range', 'unbounded'})
```
Keywords inside an analytic ``over (...)`` clause.
## `dpmxl.grammar.Signature`
```python
class Signature(minimum: int, maximum: int | None, group_by: bool = False, key_operand: int | None = None, period_operand: int | None = None)
```
How many operands a function takes, and where it takes a key name.
Both evaluators check calls against these signatures, and
`diagnose` reports the same problems before
evaluation, so the three never disagree.
Attributes:
minimum: The fewest operands.
maximum: The most operands, or ``None`` for no limit.
group_by: Whether ``group by`` may follow the (single) operand.
key_operand: Zero-based position of an operand that must be a key
name, such as ``qDATE`` in ``annualise(x, 12, qDATE)``; ``None``
when there is none.
period_operand: Zero-based position of an operand that is a period
code from `TIME_PERIODS`, such as ``Q`` in ``time_shift(x, Q, 1)``;
``None`` when there is none.
Examples:
>>> SIGNATURES["substr"].problem("substr", 4)
'substr takes 1 to 3 operands, not 4'
>>> SIGNATURES["sum"].problem("sum", 1, group_by=True) is None
True
>>> SIGNATURES["abs"].problem("abs", 1, group_by=True)
'group by is not allowed in abs'
### `dpmxl.grammar.Signature.operands`
```python
def operands() -> str
```
Describe the accepted operand count.
Returns:
str: For example ``1 operand``, ``3 or 4 operands``, ``1 to 3
operands`` or ``2 or more operands``.
### `dpmxl.grammar.Signature.problem`
```python
def problem(name: str, count: int, *, group_by: bool = False) -> str | None
```
Return what is wrong with a call's operand count, if anything.
Args:
name: The function's ``Call.name``, for the message.
count: The number of operands written.
group_by: Whether the call has a ``group by`` clause.
Returns:
str | None: A message such as ``abs takes 1 operand, not 2``, or
``None`` when the call is well formed.
### `dpmxl.grammar.Signature.key_problem`
```python
def key_problem(name: str) -> str
```
Describe a call whose key operand is not a key name.
Returns:
str: For example ``annualise needs a key name as its third
operand``.
## `dpmxl.grammar.canonical_function`
```python
def canonical_function(name: str) -> str
```
Map a function's source spelling to the ``Call.name`` the parser gives it.
Args:
name: A function name as written, in any case.
Returns:
str: The lower-case name, with ``max_aggr`` and ``min_aggr`` read as
``max`` and ``min``.
Examples:
>>> canonical_function("max_aggr"), canonical_function("SUM")
('max', 'sum')
## `dpmxl.grammar.textmate_grammar`
```python
def textmate_grammar() -> dict[str, Any]
```
Build a TextMate grammar for DPM-XL syntax highlighting.
``editors/dpm-xl.tmLanguage.json`` in the repository is this grammar;
the tests keep the two in step.
Returns:
dict[str, Any]: The grammar, ready for ``json.dump``.
Examples:
>>> textmate_grammar()["scopeName"]
'source.dpm-xl'
# `dpmxl.values`
Values and their comparison. See [Interval arithmetic](../../concepts/intervals.md).
Runtime values: numbers with interval thresholds, items, text, dates.
Numbers follow the Eurofiling interval arithmetic functions (``iaf:*``) that
EBA validation rules are based on: a number is a value with a symmetric
threshold, so it stands for the interval ``[value - t, value + t]``. Facts
reported with XBRL ``decimals`` get ``t = 0.5 * 10**-decimals`` (``INF`` is
exact); constants and defaults are exact. Sums and differences add
thresholds, products use ``|A*dB| + |B*dA| + dA*dB``, quotients take the
largest deviation of the endpoint quotients, ``abs`` and negation keep the
threshold, and ``min``/``max`` keep the chosen value's threshold. Two numbers
are equal when ``|a - b| <= ta + tb``, i.e. when their intervals overlap.
Selections evaluated with ``interval: false`` drop the threshold, so their
comparisons are exact.
Arithmetic uses `DECIMAL_CONTEXT` (34 significant
digits, as IEEE decimal128), whatever the caller's thread decimal context.
## `dpmxl.values.DECIMAL_CONTEXT`
```python
DECIMAL_CONTEXT = Context(prec=34, rounding=ROUND_HALF_EVEN, Emax=6144, Emin=-6143, traps=[DivisionByZero, InvalidOperation, Overflow])
```
The decimal context of every operation on numbers (IEEE decimal128).
## `dpmxl.values.Number`
```python
class Number(value: Decimal, tolerance: Decimal = ZERO)
```
A decimal value with a symmetric tolerance for interval arithmetic.
A number stands for the interval
``[value - tolerance, value + tolerance]``. Build reported facts with
`Number.of`, which derives the tolerance from
XBRL ``decimals``. Arithmetic operators
(``+``, ``-``, ``*``, ``/``, unary ``-`` and ``abs``) follow the interval
rules in the module documentation.
Attributes:
value: The amount.
tolerance: Half-width of the interval; 0 for an exact number.
Examples:
>>> Number.of(2, 0) + Number.of(3, 1)
Number(value=Decimal('5'), tolerance=Decimal('0.55'))
### `dpmxl.values.Number.of`
```python
def of(value: int | str | Decimal, decimals: int | str | None = None) -> Number
```
Build a number, deriving tolerance from XBRL ``decimals``.
Args:
value: The amount.
decimals: XBRL ``decimals``: an integer, ``"INF"`` (in any case)
or ``None``, which both mean exact.
Returns:
Number: The value with ``0.5 * 10**-decimals`` tolerance, or an
exact value.
Raises:
ValueError: If ``decimals`` is text other than ``INF`` or an
integer, or puts the tolerance outside the range of
`DECIMAL_CONTEXT` (beyond about ±6,100).
Examples:
>>> Number.of(1204, 0)
Number(value=Decimal('1204'), tolerance=Decimal('0.5'))
>>> Number.of("250.4", 1).tolerance, Number.of(1000, -3).tolerance
(Decimal('0.05'), Decimal('5E+2'))
>>> Number.of(5, "INF").tolerance
Decimal('0')
### `dpmxl.values.Number.exact`
```python
def exact() -> Number
```
Return the same value without tolerance.
Returns:
Number: An exact copy.
### `dpmxl.values.Number.low`
```python
low: Decimal
```
Lower interval bound, ``value - tolerance``.
Examples:
>>> Number.of(1204, 0).low, Number.of(1204, 0).high
(Decimal('1203.5'), Decimal('1204.5'))
### `dpmxl.values.Number.high`
```python
high: Decimal
```
Upper interval bound, ``value + tolerance``.
## `dpmxl.values.Value`
```python
Value: TypeAlias = Number | Item | str | bool | date | ValueSet | None
```
A runtime value: `Number`, `Item`,
text, Boolean, date (or date-time), `ValueSet` or
``None`` (null).
## `dpmxl.values.ValueSet`
```python
class ValueSet(members: tuple[Value, ...] = ())
```
A set of distinct scalar values.
Set literals, ``set-*`` parameters and set operators yield sets. Members are kept in
first-seen order; numbers are exact, so ``1`` and ``1.0`` are one member. Build sets
with `ValueSet.of`, which removes duplicates by DPM-XL
equality.
### `dpmxl.values.ValueSet.members`
```python
members: tuple[Value, ...] = ()
```
The members in first-seen order.
### `dpmxl.values.ValueSet.of`
```python
def of(values: Iterable[Value]) -> ValueSet
```
Collect distinct non-null values.
Returns:
ValueSet: The set, in first-seen order.
Raises:
TypeError: If a value is itself a set.
Examples:
>>> values = ValueSet.of([Number.of(1), Number.of("1.0"), None, "a"])
>>> len(values), Number.of(1) in values, "b" in values
(2, True, False)
## `dpmxl.values.compare`
```python
def compare(op: str, left: Value, right: Value) -> bool | None
```
Compare two scalar values with DPM-XL semantics.
Numbers use interval comparison: ``a = b`` holds when the tolerance
intervals overlap. Items compare by category and code, ignoring a missing
category. Null on either side yields null.
Returns:
bool | None: The comparison result, or ``None`` for null operands.
Raises:
TypeError: When the operands cannot be compared.
ValueError: When ``op`` is not a comparison operator.
Examples:
>>> compare("=", Number.of(1204, 0), Number.of(1205, 0)) # overlap
True
>>> compare("=", Number.of(1204, 0), Number.of(1206, 0))
False
>>> compare("<", None, Number.of(1)) is None
True
>>> compare("=", Item.parse("eba_qEC:qx01"), Item.parse("qx01"))
True
## `dpmxl.values.format_computed`
```python
def format_computed(value: Decimal) -> str
```
Format a computed decimal, rounding a long fraction for display.
A division yields up to 34 significant digits. Beyond ten decimals, the
number shows six significant digits, but keeps every integer digit and
two decimals:
``1.852935352358765...`` reads ``1.85294`` and ``12345.67890123456``
``12,345.68``. Shorter fractions, such as the exact difference of two
reported values, are shown in full, as reported values always are.
Returns:
str: The formatted number.
Examples:
>>> format_computed(Decimal("1.852935352358765123"))
'1.85294'
>>> format_computed(Decimal("12345.67890123456"))
'12,345.68'
## `dpmxl.values.format_decimal`
```python
def format_decimal(value: Decimal) -> str
```
Format a decimal with thousands separators and no exponent.
Returns:
str: For example ``1,204`` or ``-0.075``.
Examples:
>>> format_decimal(Decimal("1234567.8900"))
'1,234,567.89'
## `dpmxl.values.format_value`
```python
def format_value(value: Value) -> str
```
Format a value for an explanation.
Numbers use thousands separators and drop trailing zeros; items keep their
``category:code`` form in brackets.
Returns:
str: Human-readable text.
Examples:
>>> from datetime import date
>>> format_value(Number.of("1234567.50")), format_value(Item.parse("a:b"))
('1,234,567.5', '[a:b]')
>>> format_value(date(2026, 12, 31)), format_value(True), format_value(None)
('2026-12-31', 'true', 'null')
>>> format_value("text")
'"text"'
## `dpmxl.values.type_name`
```python
def type_name(value: Value) -> str
```
Return a readable type name for error messages.
Returns:
str: For example ``number`` or ``item``.
Examples:
>>> type_name(Number.of(1)), type_name("x"), type_name(None)
('number', 'string', 'null')
# `dpmxl.recordset`
Keyed record sets and the natural join every operator uses. See [Records and keys](../../concepts/records-and-keys.md).
Keyed record sets, the value domain of DPM-XL operators.
A selection evaluates to records keyed by its varying components: closed axes
that select more than one coordinate (``r``, ``c``, ``s``) and open key
components such as ``qEBB``. Operators combine operands with a natural join on
shared keys, so ``{r0010} = {r0020}`` under ``with {c*}`` compares column by
column, and a scalar broadcasts to every record. An operand with a
`default` stands for that default where it lacks a
record another operand has.
## `dpmxl.recordset.STANDARD_KEYS`
```python
STANDARD_KEYS = frozenset({'r', 'c', 's'})
```
Keys of the closed axes: rows, columns and sheets.
## `dpmxl.recordset.EvaluationError`
```python
class EvaluationError(Exception)
```
Raised when an expression cannot be evaluated against the data.
For example a type mismatch, an unknown table or incompatible keys.
`evaluate_rule` catches it and returns
status ``error`` with the message.
## `dpmxl.recordset.Record`
```python
class Record(value: Value, points: tuple[Any, ...] = ())
```
One keyed value and the data points it was computed from.
``points`` may nest the operands' point tuples instead of copying them: joins over
large open tables would otherwise copy every record's growing point list at each
operator. Use `unique_points` to flatten.
Attributes:
value: The record's value.
points: The data points it was computed from, possibly nested.
## `dpmxl.recordset.Recordset`
```python
class Recordset(keys: tuple[str, ...], records: dict[tuple[Hashable, ...], Record] = dict(), default: Record | None = None)
```
Records indexed by a tuple of key values aligned with ``keys``.
A selection with a ``default`` keeps it as ``default``: the record of every
key the set lacks. `combine` then fills in the
default where another operand has a record, and
`lookup` returns it for a missing key.
Per-record operations carry it along (see
`map`), as do ``where``, ``sub``,
``filter``, ``rename``, a ``time_shift`` of a date key and an ``if``
whose condition has one; ``get``, aggregates and analytic functions drop
it.
Attributes:
keys: Key names, such as ``("r", "qCPI")``; empty for a scalar.
records: Records by key values, aligned with ``keys``.
default: The record of a key the set lacks, or ``None`` when a
missing key has no record.
### `dpmxl.recordset.Recordset.scalar`
```python
def scalar(value: Value, points: tuple[DataPoint, ...] = ()) -> Recordset
```
Build a key-less record set holding one value.
Returns:
Recordset: A scalar.
### `dpmxl.recordset.Recordset.is_scalar`
```python
is_scalar: bool
```
Whether the set has no key components.
### `dpmxl.recordset.Recordset.single`
```python
def single() -> Record | None
```
Return the only record of a scalar.
Returns:
Record | None: The record, or ``None`` for an empty set.
Raises:
EvaluationError: If the set is keyed.
### `dpmxl.recordset.Recordset.lookup`
```python
def lookup(binding: Mapping[str, Hashable]) -> list[Record]
```
Return records consistent with a partial key binding.
Returns:
list[Record]: Matching records (one when all keys are bound; the
``default`` when the set lacks that key and has one).
### `dpmxl.recordset.Recordset.map`
```python
def map(function: Callable[[Value], Value]) -> Recordset
```
Apply a function to every value, the ``default`` included.
Returns:
Recordset: Same keys, transformed values.
## `dpmxl.recordset.UnsupportedError`
```python
class UnsupportedError(EvaluationError)
```
Raised for valid DPM-XL that this evaluator cannot resolve.
Raised when a rule needs an optional provider method (``time_shift``,
``{o_...}``, ``{g_...}``) that the provider lacks or that returns
``None``, and for a function name the evaluator does not know.
`evaluate_rule` catches it and returns
status ``unsupported``.
## `dpmxl.recordset.check_structure`
```python
def check_structure(operands: Sequence[Recordset]) -> None
```
Enforce DPM-XL 2.2.3.1 for operators over several record sets.
Keyed operands must have exactly the same keys, or one operand's keys
(the reference) must be a superset of every other's. Standard keys
(``r``, ``c``, ``s``) shared with the reference must contain the same
values: ``{tT1, r010-020} + {tT1, r030-040}`` is an error, not empty.
Raises:
EvaluationError: When the structures are incompatible.
## `dpmxl.recordset.combine`
```python
def combine(operands: Sequence[Recordset], function: Callable[[list[Value]], Value]) -> Recordset
```
Natural-join operands on shared keys and combine their values.
Records without a partner on a shared key are dropped, unless the operand
lacking the partner has a ``default``: it then takes part with its default,
as a fact variable with a fallback value does. When every keyed operand
has a default, the result has a record for each key any operand with all
the keys has, and its own default combines theirs. Operands without any
shared key form a cross product, which is only allowed when at most one
of them is keyed (scalar broadcast) to avoid accidental explosions.
Returns:
Recordset: Keys in first-seen order across operands.
Raises:
EvaluationError: On incompatible key structures or oversized joins.
Examples:
>>> def numbers(values, default=None):
... records = {(key,): Record(value) for key, value in values}
... return Recordset(("qKEY",), records, default)
>>> def total(values):
... return sum(values)
>>> left = numbers([("a", 1), ("b", 2)])
>>> right = numbers([("b", 3), ("c", 4)], Record(0))
>>> def values(result):
... return {key: record.value for key, record in result.records.items()}
>>> values(combine([left, right], total))
{('a',): 1, ('b',): 5}
>>> left.default = Record(0)
>>> values(combine([left, right], total))
{('a',): 1, ('b',): 5, ('c',): 4}
## `dpmxl.recordset.iter_points`
```python
def iter_points(points: Iterable[Any]) -> Iterator[DataPoint]
```
Flatten nested point tuples (see `Record`) in order.
Yields:
DataPoint: Each point, duplicates included.
## `dpmxl.recordset.resolve_key`
```python
def resolve_key(name: str, keys: Iterable[str]) -> str | None
```
Resolve a key name written in a rule against available key components.
Rules refer to DPM 2.0 key properties with or without their ``q`` prefix
(``qEBB`` and ``EBB``) and in any case.
Returns:
str | None: The matching key, or ``None`` when absent or ambiguous.
Examples:
>>> resolve_key("EBB", ["qEBB", "r"]), resolve_key("qebb", ["qEBB"])
('qEBB', 'qEBB')
>>> resolve_key("CPI", ["r"]) is None
True
## `dpmxl.recordset.unique_points`
```python
def unique_points(points: Iterable[Any]) -> tuple[DataPoint, ...]
```
Flatten and deduplicate data points by provider reference, keeping order.
Returns:
tuple[DataPoint, ...]: Distinct points.
# `dpmxl.references`
Static analysis of what a rule reads. See [Analyse rules statically](../../guides/analyse.md).
Static analysis: the selections and templates an expression refers to.
## `dpmxl.references.Reference`
```python
class Reference(spec: SelectionSpec, span: tuple[int, int])
```
A selection completed by its inherited ``with`` context.
``spec.table`` is the table code of a cell selection; for a table group
(``{g_Group, ...}``) or operation (``{o_Code, ...}``) reference it is
``None`` and `kind` and
`code` say what is referenced. Typed
parameters (``{p_Name, ...}``) are not selections and are not listed.
Attributes:
spec: The selection with its ``with`` context applied.
span: The selection's source span.
### `dpmxl.references.Reference.kind`
```python
kind: Literal['table', 'table_group', 'operation']
```
What the selection reads: a table, a table group or an operation.
### `dpmxl.references.Reference.code`
```python
code: str | None
```
The table, table group or operation code.
## `dpmxl.references.references`
```python
def references(node: Node) -> tuple[Reference, ...]
```
Return every selection with its inherited context applied.
Table-group and operation references are included with ``spec.table`` set to
``None``: check `Reference.kind` and read the
group or operation code from `Reference.code`. A
group read reads every table the provider's ``table_group(code)`` lists;
`tables` expands groups that way when given the provider.
Targets of persistent assignments are included too.
Returns:
tuple[Reference, ...]: Selections in source order.
Examples:
>>> from dpmxl import parse
>>> node = parse("with {tC_01.00, c0010}: {r0010} = {r0020} + {g_G1, r0010}")
>>> [(ref.spec.text(), ref.kind, ref.code) for ref in references(node)]
... # doctest: +NORMALIZE_WHITESPACE
[('{tC_01.00, r0010, c0010}', 'table', 'C_01.00'),
('{tC_01.00, r0020, c0010}', 'table', 'C_01.00'),
('{gG1, r0010, c0010}', 'table_group', 'G1')]
## `dpmxl.references.reported_templates`
```python
def reported_templates(node: Node) -> tuple[str, ...]
```
Return templates tested with ``{v_...}`` filing-indicator checks.
Returns:
tuple[str, ...]: Template codes in first-seen order.
Examples:
>>> from dpmxl import parse
>>> reported_templates(parse("{v_C_01.00} and not {v_C_02.00}"))
('C_01.00', 'C_02.00')
## `dpmxl.references.tables`
```python
def tables(node: Node, provider: object | None = None) -> tuple[str, ...]
```
Return the distinct table codes referenced by cell selections.
Args:
node: The expression tree.
provider: Optional provider whose ``table_group(code)`` method
resolves table groups; each group read then counts as reading
all its tables. Without it, groups are left out.
Returns:
tuple[str, ...]: Table codes in first-seen order.
Examples:
>>> from dpmxl import MemoryProvider, parse
>>> node = parse("{tC_01.00, r0010} = {g_G1, r0010}")
>>> tables(node)
('C_01.00',)
>>> tables(node, MemoryProvider([], {}, table_groups={"G1": ("C_05.01",)}))
('C_01.00', 'C_05.01')
# `dpmxl.diagnostics`
Static checks for editors. See [Build editor support](../../guides/editor.md#find-every-problem).
Static diagnostics: positioned problems in an expression, for editors.
`diagnose` reports what can be found without
report data. On its own it finds every syntax error, unknown functions,
calls with the wrong operands and selections without a table. Given table
layouts it also checks tables, rows, columns and sheets, open-table keys
and, when the layouts carry cell data types, operands of the wrong type.
Given the report parameters, it also reports names that resolve to nothing.
## `dpmxl.diagnostics.DIAGNOSTIC_CODES`
```python
DIAGNOSTIC_CODES = frozenset({'syntax', 'unknown-function', 'operand-count', 'group-by', 'key-operand', 'missing-table', 'unknown-table', 'unknown-row', 'unknown-column', 'unknown-sheet', 'missing-row', 'missing-column', 'missing-sheet', 'open-row', 'open-column', 'open-sheet', 'empty-range', 'type-mismatch', 'unknown-key', 'unknown-name'})
```
Every `Diagnostic.code`. Codes are stable;
messages may be reworded in minor releases.
## `dpmxl.diagnostics.Diagnostic`
```python
class Diagnostic(severity: Severity, code: str, message: str, start: int, end: int, subject: str | None = None, suggestion: str | None = None)
```
One problem in an expression, with the source text it concerns.
Attributes:
severity: ``error`` when evaluation would fail or select nothing as
written; ``warning`` when the expression evaluates but probably
does not mean what it says.
code: A stable identifier from
`DIAGNOSTIC_CODES`:
* ``syntax``: the text does not parse;
* ``unknown-function``; ``operand-count`` and ``group-by``: a
call with the wrong number of operands or a ``group by`` the
function does not take; ``key-operand``: no key name where
``time_shift`` or ``annualise`` needs one;
* ``missing-table``: a selection without a table;
* ``unknown-table``: a table that does not exist;
* ``unknown-row``, ``unknown-column`` and ``unknown-sheet``: a
coordinate the table does not have, an
error when no part of the selector matches and a warning when
another part does; ``empty-range``: the same for a range;
* ``missing-row``, ``missing-column`` and ``missing-sheet``: a
code on an axis the table does not have; ``open-row``,
``open-column`` and ``open-sheet``: a code on an open axis,
whose rows, columns or sheets have keys instead of codes. What
a provider selects for such a code is up to it (a
`MemoryProvider` ignores the
selector), so these are warnings, and they are not reported
for a selector a shared ``with`` context applies;
* ``type-mismatch``: an operand of the wrong data type;
* ``unknown-key``: a name in a clause that is not a key of what
the clause applies to (a warning where a report parameter may
stand in and the parameters are not known); ``unknown-name``: a
name that is no report parameter or earlier assignment.
message: What is wrong, in plain words.
start: Zero-based offset of the first character concerned.
end: Offset just past the last character; equals ``start`` for a
problem at a position, such as the end of the expression.
subject: What the problem is about, when it names something: the
function, table, coordinate, key or name as written, or the data
type found; ``None`` otherwise.
suggestion: A replacement for ``expression[start:end]`` that likely
fixes the problem, such as ``sum`` for ``summ``; ``None`` when
there is no likely fix.
### `dpmxl.diagnostics.Diagnostic.pointer`
```python
def pointer(expression: str) -> str
```
Render the source line and mark the text concerned.
Args:
expression: The expression the diagnostic was found in.
Returns:
str: Two lines for terminal output, as
`DpmXlSyntaxError.pointer`.
Examples:
>>> source = "summ({tC_01.00, r0010}) > 0"
>>> print(diagnose(source)[0].pointer(source))
summ({tC_01.00, r0010}) > 0
^~~~
## `dpmxl.diagnostics.LayoutLookup`
```python
LayoutLookup = Callable[[str], TableLayout | None]
```
Returns a table's `TableLayout`, or ``None``
for a table that does not exist, such as ``layouts.get`` for a dict.
## `dpmxl.diagnostics.Severity`
```python
Severity = Literal['error', 'warning']
```
``error`` or ``warning``; see `Diagnostic`.
## `dpmxl.diagnostics.diagnose`
```python
def diagnose(expression: str, layout: LayoutLookup | None = None, *, tables: Iterable[str] | None = None, parameters: Iterable[str] | None = None, program: bool = False) -> tuple[Diagnostic, ...]
```
Return the problems found in an expression without evaluating it.
Every syntax error is reported: after one, parsing resumes where the
enclosing construct can carry on, and an error caused by an earlier one
is left out. A construct that contains a syntax error is not checked
further. Selections inside ``with`` are checked with their context, and
a problem in the context is reported once, at the context.
Args:
expression: A DPM-XL rule expression, or a script with
``program=True``.
layout: Looks up a table's `TableLayout`,
or returns ``None`` for a table that does not exist. Without it,
tables, coordinates, keys and data types are not checked. Table
groups and operations are never checked.
tables: Every table code, to suggest one for an unknown table.
parameters: Every report parameter name. With them, a name that is
neither a parameter, nor assigned by an earlier statement, nor a
key in a clause is an error; without them names are not checked
outside clauses.
program: Accept a script (``;``, ``:=`` and ``<-``), as
`parse_program` does; otherwise the
text must be one expression, as for
`parse`.
Returns:
tuple[Diagnostic, ...]: Problems in source order; empty when none
were found.
Examples:
>>> for d in diagnose("summ({tC_01.00, r0010}) > 0 and"):
... print(d.start, d.end, d.code, d.message)
0 4 unknown-function unknown function 'summ'; did you mean 'sum'?
31 31 syntax expression ends where an operand is expected
>>> from dpmxl import TableLayout
>>> layouts = {"C_01.00": TableLayout("C_01.00", rows=("0010", "0020"))}
>>> for d in diagnose("{tC_01.00, r0011} = {tC_02.00, r0010}", layouts.get):
... print(d.start, d.end, d.message, d.suggestion)
11 16 row 0011 is not in table C_01.00; did you mean r0010? r0010
21 29 unknown table C_02.00 None
# Using dpm-xl with AI agents
This page is for AI coding agents (and the people prompting them) writing
code that *uses* dpm-xl. It holds the essential API, the contracts that
matter and the mistakes to avoid. Agents working *on* the dpm-xl repository
should read
[`AGENTS.md`](https://github.com/luukhopman/dpm-xl/blob/main/AGENTS.md)
instead.
Machine-readable documentation, following [llmstxt.org](https://llmstxt.org):
- [`llms.txt`](https://luukhopman.github.io/dpm-xl/llms.txt): an index of
every page with a one-line description.
- [`llms-full.txt`](https://luukhopman.github.io/dpm-xl/llms-full.txt): every
page, API reference included, as one Markdown file.
- Each page as Markdown: replace the trailing `/` of a page URL with `.md`,
for example
[`guides/provider.md`](https://luukhopman.github.io/dpm-xl/guides/provider.md).
## The essential API
```python
from dpmxl import (
DataPoint,
CellQuery,
MemoryProvider,
TableLayout,
Number, # data
evaluate_rule,
explain,
SelectionCache, # evaluation
parse,
parse_program,
DpmXlSyntaxError, # parsing
references,
tables, # analysis
axis_matcher, # providers
)
```
| Call | Returns | Raises |
| --- | --- | --- |
| `evaluate_rule(expression, provider, *, precondition=None, trace=True, cache=None, only_keys=None)` | `RuleResult` with `status`, `message`, `outcomes`, `failures`, `counts`, `evaluation`, `assignments` | Nothing for bad rules or data: those are statuses |
| `explain(result.evaluation, expression, outcome.key)` | `Explanation` with `summary`, `filled`, `difference`, `gap`, `direction`, `left`, `right`, `primary`, `kind` | |
| `parse(expression)` / `parse_program(source)` | Expression tree / `Program` | `DpmXlSyntaxError` (`message`, `position`, `end`, `pointer()`) |
| `references(tree)`, `tables(tree, provider=None)` | Selections with context applied; table codes | |
| `dpmxl.columnar.evaluate_rule_columnar(expression, frame_provider, *, precondition=None, cache=None)` | `ColumnarResult` with `status`, `passed`, `failed`, `null`, `failing_keys`, `failing_frame` | `ColumnarUnsupported`: evaluate with `evaluate_rule` instead |
Statuses: `passed`, `failed`, `not_applicable`, `error`, `unsupported`.
## A complete, correct example
```python
from dpmxl import (
MemoryProvider,
Number,
SelectionCache,
TableLayout,
evaluate_rule,
explain,
)
provider = MemoryProvider(
[TableLayout("C_01.00", rows=("0010", "0020", "0030"), columns=("0010",))],
{
("C_01.00", "0010", "0010", None, ()): Number.of(1204, 0),
("C_01.00", "0020", "0010", None, ()): Number.of(1180, 0),
("C_01.00", "0030", "0010", None, ()): Number.of(31, 0),
},
)
rules = [
(
"v1",
"with {tC_01.00, c0010, interval: true}: {r0010} = {r0020} + {r0030}",
"{v_C_01.00}",
)
]
cache = SelectionCache()
for code, expression, precondition in rules:
result = evaluate_rule(expression, provider, precondition=precondition, cache=cache)
print(code, result.status, result.message or "")
for outcome in result.failures:
print(" ", explain(result.evaluation, expression, outcome.key).summary)
```
```text
v1 failed
1,204 should equal 1,211 (1,180 + 31): difference 7, at least 5.5 given the reported precision
```
## Contracts
- **Providers** implement `select(query) -> Sequence[DataPoint]`,
`table_reported(table) -> bool | None` and `parameter(name) -> Value`;
optionally `open_keys`, `table_group`, `operation`, `earlier_report`
([Connect your data](guides/provider.md)).
- **`select` returns every defined cell of a closed selection**, with
`value=None` for unreported ones, and one point per reported key
combination for open tables (keys in `DataPoint.keys`). It raises
`LookupError` for an unknown table.
- **Numbers** are `Number.of(value, decimals)` with the fact's XBRL
`decimals` (`None` or `"INF"` for exact). Pass strings or `Decimal`, not
floats, for exact values.
- **`MemoryProvider` value keys** are `(table, row, column, sheet, keys)`;
`keys` is a tuple of `(name, value)` pairs for open tables and `()` for
closed ones. Absent coordinates are `None`.
- **Thread safety:** `evaluate_rule` and `evaluate_rule_columnar` may run
concurrently on one report; `parse_cached`, `SelectionCache`, `FrameCache`
and `MemoryProvider` may be shared ([Run rules in threads](guides/threads.md)).
## Common pitfalls
| Mistake | Consequence | Do instead |
| --- | --- | --- |
| Returning only reported cells from `select` for a closed table | `default` and `isnull` never apply; presence checks pass wrongly | Return every defined cell with `value=None` |
| `Number.of(0.1, 2)` with a float | Binary float noise in the value | `Number.of("0.1", 2)` |
| Treating `not_applicable` or null records as failures | False alarms | Only `status == "failed"` / `outcome.passed is False` are failures |
| Expecting `evaluate_rule` to raise on a bad rule | Errors go unnoticed | Check `result.status in {"error", "unsupported"}` and `result.message` |
| Calling `explain` on a `trace=False` result or a `ColumnarResult` | Wrong or no explanation | Re-run `evaluate_rule` with the default `trace=True` for failing rules |
| Not catching `ColumnarUnsupported` | Rules with scripts, `time_shift`, sets and more crash | Catch it and call `evaluate_rule` |
| `parse` on a script with `;`, `:=` or `<-` | `DpmXlSyntaxError` | `parse_program` |
| Writing `a<-1` to compare with -1 | Parsed as a persistent assignment | `a < -1` |
| `{vX}` for a filing indicator | Reads parameter `X` | `{v_X}` tests the filing indicator |
| Keywords run together, such as `ornot` | One unknown name: a syntax error | Separate words: `or not` |
| A new `SelectionCache` per rule | Every selection is read again | One cache per report |
## Where to look next
- [Quick reference](reference/quick-reference.md): the language and its
grammar on one page.
- [Language](reference/language.md): every construct with its syntax,
types, null behaviour, errors and tested examples.
- [Concepts](concepts/index.md): records and keys, null logic, intervals.
- [Python API](reference/api/index.md): every public name.
# Changelog
All notable changes to this project are documented here. The project follows
[semantic versioning](https://semver.org).
## 1.2.1 - 2026-10-25
### Fixed
- `date()` with a day, month or year that does not exist raises `invalid
date: 2026-02-30` on every supported Python version; the message used to
repeat Python's own wording, which differs between versions.
## 1.2.0 - 2026-10-25
### Added
- The language reference documents every construct with its syntax, types,
null and interval behaviour, errors and examples whose results the tests
compute, and a new quick reference has the language and its grammar on
one page. The tests check that the reference covers `dpmxl.grammar`.
- `Recordset.default`: the record of a key a record set lacks, from a
selection's `default`. `Recordset.map` carries it and `Recordset.lookup`
returns it for a missing key.
- `Frame.has_default`: whether a columnar frame comes from a selection with
a `default`.
### Fixed
- An operand with a `default` stands for its default where another operand
has a record it lacks, as a fact variable's fallback value does, instead
of dropping that record from the join. `with {default: null}: if not
isnull({tA, c0010}) then not isnull({tB, c0010}) endif` over two parts of
an open table now fails for a key filed only in `A`; it was not evaluated.
Operands without a default still decide the keys and join on the keys
they share. An `if` condition with a default also covers keys only a
branch has. `where`, `sub`, `filter`, a `with` context's `where`,
`rename`, a `time_shift` of a date key and `if` results keep the default,
so a filtered operand falls back to it for keys the other operands bring;
`get`, aggregates and analytic functions do not. The columnar evaluator
leaves these cases to the reference evaluator.
- `[rename a to b]` on a record set that already has a key `b` is an error;
it produced two keys of the same name.
- A `with` context's `where` applies only to the selections that have the
keys it names. A body selection without them, key-less ones included,
was an error ("unknown key component" or "where needs a keyed operand").
## 1.1.0 - 2026-10-25
### Added
- `diagnose()` finds problems without report data, as positioned
`Diagnostic`s with a stable `code`, the `subject` concerned and a
`suggestion` to replace the text with: every syntax error (parsing
resumes after one), keywords run together (`ornot`), unknown functions,
wrong operand counts and selections without a table. Given table layouts
it also reports unknown tables, rows, columns and sheets, empty ranges,
clause names that are not keys and, with cell data types, operands of the
wrong type; given the report parameters, names that resolve to nothing.
An error means the rule fails, is inapplicable or cannot be evaluated as
written; a code on an open axis or one the table lacks, whose outcome is
up to the provider, is a warning.
`Diagnostic.pointer()` renders one; `DIAGNOSTIC_CODES` lists the codes.
- `dpmxl.grammar.SIGNATURES`: each function's `Signature` (operand counts,
`group by`, key and period operands), which both evaluators and
`diagnose()` check calls against.
- `TableLayout.open_axes` and `TableLayout.data_types` describe open axes
and cell data types (`dpmxl.provider.CELL_TYPES`; it may be partial); data
files accept them.
- Parsed selections keep source positions: `AxisSelector.span` and `spans`,
and `SelectionSpec.source_span`.
- `dpm-xl check` reports every problem `diagnose()` finds, also as
`diagnostics` in `--json`. `--layouts DATA` checks against a data file's
tables and parameters; `--strict` fails on any error, not only syntax
errors.
### Changed
- Calls with the wrong operands fail with one wording in both evaluators,
such as `abs takes 1 operand, not 2`, before their operands are evaluated.
- `dpm-xl check --json` lists every syntax error, so `errors` may hold
several entries for one expression: count failing expressions with
`total - parsed`, not `len(errors)`.
- The errors for `max_aggr` and `min_aggr` with several operands and for
`rank` without `over` point at the function name; an unknown selection
component's error spans the whole component.
### Fixed
- A range that repeats its axis letter, such as `r0440-r0480`, selected every
coordinate from its lower bound up; it now selects what `r0440-0480` does.
The letter is dropped only when the lower bound does not start with it, so
`rr1-rr10` reads as written (see `AxisSelector`).
## 1.0.0 - 2026-10-24
### Added
- Python 3.15 support, tested in CI on 3.11 to 3.15 with and without extras.
- Documentation site at https://luukhopman.github.io/dpm-xl/, with
`llms.txt` and `llms-full.txt`; every example in it is tested.
- `examples/rules.txt` and `examples/xbrl-report/`, a small xBRL-CSV report
for the end-to-end guide.
- `dpm-xl eval` takes a rule file as well as one expression and summarizes
every rule; `check` and `eval` accept `--json`; `--version` and examples in
`--help`.
- Runnable examples in `examples/`, run by the tests.
- `dpmxl.__version__`; `--json` output carries `schema_version`.
- The public API is documented: every name in a module's `__all__`
(including the `dpmxl.ast` tree and `VALUE_COLUMNS`), the columnar frame
schema and which caches are thread-safe.
- `tokenize()` and `Token(kind, text, start, end)` are public, with kinds
for syntax highlighting (keyword, function, operator, brace, table and axis
references, literal kinds, item, identifier, punctuation and more); every
AST node's source `span` is documented.
- `DpmXlSyntaxError.end`, and `pointer()` underlines the offending text.
- Text concatenation `&`, `substr`, `log`, the date functions `year`,
`semester`, `quarter`, `month`, `week`, `day` and `date(y, m, d)`, and date
literals with a time part (`#2026-12-31T10:00:00#`).
- Sets: literals such as `{1, 2}` and `{}`, `set_of`, `union`, `intersect`,
`setdiff` and `symdiff`, `in` with any set operand and `default: {…}`;
the `ValueSet` value type.
- Typed parameters `{p_Name, type[, default: v]}`, operation references
`{o_Code, …}` and table groups `{g_Group, …}`, resolved through optional
`operation(code)` and `table_group(code)` provider methods (and
`MemoryProvider`'s `operations` and `table_groups`, also in the CLI data
file); `Reference.kind` and `Reference.code`, and `tables(node, provider)`
expands table groups.
- Statements separated by `;` with temporary (`name := e`) and persistent
(`{target} <- e`) assignments: `parse_program`, the `Program`,
`Assignment` and `PersistentAssignment` nodes, and
`RuleResult.assignments`. `evaluate_rule`, the columnar fallback and
`dpm-xl check` accept programs; `parse` still returns one expression and
allows a trailing `;`.
- `time_shift` evaluates its operand on another period's report from the
optional `earlier_report(period, count)` provider method (or
`MemoryProvider(earlier_reports=...)`, or `earlier` in the CLI data file),
or moves a date key; its points carry `provider.EarlierRef` refs, and
explanations show the earlier report's values;
`annualise(x, fiscalYearEnd, key)`.
- Analytic functions: `rank` and aggregates with `over (partition by …
order by … window)`, parsed to the `Analytic` node with `Window` and
`WindowBound`.
- `tests/test_grammar_coverage.py`: an example for every alternative of the
DPM-XL grammar, each parsed and, where it yields a value, evaluated.
- `dpmxl.grammar`: keywords, operators, function names and parameter types as
frozensets (`AXIS_LETTERS` for `r`, `c`, `s`), `canonical_function` and
`CALL_NAMES` for the `Call.name` of each spelling, and a TextMate grammar in
`editors/dpm-xl.tmLanguage.json`.
### Changed
- **Breaking:** `dpm-xl` exits with status 3, not 2, for an evaluation error
or unreadable input, so 2 means a usage error only.
- The tokenizer splits selections into tokens; `Token.text` is the exact
source text (quotes, brackets and backticks included) and `Token` is a
named tuple. Error positions inside selections point at the component.
- `{}` is the empty set rather than an empty selection, and `In.values` may
be any set-valued node, not only a `SetLiteral`; `Value` includes
`ValueSet`.
- `time_shift` is no longer always `unsupported`: with a provider that has
`earlier_report` it is evaluated, and `MemoryProvider` without the
requested report reports it with a new message. The columnar evaluator
hands every `time_shift` rule to the reference evaluator (raising
`ColumnarUnsupported`) instead of reporting `unsupported` itself.
- A `_` after the `t`, `g`, `o` and `p` prefixes is cosmetic, as the
DPM-XL grammar says: `{t_T1}` selects table `T1`, not `_T1`. `{v_X}`
stays a filing-indicator test. `SelectionSpec` has
`table_group` and `operation` fields.
- `children()` and `walk()` reach the `default` of a selection, a `with`
context and a typed parameter.
- `with`, `to`, `where`, `get`, `sub`, `rename` and `over` are reserved
words; escape a key of that name as `` `to` ``. `a<-1` is now a persistent
assignment; write `a < -1` for the comparison.
- `Number.of` raises `ValueError` for `decimals` beyond the decimal128 range
(about -6,100 to 6,100, such as -7000), which 0.1.1 accepted; callers that
pass report-level decimals should catch it. The CLI reports it.
- Syntax errors read in plain words, such as "expression ends where an
operand is expected" and "expected a name, found ')'".
- A `where` or `sub` on a selection inside a `with` body overrides only the
context's `where` terms on the same key; the other terms still apply.
- Expressions nested more than 50 deep (chained clauses included), and
numbers outside the decimal128 range, are syntax errors; `4.5%%` is no
longer read as 0.00045.
- Explanations round the remaining gap down, so "at least" never overstates
it, and keep only the parentheses the rule writes.
### Fixed
- `dpm-xl check` and `eval` report a rule file path that does not exist
instead of reading it as an expression; `check --json` errors carry `end`.
- A division by zero or type error fails only its own record: a rule is an
error only if that record reaches the result, so guards such as
`if {x} != 0 then {y} / {x} > 1 endif` work.
- `evaluate_rule` reports deep recursion, decimal overflow and type errors in
`[sub]` as status `error` instead of raising.
- Arithmetic no longer depends on the caller's decimal context.
- Explanations show the right values after `[rename]`.
- Explanations of values around 1e33 and larger with `interval: true` no
longer raise.
- `if` without `else` treats filing indicators (`{v_…}`) and Boolean facts as
Boolean, as for other conditions.
- Items built with an `eba_` category match parsed items.
- The columnar evaluator hands every case it cannot match exactly to the
reference evaluator: any polars failure, values outside `Decimal(38, 12)`,
numeric open keys, ordering items and text that may be an item.
- The CLI accepts `"decimals": "INF"` and reports malformed data without a
traceback.
- The columnar evaluator hands record-level type errors (comparing text with
numbers, `and`/`not`/`abs` on the wrong type, mixed `max`/`min` groups) to
the reference evaluator, which fails only the records concerned.
- A non-Boolean condition of a keyed `if` fails only its records.
- The columnar evaluator no longer orders items without a category as text;
item literals in `<`, `<=`, `>`, `>=` and such items in the data go to the
reference evaluator, which reports the error.
### Performance
- `if` conditions whose branches add keys look records up by key instead of
scanning every branch record; repeated `where` verdicts are reused.
## 0.1.1 - 2026-10-23
### Performance
- Faster explanations and selections when many rules run against large
reports.
## 0.1.0 - 2026-10-23
First release on PyPI.
# Stability and versioning
dpm-xl follows [semantic versioning](https://semver.org). From 1.0.0, a
change that breaks the public API needs a new major version.
## The public API
- Every name in the `__all__` of the modules in the
[Python API](../reference/api/index.md), including the expression tree in
`dpmxl.ast` (frozen node classes with source `span`s, `walk` and
`children`) for callers that analyse rules.
- The [columnar frame schema](../reference/api/columnar.md#dpmxl.columnar--frame-schema)
that a `FrameProvider` returns.
- The thread-safety guarantees in [Run rules in threads](../guides/threads.md).
- The CLI: its commands, options, [exit statuses](../reference/cli.md#exit-statuses)
and the `--json` layout, versioned by `schema_version` (1), and the
[data formats](../reference/data-formats.md) it reads.
Names starting with an underscore are internal, as are modules whose names
start with one. `dpmxl.__version__` gives the installed version.
## What may change in a minor release
- The wording of explanations (`summary`, `filled` and the other text
fields), of syntax error messages and of `RuleResult.message`.
- New constructs, functions, token kinds and optional provider methods.
- Constructs the columnar evaluator handles itself instead of raising
`ColumnarUnsupported`.
- Bug fixes that change a result to what the specification and these
docs say it should be. They are listed under "Fixed" in the
[changelog](changelog.md).
Statuses, record keys, field names, `Explanation.kind` values and the JSON
field names do not change outside a major release.
## Supported Python versions
dpm-xl supports the Python versions listed in its package metadata,
currently 3.11 to 3.15.
# Benchmarks
Three Python packages parse DPM-XL. pydpm-xl and dpmcore validate rules
against a DPM database; dpm-xl evaluates rules against report data and
explains failures. The table shows where they overlap and where they differ.
## Capabilities
| | dpm-xl | pydpm-xl | dpmcore |
| --- | --- | --- | --- |
| Licence | MIT | GPL-3.0-or-later | Apache-2.0 |
| Runtime dependencies | none | 8 (pandas, SQLAlchemy, database drivers) | antlr4, SQLAlchemy |
| Parses every EBA 3.4–4.4 rule | yes | yes | yes |
| Syntax errors | plain words with position | ANTLR messages | ANTLR messages |
| Evaluates rules against report data | yes | no | no |
| Explains failing records | yes | no | no |
| Columnar evaluation for millions of rows | yes | no | no |
| CLI to evaluate rules on a data file | yes | no | no |
| Tokenizer and editor grammar | yes | no | no |
| *Outside dpm-xl's scope:* | | | |
| Semantic validation against a DPM release | no | yes | yes |
| DPM database migration, REST API | no | migration | yes |
| Reads report files | via [dpm-xbrl](https://luukhopman.github.io/dpm-xbrl/) | no | no |
Use pydpm-xl or dpmcore to check that a rule fits a DPM release; use dpm-xl
to check that a report satisfies the rules, and why not. The two combine.
## Results
The corpus holds the 14,174 distinct rule and precondition expressions of
EBA DPM releases 3.4 to 4.4. All three packages reject the same two, which
contain typos in the EBA source.
Measured on 2026-10-24 on AMD Ryzen 7 4700U with Radeon Graphics, 8 cores, Linux 7.0.0-38-generic; Python 3.12.3. Best of 3 warm runs; peak RSS over those runs.
#### Speed and footprint
| | dpm-xl 1.0.0 | pydpm-xl 0.3.12 | dpmcore 0.2.1 |
| --- | --- | --- | --- |
| Syntax check of the corpus (whole process) | 1.61 s, 23 MB | 14.85 s, 109 MB | 15.23 s, 113 MB |
| Per expression | 113 µs | 1,047 µs | 1,075 µs |
| `import` (whole process) | 0.09 s, 18 MB | 0.75 s, 97 MB | 0.85 s, 101 MB |
| Installed environment | 1 distributions, 1 MB | 17 distributions, 142 MB | 15 distributions, 143 MB |
#### Coverage of the EBA corpus
| | dpm-xl 1.0.0 | pydpm-xl 0.3.12 | dpmcore 0.2.1 |
| --- | --- | --- | --- |
| Expressions accepted | 14,172 of 14,174 | 14,172 of 14,174 | 14,172 of 14,174 |
| Rejected | `with {default: 0, interval: true}:if sum({tC_08.01.a, r0070…`
`with {tV_03.01, default: null, interval: false}: if not(isn…` | `with {default: 0, interval: true}:if sum({tC_08.01.a, r0070…`
`with {tV_03.01, default: null, interval: false}: if not(isn…` | `with {default: 0, interval: true}:if sum({tC_08.01.a, r0070…`
`with {tV_03.01, default: null, interval: false}: if not(isn…` |
#### DPM-XL constructs beyond the EBA rules
| | dpm-xl 1.0.0 | pydpm-xl 0.3.12 | dpmcore 0.2.1 |
| --- | --- | --- | --- |
| rank over a partition | accepted | rejected | accepted |
| windowed aggregate | accepted | rejected | accepted |
| statements with := | accepted | rejected | rejected |
| persistent assignment <- | accepted | rejected | accepted |
| operation reference {o...} | accepted | accepted | accepted |
| set operators and literals | accepted | rejected | accepted |
| typed parameter {p...} | accepted | rejected | accepted |
| text concatenation & | accepted | accepted | accepted |
| date functions | accepted | rejected | accepted |
| date() and date literals | accepted | rejected | accepted |
| table group {g...} | accepted | accepted | accepted |
| annualise | accepted | rejected | accepted |
#### Syntax error messages
`{tC_01.00, r0010, c0010} >=`
- dpm-xl: expression ends where an operand is expected at position 27
- pydpm-xl: offendingSymbol: [@11,27:26='',<-1>,1:27] msg: mismatched input '' expecting {BOOLEAN_LITERAL, NOT,…
- dpmcore: Unexpected error: "offendingSymbol: [@11,27:26='',<-1>,1:27] msg: mismatched input '' expecting {BO…
`with {tC_01.00, c0010}: {r0010} = {r0020} +`
- dpm-xl: expression ends where an operand is expected at position 43
- pydpm-xl: offendingSymbol: [@21,43:42='',<-1>,1:43] msg: mismatched input '' expecting {BOOLEAN_LITERAL, NOT,…
- dpmcore: Unexpected error: "offendingSymbol: [@21,43:42='',<-1>,1:43] msg: mismatched input '' expecting {BO…
## Method
Each package runs in its own virtual environment from PyPI (dpm-xl from this
repository), as a separate process through its public syntax-check API: one
warm-up, then the best of three runs, with peak memory. Ratios between the
packages are more stable than absolute times. Scripts and instructions are in
[`benchmarks/`](https://github.com/luukhopman/dpm-xl/tree/main/benchmarks).
# Security
## Reporting a vulnerability
Report vulnerabilities privately through
[GitHub security advisories](https://github.com/luukhopman/dpm-xl/security/advisories/new),
not in a public issue.
## Threat model
dpm-xl parses and evaluates expressions; it reads no files, opens no network
connections and never executes rule text as Python. What it does with
untrusted input:
| Input | Treatment |
| --- | --- |
| Expression text | Parsed by a hand-written recursive-descent parser. Nesting deeper than 50 levels and numbers outside the decimal128 range are syntax errors, so a hostile expression cannot exhaust the stack. |
| Report data | Values are typed data (`Number`, `str`, dates, items, sets); nothing is evaluated. Joins that would produce more than 2,000,000 records are an error instead of a memory spike. |
| `decimals` | `Number.of` rejects decimals beyond the decimal128 range, which would otherwise make tolerances overflow. |
| CLI input files | Read as JSON or text; malformed input is reported with exit status 3, without a traceback. |
What it does not protect against:
- **Regular expressions.** `match(s, pattern)` runs the rule's pattern with
Python's `re` module. A pattern written to backtrack catastrophically can
take very long on a long text. Treat rule text as code from a trusted
source, or evaluate rules from untrusted sources with a time limit (for
example in a separate process).
- **Rule cost.** Rules over very large selections take time and memory in
proportion to the data. Set limits in your application when users can
submit rules.
- **Providers.** Your provider runs inside the evaluation; it must protect
its own data source (SQL parameters, access control).
For untrusted report *files*, see the security model of
[dpm-xbrl](https://luukhopman.github.io/dpm-xbrl/), which reads them.
# Contributing
Issues and pull requests are welcome at
[github.com/luukhopman/dpm-xl](https://github.com/luukhopman/dpm-xl).
[`AGENTS.md`](https://github.com/luukhopman/dpm-xl/blob/main/AGENTS.md) in
the repository has the same conventions in a form for coding agents.
## Set up
dpm-xl uses [uv](https://docs.astral.sh/uv/):
```bash
git clone https://github.com/luukhopman/dpm-xl
cd dpm-xl
uv sync
```
`uv sync` installs the package with the test, lint and docs tools.
## Checks
Every change must keep these green:
```bash
uv run pytest
uv run ruff check . && uv run ruff format --check . && uv run ty check
uv run python scripts/build_docs.py
```
The docs build is strict: a broken link, an unresolved API reference or a
missing page fails it. `uv run zensical serve` previews the site while you
write.
## Conventions
- **Tests for every behaviour.** `tests/test_columnar.py` checks that both
evaluators agree case by case; add a case there when you touch either.
`tests/test_grammar_coverage.py` has an example for every alternative of
the DPM-XL grammar, and `tests/data/eba_expressions.txt` samples about 300
EBA rule expressions; every line must parse.
- **No new runtime dependencies.** The package has none; polars is the only
optional one.
- **Google-style docstrings** for every public name, with an `Examples:`
section where it helps. They feed the API reference and are run as
doctests.
- **Keep `__all__` and the docs in step.** A new public name goes in its
module's `__all__`, gets a docstring, and appears in the guide or concept
page it belongs to.
- **Docs examples run.** Every `python` and `console` block in `docs/` is
executed by `tests/test_docs.py`, and a `text` block right after a
`python` block must match its output exactly. Every `dpm-xl` block must
parse, and every row of an `| Expression | Result |` table is evaluated
against `examples/language_report.py`; see that file for the
conventions.
- **Language reference.** A new keyword, function or other name in
`dpmxl.grammar` needs an entry and an index row in
`docs/reference/language.md`, and an example that uses it; the tests
fail otherwise.
- **Grammar changes.** After changing `dpmxl.grammar`, regenerate
`editors/dpm-xl.tmLanguage.json`:
```bash
uv run python -c "import json; from dpmxl.grammar import textmate_grammar; print(json.dumps(textmate_grammar(), indent=2))" > editors/dpm-xl.tmLanguage.json
```
- **Commits** follow [Conventional Commits](https://www.conventionalcommits.org)
(`fix(evaluator): …`, `docs: …`), in the imperative, under 72 characters.
- **Changelog.** Add user-visible changes to `CHANGELOG.md` under the next
version, one short bullet per change.
## Releases
1. Update the version in `pyproject.toml` (`uv version X.Y.Z`) and the
changelog heading, and commit as `build: release X.Y.Z`.
2. Tag `vX.Y.Z` and publish a GitHub release. The `Publish` workflow tests,
builds and uploads to PyPI through trusted publishing.
3. The docs site deploys from `main` on every push.
# Licence
dpm-xl is released under the MIT licence. dpm-xl is an independent project,
not affiliated with or endorsed by the European Banking Authority.
```text
MIT License
Copyright (c) 2026 Luuk Hopman
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```