Skip to content

dpmxl.columnar

The columnar evaluator (the columnar extra). The module docstring below defines the frame schema a FrameProvider returns and the thread-safety guarantees. See Evaluate large reports.

dpmxl.columnar

Vectorized DPM-XL evaluation on polars data frames.

Needs the columnar extra (pip install "dpm-xl[columnar]"). The record-by-record evaluator (dpmxl.evaluator) is the reference implementation; this module evaluates the same language with the same semantics on columnar data, so open tables with millions of rows evaluate in milliseconds per rule.

Anything this module does not implement raises ColumnarUnsupported, and callers of evaluate_rule_columnar then fall back to the reference evaluator, so coverage never trades off correctness. That covers statements and assignments, set operators such as union, analytic functions, the & operator, functions other than the aggregates, filter, abs, isnull, nvl, match, len and min/max (so time_shift, annualise, substr, the mathematical and the date functions), typed parameter, operation and table-group references, values that do not fit Decimal(38, 12), arithmetic that would overflow it, any record error (a zero divisor or a type error, which the reference evaluator confines to its record), ordering items and text that may be an item, and a default standing in for a record another operand has.

Values are Decimal(38, 12): sums and differences are exact, products and quotients round to 12 decimals (thresholds of quotients round up), where the reference evaluator keeps 34 significant digits, so an exact comparison that hinges on the 13th decimal can differ.

Frame schema

A FrameProvider returns one row per selected data point (closed cells times reported open key combinations) with these columns:

Column Type Content
cell (CELL) any Opaque cell reference, returned in outcomes
r, c, s text or null Raw closed coordinates
k:<code> text, may be categorical One per open key (see below)
num (NUM) DEC = Decimal(38, 12) Numeric value
thr (THR) DEC Threshold, 0.5·10^-decimals (0 if exact)
txt (TXT) text Text, canonical item or ISO date value
flag (FLAG) Boolean Boolean value

Open key columns hold text, canonical items (category:code, see canonical_item) or ISO dates.

VALUE_COLUMNS is (num, thr, txt, flag); all four are null for a cell with no fact, and value_columns maps one Python value to them. A cell and open-key combination appears at most once, and a cell's coordinates are the same on every row. String columns may be categorical, which keeps large selections compact. PointsFrameProvider builds such frames from any DataProvider.

Inside the evaluator a record set is a Frame: key columns k:<name> (k:r, k:c, k:s for varying closed axes, open keys such as k:qEBB) as canonical strings, plus the same value columns. Operators natural-join their operands on shared keys exactly like combine. Where an operand with a default lacks a record another operand has, or an if condition with a default lacks a key a branch has, which the reference evaluator fills in with the default, this module raises ColumnarUnsupported.

Threads

evaluate_rule_columnar may run concurrently on one report from several threads (polars releases the GIL). FrameCache is thread-safe and builds each selection once while other threads asking for it wait for that build. A provider shared across threads must be safe to read concurrently; PointsFrameProvider over a MemoryProvider is.

Examples:

>>> from dpmxl import MemoryProvider, Number, TableLayout
>>> from dpmxl.columnar import PointsFrameProvider, evaluate_rule_columnar
>>> layout = TableLayout("T", rows=("0010", "0020"), columns=("0010",))
>>> provider = MemoryProvider([layout], {
...     ("T", "0010", "0010", None, ()): Number.of(5, 0),
...     ("T", "0020", "0010", None, ()): Number.of(-2, 0),
... })
>>> result = evaluate_rule_columnar(
...     "{tT, r*, c0010} >= 0", PointsFrameProvider(provider)
... )
>>> result.status, result.passed, result.failed, result.failing_keys
('failed', 1, 1, [{'r': '0020'}])

DEC module-attribute

DEC = pl.Decimal(38, 12)

The polars type of the num and thr columns: Decimal(38, 12).

NUM module-attribute

NUM = 'num'

Name of the numeric value column.

THR module-attribute

THR = 'thr'

Name of the column holding the interval threshold of num.

TXT module-attribute

TXT = 'txt'

Name of the text column: text, canonical items and ISO dates.

FLAG module-attribute

FLAG = 'flag'

Name of the Boolean value column.

VALUE_COLUMNS module-attribute

VALUE_COLUMNS = (NUM, THR, TXT, FLAG)

The four value columns, ("num", "thr", "txt", "flag"); all null for a cell with no fact.

CELL module-attribute

CELL = 'cell'

Name of the opaque cell-reference column.

ColumnarUnsupported

Bases: Exception

The expression needs the reference evaluator.

Raised by evaluate_rule_columnar and the frame helpers for anything this module does not evaluate exactly; catch it and call evaluate_rule instead.

FrameProvider

Bases: Protocol

Columnar report data for the columnar evaluator.

The columnar counterpart of DataProvider. select_frame returns one row per selected data point (closed cells times reported open key combinations) with columns cell (an opaque cell reference), r, c, s (raw coordinates or null), k:<code> for each open key (text, items or ISO dates), and the value columns num, thr, txt, flag (all null when the cell was not reported). A cell and open key combination appears at most once, and a cell's coordinates are the same on every row. String columns may be categorical, which keeps large selections compact. See the frame schema in the dpmxl.columnar module documentation.

select_frame

select_frame(query: CellQuery) -> DataFrame

Return the selected data points as a frame.

Raises:

  • LookupError –

    If the table does not exist in the release.

table_reported

table_reported(table: str) -> bool | None

Answer {v_TABLE}: whether the report's filing indicators include it.

Returns None when the provider cannot tell.

parameter

parameter(name: str) -> Value

Return a report parameter such as refPeriod, or None.

Frame dataclass

Frame(
    keys: tuple[str, ...],
    df: DataFrame,
    has_default: bool = False,
)

A record set: key names plus a frame of key and value columns.

Attributes:

  • keys (tuple[str, ...]) –

    Key names, such as ("c",) or ("qCPI",).

  • df (DataFrame) –

    One row per record: a k:<name> column per key and the value columns num, thr, txt and flag.

  • has_default (bool) –

    Whether the records come from a selection with a default through per-record operations, so that the reference evaluator fills in a default for keys they lack (see Recordset).

keys instance-attribute

keys: tuple[str, ...]

df instance-attribute

df: DataFrame

has_default class-attribute instance-attribute

has_default: bool = False

kinds cached property

kinds: frozenset[str]

Value columns holding at least one non-null value (one scan).

key_columns property

key_columns: list[str]

Frame column names of the keys.

PointsFrameProvider

PointsFrameProvider(provider: Any)

Adapt any DataProvider to FrameProvider.

Each select_frame call converts the wrapped DataProvider's points with frame_from_points; the other methods delegate. It is as thread-safe as the wrapped provider.

Examples:

>>> from dpmxl import CellQuery, MemoryProvider, Number, TableLayout
>>> layout = TableLayout("T", columns=("0010",), open_keys=("qCPI",))
>>> provider = MemoryProvider(
...     [layout], {("T", None, "0010", None, (("qCPI", "A"),)): Number.of(7, 0)}
... )
>>> frame = PointsFrameProvider(provider).select_frame(CellQuery("T"))
>>> frame.columns
['cell', 'r', 'c', 's', 'k:qCPI', 'num', 'thr', 'txt', 'flag']
>>> frame.select("k:qCPI", "num", "thr").row(0)
('A', Decimal('7.000000000000'), Decimal('0.500000000000'))

Wrap a record provider.

Parameters:

provider instance-attribute

provider = provider

select_frame

select_frame(query: CellQuery) -> DataFrame

Return the provider's points as a frame.

Returns:

  • DataFrame –

    pl.DataFrame: The selection.

table_reported

table_reported(table: str) -> bool | None

Delegate to the wrapped provider.

Returns:

  • bool | None –

    bool | None: Filing-indicator state.

parameter

parameter(name: str) -> Value

Delegate to the wrapped provider.

Returns:

  • Value ( Value ) –

    The parameter.

FrameCache

FrameCache(max_rows: int = 20000000)

Selection frames shared by every rule evaluated against one report.

Rules reuse the same selections, but on massive open tables a selection can hold millions of rows, so the cache keeps at most max_rows rows and evicts the least recently used selections beyond that.

Rules may be evaluated concurrently from several threads (polars releases the GIL): the cache is thread-safe, and raw_frame and selection build each missing entry once while other threads asking for it wait for that build.

Create an empty cache.

Parameters:

  • max_rows (int, default: 20000000 ) –

    Total rows kept across cached selection frames.

max_rows instance-attribute

max_rows = max_rows

raw instance-attribute

raw: OrderedDict[CellQuery, DataFrame] = OrderedDict()

frames instance-attribute

frames: OrderedDict[SelectionSpec, Frame] = OrderedDict()

get_raw

get_raw(query: CellQuery) -> DataFrame | None

Return a cached provider frame.

Returns:

  • DataFrame | None –

    pl.DataFrame | None: The frame, or None.

get_frame

get_frame(spec: SelectionSpec) -> Frame | None

Return a cached selection record set.

Returns:

  • Frame | None –

    Frame | None: The frame, or None.

put_raw

put_raw(query: CellQuery, frame: DataFrame) -> None

Cache a provider frame within the row budget.

put_frame

put_frame(spec: SelectionSpec, frame: Frame) -> None

Cache a selection record set within the row budget.

raw_frame

raw_frame(
    query: CellQuery, build: Callable[[], DataFrame]
) -> DataFrame

Return a provider frame, building and caching it once.

Returns:

  • DataFrame –

    pl.DataFrame: The cached or newly built frame.

selection

selection(
    spec: SelectionSpec, build: Callable[[], Frame]
) -> Frame

Return a selection record set, building and caching it once.

Returns:

  • Frame ( Frame ) –

    The cached or newly built record set.

ColumnarEvaluator

ColumnarEvaluator(
    provider: FrameProvider, cache: FrameCache | None = None
)

Evaluate one expression tree on frames.

Most callers want evaluate_rule_columnar, which also applies the precondition and turns the result into counts. evaluate raises ColumnarUnsupported for anything it does not implement. Use one evaluator per expression; the FrameCache is what rules share.

Bind the provider.

Parameters:

  • provider (FrameProvider) –

    Columnar report data.

  • cache (FrameCache | None, default: None ) –

    Selections shared across rules of one report.

provider instance-attribute

provider = provider

cache instance-attribute

cache = cache or FrameCache()

evaluate

evaluate(
    node: Node, context: SelectionSpec | None = None
) -> Frame

Evaluate a node under an inherited with context.

Returns:

  • Frame ( Frame ) –

    The node's records.

Raises:

ColumnarResult dataclass

ColumnarResult(
    status: Status,
    message: str | None = None,
    passed: int = 0,
    failed: int = 0,
    null: int = 0,
    failing_keys: list[dict[str, str | None]] | None = None,
    failing_frame: DataFrame | None = None,
)

Outcome of one rule evaluated on frames.

Unlike RuleResult it carries no evaluation trace: to explain a failure, evaluate the rule with evaluate_rule (only_keys limits it to the failing keys) and call explain.

Attributes:

  • status (Status) –

    passed, failed, not_applicable, error or unsupported, as for RuleResult.

  • message (str | None) –

    Why the status is error, unsupported or not_applicable.

  • passed (int) –

    Number of records that hold.

  • failed (int) –

    Number of records that fail.

  • null (int) –

    Number of records whose result is null (not evaluated).

status instance-attribute

status: Status

message class-attribute instance-attribute

message: str | None = None

passed class-attribute instance-attribute

passed: int = 0

failed class-attribute instance-attribute

failed: int = 0

null class-attribute instance-attribute

null: int = 0

failing_keys class-attribute instance-attribute

failing_keys: list[dict[str, str | None]] | None = None

Keys of the first failing records, up to max_failing_keys.

failing_frame class-attribute instance-attribute

failing_frame: DataFrame | None = None

Keys of every failing record, one string column per key name; None when the result has no keys.

canonical_item

canonical_item(item: Item) -> str

Return the canonical text of an item: category:code or code.

Returns:

  • str ( str ) –

    The canonical form used in txt and open key columns.

Examples:

>>> from dpmxl import Item
>>> canonical_item(Item.parse("eba_qEC:qx01"))
'qEC:qx01'

value_columns

value_columns(value: Value) -> dict[str, Any]

Split a scalar value into the four value columns.

Numbers fill num and thr; Booleans flag; text, items (canonical, see canonical_item) and dates (ISO) txt. None leaves all four null.

Returns:

  • dict[str, Any] –

    dict[str, Any]: num, thr, txt and flag entries.

Examples:

>>> from dpmxl import Number
>>> value_columns(Number.of("1.5", 1))
{'num': Decimal('1.5'), 'thr': Decimal('0.05'), 'txt': None, 'flag': None}
>>> value_columns(True)
{'num': None, 'thr': None, 'txt': None, 'flag': True}

Raises:

  • ColumnarUnsupported –

    For a number or threshold that Decimal(38, 12) does not hold exactly.

scalar_frame

scalar_frame(value: Value) -> Frame

Build a key-less one-row frame holding one value.

Returns:

frame_from_points

frame_from_points(
    points: Sequence[DataPoint],
    open_keys: Sequence[str] = (),
) -> DataFrame

Adapt data points to the frame layout of a FrameProvider.

Builds the columns described by FrameProvider from DataPoint objects; cell holds repr(point.ref).

Parameters:

  • points (Sequence[DataPoint]) –

    Selected data points.

  • open_keys (Sequence[str], default: () ) –

    The table's open key names, kept even without points.

Returns:

  • DataFrame –

    pl.DataFrame: One row per point.

Raises:

  • ColumnarUnsupported –

    For numeric or Boolean key values, items without a category, or numbers Decimal(38, 12) does not hold.

normalize_code_expr

normalize_code_expr(column: Expr) -> Expr

Vectorized normalize_code.

Returns:

  • Expr –

    pl.Expr: Numeric codes zero-padded to four digits, others upper-case.

evaluate_rule_columnar

evaluate_rule_columnar(
    expression: str | Node,
    provider: FrameProvider,
    *,
    precondition: str | Node | None = None,
    cache: FrameCache | None = None,
    max_failing_keys: int = 1000,
) -> ColumnarResult

Evaluate a validation rule on frames, honouring its precondition.

The columnar counterpart of evaluate_rule: the same statuses and counts, and failing keys in canonical form (see canonical_key). Syntax and evaluation errors are returned as status error, not raised. It may run concurrently from several threads with a shared cache.

Parameters:

  • expression (str | Node) –

    Rule expression text or a parsed tree.

  • provider (FrameProvider) –

    Columnar report data.

  • precondition (str | Node | None, default: None ) –

    Optional precondition; the rule is not applicable unless it yields at least one true record.

  • cache (FrameCache | None, default: None ) –

    Selections shared by the rules of one report; a new cache per call when omitted.

  • max_failing_keys (int, default: 1000 ) –

    How many failing keys to list in failing_keys; failing_frame has them all.

Returns:

  • ColumnarResult ( ColumnarResult ) –

    Status, record counts and the keys of failing records.

Raises:

Examples:

Fall back to the reference evaluator when needed:

>>> from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule
>>> provider = MemoryProvider(
...     [TableLayout("T", rows=("0010",), columns=("0010",))],
...     {("T", "0010", "0010", None, ()): Number.of(3, 0)},
... )
>>> def run(expression):
...     try:
...         return evaluate_rule_columnar(
...             expression, PointsFrameProvider(provider)
...         ).status
...     except ColumnarUnsupported:
...         return evaluate_rule(expression, provider).status
>>> run("{tT, r0010, c0010} > 0")
'passed'
>>> run("year(#2026-12-31#) = 2026")  # dates: reference evaluator
'passed'

canonical_key

canonical_key(
    key: Mapping[str, Any],
) -> dict[str, str | None]

Canonicalize a reference outcome key for comparison with columnar keys.

Closed coordinates are normalized with normalize_code; other key values become the text the columnar evaluator uses (items as category:code).

Returns:

  • dict[str, str | None] –

    dict[str, str | None]: Key name to canonical string.

Examples:

>>> canonical_key({"r": "10", "qCPI": "LEI-A"})
{'r': '0010', 'qCPI': 'LEI-A'}

selected_axes

selected_axes(
    query: CellQuery,
    row: str | None,
    column: str | None,
    sheet: str | None,
) -> bool

Return whether a cell's coordinates are selected (for providers).

Returns:

  • bool ( bool ) –

    Whether every axis matches.