dpmxl.columnar¶
The columnar evaluator (the columnar extra). The module docstring below defines the frame schema a FrameProvider returns and the thread-safety guarantees. See Evaluate large reports.
dpmxl.columnar
¶
Vectorized DPM-XL evaluation on polars data frames.
Needs the columnar extra (pip install "dpm-xl[columnar]"). The
record-by-record evaluator (dpmxl.evaluator) is the
reference implementation; this module evaluates the same language with the
same semantics on columnar data, so open tables with millions of rows evaluate
in milliseconds per rule.
Anything this module does not implement raises
ColumnarUnsupported, and callers of
evaluate_rule_columnar then fall
back to the reference evaluator, so coverage never trades off correctness.
That covers statements and assignments, set operators such as union,
analytic functions, the & operator, functions other than the aggregates,
filter, abs, isnull, nvl, match, len and
min/max (so time_shift, annualise, substr, the
mathematical and the date functions), typed parameter, operation and
table-group references, values that do not fit
Decimal(38, 12), arithmetic that would overflow it, any record error (a
zero divisor or a type error, which the reference evaluator confines to its
record), ordering items and text that may be an item, and a default
standing in for a record another operand has.
Values are Decimal(38, 12): sums and differences are exact, products and
quotients round to 12 decimals (thresholds of quotients round up), where the
reference evaluator keeps 34 significant digits, so an exact comparison that
hinges on the 13th decimal can differ.
Frame schema¶
A FrameProvider returns one row per selected
data point (closed cells times reported open key combinations) with these
columns:
| Column | Type | Content |
|---|---|---|
cell (CELL) |
any | Opaque cell reference, returned in outcomes |
r, c, s |
text or null | Raw closed coordinates |
k:<code> |
text, may be categorical | One per open key (see below) |
num (NUM) |
DEC = Decimal(38, 12) |
Numeric value |
thr (THR) |
DEC |
Threshold, 0.5·10^-decimals (0 if exact) |
txt (TXT) |
text | Text, canonical item or ISO date value |
flag (FLAG) |
Boolean | Boolean value |
Open key columns hold text, canonical items (category:code, see
canonical_item) or ISO dates.
VALUE_COLUMNS is (num, thr, txt, flag);
all four are null for a cell with no fact, and
value_columns maps one Python value to them.
A cell and open-key combination appears at most once, and a cell's
coordinates are the same on every row. String columns may be categorical,
which keeps large selections compact.
PointsFrameProvider builds such frames
from any DataProvider.
Inside the evaluator a record set is a Frame: key
columns k:<name> (k:r, k:c, k:s for varying closed axes, open
keys such as k:qEBB) as canonical strings, plus the same value columns.
Operators natural-join their operands on shared keys exactly like
combine. Where an operand with a default
lacks a record another operand has, or an if condition with a default
lacks a key a branch has, which the reference evaluator fills in with the
default, this module raises
ColumnarUnsupported.
Threads¶
evaluate_rule_columnar may run
concurrently on one report from several threads (polars releases the GIL).
FrameCache is thread-safe and builds each
selection once while other threads asking for it wait for that build. A
provider shared across threads must be safe to read concurrently;
PointsFrameProvider over a
MemoryProvider is.
Examples:
>>> from dpmxl import MemoryProvider, Number, TableLayout
>>> from dpmxl.columnar import PointsFrameProvider, evaluate_rule_columnar
>>> layout = TableLayout("T", rows=("0010", "0020"), columns=("0010",))
>>> provider = MemoryProvider([layout], {
... ("T", "0010", "0010", None, ()): Number.of(5, 0),
... ("T", "0020", "0010", None, ()): Number.of(-2, 0),
... })
>>> result = evaluate_rule_columnar(
... "{tT, r*, c0010} >= 0", PointsFrameProvider(provider)
... )
>>> result.status, result.passed, result.failed, result.failing_keys
('failed', 1, 1, [{'r': '0020'}])
DEC
module-attribute
¶
The polars type of the num and thr columns: Decimal(38, 12).
VALUE_COLUMNS
module-attribute
¶
The four value columns, ("num", "thr", "txt", "flag"); all null for a
cell with no fact.
ColumnarUnsupported
¶
Bases: Exception
The expression needs the reference evaluator.
Raised by evaluate_rule_columnar
and the frame helpers for anything this module does not evaluate exactly;
catch it and call evaluate_rule
instead.
FrameProvider
¶
Bases: Protocol
Columnar report data for the columnar evaluator.
The columnar counterpart of DataProvider.
select_frame returns one row per selected data point (closed cells
times reported open key combinations) with columns cell (an opaque
cell reference), r, c, s (raw coordinates or null),
k:<code> for each open key (text, items or ISO dates), and the value
columns num, thr, txt, flag (all null when the cell was
not reported). A cell and open key combination appears at most once, and
a cell's coordinates are the same on every row. String columns may be
categorical, which keeps large selections compact. See the frame schema
in the dpmxl.columnar module documentation.
Frame
dataclass
¶
A record set: key names plus a frame of key and value columns.
Attributes:
-
keys(tuple[str, ...]) –Key names, such as
("c",)or("qCPI",). -
df(DataFrame) –One row per record: a
k:<name>column per key and the value columnsnum,thr,txtandflag. -
has_default(bool) –Whether the records come from a selection with a
defaultthrough per-record operations, so that the reference evaluator fills in a default for keys they lack (seeRecordset).
kinds
cached
property
¶
Value columns holding at least one non-null value (one scan).
PointsFrameProvider
¶
Adapt any DataProvider to FrameProvider.
Each select_frame call converts the wrapped
DataProvider's points with
frame_from_points; the other methods
delegate. It is as thread-safe as the wrapped provider.
Examples:
>>> from dpmxl import CellQuery, MemoryProvider, Number, TableLayout
>>> layout = TableLayout("T", columns=("0010",), open_keys=("qCPI",))
>>> provider = MemoryProvider(
... [layout], {("T", None, "0010", None, (("qCPI", "A"),)): Number.of(7, 0)}
... )
>>> frame = PointsFrameProvider(provider).select_frame(CellQuery("T"))
>>> frame.columns
['cell', 'r', 'c', 's', 'k:qCPI', 'num', 'thr', 'txt', 'flag']
>>> frame.select("k:qCPI", "num", "thr").row(0)
('A', Decimal('7.000000000000'), Decimal('0.500000000000'))
Wrap a record provider.
Parameters:
-
provider(Any) –A
DataProvider.
FrameCache
¶
Selection frames shared by every rule evaluated against one report.
Rules reuse the same selections, but on massive open tables a selection
can hold millions of rows, so the cache keeps at most max_rows rows
and evicts the least recently used selections beyond that.
Rules may be evaluated concurrently from several threads (polars
releases the GIL): the cache is thread-safe, and
raw_frame and
selection build each missing
entry once while other threads asking for it wait for that build.
Create an empty cache.
Parameters:
-
max_rows(int, default:20000000) –Total rows kept across cached selection frames.
get_raw
¶
get_raw(query: CellQuery) -> DataFrame | None
Return a cached provider frame.
Returns:
-
DataFrame | None–pl.DataFrame | None: The frame, or
None.
get_frame
¶
get_frame(spec: SelectionSpec) -> Frame | None
put_raw
¶
put_raw(query: CellQuery, frame: DataFrame) -> None
Cache a provider frame within the row budget.
put_frame
¶
put_frame(spec: SelectionSpec, frame: Frame) -> None
Cache a selection record set within the row budget.
raw_frame
¶
raw_frame(
query: CellQuery, build: Callable[[], DataFrame]
) -> DataFrame
Return a provider frame, building and caching it once.
Returns:
-
DataFrame–pl.DataFrame: The cached or newly built frame.
selection
¶
selection(
spec: SelectionSpec, build: Callable[[], Frame]
) -> Frame
Return a selection record set, building and caching it once.
Returns:
-
Frame(Frame) –The cached or newly built record set.
ColumnarEvaluator
¶
ColumnarEvaluator(
provider: FrameProvider, cache: FrameCache | None = None
)
Evaluate one expression tree on frames.
Most callers want
evaluate_rule_columnar, which
also applies the precondition and turns the result into counts.
evaluate raises ColumnarUnsupported
for anything it does not implement. Use one evaluator per expression; the
FrameCache is what rules share.
Bind the provider.
Parameters:
-
provider(FrameProvider) –Columnar report data.
-
cache(FrameCache | None, default:None) –Selections shared across rules of one report.
evaluate
¶
evaluate(
node: Node, context: SelectionSpec | None = None
) -> Frame
Evaluate a node under an inherited with context.
Returns:
-
Frame(Frame) –The node's records.
Raises:
-
ColumnarUnsupported–For constructs the reference engine must run.
-
UnsupportedError–For constructs no engine supports.
-
EvaluationError–For invalid expressions or data.
ColumnarResult
dataclass
¶
ColumnarResult(
status: Status,
message: str | None = None,
passed: int = 0,
failed: int = 0,
null: int = 0,
failing_keys: list[dict[str, str | None]] | None = None,
failing_frame: DataFrame | None = None,
)
Outcome of one rule evaluated on frames.
Unlike RuleResult it carries no
evaluation trace: to explain a failure, evaluate the rule with
evaluate_rule (only_keys limits it
to the failing keys) and call explain.
Attributes:
-
status(Status) –passed,failed,not_applicable,errororunsupported, as forRuleResult. -
message(str | None) –Why the status is
error,unsupportedornot_applicable. -
passed(int) –Number of records that hold.
-
failed(int) –Number of records that fail.
-
null(int) –Number of records whose result is null (not evaluated).
value_columns
¶
value_columns(value: Value) -> dict[str, Any]
Split a scalar value into the four value columns.
Numbers fill num and thr; Booleans flag; text, items
(canonical, see canonical_item) and
dates (ISO) txt. None leaves all four null.
Returns:
-
dict[str, Any]–dict[str, Any]:
num,thr,txtandflagentries.
Examples:
>>> from dpmxl import Number
>>> value_columns(Number.of("1.5", 1))
{'num': Decimal('1.5'), 'thr': Decimal('0.05'), 'txt': None, 'flag': None}
>>> value_columns(True)
{'num': None, 'thr': None, 'txt': None, 'flag': True}
Raises:
-
ColumnarUnsupported–For a number or threshold that
Decimal(38, 12)does not hold exactly.
scalar_frame
¶
Build a key-less one-row frame holding one value.
Returns:
frame_from_points
¶
frame_from_points(
points: Sequence[DataPoint],
open_keys: Sequence[str] = (),
) -> DataFrame
Adapt data points to the frame layout of a FrameProvider.
Builds the columns described by
FrameProvider from
DataPoint objects; cell holds
repr(point.ref).
Parameters:
-
points(Sequence[DataPoint]) –Selected data points.
-
open_keys(Sequence[str], default:()) –The table's open key names, kept even without points.
Returns:
-
DataFrame–pl.DataFrame: One row per point.
Raises:
-
ColumnarUnsupported–For numeric or Boolean key values, items without a category, or numbers
Decimal(38, 12)does not hold.
normalize_code_expr
¶
Vectorized normalize_code.
Returns:
-
Expr–pl.Expr: Numeric codes zero-padded to four digits, others upper-case.
evaluate_rule_columnar
¶
evaluate_rule_columnar(
expression: str | Node,
provider: FrameProvider,
*,
precondition: str | Node | None = None,
cache: FrameCache | None = None,
max_failing_keys: int = 1000,
) -> ColumnarResult
Evaluate a validation rule on frames, honouring its precondition.
The columnar counterpart of evaluate_rule:
the same statuses and counts, and failing keys in canonical form (see
canonical_key). Syntax and evaluation
errors are returned as status error, not raised. It may run
concurrently from several threads with a shared cache.
Parameters:
-
expression(str | Node) –Rule expression text or a parsed tree.
-
provider(FrameProvider) –Columnar report data.
-
precondition(str | Node | None, default:None) –Optional precondition; the rule is not applicable unless it yields at least one true record.
-
cache(FrameCache | None, default:None) –Selections shared by the rules of one report; a new cache per call when omitted.
-
max_failing_keys(int, default:1000) –How many failing keys to list in
failing_keys;failing_framehas them all.
Returns:
-
ColumnarResult(ColumnarResult) –Status, record counts and the keys of failing records.
Raises:
-
ColumnarUnsupported–When the reference evaluator must run instead.
Examples:
Fall back to the reference evaluator when needed:
>>> from dpmxl import MemoryProvider, Number, TableLayout, evaluate_rule
>>> provider = MemoryProvider(
... [TableLayout("T", rows=("0010",), columns=("0010",))],
... {("T", "0010", "0010", None, ()): Number.of(3, 0)},
... )
>>> def run(expression):
... try:
... return evaluate_rule_columnar(
... expression, PointsFrameProvider(provider)
... ).status
... except ColumnarUnsupported:
... return evaluate_rule(expression, provider).status
>>> run("{tT, r0010, c0010} > 0")
'passed'
>>> run("year(#2026-12-31#) = 2026") # dates: reference evaluator
'passed'
canonical_key
¶
Canonicalize a reference outcome key for comparison with columnar keys.
Closed coordinates are normalized with
normalize_code; other key values become
the text the columnar evaluator uses (items as category:code).
Returns:
-
dict[str, str | None]–dict[str, str | None]: Key name to canonical string.
Examples: