# dpm-synth dpm-synth generates synthetic EBA xBRL-CSV report packages that satisfy a module's DPM-XL validation rules. Give it a template that lists a module's tables, cells and rules; it fills every cell with a plausible value, repairs the values until the rules pass, and writes a report package that reads and validates like a real submission. dpm-synth is an independent project, not affiliated with or endorsed by the European Banking Authority. It builds on two sibling packages: [dpm-xl] evaluates the rules and [dpm-xbrl] writes the package. It needs Python 3.11 or newer. The reports it writes are synthetic: the entity is a fictional LEI and every value is generated. ## Who it is for - Teams that build loaders, data warehouses or validators for EBA reports and need realistic, internally consistent test submissions for any module, without using real supervisory data. - Performance tests that need report packages with millions of facts. - Rule authors who want to see a module's rules satisfied, or find the rules that cannot be satisfied together. ## In 30 seconds ```python from dpmxbrl import write_xbrl_csv from dpmsynth import generate, load_template template = load_template("minimal.json") # a module's tables, cells and rules result = generate(template, seed=7) print(result.passed, "of", result.rules, "rules pass; failing:", result.failed) write_xbrl_csv(result.report, "report.zip") for fact in result.report.facts: print(fact.template, fact.datapoint, fact.value) ``` ```text 2 of 2 rules pass; failing: [] t_01.00 dp1001 1126000 t_01.00 dp1002 971000 t_01.00 dp1003 155000 ``` The template [`minimal.json`](samples/minimal.json) has one table of three monetary cells and two rules: row 0010 is the sum of rows 0020 and 0030, and both components are not negative. The same from the command line: ```console $ dpm-synth generate minimal.json -o report.zip --seed 7 rules: 2, passed: 2, failed: 0, not evaluated: 0, not applicable: 0 rounds: 2 (failing rules per round: 1, 0) wrote report.zip (3 facts) ``` Every example in these pages runs against the templates in [`docs/samples`](https://github.com/luukhopman/dpm-synth/tree/main/docs/samples), and the test suite checks that each prints what is shown. ## Install ```text pip install dpm-synth ``` This installs dpm-xl and dpm-xbrl too, and the `dpm-synth` command. ## Key features - **Any module.** The template format describes tables, open keys, cells, data types, allowed items and rules, so one tool covers COREP, FINREP and every other DPM framework; see the [template format](reference/template-format.md). - **Rules satisfied, not just shapes.** Sums, differences, products of two cells, sign rules, `isnull`, `in` and `match` are repaired; on EBA DPM 4.3 COREP_OF, 999 of the 1,051 rules that apply pass. See [the evaluate-and-repair loop](concepts/repair-loop.md) and [its limits](concepts/limits.md). - **Valid packages.** Packages are written by dpm-xbrl and read back without a single broken EBA filing rule that the file can show, other than the synthetic file name. - **Reproducible.** The same template, seed, options and versions give the same bytes; see [reproduce a report](guides/reproducible-seeds.md). - **Large.** Scaled packages stream open tables with millions of rows in seconds; see [load-test with scaled packages](guides/load-testing.md). ## Toolkit **Part of the dpm toolkit**, three Python packages that work together: | Package | What it does | | :-- | :-- | | | Parse and evaluate DPM-XL validation rules | | | Read, write and check xBRL-CSV and XBRL-XML reports | | | Generate test reports that satisfy DPM-XL rules **(this package)** | The [toolkit tour](https://luukhopman.github.io/dpm-xl/guides/toolkit-tour/) takes one report through all three: generated with dpm-synth, read with dpm-xbrl, then evaluated and explained with dpm-xl. ## Where next - New here: the [getting started](getting-started.md) tutorial. - A task to do: the [how-to guides](guides/index.md). - How it works: the [concepts](concepts/index.md). - Exact behaviour: the [reference](reference/index.md), including the [template format](reference/template-format.md), the [API](reference/api.md) and the [command line](reference/cli.md). - Working with an AI agent: [dpm-synth for AI agents](agents.md), [`llms.txt`](llms.txt) and [`llms-full.txt`](llms-full.txt). - The sibling packages: [dpm-xl] for DPM-XL rules and [dpm-xbrl] for reading and writing reports. [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Getting started This tutorial generates a synthetic report for a small template, reads the package back with dpm-xbrl and checks the rules with dpm-xl. It takes about ten minutes. ## Install ```text pip install dpm-synth ``` dpm-synth needs Python 3.11 or newer. It installs [dpm-xl], which evaluates DPM-XL rules, and [dpm-xbrl], which reads and writes EBA reports. ## The template A template describes one module: its tables, the cells of each table, and the rules a report must satisfy. It is a JSON document in the `dpm-synth-template/1` format. This is [`minimal.json`](samples/minimal.json): ```json { "format": "dpm-synth-template/1", "release": "1.0", "module": "DEMO", "entry_point": "https://example.com/taxonomy/demo/1.0/mod/demo.json", "reference_date": "2026-12-31", "tables": [ { "code": "T_01.00", "cells": [ {"id": "T_01.00/r0010", "row": "0010", "column": "0010", "datapoint": "dp1001", "data_type": "m"}, {"id": "T_01.00/r0020", "row": "0020", "column": "0010", "datapoint": "dp1002", "data_type": "m"}, {"id": "T_01.00/r0030", "row": "0030", "column": "0010", "datapoint": "dp1003", "data_type": "m"} ] } ], "rules": [ {"code": "v1", "expression": "with {tT_01.00, c0010, default: 0, interval: true}: {r0010} = {r0020} + {r0030}"}, {"code": "v2", "expression": "with {tT_01.00, c0010}: {(r0020, r0030)} >= 0"} ] } ``` - Table `T_01.00` has three monetary cells (`data_type` `m`) in rows 0010 to 0030 of column 0010. Each cell names the `datapoint` that carries its value in the package. - Rule `v1` says row 0010 is the sum of rows 0020 and 0030; rule `v2` says both components are zero or more. For real modules you do not write templates by hand: any tool that reads a DPM release can export them; see [build a template from a DPM release](guides/from-a-dpm-release.md). To write your own, see [write a template by hand](guides/write-a-template.md). ## Generate from the command line ```console $ dpm-synth generate minimal.json -o report.zip --seed 7 rules: 2, passed: 2, failed: 0, not evaluated: 0, not applicable: 0 rounds: 2 (failing rules per round: 1, 0) wrote report.zip (3 facts) $ echo $? 0 ``` The first round found one failing rule (the random total did not match its components), repaired it, and the second round found none. Exit status 0 means every rule that could be evaluated passes; 1 would mean some still fail. The package is written either way. ## Read the package back `report.zip` is an EBA xBRL-CSV report package. Read it with dpm-xbrl: ```python from dpmxbrl import read_report report = read_report("report.zip") print(report.entity_id, report.reference_date, report.filing_indicators) for fact in report.facts: print(fact.template, fact.datapoint, fact.value) print([finding.code for finding in report.filing]) ``` ```text SYNTHETICDPMXL000002.CON 2026-12-31 {'T_01.00': True} t_01.00 dp1001 1126000 t_01.00 dp1002 971000 t_01.00 dp1003 155000 ['invalidFileName'] ``` The entity is the fictional LEI `SYNTHETICDPMXL000002` at the consolidated level. The only EBA filing rule the package breaks is the file name, which for a real submission encodes the entity, module and dates; name the output accordingly if that matters to you. ## Generate from Python The same from Python, with the result of the repair loop: ```python from dpmxbrl import write_xbrl_csv from dpmsynth import generate, load_template template = load_template("minimal.json") result = generate(template, seed=7) print(f"{result.passed} passed, failed: {result.failed}") print(f"rounds: {result.rounds}, failing per round: {result.history}") write_xbrl_csv(result.report, "report.zip") ``` ```text 2 passed, failed: [] rounds: 2, failing per round: [1, 0] ``` `result.report` is a `dpmxbrl.Report`; `result.data` holds the generated values and is a dpm-xl data provider, so you can evaluate any rule on it: ```python from dpmxl import evaluate_rule rule = "with {tT_01.00, c0010}: {r0010} >= {r0020}" print(evaluate_rule(rule, result.data).status) ``` ```text passed ``` ## A bigger template [`sample.json`](samples/sample.json) has a closed table with row and column totals, a ratio and an enumerated cell, an open table keyed by an enumerated property, an open table keyed by LEIs, and a `time_shift` rule that needs last year's report: ```console $ dpm-synth generate sample.json -o sample.zip --seed 7 rules: 9, passed: 8, failed: 0, not evaluated: 1, not applicable: 0 rounds: 5 (failing rules per round: 5, 3, 1, 1, 0) wrote sample.zip (20 facts) ``` The `time_shift` rule is "not evaluated": a single report has no previous period. [The evaluate-and-repair loop](concepts/repair-loop.md) explains the rounds. ## Next steps - [Validate the output](guides/validate-output.md) independently with dpm-xbrl and dpm-xl. - [Load-test](guides/load-testing.md) with millions of open-table rows. - Read [the limits of the loop](concepts/limits.md) before relying on a module's every rule passing. [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # How-to guides Recipes for specific tasks. Each assumes you have dpm-synth installed and have generated a first report; the [getting started](../getting-started.md) tutorial covers the basics. Every example runs against the templates in [`docs/samples`](https://github.com/luukhopman/dpm-synth/tree/main/docs/samples). | Guide | Use it to | | --- | --- | | [Write a template by hand](write-a-template.md) | Describe your own tables, keys, data types and rules, and check the template. | | [Build a template from a DPM release](from-a-dpm-release.md) | Export a real module's template from a DPM release, choose its rules, generate and read the results. | | [Load-test with scaled packages](load-testing.md) | Write packages with millions of open-table rows, and know which rules still hold. | | [Reproduce a report with seeds](reproducible-seeds.md) | Get the same bytes again, or different reports on purpose. | | [Validate the output with dpm-xbrl and dpm-xl](validate-output.md) | Check a package independently, as a downstream consumer would. | [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Write a template by hand Templates for real EBA modules come from a DPM tool (see [build a template from a DPM release](from-a-dpm-release.md)). Writing one by hand is useful for your own tables, for a few tables of a module, or for a test fixture. This guide builds a template in Python and writes it as JSON; the [template format](../reference/template-format.md) lists every field. ## The module Start with the format, the release and module codes, and optionally the entry point and the reference date that the package should carry: ```python template = { "format": "dpm-synth-template/1", "release": "1.0", "module": "LOANS", "entry_point": "https://example.com/taxonomy/loans/1.0/mod/loans.json", "reference_date": "2026-06-30", "decimals": {"m": -3}, # monetary amounts in thousands "tables": [], "rules": [], } ``` ## A closed table A closed table has a fixed set of cells. Give each cell an `id`, its row and column codes (without the `r` and `c`), the `datapoint` that carries its value in the package, and its data type: `m` monetary, `p` percentage, `i` integer, `r` decimal, `e` enumerated, `b` boolean, `d` date, anything else text. ```python def cell(table, row, column, datapoint, data_type, **more): return { "id": f"{table}/r{row}/c{column}", "row": row, "column": column, "datapoint": datapoint, "data_type": data_type, **more, } template["tables"].append( { "code": "L_01.00", "cells": [ cell("L_01.00", "0010", "0010", "dp101", "m"), # total loans cell("L_01.00", "0020", "0010", "dp102", "m"), # of which secured cell("L_01.00", "0030", "0010", "dp103", "m"), # of which unsecured cell("L_01.00", "0040", "0010", "dp104", "p"), # share secured cell( "L_01.00", "0050", "0010", "dp105", "e", items=["eba_CT:x1", "eba_CT:x2"], # allowed items, first is the start ), ], } ) ``` ## An open table An open table has one row per key value. Each `open_keys` entry becomes a column of the table's file. A key with `items` is enumerated and takes its items in order; a key without is typed, and gets generated LEI-shaped identifiers. Cells of an open table usually have row `*`, for every row: ```python template["tables"].append( { "code": "L_02.00", "open_keys": [ {"code": "LEI", "column": "c0010"}, # typed: generated identifiers {"code": "qCT", "column": "c0020", "items": ["eba_CT:x1", "eba_CT:x2"]}, ], "cells": [ cell("L_02.00", "*", "0030", "dp201", "m", sheet="*"), # amount cell("L_02.00", "*", "0040", "dp202", "s", sheet="*"), # name ], } ) ``` ## Rules Rules are DPM-XL expressions, as [dpm-xl] parses them. A rule may have a precondition, usually the table's filing indicator; every table of the template counts as reported. ```python template["rules"] = [ { "code": "l1", "expression": "with {tL_01.00, c0010, default: 0, interval: true}: " "{r0010} = {r0020} + {r0030}", "precondition": "{v_L_01.00}", }, {"code": "l2", "expression": "with {tL_01.00, c0010}: {(r0020, r0030)} >= 0"}, { "code": "l3", "expression": "with {default: 0, interval: true}: " "{tL_01.00, r0040, c0010} * {tL_01.00, r0010, c0010} " "= {tL_01.00, r0020, c0010}", }, {"code": "l4", "expression": "{tL_01.00, r0050, c0010} in {[eba_CT:x2]}"}, { "code": "l5", "expression": "with {default: 0, interval: true}: " "sum({tL_02.00, c0030}) <= {tL_01.00, r0010, c0010}", }, {"code": "l6", "expression": "{tL_02.00, c0030} >= 0"}, { "code": "l7", "expression": 'match({tL_02.00, c0030}[get LEI], "^[A-Z0-9]{18}[0-9]{2}$")', }, ] ``` ## Check and generate Write the JSON and load it; `load_template` validates every field and names the first wrong one: ```python import json from pathlib import Path from dpmsynth import generate, load_template Path("loans.json").write_text(json.dumps(template, indent=2), encoding="utf-8") loaded = load_template("loans.json") result = generate(loaded, seed=1) print(result.passed, "passed; failed:", result.failed) for fact in result.report.facts: print(fact.template, fact.datapoint, fact.value, dict(fact.keys)) ``` ```text 7 passed; failed: [] l_01.00 dp101 1451000 {} l_01.00 dp102 583000 {} l_01.00 dp103 868000 {} l_01.00 dp104 0.4018 {} l_01.00 dp105 eba_CT:x2 {} l_02.00 dp201 993000 {'c0010': 'SYNTHEQH524YNG5BY106', 'c0020': 'eba_CT:x1'} l_02.00 dp202 SYNTHB7O259OWOO3SB53 {'c0010': 'SYNTHEQH524YNG5BY106', 'c0020': 'eba_CT:x1'} l_02.00 dp201 433000 {'c0010': 'SYNTHA2ROGUBBB8AYN47', 'c0020': 'eba_CT:x2'} l_02.00 dp202 SYNTH09GLSHV616MTS07 {'c0010': 'SYNTHA2ROGUBBB8AYN47', 'c0020': 'eba_CT:x2'} ``` Monetary amounts are multiples of 1,000, as `"decimals": {"m": -3}` asks, the share in row 0040 times the total gives the secured loans within the precision the package reports, and the item rule picked `eba_CT:x2`. Text cells and typed keys get visibly fictional LEI-shaped values starting with `SYNTH`; only `match` rules with the LEI or `dd/mm/yyyy` pattern ask for anything else. The same file works from the command line: ```console $ dpm-synth generate loans.json -o loans.zip --seed 1 rules: 7, passed: 7, failed: 0, not evaluated: 0, not applicable: 0 rounds: 2 (failing rules per round: 4, 0) wrote loans.zip (9 facts) ``` ## Mistakes the loader catches ```python from dpmsynth import TemplateError broken = json.loads(json.dumps(template)) broken["tables"][1]["open_keys"][0]["column"] = "datapoint" try: load_template(broken) except TemplateError as error: print(error) ``` ```text tables[1].open_keys[0].column: 'datapoint' is a fact column of the table ``` The loader does not check the rules: a rule that does not parse is reported as not evaluated when you generate, with the syntax error, so one bad rule does not stop a module of thousands. [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Build a template from a DPM release A real module has thousands of cells and rules, so its template is exported from a DPM release rather than written by hand. The [template format](../reference/template-format.md) is the interface: dpm-synth does not read DPM databases itself, and any tool that can read a release, such as a script over the EBA's DPM database, can write templates. This guide shows what such an exporter must fill in, with a small exporter you can adapt. ## What an exporter fills in For one module of one release, the exporter writes: | Template field | From the DPM release | | --- | --- | | `release`, `module` | The release code, such as `4.3`, and the module code, such as `COREP_OF`. | | `entry_point` | The module's xBRL-CSV entry point (its `.json` URL in the taxonomy), if the consumer of the package needs it. | | `tables[].code` | Every non-abstract table version of the module, such as `C_07.00.a`. | | `tables[].filing_indicator` | The table's template code: the table code without single-letter parts, such as `C_07.00` for `C_07.00.a`. | | `tables[].open_keys` | The table's open-axis properties: their codes, the CSV column of each, and for an enumerated property the item qnames of its domain or subdomain. | | `tables[].cells` | Every reportable cell: its row, column and sheet codes, the datapoint (variable) code that carries its value, the variable's data type code and, for enumerated variables, the allowed item qnames. | | `rules` | The module's validation rules: code, DPM-XL expression and precondition, with a `source` such as `eba`. | | `decimals` | The reporting decimals per data type, if the defaults (`m` 0, `p` 4, `i` 0, `r` 2) do not suit. | Leave out abstract tables, cells without a variable and rules of other modules. Item qnames are written as they appear in a report, such as `eba_qEC:qx16`; dpm-synth writes them back unchanged. ## A small exporter An exporter is a query per field. Here the release's rows are plain Python lists, standing in for the query results of your DPM database: ```python import json from pathlib import Path # Query results of a release, one tuple per row. tables = [("X_01.00", False), ("X_02.00.a", False), ("X_99.00", True)] # code, abstract cells = [ # table, row, column, sheet, variable, data type ("X_01.00", "0010", "0010", None, "500", "m"), ("X_01.00", "0020", "0010", None, "501", "m"), ("X_01.00", "0030", "0010", None, "502", "e"), ("X_02.00.a", "*", "0010", "*", "510", "m"), ] open_axes = [("X_02.00.a", "qCP", None)] # table, property, domain (None: typed) domains = {"CT": ["x1", "x2"]} # domain code -> item codes enumerated = {"502": "CT"} # variable -> domain of an enumerated variable rules = [ # code, expression, precondition ("v1", "with {tX_01.00, c0010, default: 0}: {r0010} >= {r0020}", "{v_X_01.00}"), ( "v2", "with {default: 0}: sum({tX_02.00.a, c0010}) = {tX_01.00, r0020, c0010}", None, ), ("v3", "{tX_01.00, r0030, c0010} in {[eba_CT:x2]}", None), ] def items(domain): return [f"eba_{domain}:{code}" for code in domains[domain]] if domain else None def filing_indicator(code): return ".".join(part for part in code.split(".") if len(part) > 1) template = { "format": "dpm-synth-template/1", "release": "1.0", "module": "X", "reference_date": "2026-12-31", "tables": [ { "code": code, "filing_indicator": filing_indicator(code), "open_keys": [ {"code": prop, "column": prop, "items": items(domain)} for table, prop, domain in open_axes if table == code ], "cells": [ { "id": f"{code}/r{row}/c{column}", "row": row, "column": column, "sheet": sheet, "datapoint": f"dp{variable}", "data_type": data_type, "items": items(enumerated.get(variable)), } for table, row, column, sheet, variable, data_type in cells if table == code ], } for code, abstract in tables if not abstract ], "rules": [ {"source": "eba", "code": code, "expression": expression, "precondition": pre} for code, expression, pre in rules ], } Path("x.json").write_text(json.dumps(template, indent=1), encoding="utf-8") ``` Load the result to check it; `load_template` names the first field that is wrong: ```python from dpmsynth import generate, load_template module = load_template("x.json") print([table.code for table in module.tables], len(module.rules)) result = generate(module, seed=1) print(result.passed, result.failed) ``` ```text ['X_01.00', 'X_02.00.a'] 3 3 [] ``` ## Choose the rules Each rule carries a `source`. If the release, or your own rule set, has rules from more than one source, keep the ones you want before generating. `ModuleTemplate` is a frozen dataclass, so `dataclasses.replace` gives a copy with other rules, or other tables: ```python from dataclasses import replace own = replace(module, rules=tuple(r for r in module.rules if r.source == "eba")) print(len(own.rules)) ``` ```text 3 ``` ## Generate ```console $ dpm-synth generate x.json -o x.zip --seed 1 rules: 3, passed: 3, failed: 0, not evaluated: 0, not applicable: 0 rounds: 2 (failing rules per round: 3, 0) wrote x.zip (5 facts) ``` On EBA DPM 4.3 COREP_OF, with its 1,205 EBA rules, generation takes 15 rounds (about 15 seconds on a laptop), and 999 of the 1,051 rules that apply pass. The other 154 rules do not apply to the generated report: they select no generated cell, such as rows filtered on key values that the generated rows do not have. [Limits of the loop](../concepts/limits.md) has the figures for other modules and explains what remains. ## Read the results The summary names up to 20 failing rules; `--json` lists them all. To see why a rule fails, evaluate it on the generated values and explain the failure with dpm-xl: ```python from dpmxl import evaluate_rule, explain from dpmsynth import generate, load_template template = load_template("sample.json") result = generate(template, seed=7, max_rounds=1) rules = {rule.code: rule for rule in template.rules} for code in result.failed: rule = rules[code] outcome = evaluate_rule( rule.expression, result.data, precondition=rule.precondition ) failure = outcome.failures[0] print(code, explain(outcome.evaluation, rule.expression, failure.key).summary) ``` ```text open 39,000 should not exceed -600,000: difference 639,000, at least 638,999 given the reported precision open-total 332,000 should equal 1,135,000: difference 803,000 rows 1,135,000 should equal -180,000 (717,000 + 75,000 − 972,000): difference 1,315,000, at least 1,314,998 given the reported precision ``` Here the loop was stopped after one round, so rules that more rounds would settle still fail. On a real module, a rule that still fails after the loop stops is usually coupled to another rule that defines the same cell differently; see [limits of the loop](../concepts/limits.md). [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Load-test with scaled packages Loaders and validators need to be tested on reports far larger than any that the repair loop could solve row by row. A scaled package keeps the solved values and multiplies the open tables: every open table gets as many key rows as you ask for, whose values repeat the solved rows under new keys. Tables are streamed into the ZIP file, so memory stays flat however many rows you write. ## From the command line ```console $ dpm-synth generate sample.json -o big.zip --seed 7 --scale-rows 100000 rules: 9, passed: 8, failed: 0, not evaluated: 1, not applicable: 0 rounds: 5 (failing rules per round: 5, 3, 1, 1, 0) wrote big.zip (100018 facts) ``` The summary is that of the solved report; the scaled rows are not evaluated. Table `S_03.00`, keyed by LEIs, got 100,000 rows. Table `S_02.00.a` is keyed only by an enumerated property, so it keeps its two solved rows. On a laptop, a million rows take about six seconds and 35 MB of memory. ## From Python `write_scaled_package` takes one row count for every open table, or a mapping of table code to rows; tables left out of the mapping keep their generated rows: ```python from dpmsynth import generate, load_template, write_scaled_package result = generate(load_template("sample.json"), seed=7) written = write_scaled_package(result.data, "big.zip", rows={"S_03.00": 50_000}) print(written) ``` ```text {'S_01.00': 14, 'S_02.00.a': 4, 'S_03.00': 50000} ``` It returns the fact rows written per table. Read the package back with dpm-xbrl; for millions of facts its `frames` extra reads them into polars frames in seconds: ```python from dpmxbrl import read_report report = read_report("big.zip") print(len(report.facts), report.facts[-1].keys) ``` ```text 50018 {'c0010': 'SYNTHERI000004999908'} ``` ## New keys - A typed key, such as an LEI column, gets identifiers made of `SYNTH`, the first three letters and digits of the key code (padded with `X`), a 10-digit row number and two ISO 17442 check digits, so they are unique, pass LEI validation and are visibly fictional. A table can have up to 9,999,999,999 rows. - An enumerated key keeps the value of the solved row that the scaled row repeats, so the scaled rows have the same mix of items as the solved ones. When every key of a table is enumerated, the table keeps its solved rows; generate with more `--open-rows` to have more item combinations. - Closed tables are written as generated. ## Which rules still hold Row-wise rules, such as "amount at most the exposure" within one row, a `match` on the key, or a rule on the rows with one enumerated key value (`[where CT = [eba_CT:x1]]`), hold on every scaled row, because each row repeats a solved one with its enumerated key values. Rules that add up an open table, such as "the sum over all counterparties equals the total in another table", are not re-solved and will fail: the totals are those of the solved rows. Use the scaled package to test volume, and the unscaled package to test validation results. The package is the same as `dpmxbrl.write_xbrl_csv` would write: the same report package and metadata files, the root folder named as the ZIP file, and the same CSV layout. [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Reproduce a report with seeds Every random choice dpm-synth makes, starting values and generated identifiers alike, comes from one random generator seeded with `--seed` (default 0). The written package has fixed ZIP timestamps and a fixed order of files and facts. So the same inputs give the same bytes: ```python import hashlib from pathlib import Path from dpmxbrl import write_xbrl_csv from dpmsynth import generate, load_template template = load_template("sample.json") for folder, seed in (("first", 7), ("second", 7), ("other", 8)): Path(folder).mkdir() write_xbrl_csv(generate(template, seed=seed).report, f"{folder}/report.zip") def digest(folder): return hashlib.sha256(Path(folder, "report.zip").read_bytes()).hexdigest() print(digest("first") == digest("second")) print(digest("first") == digest("other")) ``` ```text True False ``` ## What the output depends on The package is a function of: - the template, including the order of its tables, cells and rules; - the seed and the other options: `--open-rows`, `--max-rounds`, `--entity`, `--level`, `--reference-date` and `--scale-rows`; - the package's file name, which names its root folder (the example writes each package as `report.zip`, in its own folder); - the versions of dpm-synth, dpm-xl and dpm-xbrl. It does not depend on the machine, the time, the working directory or Python's hash seed. A patch or minor release of any of the three packages may change the values generated for a seed, for example when a repair or a rule's evaluation improves; the [changelog](../project/changelog.md) says so when it does. ## Several different reports Use different seeds for reports that differ, such as one report per test case or a set of reports for a performance test: ```console $ dpm-synth generate minimal.json -o report-1.zip --seed 1 rules: 2, passed: 2, failed: 0, not evaluated: 0, not applicable: 0 rounds: 2 (failing rules per round: 1, 0) wrote report-1.zip (3 facts) $ dpm-synth generate minimal.json -o report-2.zip --seed 2 rules: 2, passed: 2, failed: 0, not evaluated: 0, not applicable: 0 rounds: 2 (failing rules per round: 1, 0) wrote report-2.zip (3 facts) ``` The values differ; on a large module, the number of rounds and the rules that still fail may differ from seed to seed too. ## Stable test fixtures To keep a test fixture stable across upgrades, commit the generated package rather than regenerating it in every test run, and regenerate it on purpose, with the same seed, when you upgrade. A test that only needs a valid report of a module can regenerate it each time and check the rule results instead of the bytes. [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Validate the output with dpm-xbrl and dpm-xl dpm-synth evaluates the rules on its own values while it generates. To be sure the package says the same, check it as a downstream consumer would: read the written ZIP file with [dpm-xbrl], and evaluate the template's rules on what was read with [dpm-xl]. The test suite does exactly this for every sample template. ## Generate and read back ```python from dpmxbrl import read_report, write_xbrl_csv from dpmsynth import generate, load_template template = load_template("sample.json") result = generate(template, seed=7) write_xbrl_csv(result.report, "sample.zip") report = read_report("sample.zip") print(report.format, report.entity_id, len(report.facts)) for finding in report.filing: print(finding.severity, finding.rule, finding.code) ``` ```text xbrl-csv SYNTHETICDPMXL000002.CON 20 warning File naming invalidFileName ``` dpm-xbrl records every [EBA filing rule][dpm-xbrl-filing] the file breaks. The only one is the file name, which for a real submission encodes the entity, module and creation time. ## A provider over the package dpm-xl evaluates rules against a data provider: an object that answers a selection such as `{tS_01.00, r0010, c0010}` with data points. The facts of an EBA xBRL-CSV package carry a datapoint, not row and column codes, so the template's cells place them. This provider knows nothing of dpm-synth's own values; everything comes from the package (see [writing a provider][dpm-xl-provider] for the protocol): ```python from datetime import date from decimal import Decimal from dpmxl import DataPoint, Item, Number, axis_matches DECIMALS = { "m": "decimalsMonetary", "p": "decimalsPercentage", "i": "decimalsInteger", "r": "decimalsDecimal", } def selected(code, selector): if selector is None or selector.wildcard: return True return code not in (None, "*") and axis_matches(code, selector) class PackageProvider: """Facts of a report read with dpm-xbrl, placed by a template's cells.""" def __init__(self, template, report): self.template, self.report = template, report self.rows = {} # table code -> key row -> datapoint -> value for fact in report.facts: table = template.table(fact.template) cell = next(c for c in table.cells if c.datapoint == fact.datapoint) columns = {key.column: key for key in table.open_keys} keys = tuple( (columns[name].code, Item.parse(text) if columns[name].items else text) for name, text in fact.keys.items() ) row = self.rows.setdefault(table.code, {}).setdefault(keys, {}) row[fact.datapoint] = self.value(cell.data_type, fact.value) def value(self, data_type, text): if text is None: return None if data_type in DECIMALS: # reported decimals become the tolerance decimals = self.report.decimals_parameters.get(DECIMALS[data_type]) return Number.of(Decimal(text), decimals) if data_type == "e": return Item.parse(text) if data_type == "b": return text == "true" if data_type == "d": return date.fromisoformat(text) return text def select(self, query): table = self.template.table(query.table) if table is None: raise LookupError(f"table {query.table} is not in the template") return [ DataPoint( table=table.code, row=cell.row, column=cell.column, sheet=cell.sheet, keys=dict(keys), value=values.get(cell.datapoint), ref=(table.code, cell.datapoint, keys), ) for keys, values in self.rows.get(table.code, {}).items() for cell in table.cells if selected(cell.row, query.rows) and selected(cell.column, query.columns) and selected(cell.sheet, query.sheets) ] def table_reported(self, table): folded = table.casefold() return any( reported and code.casefold() == folded for code, reported in self.report.filing_indicators.items() ) or any( t.code.casefold() == folded and t.code in self.rows for t in self.template.tables ) def parameter(self, name): if name.casefold() == "refperiod": return self.report.reference_date if name.casefold() == "reportinglevel": return self.report.entity_id.rsplit(".", 1)[-1].casefold() return None def open_keys(self, table): found = self.template.table(table) return tuple(key.code for key in found.open_keys) if found else () ``` ## Evaluate the rules ```python from collections import Counter from dpmxl import evaluate_rule provider = PackageProvider(template, report) statuses = { rule.code: evaluate_rule( rule.expression, provider, precondition=rule.precondition ).status for rule in template.rules } print(Counter(statuses.values())) failed = [code for code, status in statuses.items() if status == "failed"] print(failed == result.failed) ``` ```text Counter({'passed': 8, 'unsupported': 1}) True ``` The package passes the same eight rules. The ninth, a `time_shift`, is `unsupported` here, as it is "not evaluated" in the generator: a single report has no previous period. To evaluate it, give the provider an `earlier_report(period, count)` method that returns a provider over last year's package. For a large module, use dpm-xl's columnar evaluator and its [end-to-end guide](https://luukhopman.github.io/dpm-xl/guides/end-to-end/), which reads a package with dpm-xbrl in the same way. [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Concepts These pages explain how dpm-synth works and why it behaves as it does. For tasks, see the [how-to guides](../guides/index.md); for exact behaviour, the [reference](../reference/index.md). | Page | Explains | | --- | --- | | [The evaluate-and-repair loop](repair-loop.md) | Starting values, bounds and definitions, the repairs, rounds and rounding. | | [Limits of the loop](limits.md) | Results on real EBA modules, the rules that stay failing or do not apply, and why. | [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # The evaluate-and-repair loop `generate` does not solve a module's rules as a system of equations. It starts from plausible values, evaluates every rule with [dpm-xl], and changes one cell for each failing record so that the failing comparison holds; then it evaluates again. Most rules in EBA modules are local (a total is the sum of its components, an amount is not negative), so this converges in a few rounds for most of a module. ## 1. Starting values Every cell gets a value for its [data type](../reference/template-format.md#data-types): monetary amounts in thousands, ratios between 0 and 1, small integers, the first allowed item of an enumerated cell, the reference date for dates and an LEI-shaped identifier for text. Every open table gets `open_rows` key rows: allowed items for enumerated keys, LEI-shaped identifiers for typed keys. Every generated LEI-shaped identifier starts with `SYNTH` (`LEI_PREFIX`), followed by 13 random letters and digits and two valid ISO 17442 check digits: it passes LEI validation, but is visibly fictional and cannot coincide with a real entity's LEI. The reporting entity is the fictional `SYNTHETICDPMXL000002` unless you pass another. With `max_rounds=0` you see the starting values unrepaired: ```python from dpmsynth import generate, load_template template = load_template("minimal.json") start = generate(template, seed=7, max_rounds=0) print([fact.value for fact in start.report.facts], start.failed) ``` ```text ['332000', '971000', '155000'] ['v1'] ``` The total in row 0010 is not the sum of rows 0020 and 0030, so rule `v1` fails. The first round repairs it by setting the total, the cell that `v1` defines, to 1,126,000; the second round finds no failing rule. ## 2. Analysis Before the first round, the rules are read once: - **Bounds.** A rule that compares cells with a constant, such as `{(r0020, r0030)} >= 0` or `{c0020} <= 0`, becomes a bound on each of those cells, and so does such a comparison joined to the rest of a rule by `and`. Starting values outside a bound change sign, or take the bound; repairs never move a cell outside its bounds. A strict bound (`> 0`) is one reporting step past the constant, such as 1 for an amount or 0.0001 for a percentage. - **Definitions.** A rule whose `=` has a single cell on one side, such as `{r0010} = {r0020} + {r0030}`, *defines* that cell. Repairs of other rules prefer to change cells that no other rule defines, so a total that one rule owns is not pulled around by rules that merely use it. - **Order.** Rules are sorted so that a rule defining a component runs before the rule that sums the components: totals are then settled from components that no longer change in that round. ## 3. Repairs Each failing record of a rule is traced to the comparison that fails, using the evaluation trace that dpm-xl keeps. Its repair depends on the construct: | Construct | Repair | | --- | --- | | `a = b`, `a <= b`, `a > b`, ... on numbers | Expand `a - b` into terms `coefficient × cell`: sums, differences, unary minus, constants, `sum(...)`, `abs` (at its current sign), a product of two cells (each at the other's current value) and division by a constant or cell. One cell absorbs the whole difference, the first that no other rule defines and whose new value respects its bounds. Strict comparisons are met by one reporting step. | | `a = b` on items, text or dates | Copy the known side into the cells of the other. | | `and`, `or`, `not` | Repair both sides of `and`; the first side of `or` that can be changed; `not` repairs its operand towards the opposite. | | `if c then a else b endif` | Repair the branch that the condition selects. | | `isnull(x)` | Clear `x`; `not isnull(x)` gives a cleared cell a starting value again. | | `x in {…}` or `x in s` | Set `x` to the first member of the set that fits its cell: a number for a numeric cell, an item for an enumerated one. Any set works: a literal, a `set-*` parameter, `union(...)`, or a selection of cells. | | `match(x, pattern)` | For the LEI pattern `[A-Z0-9]{18}[0-9]{2}`, a checksum-valid LEI-shaped identifier starting with `SYNTH`; for `dd/mm/yyyy`, the reference date. | Numbers never go into text or enumerated cells and nothing else goes into a numeric cell. A repair that would need a number beyond 10^15, as a ratio dividing by an almost-zero cell can, is not made. ## 4. Rounds and stopping A round evaluates every rule and repairs every failing record. The loop stops when no rule fails, when a round changes nothing, after `max_rounds` (default 40), or when the failing count has not reached a new low for six rounds: rules that keep undoing each other's repairs. The failing count per round is `GenerationResult.history`: ```python result = generate(load_template("sample.json"), seed=7) print(result.history, result.rounds, result.repairs) ``` ```text [5, 3, 1, 1, 0] 5 16 ``` ## 5. Rounding and the final evaluation Repairs work on values with eight decimals. At the end, every number is rounded to its reporting decimals (`m` 0, `p` 4, `i` 0, `r` 2 unless the template says otherwise), and every rule is evaluated once more on the rounded values. That final evaluation is what `passed`, `failed`, `not_evaluated` and `not_applicable` report. The generated values are evaluated as the written package will be: each number carries its reporting decimals, which dpm-xl turns into a tolerance for rules with `interval: true`. A ratio rounded to four decimals is judged as a validator would judge the reported ratio, so a rule that passes here passes on the package; the [validation guide](../guides/validate-output.md) checks this. ## Why not a solver A complete approach would solve a module's linear equalities jointly, for example with a sparse least-squares or a mixed-integer solver, and then repair the rest. The repair loop needs no solver dependency, handles the non-linear and non-numeric constructs uniformly, and satisfies most rules of real modules in seconds. The template carries everything a solver would need, so one can be added without changing the format. [Limits of the loop](limits.md) shows what remains. [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Limits of the loop The [evaluate-and-repair loop](repair-loop.md) changes one cell at a time to fix one failing comparison. That settles most rules of real modules, but not all: some rules constrain the same cells in ways that local repairs cannot reconcile. A report that passes every rule is therefore not guaranteed; the result always says which rules still fail. ## Results on EBA modules EBA DPM 4.3 modules, their EBA validation rules only, seed 7, default options, dpm-synth 1.0.0 with dpm-xl and dpm-xbrl 1.0.0. Times are wall clock on a laptop (AMD Ryzen 7 4700U) and only indicate the scale; the other columns do not depend on the machine: | Module | Tables | Cells | Rules | Apply | Pass | Fail | Not applicable | Not evaluated | Rounds | Time | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | COREP_OF | 81 | 21,339 | 1,205 | 1,051 | 999 | 52 | 154 | 0 | 15 | 13 s | | FINREP9 | 124 | 13,834 | 2,260 | 2,258 | 1,929 | 329 | 0 | 2 | 14 | 10 s | | AE | 23 | 1,306 | 218 | 218 | 207 | 11 | 0 | 0 | 10 | < 1 s | | COREP_LCR_DA | 14 | 5,119 | 328 | 328 | 321 | 7 | 0 | 0 | 9 | 2 s | | COREP_LE | 5 | 79 | 25 | 25 | 24 | 1 | 0 | 0 | 9 | < 1 s | "Apply" counts the rules that are evaluated and whose precondition holds. On each of these modules, the rules that pass and fail on the generated values are exactly those that pass and fail when the written package is read back with dpm-xbrl and evaluated with dpm-xl, as in [validate the output](../guides/validate-output.md). ## Rules that still fail - **A cell defined twice.** Two rules give the same cell different definitions, such as a capital amount as the sum of its components and as a ratio times the total risk exposure, while the ratio has its own minimum. Each repair undoes the other; the loop stops when the failing count no longer improves. - **Totals over open tables that are also constrained per row.** A sum over all rows of an open table must equal a total, and each row is bounded by other rules; the repair of the sum moves one row past its own bounds, or cannot. - **Chains of inequalities.** `a <= b`, `b <= c` and `c <= a + d`, with some of the cells defined by sums elsewhere, can need several cells to move together. - **Precision.** A rule without `interval: true` that multiplies a ratio by an amount holds before rounding but can miss by a few units after the ratio is rounded to its reporting decimals. [Build a template from a DPM release](../guides/from-a-dpm-release.md#read-the-results) shows how to explain each failing rule with dpm-xl. ## Rules that are not evaluated `not_evaluated` holds the rules dpm-xl reports as an error or as unsupported, with the reason: - `time_shift` compares with an earlier period, and a single report has none; - a rule on a table that is not in the module's template; - a rule that does not parse, or uses a construct dpm-xl does not support. They do not count towards the exit status. ## Rules that do not apply `not_applicable` holds the rules that the generated report gives nothing to check: the precondition is false, or the rule selects no generated cell. On COREP_OF these are mostly rules that filter an open table on key values, such as `[where CEG = [eba_GA:qx2014]]`, when the generated key rows use other items. More rows (`--open-rows`) cover more items, at the cost of time: enumerated keys take their items in order. ## Regular expressions `match` is repaired for the two patterns EBA rules use, LEIs and `dd/mm/yyyy` dates. Any other pattern is evaluated but not repaired. ## What would do better Solving a module's linear equalities jointly, with a sparse least-squares or mixed-integer solver, before repairing the rest would settle most cells defined twice consistently, or prove that the rules contradict each other. The template already carries what such a solver needs. [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Reference Exact, complete descriptions. For explanations, see the [concepts](../concepts/index.md); for recipes, the [how-to guides](../guides/index.md). | Page | Describes | | --- | --- | | [Template format](template-format.md) | The `dpm-synth-template/1` JSON format, field by field, and how it is validated. | | [CLI](cli.md) | The `dpm-synth generate` command and every option. | | [Exit statuses](exit-statuses.md) | What each exit status means, and when the package is written. | | [JSON output](json-output.md) | The layout of `dpm-synth generate --json`, with `schema_version`. | | [Python API](api.md) | Every public name, generated from the docstrings. | [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Template format A template describes one module of a DPM release: its tables, the cells of each table, and the rules a report must satisfy. It is a JSON document (UTF-8) in the `dpm-synth-template/1` format, read by `load_template`. Any DPM source can produce one; see [build a template from a DPM release](../guides/from-a-dpm-release.md) and [write a template by hand](../guides/write-a-template.md). ## Example ```json { "format": "dpm-synth-template/1", "release": "4.3", "module": "COREP_OF", "entry_point": "http://www.eba.europa.eu/eu/fr/xbrl/crr/fws/corep/4.3/mod/corep_of.json", "reference_date": "2026-12-31", "decimals": {"m": 0, "p": 4, "i": 0, "r": 2}, "tables": [ { "code": "C_07.00.a", "filing_indicator": "C_07.00", "open_keys": [ {"code": "qEBB", "column": "qEBB", "items": ["eba_qEC:qx16", "eba_qEC:qx23"]} ], "cells": [ {"id": "table-cell:5", "row": "0010", "column": "0010", "sheet": "*", "datapoint": "dp12345", "data_type": "m"} ] } ], "rules": [ {"source": "eba", "code": "v0001_m", "expression": "…", "precondition": "{v_C_07.00}"} ] } ``` ## The module | Field | Type | Required | Meaning | | --- | --- | --- | --- | | `format` | string | yes | Exactly `"dpm-synth-template/1"`. | | `release` | string | yes | The DPM release, such as `"4.3"`. Informative. | | `module` | string | yes | The module code, such as `"COREP_OF"`. Named in error messages and the JSON output. | | `entry_point` | string or null | no | The module's xBRL-CSV entry point, written to `reports/report.json` as `extends`. | | `reference_date` | string or null | no | The reference date, `YYYY-MM-DD`; it is the `refPeriod` parameter, the value of `d` cells and the rules' `{refPeriod}`. Default `2026-12-31`; the `--reference-date` option overrides it. | | `decimals` | object | no | Reporting decimals by data type code, such as `{"m": -3}`; whole numbers from -20 to 20. Unlisted types keep the defaults `m` 0, `p` 4, `i` 0, `r` 2. | | `tables` | array | yes | At least one [table](#tables). | | `rules` | array | no | The [rules](#rules); none means every value is simply generated. | ## Tables | Field | Type | Required | Meaning | | --- | --- | --- | --- | | `code` | string | yes | The table code, such as `"C_07.00.a"`, as rules name it. Lower-cased, it names the table's file, `reports/c_07.00.a.csv`, so it is letters, digits, `_`, `.` and `-`, starting with a letter or digit, and not `parameters` or `FilingIndicators`. Unique within the template, ignoring case. | | `filing_indicator` | string or null | no | The filing indicator the table is reported under, such as `"C_07.00"`. Default: the code. Every filing indicator of the template is reported as `true`. | | `open_keys` | array | no | The table's [open keys](#open-keys); none for a closed table. | | `cells` | array | yes | At least one [cell](#cells). | ## Open keys An open table has one row per combination of key values, such as one row per counterparty. Each key becomes a column of the table's file. | Field | Type | Required | Meaning | | --- | --- | --- | --- | | `code` | string | yes | The key's property code, as rules name it, such as `"qEBB"` in `[where qEBB = …]`. | | `column` | string or null | no | The key's column in the table's file. Default: the code. Unique within the table, and not `datapoint` or `factValue`. | | `items` | array of strings or null | no | The allowed item qnames of an enumerated key, such as `"eba_qEC:qx16"`. Null or empty for a typed key, whose values are generated LEI-shaped identifiers starting with `SYNTH`. | The generator gives every open table `open_rows` key rows (default 2). Enumerated keys take their items in order, row by row, so an enumerated key with fewer items than rows gives fewer distinct rows. ## Cells | Field | Type | Required | Meaning | | --- | --- | --- | --- | | `id` | string | yes | An identifier of the cell, unique within the template. Only used in error messages. | | `datapoint` | string | yes | The datapoint code written in the table's `datapoint` column, such as `"dp12345"`. Two tables may share a datapoint; they then share its value. | | `data_type` | string or null | no | The [data type](#data-types). | | `row` | string or null | no | The row code, such as `"0010"`, without the `r`. Null for a table without rows; `"*"` for a cell of every row of an open table, which only a selection without rows, or with `r*`, selects. | | `column` | string or null | no | The column code, such as `"0010"`. | | `sheet` | string or null | no | The sheet code; `"*"` for a cell on every sheet, as in open tables. | | `items` | array of strings or null | no | The allowed item qnames of an enumerated cell. The first is its starting value. | Rules select cells by table, row, column and sheet, as in `{tC_07.00.a, r0010, c0010}`; the generator answers such a selection with the template's cells, for every key row of an open table. ## Data types `data_type` is a DPM data type code. It decides the starting value, which repairs apply, how the value is rounded and how it is written. | Code | Type | Starting value | Written as | | --- | --- | --- | --- | | `m` | monetary | a multiple of 1,000 from 1,000 to 1,000,000 | a decimal with `decimals.m` places | | `p` | percentage | 0.01 to 0.99 | a decimal with `decimals.p` places | | `i` | integer | 1 to 50 | a decimal with `decimals.i` places | | `r` | decimal | 1.00 to 9.99 | a decimal with `decimals.r` places | | `e` | enumerated | the first of `items`, else `eba_qx:qx0` | the item qname as the template writes it | | `b` | boolean | `true` | `true` or `false` | | `d` | date | the reference date | `YYYY-MM-DD` | | anything else, or null | text | a random LEI-shaped identifier starting with `SYNTH` | the text | Only numeric cells (`m`, `p`, `i`, `r`) take part in arithmetic repairs. Negative decimals round to tens, hundreds or thousands: with `"m": -3` a monetary value is written as a multiple of 1,000. ## Rules | Field | Type | Required | Meaning | | --- | --- | --- | --- | | `code` | string | yes | The rule code, such as `"v0001_m"`. Unique within the template: results list rules by code. | | `expression` | string | yes | The DPM-XL expression, as [dpm-xl] parses it. | | `precondition` | string or null | no | A DPM-XL precondition, such as `"{v_C_07.00}"`; the rule applies only when it holds. | | `source` | string or null | no | Who defines the rule, such as `"eba"`. Default `"eba"`. Informative: filter the rules before generating if you want only some sources. | Every table of the template counts as reported, so a precondition such as `{v_C_07.00}` holds when the template has that table or filing indicator. `{vReportingLevel}` is `con` or `ind` and `{refPeriod}` is the reference date. A rule that does not parse is not an error of the template: it is reported as not evaluated, with the syntax error. ## Validation `load_template` checks every field above and raises `TemplateError` for the first problem, naming the field by its path. Text that cannot be written as UTF-8, such as a lone surrogate escape `\ud800` in the JSON, is rejected too, since the package is UTF-8: ```python import json from pathlib import Path from dpmsynth import TemplateError, load_template document = json.loads(Path("minimal.json").read_text(encoding="utf-8")) document["tables"][0]["cells"][1]["datapoint"] = 1002 try: load_template(document) except TemplateError as error: print(error) ``` ```text tables[0].cells[1].datapoint: expected a non-empty string, got the number 1002 ``` A file that cannot be read or is not JSON raises `TemplateError` too, and the command line reports any of them with exit status 3. ## Compatibility Fields a version does not know are ignored, so a template with extra fields loads, and exports may carry more information than dpm-synth uses. A change that a reader of `dpm-synth-template/1` cannot ignore gets a new format version, `dpm-synth-template/2`; see [stability and versioning](../project/stability.md). [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # CLI Installing dpm-synth installs the `dpm-synth` command, with one subcommand, `generate`. `uvx dpm-synth` runs it without installing. ```console $ dpm-synth --help usage: dpm-synth [-h] [--version] {generate} ... Generate synthetic EBA xBRL-CSV reports that satisfy a module's DPM-XL validation rules. positional arguments: {generate} generate write an xBRL-CSV report package for a template options: -h, --help show this help message and exit --version show program's version number and exit examples: dpm-synth generate template.json -o report.zip dpm-synth generate template.json -o report.zip --seed 7 --open-rows 3 dpm-synth generate template.json -o report.zip --json dpm-synth generate template.json -o big.zip --scale-rows 1000000 exit statuses: 0 every evaluable rule passes, 1 some rules still fail (the package is written), 2 usage error, 3 unreadable or invalid template, or the package cannot be written. The template format: https://luukhopman.github.io/dpm-synth/reference/template-format/ ``` ## `dpm-synth generate` ```console $ dpm-synth generate --help usage: dpm-synth generate [-h] -o ZIP [--seed N] [--open-rows N] [--max-rounds N] [--scale-rows N] [--reference-date YYYY-MM-DD] [--entity LEI] [--level {CON,IND}] [--json] template Generate values for every cell of a template, repair them until the rules pass, and write an xBRL-CSV report package. positional arguments: template dpm-synth-template/1 JSON file options: -h, --help show this help message and exit -o, --output ZIP report package to write (.zip) --seed N random seed (default 0) --open-rows N key rows per open table, at most 1000 (default 2) --max-rounds N most evaluate-and-repair rounds (default 40) --scale-rows N after solving, write every open table with N key rows (load testing; rules over open-table totals will fail) --reference-date YYYY-MM-DD reference date (default: the template's, else 2026-12-31) --entity LEI LEI of the reporting entity (default SYNTHETICDPMXL000002) --level {CON,IND} reporting level, the entity suffix (default CON) --json print a JSON summary ``` `generate` loads the [template](template-format.md), runs `generate` and writes the report with `dpmxbrl.write_xbrl_csv`, or with `write_scaled_package` for `--scale-rows`. It prints a summary: ```console $ dpm-synth generate sample.json -o sample.zip --seed 7 rules: 9, passed: 8, failed: 0, not evaluated: 1, not applicable: 0 rounds: 5 (failing rules per round: 5, 3, 1, 1, 0) wrote sample.zip (20 facts) ``` - `rules` counts the template's rules; `passed`, `failed`, `not evaluated` (dpm-xl could not evaluate the rule, such as a `time_shift`) and `not applicable` (its precondition is false, or it selects no cell) add up to it. - `rounds` lists the failing rules at the start of each round. - When rules still fail, a line `still failing: ...` names up to 20 of them. - `wrote` names the package and the facts in it. | Option | Effect | | --- | --- | | `-o`, `--output ZIP` | The package to write; required, and must end in `.zip`. Its name without `.zip` names the package's root folder. An existing file is replaced. | | `--seed N` | Random seed, any whole number. The same template, seed, options and versions give the same package; see [reproduce a report](../guides/reproducible-seeds.md). | | `--open-rows N` | Key rows per open table while solving, from 1 to 1,000 (`MAX_OPEN_ROWS`). A table whose keys are all enumerated gets at most as many rows as its item lists repeat after. | | `--max-rounds N` | Most evaluate-and-repair rounds, 0 or more; 0 writes the starting values. | | `--scale-rows N` | Write every open table with `N` key rows, from 1 to 9,999,999,999, by repeating the solved rows under new keys; see [load-test with scaled packages](../guides/load-testing.md). The summary and the exit status are those of the unscaled report. | | `--reference-date YYYY-MM-DD` | Overrides the template's reference date; exactly `YYYY-MM-DD`, as in the template. | | `--entity LEI` | The reporting entity, letters and digits (`.`, `_` and `-` allowed). The entity identifier is `.` in the reporting-subject scheme. | | `--level CON\|IND` | The reporting level, consolidated or individual. | | `--json` | Print the summary as JSON instead; see [JSON output](json-output.md). | | `--version` | Print the version and exit. | ## Errors Errors are one line on standard error, starting with `error:`, and never a traceback: ```console $ dpm-synth generate missing.json -o report.zip error: missing.json: cannot read: No such file or directory $ echo $? 3 $ dpm-synth generate minimal.json -o report.csv usage: dpm-synth generate [-h] -o ZIP [--seed N] [--open-rows N] [--max-rounds N] [--scale-rows N] [--reference-date YYYY-MM-DD] [--entity LEI] [--level {CON,IND}] [--json] template dpm-synth generate: error: argument -o/--output: expected a .zip file, got 'report.csv' $ echo $? 2 ``` See [exit statuses](exit-statuses.md) for every status. [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Exit statuses | Status | Meaning | Package written | | --- | --- | --- | | 0 | Every rule that could be evaluated passes. Rules that are not evaluated (such as `time_shift`) or not applicable do not count. | yes | | 1 | Some rules still fail; the summary names them. | yes | | 2 | Usage error: an unknown option, a missing argument, or a bad option value, such as `--open-rows 0`, `--open-rows 5000`, a reference date not written as `YYYY-MM-DD` or an output not ending in `.zip`. argparse prints the usage and the error. | no | | 3 | The template cannot be read, is not JSON or is not a valid [template](template-format.md); or the package cannot be written. One line `error: ...` on standard error. | no | The statuses match the sibling commands: 2 is always a usage error, and neither tool prints a traceback for bad input. With `--scale-rows`, the status is that of the unscaled report, since the scaled rows are not evaluated. A rule that cannot be satisfied makes status 1, not a failure of the run: the package is still useful for testing. To accept a known set of failing rules in CI, compare the `failed` list of the [JSON output](json-output.md) with the list you expect instead of relying on the status: ```console $ dpm-synth generate sample.json -o sample.zip --seed 7 --json --max-rounds 1 { "schema_version": 1, "output": "sample.zip", "release": "1.0", "module": "SAMPLE", "seed": 7, "rules": 9, "passed": 5, "failed": [ "open", "open-total", "rows" ], "not_evaluated": { "history": "The rule compares with the previous reporting period (time_shift), which a single report does not include" }, "not_applicable": [], "rounds": 1, "history": [ 5 ], "repairs": 10, "facts": 20 } $ echo $? 1 ``` [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # JSON output `dpm-synth generate --json` prints one JSON object (UTF-8, two-space indentation) instead of the text summary. The package is written and the [exit status](exit-statuses.md) is the same as without `--json`. ```console $ dpm-synth generate sample.json -o sample.zip --seed 7 --json { "schema_version": 1, "output": "sample.zip", "release": "1.0", "module": "SAMPLE", "seed": 7, "rules": 9, "passed": 8, "failed": [], "not_evaluated": { "history": "The rule compares with the previous reporting period (time_shift), which a single report does not include" }, "not_applicable": [], "rounds": 5, "history": [ 5, 3, 1, 1, 0 ], "repairs": 16, "facts": 20 } ``` ## Fields | Field | Type | Meaning | | --- | --- | --- | | `schema_version` | integer | Version of this layout, currently `1`. | | `output` | string | The package written, as given to `--output`. | | `release` | string | The template's `release`. | | `module` | string | The template's `module`. | | `seed` | integer | The random seed. | | `rules` | integer | Rules in the template. | | `passed` | integer | Rules that pass on the written values. | | `failed` | array of strings | Codes of the rules that still fail, in the order they were last evaluated. | | `not_evaluated` | object | Rule code to the reason dpm-xl could not evaluate the rule, in template order. | | `not_applicable` | array of strings | Codes of the rules whose precondition is false or that select no cell. | | `rounds` | integer | Evaluate-and-repair rounds run. | | `history` | array of integers | Failing rules at the start of each round. | | `repairs` | integer | Failing records repaired over all rounds. | | `facts` | integer | Facts written; with `--scale-rows`, those of the scaled package. | `passed`, the lengths of `failed`, `not_evaluated` and `not_applicable` add up to `rules`. The fields are those of `GenerationResult`. ## Compatibility `schema_version` changes only with a new major version of dpm-synth. Within a major version, fields are not removed or renamed and their types do not change; new fields may be added, so ignore fields you do not know. The `not_evaluated` reasons come from dpm-xl and may be reworded. See [stability and versioning](../project/stability.md). [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Python API The public API is the names in `dpmsynth.__all__`, all on this page and covered by the [stability policy](../project/stability.md). They are defined in `dpmsynth.template`, `dpmsynth.generator` and `dpmsynth.scale`; import them from `dpmsynth`. Names that start with an underscore, such as the private methods of `SyntheticData`, are internal. ```python from dpmxbrl import write_xbrl_csv from dpmsynth import generate, load_template, write_scaled_package result = generate(load_template("sample.json"), seed=7) write_xbrl_csv(result.report, "report.zip") written = write_scaled_package(result.data, "big.zip", rows=10_000) print(result.passed, result.failed, written) ``` ```text 8 [] {'S_01.00': 14, 'S_02.00.a': 4, 'S_03.00': 10000} ``` ## Thread safety `generate` keeps no shared state, so several templates may be generated in parallel threads; it is CPU-bound, so processes run faster. A `SyntheticData` is changed while it is generated and is safe to read from several threads afterwards; templates are immutable. ## Templates ## `dpmsynth.load_template` ```python def load_template(source: str | os.PathLike[str] | Mapping[str, Any]) -> ModuleTemplate ``` Load and validate a template from a JSON file or a parsed document. Fields this version does not know are ignored, so newer exports still load. Args: source: A path to a JSON file, or the document as a mapping. Returns: ModuleTemplate: The validated template. Raises: TemplateError: If the file cannot be read, is not JSON, or the document is not a valid ``dpm-synth-template/1`` template. Examples: >>> template = load_template("minimal.json") >>> template.module, len(template.tables), len(template.rules) ('DEMO', 1, 2) ## `dpmsynth.ModuleTemplate` ```python class ModuleTemplate(release: str, module: str, entry_point: str | None, tables: tuple[TemplateTable, ...], rules: tuple[TemplateRule, ...], reference_date: date | None = None, decimals: Mapping[str, int] = dict()) ``` Everything needed to generate one module's report. Build one with `load_template`. Attributes: release: The DPM release, such as ``4.3``. module: The module code, such as ``COREP_OF``. entry_point: The module's entry point, written to ``report.json``. tables: The module's tables. rules: The rules the report should satisfy. reference_date: The report's reference date; 2026-12-31 when ``None``. decimals: Reporting decimals by data type code; ``m`` 0, ``p`` 4, ``i`` 0 and ``r`` 2 unless given. ## `dpmsynth.TemplateTable` ```python class TemplateTable(code: str, filing_indicator: str, cells: tuple[TemplateCell, ...], open_keys: tuple[OpenKey, ...] = ()) ``` A table, its filing indicator, open keys and cells. Attributes: code: The table code, such as ``C_07.00.a``; it names the CSV file. filing_indicator: The filing indicator the table is reported under. cells: The reportable cells. open_keys: The open-axis key columns; empty for a closed table. ## `dpmsynth.OpenKey` ```python class OpenKey(code: str, column: str, items: tuple[str, ...] | None = None) ``` An open-axis key column of a table. Attributes: code: The key's property code, as rules name it, such as ``qEBB``. column: The column of the key in the table's CSV file. items: Allowed item qnames for an enumerated key; ``None`` for a typed key, which gets generated identifiers. ## `dpmsynth.TemplateCell` ```python class TemplateCell(id: str, datapoint: str, data_type: str | None, row: str | None = None, column: str | None = None, sheet: str | None = None, items: tuple[str, ...] | None = None) ``` One reportable cell and the datapoint that carries its value. Attributes: id: An identifier of the cell, unique within the template. datapoint: The datapoint code written in the ``datapoint`` column. data_type: The DPM data type code, such as ``m`` or ``e``; ``None`` or an unknown code is text. row: The row code, such as ``0010``; ``None`` for a table without rows, ``*`` for every row of an open table. column: The column code. sheet: The sheet code, ``*`` for every sheet, or ``None``. items: Allowed item qnames of an enumerated cell. ## `dpmsynth.TemplateRule` ```python class TemplateRule(code: str, expression: str, precondition: str | None = None, source: str = 'eba') ``` A validation rule with its precondition. Attributes: code: The rule code, such as ``v0001_m``. expression: The DPM-XL expression. precondition: An optional DPM-XL precondition, such as ``{v_C_01.00}``. source: Who defined the rule, such as ``eba``. ## `dpmsynth.TemplateError` ```python class TemplateError(ValueError) ``` Raised when a template cannot be read or is not a valid template. The message names the offending field by its path in the document, such as ``tables[2].cells[0].datapoint``. Examples: >>> load_template({"format": "dpm-synth-template/1", "module": "M"}) Traceback (most recent call last): ... dpmsynth.template.TemplateError: release: required ## `dpmsynth.TEMPLATE_FORMAT` ```python TEMPLATE_FORMAT = 'dpm-synth-template/1' ``` The ``format`` value of the templates this version reads. ## Generating ## `dpmsynth.generate` ```python def generate(template: ModuleTemplate, *, seed: int = 0, open_rows: int = 2, max_rounds: int = 40, entity: str = DEFAULT_ENTITY, reporting_level: str = 'CON', reference_date: date | None = None) -> GenerationResult ``` Generate a report for ``template`` that satisfies as many rules as it can. Args: template: Module template, from `load_template`. seed: Random seed; the same seed, template, options and versions give the same report. open_rows: Key rows per open table, from 1 to ``MAX_OPEN_ROWS`` (1,000). Enumerated keys take their allowed items in order, so a table whose keys allow fewer combinations gets fewer rows. max_rounds: Upper bound on evaluate-and-repair rounds; 0 evaluates the initial values only. entity: LEI of the reporting entity. reporting_level: Suffix of the entity identifier, such as ``CON`` or ``IND``; also the ``ReportingLevel`` parameter, in lower case. reference_date: Overrides the template's reference date. Returns: GenerationResult: The report and the final rule status. Raises: ValueError: If ``open_rows`` or ``max_rounds`` is out of range, or ``entity`` or ``reporting_level`` is not an identifier. Examples: >>> from dpmsynth import load_template >>> result = generate(load_template("minimal.json"), seed=7) >>> result.passed, result.failed (2, []) >>> result.report.entity_id 'SYNTHETICDPMXL000002.CON' ## `dpmsynth.GenerationResult` ```python class GenerationResult(report: Report, data: SyntheticData, rules: int, passed: int, failed: list[str], not_evaluated: dict[str, str], not_applicable: list[str], rounds: int, repairs: int, history: list[int] = list()) ``` The generated report and how far the repair loop got. ``passed``, ``failed``, ``not_evaluated`` and ``not_applicable`` together count every rule of the template once. Attributes: report: The report, ready for ``dpmxbrl.write_xbrl_csv``. data: The generated values, a dpm-xl data provider; pass it to `write_scaled_package` or evaluate rules on it. rules: Rules in the template. passed: Rules that pass on the final, rounded values. failed: Codes of the rules that still fail. not_evaluated: Rules dpm-xl could not evaluate, such as ``time_shift`` without a previous report or a syntax error, with the reason, by rule code. not_applicable: Codes of the rules whose precondition is false or that select no cell of the template. rounds: Evaluate-and-repair rounds run. repairs: Failing records repaired over all rounds. history: Failing rules at the start of each round. ## `dpmsynth.SyntheticData` ```python class SyntheticData(template: ModuleTemplate, rng: random.Random, open_rows: int, *, entity: str = DEFAULT_ENTITY, reporting_level: str = 'CON') ``` Generated values for a template, readable as a dpm-xl data provider. `generate` creates and repairs one; it is `GenerationResult.data`. It implements the ``dpmxl.DataProvider`` protocol, with the optional ``open_keys``, so ``dpmxl.evaluate_rule(expression, data)`` evaluates any rule on the generated values. Attributes: template: The template, with the reference date the report uses. values: Cell values by ``(datapoint, keys)``, where ``keys`` is a tuple of ``(key code, value)`` pairs of an open table's row, and ``()`` for a closed table. Numbers are ``dpmxl.Number``, enumerated values ``dpmxl.Item``. key_rows: Key rows by table code; ``[()]`` for a closed table. entity: The LEI of the reporting entity. reporting_level: ``CON`` or ``IND``, the suffix of the entity identifier and the ``ReportingLevel`` parameter. decimals: Reporting decimals by data type code. ## `dpmsynth.DEFAULT_ENTITY` ```python DEFAULT_ENTITY = 'SYNTHETICDPMXL000002' ``` The fictional LEI that reports are generated for unless told otherwise. ## `dpmsynth.DEFAULT_REFERENCE_DATE` ```python DEFAULT_REFERENCE_DATE = date(2026, 12, 31) ``` The reference date of a template that gives none. ## `dpmsynth.LEI_PREFIX` ```python LEI_PREFIX = 'SYNTH' ``` The prefix of every LEI-shaped identifier dpm-synth generates. Generated identifiers are ``SYNTH`` followed by 13 letters and digits and two ISO 17442 check digits, so they pass LEI validation but are visibly fictional and cannot coincide with an entity's real LEI by chance. ## `dpmsynth.MAX_OPEN_ROWS` ```python MAX_OPEN_ROWS = 1000 ``` The most key rows per open table that ``generate`` solves. Every rule is evaluated on every key row in every round, so the time grows with the rows; for larger tables, write a scaled package instead. ## Writing ## `dpmsynth.build_report` ```python def build_report(data: SyntheticData) -> Report ``` Turn generated values into a report that dpm-xbrl can write. Facts follow the EBA xBRL-CSV convention: a ``datapoint`` and ``factValue`` per cell, with the open keys of the row. The entity is ``.`` in the reporting-subject scheme, every table's filing indicator is reported, and a ``decimals*`` parameter is set for each numeric data type the template uses. `generate` calls it; call it yourself after changing ``data.values``. Args: data: Generated values, ``GenerationResult.data``. Returns: Report: The report, for ``dpmxbrl.write_xbrl_csv``. Examples: >>> from dpmsynth import generate, load_template >>> data = generate(load_template("minimal.json"), seed=7).data >>> report = build_report(data) >>> [fact.datapoint for fact in report.facts] ['dp1001', 'dp1002', 'dp1003'] ## `dpmsynth.write_scaled_package` ```python def write_scaled_package(data: SyntheticData, target: str | os.PathLike[str], *, rows: int | Mapping[str, int]) -> dict[str, int] ``` Write an xBRL-CSV package with every open table scaled to ``rows`` key rows. Closed tables are written as generated. Each open table gets ``rows`` key rows, whose values repeat the generated rows in turn. Typed keys get new identifiers; enumerated keys keep the values of the generated row that is repeated, so rules that filter on them still hold, and a table with only enumerated keys keeps its generated rows. The package is laid out, named and described exactly as ``dpmxbrl.write_xbrl_csv`` writes the unscaled report, but tables are streamed to the ZIP file, so memory stays flat. Args: data: Generated values, ``GenerationResult.data``. target: ZIP file to write; its name without ``.zip`` names the package's root folder. rows: Key rows per open table, or a mapping of table code to rows; tables missing from the mapping keep their generated rows. Returns: dict[str, int]: Fact rows written per table code. Raises: ValueError: If a row count is not a whole number of 1 or more. dpmxbrl.ReportError: If dpm-xbrl cannot write the package header. OSError: If the file cannot be written. Examples: >>> from dpmsynth import generate, load_template >>> result = generate(load_template("sample.json"), seed=7) >>> written = write_scaled_package(result.data, "big.zip", rows=1000) >>> written["S_03.00"], written["S_02.00.a"] (1000, 4) ## Version ## `dpmsynth.__version__` ```python __version__ = version('dpm-synth') ``` [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Using dpm-synth with AI agents This page is for coding agents that write code *using* dpm-synth, and for people who brief them. It gives the essential API, the contracts to rely on and the mistakes to avoid. Agents working *on* the dpm-synth repository should read `AGENTS.md` in its root instead. Machine-readable copies of these docs: - [`llms.txt`](llms.txt): an index of every page with a one-line description ([llmstxt.org](https://llmstxt.org) format). - [`llms-full.txt`](llms-full.txt): every page in one Markdown file, including the API reference. - Each page as Markdown: replace the trailing `/` of a page URL with `.md`, for example `https://luukhopman.github.io/dpm-synth/guides/load-testing.md` (section index pages are `.../guides/index.md`). The sibling packages have the same files: [dpm-xl](https://luukhopman.github.io/dpm-xl/llms.txt) and [dpm-xbrl](https://luukhopman.github.io/dpm-xbrl/llms.txt). ## Essentials ```python from dpmxbrl import write_xbrl_csv from dpmxl import evaluate_rule from dpmsynth import TemplateError, generate, load_template, write_scaled_package try: template = load_template("sample.json") # a path, or the parsed JSON as a dict except TemplateError as error: # the only exception for a bad template raise SystemExit(f"bad template: {error}") result = generate(template, seed=7) # same inputs and versions, same report result.passed # 8: rules that pass on the rounded values result.failed # []: codes of rules that still fail result.not_evaluated # {"history": "...time_shift..."}: dpm-xl could not evaluate result.not_applicable # []: precondition false, or no cell selected result.report # dpmxbrl.Report, ready to write result.data # SyntheticData: a dpm-xl provider over the values write_xbrl_csv(result.report, "report.zip") write_scaled_package(result.data, "big.zip", rows=1000) # load testing evaluate_rule("{tS_01.00, r0010, c0010} >= 0", result.data).status ``` | Task | Call | | --- | --- | | Load a template | `load_template(path_or_mapping) -> ModuleTemplate`; raises `TemplateError` | | Generate | `generate(template, *, seed=0, open_rows=2, max_rounds=40, entity="SYNTHETICDPMXL000002", reporting_level="CON", reference_date=None) -> GenerationResult` | | Write the package | `dpmxbrl.write_xbrl_csv(result.report, "report.zip")` | | Millions of rows | `write_scaled_package(result.data, "big.zip", rows=N or {table: N}) -> dict[str, int]` | | Rebuild the report after changing values | `build_report(result.data) -> dpmxbrl.Report` | | Evaluate a rule on the values | `dpmxl.evaluate_rule(expression, result.data, precondition=...)` | | Keep only some rules | `dataclasses.replace(template, rules=tuple(...))` | | Command line | `dpm-synth generate TEMPLATE -o OUT.zip [--seed N] [--json]`; exit 0 all pass, 1 some fail, 2 usage, 3 bad input | ## Contracts - `load_template` raises `TemplateError` (a `ValueError`) for every unreadable, non-JSON or invalid template, naming the field by its path, such as `tables[2].cells[0].datapoint`. Unknown fields are ignored. - `generate` raises `ValueError` only for invalid options. Rules that do not parse or cannot be evaluated never raise: they are in `not_evaluated`. - `passed + len(failed) + len(not_evaluated) + len(not_applicable) == rules`. - The same template, seed, options, output file name and versions give the same bytes. - `ModuleTemplate` and its parts are frozen dataclasses; `GenerationResult` is frozen too, but `result.data.values` is a mutable dict. - `--json` output carries `schema_version` (1); it changes only with a major version. - Every generated LEI-shaped identifier starts with `SYNTH` (`LEI_PREFIX`) and has valid check digits; the default entity is the fictional `SYNTHETICDPMXL000002`. None can be a real entity's LEI. - Nothing is fetched over the network. ## Pitfalls - **Not every rule passes.** Real modules keep some failing rules; read `result.failed`, never assume it is empty. See [limits of the loop](concepts/limits.md). - **Exit status 1 is not a crash.** The package is written; compare the JSON `failed` list with what you expect. - **Scaled rows are not evaluated.** Rules that sum an open table fail on a scaled package; use the unscaled one to test validation results. - **The entity and reporting level live in the data.** Pass `entity`, `reporting_level` and `reference_date` to `generate`; `build_report` and `write_scaled_package` take them from `result.data`. - **The root folder is the file name.** Packages are named as dpm-xbrl names them: `report.zip` holds `report/...`. The EBA file-naming filing rule is a warning on synthetic names. - **Rule order matters for reproducibility.** Reordering the template's tables, cells or rules can change the values for a seed. - **Values are synthetic.** They satisfy rules, not economics; do not use them as realistic magnitudes. ## Where to look next - [Getting started](getting-started.md) for a guided first use. - [Template format](reference/template-format.md) to write or check a template. - [Python API](reference/api.md) for every signature, and [CLI](reference/cli.md), [exit statuses](reference/exit-statuses.md) and [JSON output](reference/json-output.md). [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Changelog All notable changes to this project are documented here. The project follows [semantic versioning](https://semver.org). ## 1.0.0 - 2026-10-24 The first public release. ### Generating reports - `generate` fills every cell of a `dpm-synth-template/1` template with a value for its data type and repairs the values, round by round, until the module's DPM-XL rules pass. Repairs cover sums, differences, products of two cells, `abs`, sign rules (also inside `and`), `isnull`, `in` with any set operand, `match` for LEIs and `dd/mm/yyyy` dates, `and`, `or`, `not` and `if`. Values are rounded to the reporting decimals and every rule is evaluated once more. - `GenerationResult` holds the report, ready for `dpmxbrl.write_xbrl_csv`, the generated values (`data`, a dpm-xl data provider) and every rule counted once: `passed`, `failed`, `not_evaluated` with the reason, and `not_applicable`. - Reports are reproducible: the same template, seed, options and versions give the same bytes. - Reports are synthetic: the entity is the fictional LEI `SYNTHETICDPMXL000002` (`DEFAULT_ENTITY`), and every generated LEI-shaped identifier starts with `SYNTH` (`LEI_PREFIX`) and has valid ISO 17442 check digits, so none can be a real entity's LEI. - `build_report` turns changed values into a report again. - `write_scaled_package` streams open tables with millions of rows into a package laid out as dpm-xbrl writes it, for load tests; scaled rows keep the enumerated key values of the solved rows they repeat. ### Templates - `load_template` reads and validates a template field by field, and raises `TemplateError`, naming the field such as `tables[2].cells[0].datapoint`, for an unreadable, non-JSON or invalid template, including repeated table, cell or rule codes and text that cannot be written as UTF-8. Unknown fields are ignored. ### Command line - `dpm-synth generate` with `--seed`, `--open-rows`, `--max-rounds`, `--scale-rows`, `--reference-date`, `--entity`, `--level` and `--json` (with `schema_version` 1), `--version`, and examples in `--help`. - Exit statuses: 0 every evaluable rule passes, 1 some fail (the package is written), 2 usage error, 3 unreadable or invalid template, or a package that cannot be written. ### Project - Depends on dpm-xl and dpm-xbrl 1.x, and uses only their public APIs. - Python 3.11 to 3.15, tested in CI. - A documentation site at https://luukhopman.github.io/dpm-synth/, with `llms.txt` and `llms-full.txt` for AI agents. [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Stability and versioning dpm-synth follows [semantic versioning][semver]. A major version may break the public API; minor and patch versions do not. The [changelog](changelog.md) lists every change, with breaking changes marked. ## The public API - The names in `dpmsynth.__all__`, documented in the [Python API](../reference/api.md), with the fields of `ModuleTemplate`, `TemplateTable`, `OpenKey`, `TemplateCell`, `TemplateRule` and `GenerationResult`, and the attributes of `SyntheticData` documented there. - The `TemplateError` contract: every unreadable or invalid template raises `TemplateError`, a `ValueError`. - The [`dpm-synth-template/1`](../reference/template-format.md) format. A change that a reader of version 1 cannot ignore gets a new format version; dpm-synth keeps reading version 1 for the rest of the major version. - The CLI: its commands, options, [exit statuses](../reference/exit-statuses.md) and the [`--json` layout](../reference/json-output.md), versioned by `schema_version`. Modules and names that start with an underscore are internal, as are names not in `dpmsynth.__all__`; import public names from `dpmsynth`. `dpmsynth.__version__` gives the installed version. ## What may change in a minor release - New functions, parameters with defaults, fields, CLI options, JSON fields and repairs. - **The values generated for a seed.** A better repair, or a fix in dpm-xl or dpm-xbrl, can change the values and which rules pass. The changelog notes it. Pin all three packages if a fixture must stay byte for byte the same; see [reproduce a report](../guides/reproducible-seeds.md). - The wording of the text summary, error messages and `not_evaluated` reasons. - Performance characteristics. ## Python versions dpm-synth supports Python 3.11 to 3.15. Support for a Python version is dropped only in a minor release, after its end of life. ## Dependencies dpm-synth depends on dpm-xl and dpm-xbrl, within one major version of each (`>=1.0,<2`), and uses only their public APIs. [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Security policy dpm-synth reads templates and writes report packages. Templates are usually exported by your own tools, but they may come from elsewhere, so bugs in reading them can matter. ## Supported versions Security fixes are released for the latest minor version of the latest major version. ## Reporting a vulnerability Please report vulnerabilities privately, not in a public issue: use [GitHub's private vulnerability reporting](https://github.com/luukhopman/dpm-synth/security/advisories/new) for the repository, or e-mail github@luukhopman.nl. Include the dpm-synth, dpm-xl, dpm-xbrl and Python versions, and, if you can, a small template that shows the problem. The fix is released as a patch version and described in the changelog. ## What counts - Loading a template raises anything other than `TemplateError`, or the command line prints a traceback, for any template file. - Generating writes files other than the output package, makes a network request, or runs code from the template. Rules are DPM-XL, evaluated by dpm-xl, never Python. - A written package lets a reader escape its root folder, for example through a table code; table codes are restricted to safe file names. Generation time grows with the size of a template and its rules, and scaled packages are as large as asked for; bound `--scale-rows` and the template size when they come from untrusted sources. Problems in reading or writing packages, or in evaluating rules, belong to [dpm-xbrl](https://luukhopman.github.io/dpm-xbrl/project/security/) and [dpm-xl](https://luukhopman.github.io/dpm-xl/project/security/). [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Contributing Bug reports, questions and pull requests are welcome on [GitHub](https://github.com/luukhopman/dpm-synth/issues). For a larger change, such as a new repair or a solver, open an issue first to agree on the approach. ## Set up dpm-synth uses [uv](https://docs.astral.sh/uv/): ```text git clone https://github.com/luukhopman/dpm-synth cd dpm-synth uv sync ``` ## Check a change ```text uv run pytest # tests, including every docs example uv run ruff check uv run ruff format --check uv run ty check uv run python scripts/build_docs.py # strict docs build into site/ ``` `uv run zensical serve` previews the documentation while you edit it. ## Conventions - Every behaviour has a test. Build templates in code with `tests/support.py`, or add a small one to `docs/samples`: every sample is generated, written, read back with dpm-xbrl and evaluated with dpm-xl by `tests/test_end_to_end.py`. - Public functions and classes have Google-style docstrings; they are the API reference. - dpm-synth depends on dpm-xl and dpm-xbrl only, and uses only the names in their `__all__`. New runtime dependencies need a good reason. - Keep `__all__`, the docs and the [changelog](changelog.md) in step with the code. Examples in the docs are run by `tests/test_docs.py`, so show real output. - A change to the generated values for a seed is noted in the changelog. - Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/), such as `fix(generator): keep bounds when repairing products`. `AGENTS.md` in the repository holds the same information for coding agents. [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org # Licence dpm-synth is released under the MIT licence: ```text MIT License Copyright (c) 2026 Luuk Hopman Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ``` dpm-synth is an independent project, not affiliated with or endorsed by the European Banking Authority. "EBA" and the names of EBA reporting frameworks and modules are used only to describe the reports dpm-synth generates. The reports are synthetic: their entity is a fictional LEI and their values are generated. [dpm-xl]: https://luukhopman.github.io/dpm-xl/ [dpm-xbrl]: https://luukhopman.github.io/dpm-xbrl/ [dpm-xl-provider]: https://luukhopman.github.io/dpm-xl/guides/provider/ [dpm-xbrl-write]: https://luukhopman.github.io/dpm-xbrl/guides/convert-formats/ [dpm-xbrl-filing]: https://luukhopman.github.io/dpm-xbrl/guides/check-filing-rules/ [xbrl-csv-spec]: https://www.xbrl.org/Specification/xbrl-csv/REC-2021-10-13/xbrl-csv-REC-2021-10-13.html [lei-standard]: https://www.iso.org/standard/78829.html [semver]: https://semver.org