Read a report¶

read_report reads both formats and returns a Report. This guide shows the inputs it accepts, how to label input that has no file name, and how to handle failures.

From a path¶

from dpmxbrl import read_report

package = read_report("report.zip")  # xBRL-CSV report package
instance = read_report("instance.xbrl")  # XBRL-XML instance
print(package.format, len(package.facts))
print(instance.format, len(instance.facts))

xbrl-csv 14
xbrl-xml 3

A ZIP (by content, whatever its suffix), a directory and a .json path are read as xBRL-CSV; anything else as XBRL-XML. detect_format tells you which without reading the report:

from dpmxbrl import detect_format

print(detect_format("report.zip"), detect_format("instance.xbrl"))

xbrl-csv xbrl-xml

An unpacked package works too: pass its directory, or the path of its report.json.

from zipfile import ZipFile

ZipFile("report.zip").extractall("unpacked")
print(len(read_report("unpacked/report").facts))
print(len(read_report("unpacked/report/reports/report.json").facts))

14
14

From bytes¶

Uploads and message queues hand over bytes. Pass them directly, with name= to label messages and locations the way the file would be:

from pathlib import Path

data = Path("report.zip").read_bytes()
report = read_report(data, name="SYNTHETICDPMXL000002.CON_upload.zip")
print(report.filing[0].examples)

('SYNTHETICDPMXL000002.CON_upload.zip',)

The name matters for the EBA file-naming rule, which checks the file name. Without name=, ZIP bytes are labelled <bytes> and XML bytes instance.xbrl.

Use the facts¶

Facts are frozen dataclasses. Their mappings (dimensions, typed_dimensions, keys) are read-only and may be shared between facts; copy one with dict() to change it.

fact = instance.facts[2]
print(fact.concept, fact.value)
print(fact.dimensions, fact.typed_dimensions)
labels = dict(fact.dimensions)
labels["note"] = "copied"

eba_met:qNCO Northwind Holdings
{'eba_dim:qNCO': 'eba_qCO:qx2000'} {'eba_dim:qINC': 'CP0001'}

Facts and reports pickle, copy and convert to plain data:

import dataclasses
import json

print(json.dumps(dataclasses.asdict(package.facts[4])))

{"value": "86911000", "concept": null, "datapoint": "dp32026", "template": "c_26.00", "dimensions": {}, "typed_dimensions": {}, "keys": {}, "unit": null, "decimals": null, "location": "reports/c_26.00.csv:4"}

To group facts by template, use Report.facts_for_template with the filing-indicator code. It compares codes case-insensitively; see table codes and filing indicators for how table files map to filing indicators.

Handle failures¶

Every input that cannot be read raises ReportError, a ValueError, naming the file and, where known, the line:

from dpmxbrl import ReportError

for source in ("missing.zip", b"<xbrli:xbrl"):
    try:
        read_report(source)
    except ReportError as error:
        print(error)

missing.zip: no such file or directory
instance.xbrl:1:0: malformed XML (unclosed token: line 1, column 0)

Catch ReportError alone: damaged ZIPs, unsafe entries, malformed JSON, CSV and XML, and inputs over a limit all raise it.

Its problems list every problem found. Broken xBRL-CSV tables, with rows of the wrong length, bad quoting or bad headers, are all read, up to 100 problems, before the error is raised with the first as its message:

import io
from zipfile import ZipFile

buffer = io.BytesIO()
with ZipFile("report.zip") as source, ZipFile(buffer, "w") as target:
    for name in source.namelist():
        data = source.read(name)
        if name.endswith("c_28.00.csv"):
            data += b"dp5488015,1,CP0003,extra\r\ndp5488016\r\n"
        if name.endswith("c_26.00.csv"):
            data += b'dp32027,"12"3\r\n'
        target.writestr(name, data)
try:
    read_report(buffer.getvalue(), name="report.zip")
except ReportError as error:
    print(error)
    for problem in error.problems:
        print(" ", problem)

reports/c_26.00.csv:6: ',' expected after '"' (and 2 more problems)
  reports/c_26.00.csv:6: ',' expected after '"'
  reports/c_28.00.csv:6: expected 3 cells, found 4
  reports/c_28.00.csv:7: expected 3 cells, found 1

Read the warnings¶

Problems that do not stop the read are kept in Report.warnings, in a deterministic order, and repeated findings are aggregated with a count and the first location:

for warning in package.warnings:
    print(warning)

reports/report.json: 1 extended metadata document could not be resolved offline (http://www.eba.europa.eu/eu/fr/xbrl/crr/fws/corep/4.2/mod/corep_le.json); tables without resolved metadata use the EBA datapoint/factValue convention

Warnings describe how the file was read; the EBA filing rules the report breaks are in Report.filing instead. See check EBA filing rules.

Read several reports at once¶

Readers share no state, so threads can read several reports at once. Large packages are mostly CPU-bound Python work, so processes scale better for many big reports:

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor() as pool:
    reports = list(pool.map(read_report, ["report.zip", "instance.xbrl"]))
print([len(item.facts) for item in reports])

[14, 3]

While a reader builds facts it pauses Python's cyclic garbage collector for the whole process and restores it afterwards; see performance.