Skip to content

dpmxbrl.frames

Columnar reading of large xBRL-CSV packages into polars frames. Needs the frames extra (pip install "dpm-xbrl[frames]"). Everything on this page is in dpmxbrl.frames.__all__.

See read large reports as frames.

Columnar xBRL-CSV reading: one polars data frame per table.

:func:read_xbrl_csv builds one :class:~dpmxbrl.model.Fact per row, which is convenient but costs several microseconds per fact in Python. Reports with millions of open-table rows need columnar access instead: :func:read_xbrl_csv_frames applies the same package, report.json, parameter and filing-indicator checks, then reads each EBA-convention table straight into a polars frame of strings (datapoint, factValue and the key columns), applying the same special-value and duplicate rules.

Requires the optional frames extra (polars). Tables described by resolved xBRL-CSV metadata are not supported here; use :func:read_xbrl_csv.

Filing-rule findings on values are limited to special values (#nil, #empty and others). Checks that :func:read_xbrl_csv makes on every value (the decimals suffix, spaces around key values, special key values) cost a second scan of millions of strings, so they are left to callers that already split values by data type and key column.

Table files are parsed by polars, which differs from the csv module: a row with fewer cells than the header is padded with empty cells instead of rejected, lines must end in LF or CRLF, and line numbers in findings count data rows rather than physical lines, so they differ when a file has blank lines or values spanning lines.

Rows with no fact (an empty value, #empty or #none) and exact duplicate rows are removed, so a frame's row index is not the row's place in its CSV file. :attr:ReportFrames.dropped_rows lists the removed rows by template, which maps a frame row back to its data row when a caller needs the file location.

ReportFrames dataclass 

ReportFrames(
    report: Report,
    tables: Mapping[str, DataFrame],
    dropped_rows: Mapping[str, tuple[int, ...]] = dict(),
)

Report metadata plus one string frame per table template.

Each frame has a categorical datapoint column, a factValue column of strings (null for #nil) and the table's key columns, categorical, in header order. Rows keep file order, without the rows listed in :attr:dropped_rows.

report instance-attribute 

report: Report

The report with facts == (); use :attr:tables for the facts.

tables instance-attribute 

tables: Mapping[str, DataFrame]

Template ID to a frame with datapoint, factValue (strings) and key columns; datapoint and key columns are categorical.

dropped_rows class-attribute instance-attribute 

dropped_rows: Mapping[str, tuple[int, ...]] = field(
    default_factory=dict
)

Template ID to the sorted 0-based data-row indices of the rows removed from its frame: rows without a fact (an empty value, #empty or #none) and exact duplicates of an earlier row. Indices count data rows as read, before any row is removed, and continue across the files of a template in file order, as the frame does; data row i of a file is on line i + 2 when no value spans lines. Only templates with removed rows appear.

fact_count property 

fact_count: int

Total number of fact rows across tables.

Returns:

  • int ( int ) –

    The sum of the frames' heights.

MetadataTablesError 

MetadataTablesError(
    message: str, *, problems: Sequence[str] = ()
)

Bases: ReportError

The package has tables described by xBRL-CSV metadata.

Columnar reading supports the EBA datapoint/factValue convention only; read such a package with :func:dpmxbrl.read_xbrl_csv.

read_xbrl_csv_frames 

read_xbrl_csv_frames(
    source: str | PathLike[str] | bytes,
    *,
    name: str | None = None,
    metadata_resolver: MetadataResolver | None = None,
    taxonomy_packages: str
    | PathLike[str]
    | Sequence[str | PathLike[str]] = (),
    limits: Limits = Limits(),
    on_metadata: Callable[[Report], None] | None = None,
    on_table: Callable[
        [str, DataFrame, tuple[int, ...]], None
    ]
    | None = None,
) -> ReportFrames

Read an xBRL-CSV report package into columnar table frames.

Parameters:

  • source (str | PathLike[str] | bytes) –

    Package bytes, path, or directory.

  • name (str | None, default: None ) –

    Label used in messages.

  • metadata_resolver (MetadataResolver | None, default: None ) –

    Returns the bytes for an absolute extends URL.

  • taxonomy_packages (str | PathLike[str] | Sequence[str | PathLike[str]], default: () ) –

    Taxonomy packages for offline metadata, one path or a sequence.

  • limits (Limits, default: Limits() ) –

    Resource limits for untrusted input.

  • on_metadata (Callable[[Report], None] | None, default: None ) –

    Called with the report (entry point, reference date, parameters and filing indicators, no facts) before the tables are read, so callers can start dependent work that overlaps the table parsing. Exceptions it raises abort the read.

  • on_table (Callable[[str, DataFrame, tuple[int, ...]], None] | None, default: None ) –

    Called on the calling thread with each template ID, its frame and its removed rows (as in :attr:ReportFrames.tables and :attr:ReportFrames.dropped_rows) as soon as the template's last file is read, in package order, so callers can process tables while later ones are still being read. Exceptions it raises abort the read.

Returns:

  • ReportFrames ( ReportFrames ) –

    Report metadata and the table frames.

Raises:

Examples:

>>> from dpmxbrl.frames import read_xbrl_csv_frames
>>> frames = read_xbrl_csv_frames("report.zip")
>>> frames.fact_count, sorted(frames.tables)
(14, ['c_00.01', 'c_26.00', 'c_27.00', 'c_28.00'])
>>> frames.tables["c_28.00"].columns
['datapoint', 'factValue', 'qINC']