Skip to content

dpmxl.grammar

The vocabulary of DPM-XL as frozensets, and the TextMate grammar built from it. See Build editor support.

dpmxl.grammar

Machine-readable vocabulary of DPM-XL for editors and other tooling.

The sets below are the words and symbols the tokenizer recognizes; the tests check that tokenize agrees with them. Words are lower case; the tokenizer matches keywords in any case.

KEYWORDS module-attribute 

KEYWORDS = frozenset(
    {
        "and",
        "else",
        "endif",
        "get",
        "group by",
        "if",
        "in",
        "not",
        "or",
        "over",
        "rename",
        "sub",
        "then",
        "to",
        "where",
        "with",
        "xor",
    }
)

Keywords of expressions and clauses.

WINDOW_KEYWORDS module-attribute 

WINDOW_KEYWORDS = frozenset(
    {
        "asc",
        "between",
        "current data point",
        "data points",
        "desc",
        "following",
        "order by",
        "partition by",
        "preceding",
        "range",
        "unbounded",
    }
)

Keywords inside an analytic over (...) clause.

SELECTION_KEYWORDS module-attribute 

SELECTION_KEYWORDS = frozenset({'default', 'interval'})

Option keywords inside a {...} selection.

PARAMETER_TYPES module-attribute 

PARAMETER_TYPES = frozenset(
    {
        "boolean",
        "date",
        "integer",
        "item",
        "number",
        "set-boolean",
        "set-date",
        "set-integer",
        "set-item",
        "set-number",
        "set-string",
        "string",
    }
)

Data types of a {p_Name, type} parameter reference.

LITERAL_WORDS module-attribute 

LITERAL_WORDS = frozenset({'false', 'null', 'true'})

Boolean and null literals.

TIME_PERIODS module-attribute 

TIME_PERIODS = frozenset({'A', 'S', 'Q', 'M', 'W', 'D'})

Period codes of time_shift and range window bounds (upper case).

AGGREGATE_FUNCTIONS module-attribute 

AGGREGATE_FUNCTIONS = frozenset(
    {
        "avg",
        "count",
        "max_aggr",
        "median",
        "min_aggr",
        "sum",
    }
)

Aggregate operators; max and min with one operand aggregate too.

ANALYTIC_FUNCTIONS module-attribute 

ANALYTIC_FUNCTIONS = frozenset({'rank'})

Operators that take only an analytic over (...) clause.

FUNCTIONS module-attribute 

FUNCTIONS = (
    AGGREGATE_FUNCTIONS
    | ANALYTIC_FUNCTIONS
    | frozenset(
        {
            "abs",
            "annualise",
            "date",
            "day",
            "exp",
            "filter",
            "intersect",
            "isnull",
            "len",
            "length",
            "ln",
            "log",
            "match",
            "max",
            "min",
            "month",
            "nvl",
            "power",
            "quarter",
            "semester",
            "set_of",
            "setdiff",
            "sqrt",
            "substr",
            "symdiff",
            "time_shift",
            "union",
            "week",
            "year",
        }
    )
)

Function names, recognized when followed by (.

FUNCTION_ALIASES module-attribute 

FUNCTION_ALIASES = {'max_aggr': 'max', 'min_aggr': 'min'}

Source spellings whose Call.name differs: the aggregate forms of max and min, which evaluators recognize by their single operand.

CALL_NAMES module-attribute 

CALL_NAMES = frozenset(
    canonical_function(name)
    for name in FUNCTIONS - ANALYTIC_FUNCTIONS
)

Every Call.name a known function parses to. Aggregates among them are {canonical_function(name) for name in AGGREGATE_FUNCTIONS}, which equals dpmxl.evaluator.AGGREGATES. Analytic functions (rank, and aggregates with over) parse to Analytic instead of a call.

SIGNATURES module-attribute 

SIGNATURES: Mapping[str, Signature] = MappingProxyType(
    _signatures()
)

The Signature of every name in CALL_NAMES.

OPERATORS module-attribute 

OPERATORS = frozenset(
    {
        "!=",
        "%",
        "&",
        "*",
        "+",
        "-",
        "/",
        ":=",
        "<",
        "<-",
        "<=",
        "<>",
        "=",
        ">",
        ">=",
    }
)

Operator symbols, including the assignments := and <-.

PUNCTUATION module-attribute 

PUNCTUATION = frozenset({"(", ")", ",", ":", ";", "[", "]"})

Punctuation; { and } are the separate brace token kind.

REFERENCE_PREFIXES module-attribute 

REFERENCE_PREFIXES = {
    "t": "table",
    "g": "table_group",
    "v": "variable",
    "o": "operation",
    "p": "parameter",
}

Selection reference prefix to token kind, such as tC_01.00.

AXIS_LETTERS module-attribute 

AXIS_LETTERS = frozenset({'r', 'c', 's'})

Prefixes of row, column and sheet coordinates inside a selection.

TOKEN_KINDS module-attribute 

TOKEN_KINDS = frozenset(
    {
        "axis",
        "boolean",
        "brace",
        "date",
        "eof",
        "function",
        "identifier",
        "item",
        "keyword",
        "null",
        "number",
        "operation",
        "operator",
        "parameter",
        "period",
        "punctuation",
        "string",
        "table",
        "table_group",
        "type",
        "variable",
    }
)

Every Token.kind that tokenize produces.

Signature dataclass 

Signature(
    minimum: int,
    maximum: int | None,
    group_by: bool = False,
    key_operand: int | None = None,
    period_operand: int | None = None,
)

How many operands a function takes, and where it takes a key name.

Both evaluators check calls against these signatures, and diagnose reports the same problems before evaluation, so the three never disagree.

Attributes:

  • minimum (int) –

    The fewest operands.

  • maximum (int | None) –

    The most operands, or None for no limit.

  • group_by (bool) –

    Whether group by may follow the (single) operand.

  • key_operand (int | None) –

    Zero-based position of an operand that must be a key name, such as qDATE in annualise(x, 12, qDATE); None when there is none.

  • period_operand (int | None) –

    Zero-based position of an operand that is a period code from TIME_PERIODS, such as Q in time_shift(x, Q, 1); None when there is none.

Examples:

>>> SIGNATURES["substr"].problem("substr", 4)
'substr takes 1 to 3 operands, not 4'
>>> SIGNATURES["sum"].problem("sum", 1, group_by=True) is None
True
>>> SIGNATURES["abs"].problem("abs", 1, group_by=True)
'group by is not allowed in abs'

minimum instance-attribute 

minimum: int

maximum instance-attribute 

maximum: int | None

group_by class-attribute instance-attribute 

group_by: bool = False

key_operand class-attribute instance-attribute 

key_operand: int | None = None

period_operand class-attribute instance-attribute 

period_operand: int | None = None

operands 

operands() -> str

Describe the accepted operand count.

Returns:

  • str ( str ) –

    For example 1 operand, 3 or 4 operands, ``1 to 3

  • str –

    operandsor2 or more operands``.

problem 

problem(
    name: str, count: int, *, group_by: bool = False
) -> str | None

Return what is wrong with a call's operand count, if anything.

Parameters:

  • name (str) –

    The function's Call.name, for the message.

  • count (int) –

    The number of operands written.

  • group_by (bool, default: False ) –

    Whether the call has a group by clause.

Returns:

  • str | None –

    str | None: A message such as abs takes 1 operand, not 2, or

  • str | None –

    None when the call is well formed.

key_problem 

key_problem(name: str) -> str

Describe a call whose key operand is not a key name.

Returns:

  • str ( str ) –

    For example ``annualise needs a key name as its third

  • str –

    operand``.

canonical_function 

canonical_function(name: str) -> str

Map a function's source spelling to the Call.name the parser gives it.

Parameters:

  • name (str) –

    A function name as written, in any case.

Returns:

  • str ( str ) –

    The lower-case name, with max_aggr and min_aggr read as

  • str –

    max and min.

Examples:

>>> canonical_function("max_aggr"), canonical_function("SUM")
('max', 'sum')

textmate_grammar 

textmate_grammar() -> dict[str, Any]

Build a TextMate grammar for DPM-XL syntax highlighting.

editors/dpm-xl.tmLanguage.json in the repository is this grammar; the tests keep the two in step.

Returns:

  • dict[str, Any] –

    dict[str, Any]: The grammar, ready for json.dump.

Examples:

>>> textmate_grammar()["scopeName"]
'source.dpm-xl'