For AI agents: the complete documentation index is available at https://docs.ovhcloud.com/pt/llms.txt, the full documentation bundle is available at https://docs.ovhcloud.com/pt/llms-full.txt, and this page is available as Markdown at https://docs.ovhcloud.com/pt/guides/public-cloud/data-platform/dpe-actions-data-quality.md.
Ir para o conteúdo

Check table quality with the Data quality action

Ver como Markdown

The Data quality action evaluates the quality rules of a Lakehouse Manager table and sends a report to the Lakehouse Manager

Objective

The Data quality action evaluates the quality rules of one Lakehouse Manager table and sends the result to the Lakehouse Manager as a report, one entry per rule. The report updates the quality status of the table. The action then fails the job when a rule set to Fail the job ends critical, or when the limits of When the job fails are exceeded, which lets a workflow stop before bad data goes further.

Like any other action, it can be run on demand, scheduled with a cron trigger, or placed in a workflow, for example right after the Load that fills the table.

Requirements

  • A Lakehouse Manager table with at least one active quality rule. The rules are read from the Lakehouse Manager at each run: the action has nothing to configure per rule.
  • A Python 3.11 or 3.13 environment. Python 3.11 is the default.

Configure the action

Parameters

ParameterRequiredDescription
Table to check (table)YesThe table to check, written dwh/<dataset>/<table>, for example dwh/sales/orders
Critical Results Allowed (max_criticals)NoIn When the job fails. The job fails when more rules than this end critical. Not set: only rules set to Fail the job can fail it
Warning Results Allowed (max_warnings)NoIn When the job fails. The job fails when more rules than this end in warning. Not set: warnings never fail the job
Engine (engine)Noauto (default), polars or trino. See Engine and table size
Worker Limit (Cells) (max_cells)NoSize of the table, in rows times columns, that the worker holds. Above it, the worker does not load the table. Not set: the action sets it from the worker size
Very Large From (Cells) (very_large_cells)NoSize of the table, in rows times columns, above which the report suggests the Data quality (Spark) action. Default 1,000,000,000

By default, the action runs on 2 DPU (fpu size 2). The Worker Limit follows the worker size, so a larger worker loads larger tables.

Engine and table size

The action reads the table's current snapshot once, at the start of the run, and evaluates every rule on that snapshot, so all rules see the same data even if a load runs at the same time.

Where the rules run depends on the size of the table, read from its metadata before anything is loaded:

EngineBehaviour
autoTables up to the worker limit are loaded in the worker and checked with Polars. custom_sql rules always run on the query engine. Above the worker limit, or when the table has pending deletes, the whole check runs on the query engine.
polarsAlways checks in the worker. A table above the worker limit is refused.
trinoAlways checks on the query engine.

Date and time checks read values in UTC, and count days from the day of the run.

Info

Checks that run on the query engine count in the query usage of your project.

Above Very Large From, the report carries suggested_package: data-quality-pyspark: move the table to the Data quality (Spark) action. The engine trino is a deliberate choice and sends no suggestion.

Example payload

To create the action through the API, send the action body to the Data Processing Engine API. The polar-data connector module is required next to the action:

{
  "display_name": "Quality of orders",
  "repository": "<repository id>",
  "repository_version": "<repository version id>",
  "action": "data-quality",
  "language": "python3",
  "modules": [
    {"type": "action", "id": "data-quality", "version": "latest"},
    {"type": "connector", "id": "polar-data", "version": "latest"}
  ],
  "params": {"table": "dwh/sales/orders"},
  "environment": {"fpu": {"size": 2, "instances": 1}}
}

To run on Python 3.13, add "language_version": "3.13" to environment. To let rules fail the job by count, add max_criticals or max_warnings to params, see When the job fails.

When the job fails

The report is always sent first: a failed job still has its report in the Lakehouse Manager, with the reason for each result, and the quality status of the table is updated regardless of the job result. The job result and the table status are two different things: a Report only rule ending critical makes the table critical without failing the job.

The job fails when:

  • more rules end critical than Critical Results Allowed, or, when that field is not set, a rule set to Fail the job ends critical;
  • more rules end in warning than Warning Results Allowed, when that field is set;
  • no rule at all could be evaluated, for example when the table cannot be read.

The job passes when the table has no active rule, or when the table was never loaded: each rule then reports no_data with the reason "This table has no data yet: it was never loaded".

Results that never fail the job:

  • not_evaluated and no_data results, as long as at least one rule could be evaluated;
  • a rule on a column the table no longer has. It is reported in warning, with the reason "Column <name> is no longer in the table", and it is not counted in Warning Results Allowed. Update or delete the rule.

Go further

For training or technical assistance implementing our solutions, contact your sales representative or visit our Professional Services page to request a quote and have your project analysed by our experts.

If you need support with your OVHcloud services, create a request in our Help Centre.

Join our community of users.

Esta página foi útil?