Check table quality with the Data quality action
The Data quality action evaluates the quality rules of a Lakehouse Manager table and sends a report to the Lakehouse Manager
Objective
The Data quality action evaluates the quality rules of one Lakehouse Manager table and sends the result to the Lakehouse Manager as a report, one entry per rule. The report updates the quality status of the table. The action then fails the job when a rule set to Fail the job ends critical, or when the limits of When the job fails are exceeded, which lets a workflow stop before bad data goes further.
Like any other action, it can be run on demand, scheduled with a cron trigger, or placed in a workflow, for example right after the Load that fills the table.
Requirements
- A Lakehouse Manager table with at least one active quality rule. The rules are read from the Lakehouse Manager at each run: the action has nothing to configure per rule.
- A Python 3.11 or 3.13 environment. Python 3.11 is the default.
Configure the action
Parameters
By default, the action runs on 2 DPU (fpu size 2). The Worker Limit follows the worker size, so a larger worker loads larger tables.
Engine and table size
The action reads the table's current snapshot once, at the start of the run, and evaluates every rule on that snapshot, so all rules see the same data even if a load runs at the same time.
Where the rules run depends on the size of the table, read from its metadata before anything is loaded:
Date and time checks read values in UTC, and count days from the day of the run.
Checks that run on the query engine count in the query usage of your project.
Above Very Large From, the report carries suggested_package: data-quality-pyspark: move the table to the Data quality (Spark) action. The engine trino is a deliberate choice and sends no suggestion.
Example payload
To create the action through the API, send the action body to the Data Processing Engine API. The polar-data connector module is required next to the action:
To run on Python 3.13, add "language_version": "3.13" to environment. To let rules fail the job by count, add max_criticals or max_warnings to params, see When the job fails.
When the job fails
The report is always sent first: a failed job still has its report in the Lakehouse Manager, with the reason for each result, and the quality status of the table is updated regardless of the job result. The job result and the table status are two different things: a Report only rule ending critical makes the table critical without failing the job.
The job fails when:
- more rules end critical than Critical Results Allowed, or, when that field is not set, a rule set to Fail the job ends critical;
- more rules end in warning than Warning Results Allowed, when that field is set;
- no rule at all could be evaluated, for example when the table cannot be read.
The job passes when the table has no active rule, or when the table was never loaded: each rule then reports no_data with the reason "This table has no data yet: it was never loaded".
Results that never fail the job:
not_evaluatedandno_dataresults, as long as at least one rule could be evaluated;- a rule on a column the table no longer has. It is reported in warning, with the reason "Column
<name>is no longer in the table", and it is not counted in Warning Results Allowed. Update or delete the rule.
Go further
- Check the quality of your tables with data quality rules
- Check data quality after each load
- Configure a job execution alert to be notified when a quality job fails
For training or technical assistance implementing our solutions, contact your sales representative or visit our Professional Services page to request a quote and have your project analysed by our experts.
If you need support with your OVHcloud services, create a request in our Help Centre.
Join our community of users.