Check large tables with the Data quality (Spark) action
The Data quality (Spark) action evaluates the quality rules of a large Lakehouse Manager table on Spark and sends a report to the Lakehouse Manager
Objective
The Data quality (Spark) action evaluates the quality rules of one Lakehouse Manager table on Spark executors, and sends the result to the Lakehouse Manager as a report, one entry per rule. The report updates the quality status of the table. It runs the same rules, gives the same results and fails the job under the same conditions as the Data quality action, for tables too large for a worker.
Use it when the report of a Data quality run carries suggested_package: data-quality-pyspark, or when you already know the table is too large: the Data quality action suggests it above 1,000,000,000 cells (rows times columns) by default.
Like any other action, it can be run on demand, scheduled with a cron trigger, or placed in a workflow.
Requirements
- A Lakehouse Manager table with at least one active quality rule. The rules are read from the Lakehouse Manager at each run.
- The Spark 4.1.2 image, the only one that carries the rule engine (Python 3.13).
Configure the action
Parameters
The action has no engine choice: every rule runs on Spark, custom_sql rules included. Their expression is written in the Trino SQL dialect, as for the Data quality action, and translated to Spark SQL.
The table is read at its current snapshot (VERSION AS OF), once per run, so all rules see the same data even if a load runs at the same time.
Size the Spark resources to the table, see Resources.
Example payload
To create the action through the API, send the action body to the Data Processing Engine API. The polar-data connector module is required next to the action:
When the job fails
The report is always sent first: a failed job still has its report in the Lakehouse Manager, with the reason for each result, and the quality status of the table is updated regardless of the job result. The job result and the table status are two different things: a Report only rule ending critical makes the table critical without failing the job.
The job fails when:
- more rules end critical than Critical Results Allowed, or, when that field is not set, a rule set to Fail the job ends critical;
- more rules end in warning than Warning Results Allowed, when that field is set;
- no rule at all could be evaluated, for example when the table cannot be read.
The job passes when the table has no active rule, or when the table was never loaded: each rule then reports no_data with the reason "This table has no data yet: it was never loaded".
Results that never fail the job:
not_evaluatedandno_dataresults, as long as at least one rule could be evaluated;- a rule on a column the table no longer has. It is reported in warning, with the reason "Column
<name>is no longer in the table", and it is not counted in Warning Results Allowed. Update or delete the rule.
Go further
- Check the quality of your tables with data quality rules
- Check table quality with the Data quality action
- Configure a job execution alert to be notified when a quality job fails
For training or technical assistance implementing our solutions, contact your sales representative or visit our Professional Services page to request a quote and have your project analysed by our experts.
If you need support with your OVHcloud services, create a request in our Help Centre.
Join our community of users.