Data teams using Google Cloud often face a bottleneck when implementing Great Expectations (GX) on Spark: defining valid data rules manually. A new workflow integrates the DataBuck SDK with Dataproc Serverless and Cloud Composer to automate this. The SDK profiles datasets and policy PDFs to propose candidate rules, which can then be reviewed and executed as GX validations.

Automating Rule Generation with Context

The core innovation is context-aware discovery. By providing the DataBuck SDK with a domain name (e.g., "banking") and relevant policy documents, the tool uses Gemini to propose SQL conditions. In a test run on a 10,000-row transaction dataset, a single discovery call generated 89 rules. These rules are categorized by severityβ€”fail, warn, or dropβ€”and saved as a portable JSON file for human review.

The Dataproc Serverless Workflow

Implementation requires a custom container image with pinned dependencies, including databuck-spark-sdk version 0.5.44 and Great Expectations 1.x. The workflow involves five steps: reading data from Cloud Storage, discovering rules via the SDK, reviewing and approving the JSON file, converting rules to GX Expectations, and running validations. A Dataproc Serverless batch handles the execution, with optional orchestration via Cloud Composer for scheduled pipelines.

Results and Validation Insights

In the cited example, 72 rules passed, 17 failed, and one breached the quality gate. The report highlighted specific issues, such as 25 duplicate transaction IDs violating a dataset check. Notably, 42 rules were converted to native GX Expectations, while 46 ran as custom SQL. This hybrid approach ensures performance while maintaining coverage, though manual review remains critical for catching false positives, such as a rule flagging 9,929 rows due to incorrect fee thresholds.

Key Takeaways

  • DataBuck SDK reduces manual rule creation by leveraging LLMs and policy documents.
  • The workflow integrates seamlessly with Google Cloud's serverless Dataproc and Cloud Composer.
  • Human-in-the-loop review is essential to filter out irrelevant or overly strict generated rules.
  • Custom container images are required to manage dependencies like GX and the DataBuck JAR.

The Bottom Line

Automating rule discovery cuts setup time, but human review remains non-negotiable for filtering false positives and enforcing correct severity levels.