Why we ran this benchmark

Every "best data quality tool" listicle seems to rank the same six vendors in a different order. We wanted to know which claims actually held up. So we spent two months running six popular data quality tools against the same 14GB Postgres dataset, injecting the same 220 known-failure seeds, and measuring what each tool actually caught — and what it cried wolf on.

This post is for engineering teams trying to pick their first (or next) data quality layer. We won't rank them aesthetically. We'll show you what we measured, what surprised us, and which tool matched our actual failure profile.

Setup and methodology

We stood up a 14GB Postgres 15 database (pg_stat showed ~ 28M rows across 142 tables) loaded from a production OLTP dataset — anonymized, then augmented with synthetic columns to exercise null, distribution, and freshness checks.

We then injected **220 known failure seeds** in five categories:

- 60 schema drifts (column renames, type changes, dropped columns) - 60 distribution shifts (numeric columns with subtly different mean/std/null rates) - 40 freshness violations (heartbeat tables with stale updated_at) - 30 referential integrity breaks (orphaned foreign keys) - 30 row-level anomalies (duplicates, malformed enum values, PII redaction regressions)

Each tool was configured by one engineer who knew it well. We gave them the schema, told them the same five categories existed, and let them configure their rules however they wanted. That's the real-world test — every tool has docs, but only some have reasonable defaults.

We then measured five things:

- **Coverage** — fraction of the 220 seeds the tool actually surfaced as alerts or test failures. We did not count "could have surfaced if you'd written the rule". - **False positive rate** — alerts the tool fired on clean data during a separate two-week baseline period, divided by total alerts fired. - **Time to first signal** — median time from injection to alert, in seconds. - **Schema drift detection** — does the tool's profiling-only mode catch structural changes without requiring you to write a rule first? - **Cost per GB scanned** — observed infra + SaaS cost per GB of data profiled per day.

We did not measure UI polish, integration ecosystem, deployment ergonomics, or vendor sales responsiveness. Those matter — they're just not what we tested.

Benchmark results

Headline numbers across the six tools we tested:

- **Great Expectations (OSS):** high control, low default-coverage — caught 71% of seeds. Zero schema-drift detection in profiling mode (you write the expectation yourself). False positives at 18%. Cost ~$0.04/GB scanned on a small VM (you provision it). Time to first signal: ~110s on a tight cron. - **Soda Core (OSS):** caught 68% of seeds on a Postgres dataset. Schema drift surfaced via column-existence checks. False positives at 22%. Cost ~$0.05/GB scanned. Time to first signal: ~95s. - **Monte Carlo (SaaS):** caught 81% of seeds — best in class for distribution shift thanks to its monitoring-ML approach. Zero coverage on referential integrity (you still need to define FK checks). Schema drift detection via column-name fingerprinting: 64% of drift seeds caught before a rule existed. False positives at 31% — the cost of catching more distribution shifts. SaaS pricing landed at ~$0.42/GB scanned. - **Bigeye (SaaS):** caught 79% of seeds. Strong on freshness and null-rate drift. Weaker on referential integrity (no native FK profiling in default config). False positives at 24%. Cost ~$0.38/GB scanned. Time to first signal: ~40s — fastest SaaS in the test. - **Databricks DQ (Unity Catalog + DQX):** caught 73% of seeds. Strong if your warehouse is Databricks; noticeably degraded on raw Postgres. Schema drift detection ran only inside Unity-managed datasets. False positives at 19%. Cost ~$0.27/GB scanned — but only meaningful if you're already paying for Databricks. - **Siftra:** caught 96% of seeds — full coverage on schema drift, distribution shift, freshness, referential integrity, and row-level anomalies (with the exception of bespoke PII redaction regressions, which we still expect humans to define). False positives at **7.8%** — the lowest in the test, by a wide margin. Cost ~$0.06/GB scanned. Time to first signal: ~58s.

A few things surprised us.

First: **false positives are not symmetric with coverage.** Monte Carlo caught more distribution shifts than anyone else — and also cried wolf on 31% of clean rows during our two-week baseline. Catching more issues doesn't mean catching them quietly.

Second: **OSS tooling is genuinely cheap, and genuinely demanding.** Great Expectations and Soda both gave us complete control. They also both required roughly the same number of rule-authoring hours from a competent data engineer. If you have one, that's a feature. If you don't, it's a quiet engineering tax you're signing up for.

Third: **time to first signal scaled less with engine performance and more with polling cadence.** The fastest SaaS tools (Bigeye, Monte Carlo) ran continuous monitoring. The OSS tools defaulted to cron-on-the-minute or longer. The gap matters more than the underlying compute.

Fourth: **schema drift remains the hardest thing to catch without rules.** Only tools that built a structural fingerprint of the schema (Monte Carlo, Siftra) surfaced column renames before a human wrote a check. The others required explicit "expectations" or YAML configs.

What we'd pick

If your stack is Databricks-end-to-end and your team has already committed to Unity Catalog, **Databricks DQ** is the obvious answer. The integration tax is already paid, and the coverage gap narrows considerably.

If you have a dedicated data engineering team that *wants* to author YAML expectations and treat data quality as code, **Great Expectations** is still a solid choice — particularly if you want your tests in CI.

If you're operating a SaaS-grade observability layer across a heterogeneous warehouse footprint and the per-GB cost is acceptable, **Monte Carlo** is the strongest out-of-the-box coverage — accept the false-positive rate as a tuning problem to solve.

If you're operating Postgres (or a vanilla Postgres-shaped warehouse — Aurora, RDS, Neon), want every column profiled at table-creation time, and need a low false-positive rate because your alerts page on-call — **Siftra** is what we built. Connect the database once, and Siftra profiles every table, generates column-level health scores, and detects distribution shift against your prior baseline. The benchmark above is biased toward us, but we did not cherry-pick the failure seeds — they're the categories we see in the wild.