Great Expectations Expands SQL Backend And Packaging Checks

Great Expectations is a data quality framework, and its recent develop activity is mostly about making database integrations and installed assets harder to break silently. The useful work this week is not a flashy runtime feature. It is broader SQL backend coverage, safer cloud test cleanup, and a direct check that schema catalogs and agent guidance survive packaging.

The database work starts with a shared SQL harness framework. Separate Trino coverage and ClickHouse coverage followed. That order matters. A common contract came first, then two engines exercised it.

The frequently touched tests/test_sql_backend_registry.py and tests/test_sql_backend_wiring.py point to the real scope. This is test architecture around backend registration and wiring. It should reduce the chance that an engine appears configured but misses part of the curated suite.

There is an important limit. These commits do not announce new production support or publish compatibility numbers. They make compatibility claims easier to test consistently. For teams that run the same expectation suite across several warehouses, that is valuable groundwork, but it remains groundwork until a release carries it.

Redshift also moved closer to the common dependency path. The project replaced its fork with the upstream SQLAlchemy Redshift dialect and then removed the dedicated CI marker. That reduces fork ownership, while making upstream dialect behavior the contract that users will need to watch.

The BigQuery change is small and operationally sensible. Integration tests now create tables inside the configured dataset and sweep tables rather than datasets. The affected big_query.py helper now keeps cleanup at the object type the test creates.

This narrows the likely blast radius of a failed or overly broad cleanup step. It should also fit service accounts that can manage tables but cannot delete datasets. The commit describes test infrastructure, not a production BigQuery deletion path, so users should read it as safer integration setup rather than a runtime bug fix.

PostgreSQL gained integration coverage for quoted schema names. This targets a common gap between a simple local schema and production identifiers that require quoting. Again, the evidence is coverage, not a claimed behavior change.

GCS received a four commit repair sequence. Credentials setup returned, datasource tests returned to CI, the test bucket moved to a repository variable, and documentation snippets began running again. The changed Pandas GCS example and Spark GCS example now sit closer to executable contracts than static documentation.

SingleStore gets a related reproducibility fix. The project pins the development image to version 0.2.82 instead of floating on latest. The central tasks.py changed as part of that work. A pin means a green run today exercises the same server image tomorrow. It also means maintainers must schedule image upgrades deliberately. This affects the test environment, not a documented minimum supported server version.

The largest change set ships two bundled skills named gx-configure-data-source and gx-configure-expectations. It also ships version matched indexes for expectation and datasource schemas. The catalog packaging commit supplies the machine readable context, while the agent guidance commit adds the usage layer.

The notable engineering choice is validation after installation. A new check_installed_agent_skills.py check resolves skills through the import system, confirms both catalog indexes exist, verifies that schema trees contain actual schema files, and runs the skills list command. It also compares installed content with its ownership manifest.

That catches a packaging failure that an import smoke test cannot see. A wheel can import successfully while data files selected by package_data patterns are missing. The added step in .github/workflows/ci.yml checks the artifact as users receive it, not merely the source tree. For schema driven tooling, that boundary is the correct place to fail.

The tradeoff is package size and another interface to maintain. Version matched catalogs reduce schema drift, but every catalog and skill file now has to move with the Python API it describes. The new integrity check makes that maintenance cost visible in CI.

  • Watch the next release notes for the first version that includes these develop changes. Do not base a support decision on branch activity alone.

  • Run existing Redshift validation suites before adopting the upstream dialect path. Fork removal simplifies maintenance, but dialect details can still affect reflection, quoting, and generated SQL.

  • Check whether more SQL engines join the shared harness. Wider coverage will matter only if backend specific cases remain visible instead of being flattened into the common suite.