Meltano Logging And Signal Handling Notes For Data Pipelines

Meltano is an open source ELT project used to wire extractors, loaders, and transform steps into repeatable data pipeline runs. The recent activity is not a release train, but it touches the parts operators notice under load: process shutdown, terminal logging, local secret files, and CI guardrails. The window has 19 commits across 62 files, with 1410 insertions and 398 deletions.

The most important runtime change is the SIGTERM propagation fix. Before this patch, SIGTERM handling could exit directly from the signal handler. In async job execution, that path can deny plugin subprocesses a chance to finish cleanup.

The new path cancels the task running the job when the event loop can install a signal handler. It also records that SIGTERM was seen and maps the CLI exit to 143. That status is the conventional signal derived exit code for SIGTERM, so schedulers can tell a planned stop from an application failure.

The test coverage matters here. tests/meltano/cli/test_cli.py now covers cancellation with and without SIGTERM, plus exit code recording in main. Job tests also check that cleanup can finish before the cancellation bubbles out. For long extractor or loader runs, this is the difference between a clean stop and a half closed child process.

The FORCE_COLOR logging change is small, but it removes a common CI annoyance. Meltano already had TTY based color behavior and NO_COLOR support. The new logic in src/meltano/core/logging/utils.py also honors FORCE_COLOR when stderr is not attached to a TTY.

The precedence is useful. NO_COLOR still wins if both environment variables are truthy. Non colored log formats remain unchanged, including JSON, key value, and plain output. That keeps machine parsed logs stable while allowing human oriented CI logs to keep color when the runner does not expose a TTY.

The docs in docs/docs/reference/command-line-interface.mdx now describe both variables and call out that FORCE_COLOR support lands in Meltano v4.3.0. That version note is worth tracking if teams pin Meltano in base images or pipeline templates.

The project env permission fix changes one line in src/meltano/core/settings_store.py, but it is exactly the kind of one line that belongs in a data platform changelog. When DotEnvStoreManager creates a new .env file, it now calls touch(mode=0o600).

Those files often contain source credentials, warehouse credentials, API tokens, and per project settings. Without an explicit mode, a new file can inherit process defaults through the active umask. On shared build hosts or developer machines with permissive defaults, that can expose secrets to more local users than intended.

This is not a broad secret management redesign. It is a tighter default for the local file path that Meltano already uses. The tradeoff is also clear: Unix style file modes are the target, while Windows behavior needs platform aware handling in tests.

Several commits are mostly internal refactor, but they reduce accidental public surface. The version helper refactor removes the package level meltano.__version__ variable and moves callers to meltano.core.utils.get_meltano_version. The commit also tightens mypy settings in pyproject.toml with explicit_package_bases and a src based mypy_path.

That can affect internal wrappers or plugin code if they import meltano.__version__ directly. The project points users toward get_meltano_version or importlib.metadata.version. This is not an operator visible behavior change in normal CLI use, but it is visible to code that treats Meltano as an import target.

There is also a ruamel import cleanup touching src/meltano/core/behavior/canonical.py, plus a pidfile utility removal. The pidfile module was deleted as obsolete code. That kind of deletion is useful only if the test suite and downstream import surface agree it was truly unused.

The uv hook update adds uv-sync and uv-audit to .pre-commit-config.yaml, while skipping uv-audit in hosted pre commit CI. That split is practical. Contributors get an audit hook locally, while hosted automation avoids audit behavior that may not fit an autofix job.

Dependency maintenance was active as well. The runtime dependency refresh updates uv, virtualenv, google-cloud-storage, boto3, and botocore in uv.lock. A separate actions refresh moves setup uv, CodSpeed, and CodeQL actions forward across build, benchmark, test, and security workflows.

The integration workflow also changed. In the dbt 1.12 output update, .github/workflows/integration_tests.yml broadens its path trigger from one example library subtree to all integration paths. That should catch more integration changes, but it also means more PRs may run the heavier workflow.

  • Audit internal wrappers that import meltano.__version__. The helper path is now the safer target.
  • Watch follow up work on tracker tests. Two tracking tests were marked xfail as flaky after a reverted fix, so telemetry CI signal is not fully clean yet.
  • For runners managed by Airflow, Dagster, cron, or Kubernetes Jobs, verify SIGTERM behavior after the next Meltano upgrade. The expected process status is 143, with plugin cleanup given time through task cancellation.