https://wintermutecore.com/tags/slack/Recent content in Slack on Project WintermuteHugoen-usWed, 18 Mar 2026 08:00:00 +0200https://wintermutecore.com/posts/daily-data-pipeline-dagu-python/Wed, 18 Mar 2026 08:00:00 +0200https://wintermutecore.com/posts/daily-data-pipeline-dagu-python/<p><strong>TL;DR.</strong> Three stages (ingest, index, alert), one stage per script, JSONL as the index format, a flat dedup state file, Dagu for scheduling. Boring, reliable, and dramatically less work than reaching for a heavyweight orchestrator.</p> <p>There is a class of pipeline that does not deserve a Spark cluster, an Airflow deployment, or a multi-tenant orchestrator. It is the &ldquo;fetch a few hundred records from an API every day, index them, alert on the interesting ones&rdquo; job. We have built this kind of thing dozens of times. Here is the shape that has held up.</p>