https://wintermutecore.com/tags/etl/Recent content in Etl on Project WintermuteHugoen-usWed, 18 Mar 2026 08:00:00 +0200https://wintermutecore.com/posts/daily-data-pipeline-dagu-python/Wed, 18 Mar 2026 08:00:00 +0200https://wintermutecore.com/posts/daily-data-pipeline-dagu-python/<p><strong>TL;DR.</strong> Three stages (ingest, index, alert), one stage per script, JSONL as the index format, a flat dedup state file, Dagu for scheduling. Boring, reliable, and dramatically less work than reaching for a heavyweight orchestrator.</p> <p>There is a class of pipeline that does not deserve a Spark cluster, an Airflow deployment, or a multi-tenant orchestrator. It is the &ldquo;fetch a few hundred records from an API every day, index them, alert on the interesting ones&rdquo; job. We have built this kind of thing dozens of times. Here is the shape that has held up.</p>https://wintermutecore.com/services/Sun, 13 Feb 2022 11:11:11 -0100https://wintermutecore.com/services/<p><p style="text-align:center;"> <img src="https://wintermutecore.com/services.jpg" alt="Services" /> </p> </p> <p>Below is what we actually do day to day. We try to keep the list short and the descriptions honest. If something here matches what you need, get in touch.</p> <h2 id="software-engineering" class="anchor-link"><a href="#software-engineering">Software engineering<span class="pilcrow">&nbsp;¶</span></a></h2> <p>We write backend services in Java, Kotlin, Go, and Groovy. Most of the work falls into a few buckets: REST and gRPC APIs, distributed systems handling high request volumes, and the occasional batch job that has to be reliable more than fast.</p>