The Problem
Most log-processing toolchains force a choice: a heavyweight platform (Vector, Fluentd, Logstash) with its own DSL, or a pile of shell scripts that calcify the moment they leave the author's laptop. Teams want something between — declarative enough to read at a glance, programmable enough to handle the weird cases that always show up.
The primary constraint wasn't speed — it was flexibility.
The Approach
LogBus runs event pipelines defined in YAML, flowing from sources through stages to sinks. The engine wires the DAG from each stage's inputs: declarations and validates it statically — loops and bad wiring fail before any bytes move. Every stage is an async task connected by bounded channels, and a plugin system keeps each stage swappable.
- Batteries included. Fourteen sources (journald, Kafka, OTel, HTTP, file tail & glob, S3 listings, sysinfo, a scheduler) and eleven sinks (OpenSearch, ClickHouse, CloudWatch Logs, Kafka, S3, email, OTel) cover the common wiring, plus codecs for JSON, YAML, Avro, Parquet, key-values, and lz4/deflate/snappy compression.
- Escape hatches, not lock-in. When YAML runs out, a stage can be an inline JavaScript function (embedded QuickJS) or a WASM plugin (Extism) — without making YAML the loser.
- Plugins are just functions. State lives in the plugin's closure when it needs it — the OpenSearch sink batches events until a bulk insert is worth it — with explicit start/stop hooks for the few that need lifecycle.
- Single static binary. Drop it on a host, point it at a config, done.
Backpressure without a scheduler
The interesting engineering is in what happens when a sink slows down. Stages are connected by bounded MPSC channels (default capacity 4096, tunable), so backpressure is a property of the architecture rather than plugin code: when a sink's buffer fills, the upstream send suspends, that stage stops reading its input, and the stall propagates hop by hop until the source itself pauses its I/O. No events are dropped and no CPU is burned waiting — suspended tasks are parked by the runtime. The maximum in-flight accumulation is exactly capacity × channels in the chain, so worst-case memory is a number you can compute, not a graph you discover in production.
Fan-in and fan-out fall out of the same primitives: multiple upstreams share clones of a downstream's sender (the receiver closes only when the last upstream finishes), and a lightweight fan-out task clones each event — cheap, since events are reference-counted — to every downstream.
The one deliberate exception proves the rule: the live-stats UI receives events via a non-blocking send, so a slow browser tab can never backpressure the data path. Observability should watch the pipeline, not wedge it.
Born in TypeScript, rebuilt in Rust
LogBus started as a TypeScript/Deno engine, and a real journald → OpenSearch pipeline built on it has been running in production for years — small enough to read end-to-end, which was the point. The Rust rewrite kept the YAML contract and plugin model while buying a single static binary, a stronger concurrency story, and the bounded-channel backpressure above. Porting the same design across two runtimes was its own review: the pieces that survived unchanged (the DAG, the plugin-as-function model) are the pieces that were right.
Where it stands
The TypeScript engine is the production workhorse; the Rust port carries the current development focus, including an embedded live dashboard for watching events move. Known gaps I'm working through, in the open: end-to-end acknowledgement (don't commit a Kafka offset until the final stage confirms delivery), letting plugins declare the event shapes they accept so miswired pipelines fail at validation rather than runtime, and a better dev-and-test story for pipeline-defined functions that outgrow inline YAML.