Systems Engineering ongoing open-source

Nendi

A Kafka-Free Database Streaming Daemon in Rust

Collapses the Debezium → Kafka → Consumer pipeline into a single binary. No JVM, no broker, no ops overhead — targeting sustained 100,000+ writes/sec on commodity NVMe via RocksDB's LSM-tree and Rust's zero-cost abstractions.

View on GitHub Deep Dive

100k+ writes / sec

<200µs P99 latency

1 binary — no broker

4 filter layers

Rust RocksDB gRPC / tonic PostgreSQL WAL Tokio rkyv

The Engineering Story

The traditional CDC pipeline — database → Debezium agent → Kafka cluster → consumer — introduces 2–3 network hops, JVM overhead, and significant operational complexity. For single-region deployments or latency-sensitive workloads, this architecture is overkill. Nendi collapses it into one process.

Architecture at a Glance

PostgreSQL WAL

↓

            
            Nendi Daemon
            Source Connectors → Ring Buffer → RocksDB

↙

↘

gRPC Streams

HTTP Webhooks

Key Design Decisions

Rust — No GC, Predictable Performance

Garbage collection pauses kill tail latency. Rust's ownership model gives us memory safety at zero runtime cost. Every allocation is intentional; no stop-the-world events at 100,000 messages/sec.

RocksDB — LSM-Tree Built for Sequential Writes

Events are appended as big-endian monotonic keys — physically ordered on NVMe. Write buffer at 256MB, direct I/O for flush/compaction. Target: sustained 100k+ writes/sec without compaction stalls.

Dual-Threshold Flush — Latency Without Sacrifice

A naive timer-based flush sets a 1ms latency floor. We flush when the buffer hits 4MB OR after 500µs — whichever comes first. Under light load: time wins. Under heavy load: bytes win. P99 < 200µs.

rkyv Internally, Protobuf at the Boundary

JSON at 100k/sec is a disaster. rkyv gives us zero-copy deserialization for internal RocksDB storage. Protobuf on the gRPC egress boundary ensures client compatibility without giving up wire efficiency.

Four Layers of Filtering — Store Once, Serve Variably

Nendi's filtering system operates on a single principle: filter as early and as cheaply as possible. Each layer removes work from every layer that follows it. No data is duplicated for different consumer views.

PostgreSQL Publication

Table selection, operation types, and row predicates. Filtered rows are never decoded — zero network cost on Nendi's side.

Inside PostgreSQL

Removes work from Network ingestion

Nendi Ingestion Predicate

CEL expression evaluated post-decode, pre-storage. Column projection strips fields before RocksDB write. Filtered events are never stored.

Hot path, pre-storage

Removes work from RocksDB + all consumers

Consumer Subscription Filter

Per-consumer table, column, and row filters. Ten consumers with different filters = ten evaluations per event, not ten storage writes. Same RocksDB log serves all.

Egress, per-consumer

Removes work from Network & consumer CPU

Column-Level Encryption

Sensitive columns encrypted at ingestion (AES-256-GCM or ChaCha20). Decrypted only for authorized consumers at egress. RocksDB never holds plaintext PII.

Ingestion + egress

Removes need for ACL infrastructure

The Nendi SDK — Five Lines to a Stream

The SDK owns everything below the application's event handler: reconnection, offset tracking, typed deserialization, backpressure, and TLS. A developer should not need to understand gRPC internals to consume a stream.

without sdk

// Manual setup — 20+ lines minimum
let channel = Channel::from_static("http://nendi:50051")
    .connect().await?;
let mut client = StreamServiceClient::new(channel);
let request = SubscribeRequest {
    stream_id: "orders".to_string(),
    // ... 8 more fields
};
let mut stream = client.subscribe(request).await?.into_inner();
loop {
    // Manual: offset, reconnect, backoff, schema...
}

with nendi sdk

// Five lines. That's it.
let client = NendiClient::new("http://nendi:50051").await?;
let mut stream = client.subscribe("orders").await?;

while let Some(event) = stream.next().await? {
    println!("{:?}", event);
}

Problems We Solved (and What We Learned)

Solved

WAL Slot Bloat

A slow consumer holding an unadvanced replication slot fills PostgreSQL's disk indefinitely. Our solution: autonomous slot advancement — we report min(consumer_watermark, rocksdb_persisted_lsn - lag_cap) to PostgreSQL. Consumers catch up from RocksDB, not from WAL. PostgreSQL's disk is never at risk.

Solved

Backpressure to PostgreSQL

If RocksDB compaction debt stalls disk writes, the ingestion queue fills — and then what? We gate ingestion using PostgreSQL's own replication protocol: stop sending StandbyStatusUpdate messages, causing PostgreSQL to pause WAL delivery naturally. No data loss. No queue overflow.

Solved

DDL Schema Race

A DML event can arrive before the DDL message that describes its schema. We fingerprint every event with an MD5 of the schema at the PostgreSQL trigger level, buffer events with unknown fingerprints, and drain them when the DDL arrives. pg_logical_emit_message(transactional=true) ensures WAL ordering.

Active

io_uring Integration

io_uring offers kernel-bypass I/O that could push us past 100k/sec with far lower CPU. However, SQPOLL requires CAP_SYS_ADMIN, and io_uring bugs have historically caused data corruption. We're building the epoll/tokio baseline first — profile, then layer on io_uring as an optional backend.

Performance Targets & Hardware

100k+ writes / second Single node, NVMe

<200µs P99 latency Dual-threshold flush

~35–50ms CPU / sec total filter overhead Typical 10-consumer deployment

CPU 8+ cores (high clock)

RAM 64GB recommended

Storage NVMe Gen4 (High DWPD)

OS Linux 6.x (io_uring future)

Implementation Roadmap

Nendi is built in deliberate phases — each unlocks the next layer of reliability.

done Phase 01 · Months 1–2

Core Foundation

#SourceConnector trait + PostgreSQL binary logical decoding
#RocksDB storage engine with leveled compaction
#Ring buffer + rkyv serialization pipeline

in progress Phase 02 · Months 3–4

High-Performance Egress

#gRPC bidirectional streaming server (tonic)
#Offset registry + LSN persistence
#HTTP webhook dispatcher with retry / circuit breaker

upcoming Phase 03 · Months 5–6

Reliability & Advanced Tuning

#DDL event triggers + schema fingerprinting
#io_uring async I/O integration
#Prometheus metrics + health endpoints
#Comprehensive performance benchmarks