Repository README

Lightweight telemetry agent and collector for small fleets.

A Rust agent collects host metrics, file-tailed logs, and queue-pressure alerts, then uploads framed Cap’n Proto batches over HTTPS to a single-process Rust collector. The collector validates, deduplicates, and spools batches to disk, optionally landing them in an S3-compatible object store with conditional-create acceptance, derived indexes, and configurable rollups. A Prometheus-compatible query API serves metrics from spool and object-store data, and an embedded console provides realtime fleet overview, dashboards, logs, and alerts over private SSE and query endpoints.

Key properties:

• owned binary protocol over HTTPS with at-least-once delivery and sequence-based dedupe
• local disk queue on the agent, durable disk spool on the collector
• S3-compatible object store as the durable data backend
• immutable rollups (1m/5m/1h) for long-range queries
• Prometheus-compatible query API (labels, label values, series, metadata, query, query_range) with rate(), sum(...) by (...), and histogram_quantile()
• stock Grafana Prometheus data source compatibility
• typed TOML dashboard definitions with a private dashboard API
• embedded console served by the collector
• TOML runtime config with secrets referenced by file path
• GitHub Actions CI with unit tests, clippy, and MinIO integration tests

Current implementation boundaries are tracked in docs/limitations.md.

Workspace

• crates/lm-protocol: Cap’n Proto schema bindings and wire helpers
• crates/lm-agent: agent binary with host metric collection, file log tailing, local disk queue, and HTTPS upload
• crates/lm-collector: collector binary with ingest validation, disk spool, S3 object landing, rollups, Prometheus-compatible query, logs/alerts APIs, live SSE, embedded console, and dashboard definitions
• crates/lm-config: shared TOML configuration types and validation
• schemas/telemetry.capnp: wire schema
• site/: static Astro public website package
• docs/README.md: documentation index and guide status map
• docs/design.md: design notes
• docs/limitations.md: current explicit and implicit implementation limits
• docs/PLAN.md: concise operational implementation plan and documentation backlog
• docs/worklog/: structured completed-slice history and verification ledger
• config/*.toml: runtime config examples
• config/*.cue: optional validation/generation sketches, not runtime config

Documentation

Start with docs/README.md for the current documentation map, docs/quickstart.md for the verified local demo path, docs/install.md for manual source-built binary layout, and docs/docker.md for operator-owned container usage. Planned guides are marked explicitly, and current limitations are tracked in docs/limitations.md.

The public website package lives in site/. It is a static Astro package that builds to site/dist, and reads only documented public repository inputs at build time. Public docs routes are generated from an allowlist of repository Markdown files. Cloudflare Pages preview deployment settings are documented and checked under site/; DNS/custom-domain launch and publication governance remain tracked separately in docs/PLAN.md.

First Build

cargo build
cargo run -p lm-agent

Local Demo

The demo script starts MinIO, the collector, and a local agent loop. The collector uses S3-manifest acceptance, landing raw blobs and accepted manifests in the MinIO bucket. Host metrics, logs, and alerts are queryable through the Prometheus-compatible API and visible in the embedded console.

scripts/lightmetrics-demo restart
scripts/lightmetrics-demo status
scripts/lightmetrics-demo stop

By default it writes runtime state under /tmp/lightmetrics-demo. The query token is generated there and reported as a file path by status; the console is served at http://127.0.0.1:18080/console.

The synthetic ingest smoke test runs on separate ports and checks query plus MinIO object-store landing:

tests/integration/demo_minio_synthetic_ingest.sh