scott-k.dev
SYS / LAB
Project detail — progress, post mortems & resources
projects
System Zero building
Progress — 5 / 6
spec
dev
build
test
ui
polish
Post Mortems — 8 incidents resolved
Apr 2026
Message queue publisher reconnect loop
Publisher entered an infinite reconnect cycle without backoff after broker restart — fixed with exponential retry and connection health checks.
Apr 2026
Service port not injected at container startup
Requests to a downstream service failed silently because the orchestrator wasn't injecting a required environment variable at runtime.
May 2026
Internal auth check caused infinite requeue loop
An internal service was returning 401 for valid internal calls. Jobs were nacked and requeued indefinitely with no visible error surfacing to the user.
May 2026
Workers terminated mid-job by orchestrator scale-down
Event-driven autoscaling sent SIGTERM to workers processing active jobs. No graceful drain was implemented, leaving documents stuck in a processing state.
May 2026
Document structure retrieval overflowed model context
A tool returning document headings was including full section content alongside — saturating the context window on large documents and truncating responses.
May 2026
Heading search index silently empty after ingestion
Two chained bugs: an embedding queue timeout fired silently before headers were enqueued, and a downstream consumer overwrote the index type field — preventing correct routing entirely.
May 2026
Schema draft — three independent failures masking each other
Frontend, backend, and inference layer each failed for unrelated reasons. Fixing one revealed the next: a missing stream subscription, an async type mismatch, then an insufficient token budget.
May 2026
Session token expired silently mid-session
Short-lived auth tokens expired during idle periods with no background renewal. Returning to an open page after inactivity produced a network error instead of a transparent re-auth.
Resources
Infrastructure
Kubernetes / k3s — lightweight production cluster
KEDA — event-driven autoscaling from queue depth
RabbitMQ — AMQP message broker for async pipelines
Traefik — ingress controller with automatic TLS
Prometheus + Grafana — metrics and observability
Search & Retrieval
FAISS — billion-scale dense vector index · Johnson et al., 2019
BM25 — sparse keyword retrieval · Robertson & Zaragoza
Reciprocal Rank Fusion — hybrid result merging · Cormack et al., 2009
Dense Passage Retrieval for Open-Domain QA · Karpukhin et al., 2020
SentenceTransformers — passage embeddings (all-MiniLM)
Inference & Orchestration
llama.cpp — GGUF quantised CPU inference
FastAPI + Server-Sent Events — streaming inference gateway
Model Context Protocol — tool-calling interface · Anthropic, 2024
FastApiMCP — MCP server from FastAPI routes
Further Reading
Attention Is All You Need · Vaswani et al., 2017
GGUF: efficient quantization for LLM inference on CPU
Hybrid Search: combining sparse and dense retrieval
Star Schema: The Complete Reference · Kimball & Ross
Drone RL in progress
Progress — 3 / 6
spec
dev
build
test
deploy
polish
Post Mortem
Being documented
available when complete
Resources
Being refined
papers & references coming soon
RLxNEAT planned
Progress — 1 / 6
spec
dev
build
test
deploy
polish
Post Mortem
Being documented
available when complete
Resources
Being refined
papers & references coming soon
Current Interests
LLM Architectures
Attention Is All You Need — Vaswani et al.
FlashAttention: Fast Memory-Efficient Exact Attention
Mixtral of Experts — sparse MoE routing
KV Cache compression & speculative decoding
COCONUT — Meta AI, 2024: reasoning in continuous latent space by feeding hidden states back as input, bypassing the token layer entirely
Looped Transformers — same weights, multiple passes; a smaller looped model outperforms a larger standard model
Reinforcement Learning
Soft Actor-Critic: Off-Policy Maximum Entropy RL
DroQ: Dropout Q-Functions for Doubly Efficient Reinforcement Learning
Prioritized Experience Replay — Schaul et al.
TD3: Addressing Function Approximation Error
KAN & Symbolic Networks
KAN: Kolmogorov-Arnold Networks — Liu et al. 2024
Neural-symbolic integration for interpretable reasoning
Symbolic regression with deep learning
Graph Databases & GNNs
Graph Attention Networks — Veličković et al.
Knowledge graphs for RAG & structured retrieval
GraphRAG: Microsoft's graph-based retrieval approach