Backend Engineering in Production
Practical notes from building and debugging production backends: concurrency failures, database bottlenecks, distributed-system trade-offs, and the fixes that work.
Posts
-
HTTP QUERY in RFC 10008: Go and Browser Support
RFC 10008 standardizes HTTP QUERY as a safe, idempotent method with content. Learn its semantics and current Go, framework, and browser support.
-
Goroutines Pile Up Into the Hundreds of Thousands — Even Though CPU Is Flat
CPU looks fine, memory slowly climbs, and goroutine count keeps growing. Here are the five real causes of goroutine leaks — with diagrams, pprof walkthroughs, and fixes.
-
Database Connections Exhausted at Peak Traffic — Even With SetMaxOpenConns Set
SetMaxOpenConns is set, but connections still run out under load. Here are the five real causes — with diagrams and fixes for each.
-
Availability in the Age of AI: New Failure Modes and How to Debug Them
What changes when an LLM is part of your production stack. The new failure modes nobody warns you about, and how to debug a system whose dependency answers differently every time.
-
Docker and Kubernetes Cheatsheet: Shrinking Image Size and Memory Footprint
Practical recipes for building tiny container images and running them lean in Kubernetes. Multi-stage builds, distroless, static binaries, resource tuning, and the OOMKilled traps.
-
Keeping the Domain Up: Layered HA with HAProxy, Keepalived, and Postgres Master-Slave
How a service stays online when something dies. Floating VIP at the edge, master-slave Postgres at the data tier, and the failure modes that survived the first outage.
-
HAProxy Master-Slave: Active-Passive Load Balancing with Keepalived and a Floating VIP
Building an active-passive HAProxy pair with Keepalived and a floating virtual IP. Real failover, real split-brain risks, and the gotchas that bit me in production.
-
Building an Open Service Broker in Rust: A Weekend Engineering Journey
Building a production-shape Open Service Broker (OSB) API in Rust from scratch with axum, including JSON-Schema validation, async operations, Postgres-backed storage, Docker, and a CI pipeline that catches real bugs.
-
Building a Simple RAG Stack with Redis 8 Vector Sets, Go, and Python
Build a compact RAG stack with Redis 8 Vector Sets, a Go API gateway, Python FastAPI embeddings, SSE streaming, and local Ollama support.
-
Storing Complex Python Objects in Valkey Hashes using Pickle
Learn how to serialize and deserialize complex Python objects using pickle and store them efficiently in Valkey (or Redis) Hashes using HSET and HGETALL.
-
Critical Linux Vulnerabilities Disclosed: Dirty Frag and Copy Fail
A summary of two newly discovered Linux kernel vulnerabilities, Dirty Frag and Copy Fail (CVE-2026-31431), their impact, status, and mitigations.
-
Go and Third-Party APIs: Surviving Partial Failures
How to design Go backend workflows that remain consistent when third-party API calls partially fail.
-
Go HTTP Timeouts: One Timeout Is Not Enough
A production outage caused by missing layered HTTP timeouts in Go clients and how to configure them correctly.
-
Schema Drift Between Go Services: The Silent Contract Break
A real-world microservice failure caused by schema drift and how to enforce backward-compatible contracts.
-
Transaction Isolation in Go: The Write-Skew Incident
How read-committed transactions caused write skew in production and the locking strategy that fixed it.
-
Go Cache Stampede: Fixing Redis Meltdowns with singleflight
How cache expiration spikes can flood databases, and the Go singleflight pattern that collapses duplicate work.
-
Retry Storms in Go: When Resilience Becomes an Outage
A practical guide to preventing retry storms in Go services with bounded retries, jitter, and retry budgets.
-
Go Outbox Pattern: Stop Losing Events After Commit
Why publishing events directly after DB commit causes inconsistencies, and how the outbox pattern fixes it in Go services.
-
Message Ordering Bugs in Go: Kafka Partitions Surprise
How incorrect partition keys in event-driven systems break ordering guarantees and corrupt downstream state.
-
Graceful Shutdown in Go: The In-Flight Request Trap
How a Go service can lose user requests during deploys, and the shutdown sequence that prevents dropped in-flight work.
-
Go Context Cancellation: Why Your Workers Never Stop
A production bug where background worker goroutines ignored cancellation and leaked across deploys.
-
Pessimistic vs Optimistic Locking in Concurrent Backends
How to handle high-concurrency race conditions in your database using pessimistic locking (FOR UPDATE) and optimistic locking (version numbers).
-
The Silent Go Memory Leak: Unclosed HTTP Response Bodies
A common Go bug where failing to close the HTTP response body drains connection pools, leaks memory, and crashes production services.
-
Concurrency in Go: sync.Map vs map + sync.RWMutex
When to use Go's built-in map with an RWMutex vs the specialized sync.Map for concurrent access in backend applications.
-
Handling Context Cancellations Correctly in Go Database Queries
Why passing context.Background() to database queries is a bad idea, and how to properly handle timeouts and client disconnects using context.Context.
-
API Idempotency: Why You Need Retry Keys
How to design idempotent APIs using Idempotency Keys to prevent double-charging users during network timeouts and retries.
-
Streaming AI Responses via gRPC + SSE: Python FastAPI → Go Backend → Browser
Stream AI output from Python to a Go backend with gRPC server streaming, then deliver tokens to browsers using Server-Sent Events.
-
Zero-Downtime Database Migrations on Large Tables
How to safely add columns, build indexes, and modify schema on massive PostgreSQL tables without causing production downtime or lock contention.
-
SQL Deadlocks: Why They Happen and How to Fix Them
A deep dive into why SQL deadlocks occur in concurrent backend systems and how to fix them with consistent ordering and retry mechanisms.
-
The Hidden Cost of Missing Foreign Key Indexes
Why you should almost always index your foreign keys, and the catastrophic performance hits that occur on DELETE and UPDATE when you don't.
-
Timezones and time.Time Bugs in Go
How time.Time equality works in Go, why DeepEqual fails on times, and how to safely store and transmit timestamps in backend systems.
-
The Dangers of Pagination with OFFSET and LIMIT
Why OFFSET/LIMIT pagination kills your database performance at scale, and how to implement Keyset Pagination (Cursor Pagination) instead.
-
Go Worker Pools, Job Queues, and Background Processing
Implementing safe background processing in Go: bounded worker pools, durable job queues with Asynq and Redis, cron scheduling, and graceful shutdown to avoid losing in-flight jobs.
-
gRPC in Go: Building High-Performance Service-to-Service APIs
Building high-performance service-to-service APIs in Go with gRPC and Protocol Buffers. Covers server streaming, interceptors, retry policies, and domain error mapping.
-
Observability in Go: Structured Logging, Metrics, and Distributed Tracing
The three pillars of observability for Go backends: structured logging with slog, Prometheus metrics with golden signals, and distributed tracing with OpenTelemetry.
-
Caching Strategies in Go: From In-Process to Distributed
Multi-layer caching in Go from in-process ristretto to Redis. Covers stampede protection with singleflight, probabilistic early refresh, and tag-based cache invalidation.
-
Event-Driven Architecture in Go with Kafka
Building reliable event-driven systems in Go with Kafka. Covers franz-go producer/consumer setup, dead letter queues, idempotency, and schema evolution.
-
Go Backend Engineering: Real-World Problems, Bugs, and Solutions
Real-world Go backend engineering problems and solutions: goroutine leaks, slice capacity bugs, database transaction pitfalls, and production-tested fixes.
-
Database Connection Pooling and Query Optimization in Go
Fixing the most common Go database performance problems: N+1 queries, connection pool sizing with pgx, bulk inserts with COPY, and index strategy for PostgreSQL.
-
Building Resilient Go Services: Circuit Breakers and Retry Strategies
Building resilient Go services with exponential backoff, jitter, circuit breakers, and bulkhead isolation. Includes a full resilient HTTP client implementation.
-
Rate Limiting in Go: Token Bucket, Sliding Window, and Redis
Production-grade rate limiting in Go: token bucket with golang.org/x/time/rate, distributed sliding window with Redis Lua scripts, and adaptive throttling based on system health.
-
Distributed Transactions in Go: Sagas, Outbox, and 2PC
How to handle distributed transactions in Go without losing data. Deep dive into the Saga pattern, Transactional Outbox, Two-Phase Commit, and compensating transactions.
-
Monolith vs Microservices in Go: Choosing the Right Architecture
A practical framework for choosing between monolith and microservices in Go. Covers the strangler fig pattern, go work for multi-module repos, and a real-world decision checklist.
-
Welcome to My Bugs and Solutions on Real-World Backend Engineering
An introduction to Muhammad Huzair's practical writing about Go backends, distributed systems, databases, reliability, and production debugging.
subscribe via RSS