What two years on a payments platform taught me about reliability
Lessons from building and operating Django services on a high-volume payments platform: idempotency, observability, and boring deploys.
Idempotency is a product feature
On a payments platform, a retry is not an edge case — it is the normal path. Every write endpoint needs a caller-supplied key and a stored result, so the second attempt returns the first outcome instead of creating a second charge.
Logs are not observability
Free-text logs tell you what one request did. Metrics tell you what the system is doing. Adding a handful of well-chosen counters and latency histograms per service surfaced problems long before anyone opened a log file.
Boring deploys win
Small changes, released often, with a fast rollback path, beat large releases every time. The most useful engineering work I did was often making a deploy less exciting.