An engineer who hunts assumptions

Most production failures aren’t caused by complicated bugs. They’re caused by assumptions — something everyone believed was true turns out not to be.

• A message was assumed to be delivered once.
• A queue was assumed to be ordered.
• A service was assumed to be available.
• A user was assumed to belong to the right tenant.
• A piece of state was assumed to exist.

Eventually reality asks the question anyway. I prefer asking it first.

I call these assumptions invariants. They’re the system’s safety net.

• A request should never read another tenant’s data.
• A message should never be acknowledged before its work is complete.
• Every workflow should end in a known state.
• An audit trail should exist regardless of which service performed the action.

Once the invariants are identified, architecture becomes less about opinion and more about enforcement.

The implementation evolves. The invariants tend to survive.

For several years I worked on systems that were growing continuously. Every order of magnitude exposed a different problem.

At small scale, counting views is trivial. At larger scale, every path that touches a view count becomes part of the problem: aggregation, storage, caching, replication, hot keys, recovery, and operational visibility.

The same pattern repeated everywhere.

• Analytics stopped being a reporting problem and became a streaming problem.
• Video processing stopped being a media problem and became a scheduling problem.
• Infrastructure stopped being provisioning and became systems design.
• Kafka stopped being a queue and became a state-management problem.

Every time the system grew, some design that previously worked stopped working.

Those experiences taught me to be skeptical of permanent solutions. Most architectures are temporary. The goal isn’t to find a design that lasts forever — it’s to understand which assumptions will eventually break, and make change affordable when they do.

One lesson that surprised me is how similar reliability and scalability really are.

• Both require clear ownership.
• Both require explicit state transitions.
• Both require well-defined boundaries.
• Both punish hidden assumptions.

The systems that scaled well were usually easier to reason about. The systems that were reliable were usually easier to scale.

Many of the lessons that kept systems alive under load turned out to be the same lessons that kept them correct during failures.

I’ve worked with PostgreSQL, Redis, MongoDB, Cassandra, Elasticsearch, Kafka, RabbitMQ, Kubernetes, Terraform, and a long list of other technologies. One of my favorite observations from those years is:

Every database looks good when you’re not using it.

Every tool has strengths. Every tool eventually exposes weaknesses. Most engineering discussions happen around technologies; most production problems happen around state.

• Who owns it?
• Who can modify it?
• How does it move?
• How does it recover?
• What happens when two systems disagree?

Those questions usually matter far more than framework choices.

I generally prefer explicit ownership over hidden magic, recovery over coordination, and making incorrect states structurally impossible whenever practical.

I’ve always enjoyed building things from scratch.

Many of the systems I’ve worked on started as a vague requirement and gradually became platforms: infrastructure, deployment pipelines, observability, access control, integrations, automation, operational tooling, and the product itself evolving together.

The part I enjoy most isn’t writing code. It’s taking something ambiguous and gradually making it understandable.

• Defining boundaries.
• Removing assumptions.
• Finding the simplest model that still survives reality.

I assume every component will fail eventually.

• A broker will disappear.
• A deployment will go wrong.
• A dependency will time out.
• A message will be delivered twice.
• A network partition will happen.

The goal isn’t to prevent every failure. The goal is to know what happens next.

I generally prefer systems that fail loudly and predictably over systems that continue operating quietly in a broken state.

Not perfect systems. Understandable systems — where recovery is part of the design instead of an afterthought.