Around the World in 80 Days: Building a Global delivery system

We've all been in this situation... staring at a blank whiteboard, with a problem to solve, a new system to build. It is exciting; you want to do something new, something you have not done before, something to break away from the mundane. I agree, there's a certain romanticism to it. The thrill of an adventure.

And I've given into that thrill before. I still remember the horror of deploying RethinkDB to production because it felt like the future. Or the time I decided to sneak in a gRPC service as a HTTP proxy using the gRPC Gateway — because "how hard could it be?" Those decisions cost me sleep. Literally. 3:00 AM pages from the SRE on-call are not how you want to remember a Tuesday.

So when I started building Octopus Cards, I made myself a promise: no adventures. Boring and mundane. Simple, solid foundations. The kind of stuff that lets you sleep at night.

This is the story of how we went from zero to processing millions of digital card transactions across 100+ countries — and how the most boring technology stack imaginable saved my ass every step of the way.

The Philosophy: Boring is Beautiful

I'll be honest — the temptation was real. I kept catching myself sketching out microservice boundaries, eyeing service meshes, wondering if maybe we needed an event-driven architecture from day one. I've been burned by that instinct before. I've watched teams (including ones I was on) over-engineer themselves into corners they couldn't escape.

So I fought the urge. Hard.

Our stack is deliberately boring: Go, PostgreSQL, and Valkey (the open-source Redis fork). No ORMs. No auto-migrations. No magic. Every line of SQL is written with intent. Every migration is reversible. Every dependency is explicit.

And honestly? It's the best technical decision I've ever made. The system is simple, a small team can maintain it, and it processes orders in under a second millions of times over. I sleep through the night now. Mostly.

Go: The Language That Gets Out of Your Way

I went with Go because I needed something I could trust not to surprise me at 2 AM. It's simple, it compiles to a single binary, and the compiler catches mistakes before they become incidents. The concurrency model doesn't hurt either.

Our entire application runs from a single main.go entry point:

go run main.go server        # HTTP server
go run main.go worker        # Background job processors
go run main.go cron          # Scheduled tasks
go run main.go run-all       # Everything together

One binary. One deployment artifact. I cannot tell you how many times this simplicity has saved me. When something goes wrong at midnight, I'm not debugging which of twelve services is misbehaving. I'm looking at one process, one set of logs, one thing to restart.

The CLI is built with Kong, which gives us a clean command structure without framework overhead. Server, workers, crons, migrations, seeders — all subcommands of the same binary, sharing the same initialization pipeline. It's the kind of boring that makes you grateful during an incident.

PostgreSQL: The Database That Does Everything

We use PostgreSQL for everything — and I mean everything:

• Transactional data: Orders, inventory, clients, vendors
• Critical Logs: API responses, requests, audits
• Job tracking: Execution history, cron metrics, import/export status
• Vector embeddings: Product matching via pgvector (yes, PostgreSQL does AI too)

We run primary + read replica for write/read splitting. Not at the query level — at the connection level. Write queries go to the primary. Read queries go to the replica. The repository layer decides which connection to use, keeping it dead simple.

I know this sounds like I'm putting all my eggs in one basket. And I am. But it's a really, really good basket. Postgres has been around for decades, has incredible tooling, and handles every workload I've thrown at it. Every time I've been tempted to add another database to the mix, Postgres already had a feature for it.

Why Squirrel Over an ORM

Early on, I considered GORM. It would've been faster to get started. But I'd been burned by ORM magic before — the kind where everything works beautifully until you hit scale, and then you're staring at a query plan that makes no sense because the ORM decided to do six JOINs behind your back.

So I went with Squirrel, a SQL query builder for Go. Here's what a real query looks like:

query, args, err := squirrel.
    Select("id", "product_id", "denomination", "status").
    From("inventories").
    Where(squirrel.Eq{"product_id": productID}).
    Where("deleted_at IS NULL").
    Where("status = 'active'").
    Limit(50).
    ToSql()

It's just SQL — with type-safe parameter binding and no string concatenation. You can read it. You can debug it. You can copy it into psql and run it directly. And when I'm at my desk at 11 PM trying to figure out why a query is slow, I can see exactly what's hitting the database. No N+1 surprises. No lazy loading foot-guns. No SELECT * hidden behind a method call.

This decision has saved me more debugging hours than I can count.

Migrations: Explicit, Reversible, Predictable

Every schema change is a Goose migration file with explicit Up and Down sections:

-- +goose Up
CREATE TABLE inventories (
    id BIGSERIAL PRIMARY KEY,
    product_id BIGINT NOT NULL REFERENCES products(id),
    denomination DECIMAL(12,2) NOT NULL,
    status VARCHAR(20) DEFAULT 'active',
    deleted_at TIMESTAMP,
    created_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_inventories_product ON inventories(product_id)
    WHERE deleted_at IS NULL;

-- +goose Down
DROP TABLE IF EXISTS inventories;

Every index is intentional. Every column type is chosen. Every migration is reviewed before it touches the database.

I've had to roll back a migration at 1 AM exactly once. It took thirty seconds and worked perfectly. That one moment justified every minute I'd spent writing Down sections I thought I'd never use.

Valkey: The Cache That Thinks in Data Structures

We use Valkey (the open-source Redis fork) not just as a key-value cache — we use it as a data structure server.

The most critical pattern in our system is inventory pumping: every 6 hours, a cron job reads the entire inventory table and loads it into Valkey sorted sets, keyed by product and denomination:

Key:    inventory:product:123:active
Type:   Sorted Set
Score:  denomination (e.g., 10.00, 25.00, 50.00)
Member: inventory_id

When a customer redeems a card, we don't hit PostgreSQL. We pop an item from the sorted set in Valkey — an O(log N) operation that completes in microseconds. The database is only touched to mark the item as allocated.

This single pattern reduced our p99 allocation latency from 200ms to under 5ms. I remember the day we deployed this. I sat there refreshing the Grafana dashboard, watching the latency line flatline, thinking "there's no way this actually works." But it did. And it kept working. Boring technology, used correctly, is a beautiful thing.

Observability: The Thing That Saved Us the Most

I'd love to say we built observability in from day one because we were disciplined. The truth is closer to: I'd been on teams where we didn't, and the experience was traumatic enough that I refused to repeat it.

OpenTelemetry Everywhere

Every significant operation creates a trace span:

ctx, span := utils.Tracer.Start(ctx, "inventory.allocate")
span.SetAttributes(
    attribute.Int64("product_id", productID),
    attribute.String("client_id", clientID),
)
defer span.End()

Trace context flows through the entire stack: HTTP request → service layer → database query → cache lookup → queue publish → background worker. One trace ID ties everything together.

When something goes wrong — and things always go wrong — I can pull up a single trace and see exactly what happened, in what order, and how long each step took. It's like having a flight recorder for every request. I've diagnosed issues in minutes that would have taken hours without this.

Structured Logging with Zap

Every log line is structured JSON with request context:

logger := middleware.Logger(ctx)
logger.Infow("Order created",
    "order_id", order.ID,
    "product_id", order.ProductID,
    "duration_ms", elapsed.Milliseconds(),
)

No fmt.Printf. No unstructured strings. Every log is queryable, filterable, and correlated with its trace. I used to think structured logging was overkill for small teams. Then I tried to grep through 200MB of unstructured logs at 3 AM during an incident. Never again.

The Cron System: Simple but Serious

Our scheduler runs on gocron, with a lifecycle that goes beyond "run this every N hours":

type ScheduledTask interface {
    Identifier() string
    TimePattern() string
    Execute(ctx context.Context) error
    PreExecute(ctx context.Context)
    PostExecute(ctx context.Context)
    HandleFailure(ctx context.Context, err error)
}

Each task has pre/post hooks, failure handlers, and execution tracking. Jobs run in singleton mode — if a previous execution is still running, the next one is rescheduled instead of overlapping. I learned this the hard way at a previous job, where two instances of the same cron overlapped and double-charged a batch of customers. That's not a mistake you make twice.

Schedules are defined in code but overrideable from the database. An admin can disable a job or change its frequency without a deployment. Feature flags control which jobs are even registered:

if utils.FeatureFlags.IsVouchersEnabled() {
    tasks = append(tasks,
        scheduler.NewInventoryPumpTask(),
        scheduler.NewPendingOrderRetryTask(),
    )
}

No vouchers feature? No voucher cron jobs consuming resources.

Graceful Everything

Every process — server, worker, scheduler — handles SIGTERM gracefully:

1. Stop accepting new work
2. Wait for in-flight operations to complete
3. Flush logs and metrics
4. Exit cleanly

No orphaned transactions. No half-processed messages. No lost data. The system can be restarted at any time, on any instance, without coordination. Unlike the Death Star, our shutdown sequence actually works — and I don't need an exhaust port to trigger it.

What We Didn't Build (or: How I Avoided Building a Death Star)

I have to be honest about this section. It's not that I was wise enough to avoid these things from the start. It's that I've seen — and sometimes built — the Death Star. I know what it feels like to be six months into a microservice migration, realising you've just recreated a monolith but worse, with network calls where function calls used to be.

The Empire's fatal flaw wasn't a lack of firepower — it was over-engineering. One exhaust port. One single point of failure. One proton torpedo and the whole thing is space dust. I've seen that pattern in software more times than I'd like to admit.

Here's what I chose not to build, and why I'm grateful for every one of these decisions:

• No microservices. One binary serves everything. I was tempted, believe me. But with a small team, a monolith is a superpower. One thing to deploy, one thing to monitor, one thing to reason about. No Death Star — no exhaust port.
• No Kubernetes. A single Go binary behind nginx doesn't need container orchestration. I don't need to manage a fleet when I'm running a starfighter. K8s is great — for problems I don't have.
• No GraphQL. REST with clear route groups and pagination headers has been plenty. Every time I think "maybe we should add GraphQL," I remember the last time I debugged a deeply nested resolver and I pour myself a coffee instead.
• No NoSQL. PostgreSQL handles relational data, full-text search, and vector embeddings. I was this close to adding MongoDB for "flexible schemas" early on. Thank God I didn't. Postgres does it all, and unlike the One Ring, this one actually works in your favour.
• No auto-migration. Every schema change is intentional and reviewed. The rebels won because they studied the blueprints. I review mine before they ship — because I've seen what happens when you don't.

The Result

Millions of digital cards delivered. 100+ countries. Sub-second end-to-end latency. 99.9% uptime. A codebase that any Go developer can read, understand, and contribute to on day one.

All built on PostgreSQL, Valkey, and Go. No magic. No framework lock-in. No thermal exhaust ports. Just battle-tested, boring technology — doing exactly what it was designed to do.

The Empire built a moon-sized superweapon and lost it twice. I built a monolith with sorted sets and honestly? I just sleep better now. That's the real win.

This is part one of our engineering series. Next up: how we built real-time inventory allocation with sorted sets, and why our cache layer is the most important component in the stack.