Moving 44 containerised services for 1.3 million users onto a control plane we could automate
A long-running online platform, with 1.3 million users on record, had outgrown a closed vendor control panel and a single front-door proxy. We rebuilt the control plane, split the proxy tier and kept every service online through the move.
At a glance
- 1.3M
users on record
- 44
containerised services under one control plane
- 13
container hosts
- 5
front-door proxies across two platforms
The challenge
The platform had run for years on a commercial control panel whose database was closed and undocumented. Nothing about it could be scripted, so every change meant someone clicking through screens, and the hosts underneath were getting old.
There was a harder limit too. A third-party plugin the whole platform relied on only ever supported one proxy, and its vendor had said that wasn't going to change. One proxy meant one point of failure in front of every user.
The services are game servers, spanning legacy releases of the vendor's software through to current ones, and all of them had to stay online whilst we upgraded underneath.
Architecture
Swipe to explore the diagram
Users reach the proxies only through the DDoS edge. The proxies share state in Redis, so any proxy can carry any user, and every service sits under the panel's API.
What we built
Control plane
- Migrated the whole fleet from the legacy panel to Pterodactyl and its container hosts, then retired the old panel, its database and its VM
- 21 service templates written in house, so a new service is a template choice and not a build
- Bootable install images for container hosts, the panel and container hosts, so rebuilding a host is unattended
Proxy tier
- Five Velocity proxies across two platforms, sharing state through Redis
- A Redis schema kept compatible with the legacy proxy plugin, so old and new proxies ran side by side through the cutover
- The public port admits only the DDoS edge's published address ranges
Operations
- Every service's configuration tracked in git with its own change log (6,205 files at the last count)
- A local test rig that resolves the exact plugin set from a compatibility matrix before anything reaches production
- Per-flow TCP sampling on each proxy host into TimescaleDB, so a wave of disconnects can be pinned to one side of the edge or the other
Data
- MySQL primary with a read replica. Tooling reads from the replica and only the application writes
- Nightly replica dumps to NAS, checked before anything is rotated out
Engineering decisions
Pick the panel with an API
The old panel's database was closed, so it could never be automated. Pterodactyl, kept on a frozen fork, gives every server an API and every operator a scoped file account, and nobody needs a shell on a host.
Write the proxy plugin ourselves
One in-house proxy plugin replaced five third-party ones. One vendor had ruled out multi-proxy support, and when a new release of the platform software landed first we couldn't patch code we didn't own.
No big-bang cutover
Because the new proxies spoke the old plugin's Redis schema, users moved across a proxy at a time, and rolling back meant switching one off.
The repo is the change record
Anything changed out of band is written back to that service's change log. A stale folder is treated as a bug, which is what keeps 44 services honest.
Read-only first
Every new piece of automation ships read-only. The version that changes things arrives once the read form has proved stable, and it still asks before it acts.
Outcome
The legacy panel and its database are gone and the platform has run multi-proxy in production since September. Two separate platforms now share one control plane, including a service for a second product that joined later.