// HACKER NEWS — CYBERSECURITY
Monzo Stand-In
This blog post was accurate when we published it – head to monzo.com or your Monzo app for the most up to date information.
Our customers reasonably expect to be able to spend on their card, make bank transfers and pay their bills 24 hours a day, 365 days a year. Their lives don’t have downtime for maintenance so nor should we. We dedicate a lot of our engineering effort to minimise the risk of downtime during technical migrations and other day-to-day operations, but unforeseen incidents that cause unexpected outages are impossible to eliminate entirely.
We take reliability seriously at Monzo so we built a completely separate backup banking infrastructure called Monzo Stand-in to add another layer of defence so customers can continue to use important services provided by us. We consider Monzo Stand-in to be a backup of last resort, not our primary mechanism of providing a reliable service to our customers, by providing us with an extra line of defence.
Monzo Stand-in is an independent set of systems that run on Google Cloud Platform (GCP) and is able to take over from our Primary Platform, which runs in Amazon Web Services (AWS), in the event of a major incident. It supports the most important features of Monzo like spending on cards, withdrawing cash, sending and receiving bank transfers, checking account balances and transactions, and freezing or unfreezing cards.
Our Primary and Stand-in Platforms run independently to one another, each consisting of Kubernetes clusters running a unique set of services on top of typical platform components such as a database, queueing systems and locking mechanisms. The services running in each platform are unique in the sense that services in the Stand-in Platform don’t ever run in the Primary Platform, or vice-versa, even for behaviours that are common across both like processing a card payment.
Each platform is able to make their own decisions about approving or declining transactions and can establish their own connections to payment networks via multiple physical data centres.
Monzo Stand-in also runs a limited set of API endpoints tailored to serve limited functionality while we’re using Stand-in. The Monzo App checks whether Monzo Stand-in is enabled periodically in the background, and if it is, it flips to a simplified UI that supports our most important features.
It might seem strange to build brand new services from the ground up for Monzo Stand-in rather than deploy the same services we run on the Primary Platform, but there are a number of motivations for us taking this approach.
If we tried to run the exact same set of services we would need to replicate all of our data between the two platforms. To do this well we’d have to maintain strong consistency of our data. This would mean that writes to our database would be considered successful only if the data is written to both platforms. If either platform became unavailable we’d be unable to write anything without sacrificing consistency, reducing our overall availability rather than improving it.
Instead of maintaining strong data consistency we accept that the replication is non-blocking and that it is eventually consistent, but systems like our ledger wouldn’t be able to tolerate eventually consistent data.