What is the industry/production grade solutions or if you have already any experience please share it. Thanks
In AWS, you can use Aurora to scale reads access over multiple instances. It’s easy to setup and you can use scaling policies to follow your load. You need to check your billing, it’s not the cheapest service.
Scaling horizontally is never trivial, can you share more about the requirements?
Usually application level partitioning, but it’s hardly repeatable. Cloud providers offer solutions like Spanner, that cost quite a bit. You have new SQL DBs like yugabyte or cockroach, which are not exactly MySQL, and are not really battle tested. And noSQL like MongoDB or Cassandra, which are, well, not even SQL, but are proven technologies and scale quite effortlessly.
I work for another distributed database company. I can say that it’s much harder to convert cockroachDB customers than Yugabyte customers. Given that, I’d think that CockroachDB is likely the more vetted solution. Sure it’s new (2017), but it’s not THAT new.
IIRC, MySQL (and PosgreSQL) is pretty much limitted to a write/read-replica sort of horizontal scaling. Other SQL engines have better support for multi-master configurations.
However, these types of configurations are usually tied to licensing - especially for Microsoft SQL server and OracleDB.
As another commenter suggested, there is Yugabyte and CockroachDB as well - of those two I think CockroachDB is the more mature product. And they’re one of the fiercest competitors for the company I work for too.
I cannot speak to “Battle Tested-ness” of CockroachDB, but given it’s been around for a few years now, I don’t think it’s quite as risky as other comments have indicated. Also, they’re doing something right if we haven’t been able to convert many CockroachDB customers.
If you truly need read/write to scale, multi-masrer clustering with MariaDB Galera is probably the best way to do it. They (MariaDB) also sell a load balancer/query proxy Max scale that can do a lot of surprisingly complex stuff (like publish new data to Kafka or centrally ship binlogs from one place to multiple read replicas).
However generally my advice is that if you’re finding yourself trying to build a big relational database and writing to it a significant amount of times, it might be time to consider a different or at least modified architecture. Especially if your use case starts to scale to more than what Galera can handle. At some point, all these solutions become eventually consistent the more you scale. And if you are willing to accept eventual consistency, there are some clever ways to do storage using things like queueing, batching, caching etc that can scale horizontally much further than any relational database can.
Wow, thanks 👍👍👍 this will help me a lot.
I hope so! Looking forward to the writeup at the end!
For on-prem we use a Galara cluster. It’s managed by ClusterControl because we needed something with enterprise support.
