Master/Slave Datastore vs High Replication Datastore

Question

Before starting GAE datastore, I think it will be good to know the difference b/w Master/Slave Datastore and High Replication Datastore? And what makes GAE team to migrate from Master/Slave to HRD?

Answer 1

The difference between the two (as well as the reason for the switch) is increased fault-tolerance and data consistency.

The Master/Slave Datastore implements a primary-backup protocol. Each app is served by a master (i.e. a single data center) and its data is replicated asynchronously to the slave (i.e. some other data center). The problem with this schema is that it doesn't protect your application from local failures and is more likely to lead to data inconsistencies.

The High Replication Datastore implements the Paxos consensus algorithm to ensure that a majority of data centers maintain a consistent view of your application's data. Because your data is no longer reliant on the health of a single data center, the datastore is able to function properly even in the presence of local/global failures. Google's engineers also benefit from this implementation, as it allows them to perform data center maintenance without having to enforce scheduled read-only periods for AppEngine applications.

The downside of using the High Replication Datastore is slower writes (about 2x as slow, since Paxos is inherently 2-phase). This isn't that big of a deal though, especially when compared to the increased fault tolerance and data consistency that the High Replication Datastore has to offer.

Answer 2

For the first three years of App Engine only with Master/Slave, the health of the datastore was tied to the health of a single data center. Users had low latency and strong consistency, but also transient data unavailability and planned read-only periods.

The High Replication Datastore trades small amounts of latency and consistency for significantly higher availability.