How does a random hash prefix improve S3 large-scale GET performance?

Question

I'm going to go ahead and point out this has been asked and answered here Add a random prefix to the key names to improve S3 performance? — insufficiently in my opinion.

Can someone explain in more layman's terms exactly how adding a random hash prefix to objects to be accessed at scale helps performance?

A scenario will probably help illustrate my lack of understanding:

1000 Clients all are trying (with appropriate permissions) to perform GET requests on object foo in bucket bar, so how would making foo --> 4jd8fb-foo help alleviate stress on the system? Do the clients not still all want the same object in their GET requests?

I'm obviously missing something that's probably silly, but I'd really like to wrap my head around why this helps— I'd guess my misunderstanding stems from how S3 handles indexing and partitioning but some further guidance would be greatly appreciated.

Answer 1

I would suggest that your intuition is correct: entropy in object key prefixes doesn't do anything to improve repeated reads of exactly the same one object.

That isn't the kind of performance under consideration (although if you have that kind of a workload, you should consider using CloudFront in front of S3, dividing the workload among nodes in dozens of edge locations and keeping cached copies near wherever your viewers happen to be).

Random prefixes impact horizontal scaling potential, this directly improving potential write capacity -- that is, the achievable object creation and overwrite rate, in requests per second¹ -- by reducing the incidence of hot spots in the index.

This improves potential write capacity by giving S3's partition-splitting logic something solid to work with. If you have (for example) hexadecimal object key prefixes, S3 can potentially split your bucket into up to 16 different partitions just on the first octet of the object key, 256 on the second, 4096 on the third... so with this seemingly-simple change, you give the service an easy way to cut the workload on each partition in half, again and again.

If you are creating objects with ever-incrementing keys, particularly timestamps, there isn't anything that can be done to reduce the load on one partition by splitting it into two, because wherever a split is contemplated, new objects are always going to be in the right-side (> split point) new partition, while the left-side (< split point) would be left handling little or no new object creation.

---

¹ in requests per second, not payload bandwidth, as bandwidth doesn't seem to be a concern, because S3 apparently shards its backing store independently of object keys -- the object index and the object payload appear to be stored separately, otherwise partition splits would be extremely expensive in machine terms, not to mention being a much more delicate operation, since durably-stored objects would have to be moved to new storage locations.