Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Searching for keys in a S3 bucket with prefix, suffix or regex?

I have an S3 bucket that contains a million objects, each object keys are quite different from each other and nothing standard at all.

I want to know if there's a way to search for specific key patterns and return those objects using the Amazon S3 SDK for Java?

For example, can I search for the keys using

  • Prefix
  • Suffix
  • or Regex

What are the possible ways to search for keys with S3?

like image 391
quarks Avatar asked Oct 20 '25 10:10

quarks


1 Answers

You can ListObjects() with a given Prefix. Amazon S3 does not support listing via suffix or regex.

The Prefix includes the full path of the object, so an object with a Key of 2020/06/10/foo.txt could be found with a prefix of 2020/06/10/, but not a prefix of foo.

The Java command is: ListObjects()

See also: Performing Operations on Amazon S3 Objects - AWS SDK for Java

With millions of objects, it could be quite slow to list your objects (even with a Prefix) since each API call will return a maximum of 1000 objects.

Alternatively you might want to use Amazon S3 Inventory, which can provide a daily or weekly CSV file containing a list of all objects.

like image 104
John Rotenstein Avatar answered Oct 21 '25 23:10

John Rotenstein



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!