Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Partitioning By date column in Athena

I'm using AWS Athena to query S3 bucket, that have partitioned data by day only, the partitions looks like day=yyyy/mm/dd. When I tried to us Glue to run update the partitions every day, It creates new table for each day (sync 2017, around 1500 tables).

I tried to use Partition projection with like this:

PARTITIONED BY ( 
  day string)

TBLPROPERTIES (
  'has_encrypted_data'='false', 
  'projection.day.format'='yyyy/mm/dd', 
  'projection.day.interval'='1', 
  'projection.day.interval.unit'='DAYS', 
  'projection.day.range'='2017/01/01,NOW', 
  'projection.day.type'='date', 
  'projection.enables'='true'

But the partition not updated without MSCK Repair. Any ideas? Do I miss something with the partition projection?

like image 918
Matan Stern Avatar asked Dec 06 '25 08:12

Matan Stern


1 Answers

PARTITIONED BY ( 
  day string)

TBLPROPERTIES (
  'has_encrypted_data'='false', 
  'projection.day.format'='yyyy/mm/dd', 
  'projection.day.interval'='1', 
  'projection.day.interval.unit'='DAYS', 
  'projection.day.range'='2017/01/01,NOW', 
  'projection.day.type'='date', 
  'projection.enabled'='true'            **here you have written enables , that should be enabled**

After that partition projection will work. :)

like image 83
Chaurasiya Avatar answered Dec 11 '25 07:12

Chaurasiya