I'm using AWS Athena to query S3 bucket, that have partitioned data by day only, the partitions looks like day=yyyy/mm/dd. When I tried to us Glue to run update the partitions every day, It creates new table for each day (sync 2017, around 1500 tables).
I tried to use Partition projection with like this:
PARTITIONED BY (
day string)
TBLPROPERTIES (
'has_encrypted_data'='false',
'projection.day.format'='yyyy/mm/dd',
'projection.day.interval'='1',
'projection.day.interval.unit'='DAYS',
'projection.day.range'='2017/01/01,NOW',
'projection.day.type'='date',
'projection.enables'='true'
But the partition not updated without MSCK Repair. Any ideas? Do I miss something with the partition projection?
PARTITIONED BY (
day string)
TBLPROPERTIES (
'has_encrypted_data'='false',
'projection.day.format'='yyyy/mm/dd',
'projection.day.interval'='1',
'projection.day.interval.unit'='DAYS',
'projection.day.range'='2017/01/01,NOW',
'projection.day.type'='date',
'projection.enabled'='true' **here you have written enables , that should be enabled**
After that partition projection will work. :)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With