How create a role for long running glue redshift job?

Question

A long running glue jobs for exporting data from Redshift to S3 are failed due to:

S3ServiceException:The provided token has expired

Amazon describes using a custom role as workaround (here). But they do not provide any example.

Could somebody provide a cloudformation snipped? What a role should looks like?

If I uses glue job, should I add action for dynamodb or EMR cluster into Role policy?

Answer 1

I have listed the different documentation that can help you:

1. AWS Glue permission: https://docs.aws.amazon.com/glue/latest/dg/permissions.html

2. IAM Role for Amazon Web Service: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html

3. AWS RedShift permission: https://docs.aws.amazon.com/redshift/latest/mgmt/grant-privileges.html

4. S3 permissions: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-with-s3-actions.html

5. Dynamo IAM Permission: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/iam-policy-examples.html

6. EMR IAM Permission: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-iam-roles.html

And an example of configuration in CloudFormation, I let you try it and adapt it if necessary:

Resources:
  GlueJobRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "glue.amazonaws.com"
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: "GlueJobS3Policy"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              # Permissions for Redshift
              - Effect: "Allow"
                Action:
                  - "redshift:DescribeClusters"
                  - "redshift:CopyFromS3"
                  - "redshift:Select"
                Resource: "*"
              
              # Permissions for S3
              - Effect: "Allow"
                Action:
                  - "s3:GetObject"
                  - "s3:PutObject"
                  - "s3:ListBucket"
                  - "s3:ListBucketMultipartUploads"
                Resource:
                  - "arn:aws:s3:::your-s3-bucket-name/*"
                  - "arn:aws:s3:::your-s3-bucket-name"
              
              # Permissions for Glue resources
              - Effect: "Allow"
                Action:
                  - "glue:GetTable"
                  - "glue:GetTableVersion"
                  - "glue:GetTableVersions"
                  - "glue:GetDatabase"
                  - "glue:GetPartitions"
                  - "glue:BatchGetPartition"
                  - "glue:CreateJob"
                  - "glue:GetJob"
                  - "glue:UpdateJob"
                  - "glue:StartJobRun"
                  - "glue:GetJobRun"
                Resource: "*"
              
              # Permissions for DynamoDB (optional)
              - Effect: "Allow"
                Action:
                  - "dynamodb:Scan"
                  - "dynamodb:Query"
                Resource: "*"
              
              # Permissions for EMR (optional)
              - Effect: "Allow"
                Action:
                  - "elasticmapreduce:ListClusters"
                  - "elasticmapreduce:DescribeCluster"
                  - "elasticmapreduce:DescribeStep"
                Resource: "*"