arguments error while calling an AWS Glue Pythonshell job from boto3

Question

Based on the previous post, I have an AWS Glue Pythonshell job that needs to retrieve some information from the arguments that are passed to it through a boto3 call.

My Glue job name is test_metrics

The Glue pythonshell code looks like below

import sys
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv,
                          ['test_metrics',
                           's3_target_path_key',
                           's3_target_path_value'])

print ("Target path key is: ", args['s3_target_path_key'])
print ("Target Path value is: ", args['s3_target_path_value'])

The boto3 code that calls this job is below:

glue = boto3.client('glue')

response = glue.start_job_run(
    JobName = 'test_metrics',
    Arguments = {
        '--s3_target_path_key': 's3://my_target',
        '--s3_target_path_value': 's3://my_target_value'
    } 
)

print(response)

I see a 200 response after I run the boto3 code in my local machine, but Glue error log tells me:

test_metrics.py: error: the following arguments are required: --test_metrics

What am I missing?

Answer 1

Which job you are trying to launch? Spark Job or Python shell job?

If spark job, JOB_NAME is mandatory parameter. In Python shell job, it is not needed at all.

So in your python shell job, replace

args = getResolvedOptions(sys.argv,
                          ['test_metrics',
                           's3_target_path_key',
                           's3_target_path_value'])

with

args = getResolvedOptions(sys.argv,
                           ['s3_target_path_key',
                           's3_target_path_value'])