Is it possible to update only part of a Glue Job using AWS CLI?

Question

I am trying to include in my CI/CD development the update of the script_location and only this parameter. AWS is asking me to include the required parameters such as RoleArn. How can I only update the part of the job configuration I want to change ?

This is what I am trying to use

aws glue update-job --job-name <job_name> --job-update Command="{ScriptLocation=s3://<s3_path_to_script>}

This is what happens :

An error occurred (InvalidInputException) when calling the UpdateJob operation: Command name should not be null or empty.

If I add the default Command Name glueetl, this is what happens :

An error occurred (InvalidInputException) when calling the UpdateJob operation: Role should not be null or empty.

Answer 1

An easy way to update via CLI a glue-job or a glue-trigger is using --cli-input-json option. In order to use correct json you could use aws glue update-job --generate-cli-skeleton what returns a complete structure to insert your changes.

EX:

{"JobName":"","JobUpdate":{"Description":"","LogUri":"","Role":"","ExecutionProperty":{"MaxConcurrentRuns":0},"Command":{"Name":"","ScriptLocation":"","PythonVersion":""},"DefaultArguments":{"KeyName":""},"NonOverridableArguments":{"KeyName":""},"Connections":{"Connections":[""]},"MaxRetries":0,"AllocatedCapacity":0,"Timeout":0,"MaxCapacity":null,"WorkerType":"G.1X","NumberOfWorkers":0,"SecurityConfiguration":"","NotificationProperty":{"NotifyDelayAfter":0},"GlueVersion":""}}

Well here just fill the name of the job and change the options. After this you have to transform your json into a one-line json and send into the command using ' '

aws glue update-job --cli-input-json '<one-line-json>'

I hope help someone with this problem too.

Ref:

• https://docs.aws.amazon.com/cli/latest/reference/glue/update-job.html
• https://w3percentagecalculator.com/json-to-one-line-converter/

Answer 2

I don't know whether you've solved this problem, but I managed using this command:

aws glue update-job --job-name <gluejobname> --job-update Role=myRoleNameBB,Command="{Name=<someupdatename>,ScriptLocation=<local_filename.py>}"

You don't need the the ARN of the role, rather the role name. The example above assumes that you have a role with the name myRoleNameBB and it has access to AWS Glue.

Note: I used a local file on my laptop. Also, the "Name" in "Command" part is also compulsory.

When I run it I go this output:

{
    "JobName": "<gluejobname>"
}