Azure DevOps OnPrem - Service Connection failed - Failed to obtain the Json Web Token

Question

So, a bit of a strange one and I'm hoping that someone's had this issue and found a fix (Or understands why it's happening)

So this used to work without any issue... Just started happening this week - to our knowledge

Basically, when you create a new Azure Resource Manager Service Connection within Azure DevOps you get the following error; Failed to obtain the Json Web Token(JWT) using service principal client ID. Exception Message: Object reference not set to an instance of an object.

This only happens on our On-Prem Azure DevOps server - I can create the service connection with the same settings on our Dev.Azure.com version without any issues.

Troubleshooting

• Created multiple Service Principals
• Tried different subscriptions
• Tried giving SP Owner permissions to the subscription
• Even if you try to re-verifying an existing service connection you get the same error
• Works in Dev.Azure.com
• Been no networking changes regarding our on-prem stuff
• Still able to action releases using the existing service connections (Even though you can't verify them)
• Tried creating another secret as advised my the MS troubleshooting page
• I have a PS script to gain the bearer token using the SP that I created and that works

Obviously, I've tried googling this issue, it comes up in a couple of places but none of which have a resolution or an understanding of why it's happening...

TIA

Answer 1

The solution can be found on the Troubleshooting page accessible from the "Troubleshoot" link in the Edit pane of the Service Connection: Troubleshoot ARM service connections

Relevant section pasted here for reference/persistence:

Failed to obtain the JWT by using the service principal client ID

This issue occurs when you try to verify a service connection that has an expired secret.

To resolve this issue:

1. Go to Project settings > Service connections, and then select the service connection you want to modify.

2. Select Edit in the upper-right corner, and then make any change to your service connection. The easiest and recommended change is to add a description.

3. Select Save to save the service connection.

⚠ Note

Select Save. Don't try to verify the service connection at this step.

4. Exit the service connection edit window, and then refresh the service connections page.

5. Select Edit in the upper-right corner, and now select Verify.

6. Select Save to save your service connection.