ec2 private subnet can't reach 169.254.169.254 when os is windows server 2019

Question

i've got multiple ec2 instances running in a private subnet (only traffic within the vpc is allowed). some of those instances are custom OS, some of those instances run AWS windows server 2012 ami, and some run AWS windows server 2019 ami.

on all machines, except the windows server 2019 - i can retrieve the meta data of the instance by calling "http://169.254.169.254/latest/meta-data". on windows server 2019 - it fails. firewall is down, same route table for all machines, and mostly the only diff between them is that the working instances run EC2Config, and the windows server 2019 run EC2Launch (of course, the OSes are different too).

any idea what can cause this behavior?

thanks!!

Answer 1

This recently happened to the Windows Server 2019 machine that I have on AWS. From my limited understanding, there were faulty rules in the routing tables as shown by ROUTE PRINT which for me had the following lines:

Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0      172.31.48.1    172.31.51.191     15
  169.254.169.254  255.255.255.255       172.31.0.1    172.31.51.191     30 **FAULTY**
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
  169.254.169.254  255.255.255.255       172.31.0.1      15 **FAULTY**

This means all traffic is routed thru the correct 172.31.48.1 gateway EXCEPT for 169.254.169.254 which gets routed to a black hole. So, a remedy is needed to override the faulty routes. One suggestion is to create a new routing rule for 169.254.169.254, for example (N.B. you need an administrator elevated Command Prompt to run this command):

ROUTE ADD 169.254.169.254 mask 255.255.255.255 172.31.48.1 METRIC 10

Setting the metric to 10 means this new route will have a stronger priority over the faulty routes (which had weaker priority metrics 15 and 30).

C:\Windows\system32>route add 169.254.169.254 mask 255.255.255.255 172.31.48.1 metric 10
 OK!
C:\Windows\system32\route print
...
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0      172.31.48.1    172.31.51.191     15
  169.254.169.254  255.255.255.255      172.31.48.1    172.31.51.191     10 **NEW**
  169.254.169.254  255.255.255.255       172.31.0.1    172.31.51.191     30 **FAULTY**
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
  169.254.169.254  255.255.255.255       172.31.0.1      15 **FAULTY**
...