Interface with no outbound access to IMDS will fail

[joeysk2012]

An internal customer reported a issue when the IMDS not able to reach the IMDS server if the default interface becomes an interface in which IMDS is not able to reached such as a local interface on the customer on-premise. The working fix is to use an interface which can call IMDS and use that as a parameter to curl --interface and store this interface name between calls.

Submitted a Draft PR and testing these changes:
#116

I will follow up once review and testing is completed.

UPDATE: This issue only applies to instance with multiple network interfaces. Ie Interface 1 is an ENI and Interface 2 is LNI (or something with blocked access to IMDS). Do not mistake for regular connectivity issues with single interfaces.

UPDATE1: We are holding off on merging this because of the high risk change and talking with EC2 NX to see if they can have IMDS guaranteed for a primary ENI.

[lingtjien]

Using the AL2023 image on EC2 some of my instances (in multiple regions) lose network connectivity (reboot of the instance fixes it). When I check the log files I see this message being spammed constantly when it's trying to refresh something for the network interface

Feb 12 03:23:57 ip-10-0-0-41.ec2.internal ec2net[4122880]: [get_meta] Querying IMDS for network/interfaces/macs/0a:ff:c5:48:47:e7/device-number
Feb 12 03:23:57 ip-10-0-0-41.ec2.internal setup-policy-routes[4123378]: /usr/share/amazon-ec2-net-utils/lib.sh: line 89: imds_endpoint: unbound variable
Feb 12 03:23:57 ip-10-0-0-41.ec2.internal ec2net[4122880]: [get_meta] Querying IMDS for network/interfaces/macs/0a:ff:c5:48:47:e7/device-number
Feb 12 03:23:57 ip-10-0-0-41.ec2.internal setup-policy-routes[4123385]: /usr/share/amazon-ec2-net-utils/lib.sh: line 89: imds_endpoint: unbound variable

After 60 attempts it seems to give up and try again a minute later, only to get into the same loop

Could this be related to this issue?

(I have access to more logging information if that may be of help)

[davidreedernst]

Wow, we just had exactly the same problem on a Lightsail server.

Feb 13 09:02:45 ip-172-26-6-189.ec2.internal ec2net[1193902]: [get_meta] Querying IMDS for network/interfaces/macs/12:10:01:9c:46:5d/device-number
Feb 13 09:02:45 ip-172-26-6-189.ec2.internal setup-policy-routes[1194019]: /usr/share/amazon-ec2-net-utils/lib.sh: line 89: imds_endpoint: unbound variable
Feb 13 09:02:45 ip-172-26-6-189.ec2.internal ec2net[1193902]: [get_meta] Querying IMDS for network/interfaces/macs/12:10:01:9c:46:5d/device-number
Feb 13 09:02:45 ip-172-26-6-189.ec2.internal setup-policy-routes[1194026]: /usr/share/amazon-ec2-net-utils/lib.sh: line 89: imds_endpoint: unbound variable

As with @lingtjien , we lost network connectivity, which was restored after rebooting from the Lightsail web panel. We'd love to feel like this wasn't going to happen again....

[dwasyl]

Has there been any progress on this issue? I've been running into exactly the same issues on a Linux 2023 instance. It seems to be load related as I've recently ramped up the server load (although in spikes and not sustained). Just to note, this instance only has a single interface.

[joeysk2012]

Using the AL2023 image on EC2 some of my instances (in multiple regions) lose network connectivity (reboot of the instance fixes it). When I check the log files I see this message being spammed constantly when it's trying to refresh something for the network interface

Feb 12 03:23:57 ip-10-0-0-41.ec2.internal ec2net[4122880]: [get_meta] Querying IMDS for network/interfaces/macs/0a:ff:c5:48:47:e7/device-number
Feb 12 03:23:57 ip-10-0-0-41.ec2.internal setup-policy-routes[4123378]: /usr/share/amazon-ec2-net-utils/lib.sh: line 89: imds_endpoint: unbound variable
Feb 12 03:23:57 ip-10-0-0-41.ec2.internal ec2net[4122880]: [get_meta] Querying IMDS for network/interfaces/macs/0a:ff:c5:48:47:e7/device-number
Feb 12 03:23:57 ip-10-0-0-41.ec2.internal setup-policy-routes[4123385]: /usr/share/amazon-ec2-net-utils/lib.sh: line 89: imds_endpoint: unbound variable

After 60 attempts it seems to give up and try again a minute later, only to get into the same loop

Could this be related to this issue?

(I have access to more logging information if that may be of help)

given line 89: imds_endpoint: unbound variable It seems like in this case the instance is not getting access to IMDS. It is not even getting the IMDS token so this var is not getting set. Can you provide more information @lingtjien ?

Has there been any progress on this issue? I've been running into exactly the same issues on a Linux 2023 instance. It seems to be load related as I've recently ramped up the server load (although in spikes and not sustained). Just to note, this instance only has a single interface.

A high load or spiked load should not cause this issue. Unless this load causes high connection saturation and request to IMDS times out and you are rebooting or reloading your network at the same time. This issue describes an instance where there are multiple interfaces: the primary interface cannot access the IMDS, because it is an on-premise LNI and not ENI with the possibility of accessing the IMDS through a secondary ENI. Can you file a new issue with more details @dwasyl?