Hi! Any update on this issue? any new workaround?

Hello! Same problem here. It would be nice to have a workaround on this asap.

As documented here, the workaround across our server fleet is a small change to the file /etc/nsswitch.conf. Locate this line:

hosts: files resolve [!UNAVAIL=return] dns

...and remove the "resolve" service. After the change, the line reads:

hosts: files dns

Note that this is not the final solution but only a workaround.

We are experiencing this since upgrading to Debian 12 AMIs

We're affected by this, using Debian 12 AWS EC2 AMIs. As well as agreeing that it is somehow IPv6 related, it also seems to only cause resolution failures for CNAMEs. At least in my testing with A and CNAME lookups, only CNAMEs exhibit the intermittent failures.

Our workaround has been to simply purge the systemd-resolved package entirely, which has the effect of sorting out nsswitch on its way out of the door. <edit> DO NOT USE THIS WORKAROUND, IT BREAKS DNS NAME RESOLUTION AFTER A REBOOT, SORRY 🤡

apt purge --auto-remove systemd-resolved

Thanks to @schams-net for the detailed repro details. I shamelessly grabbed the bash fragment [1] to easily reproduce the issue. Using the script to resolve a CNAME produced intermittent failures. Pointing it at an A record gave no failures. Telling netcat to use IPv4 (nc -4) also meant no failures, for A or CNAME lookups.

[1] https://github.com/schams-net/aws-ec2-rds-dns-lookup-issue/blob/main/docs/how-to-reproduce-the-issue.md

@ahmgithubahm You don't need to remove all of systemd-resolved. Just removing libnss-resolve will accomplish what you need without disabling resolv.conf management altogether. Debian 12.6 images, when they're published, will make this the default.

Thanks @nmeyerhans, that's good to know. I've edited my post now I realise that my workaround leaves you with no working /etc/resolv.conf after a reboot. Oops. 🤡

@schams-net @ahmgithubahm @jackarias @levinse @wjsc Be aware that the Debian 12 cloud images just released (version 20240415-1718) no longer enable libnss-resolve by default. So this issue should be mitigated in instances launched from these and future images.

If somebody who has encountered this issue on Debian 12 would be able to test the Debian sid images, that would help the systemd maintainers understand if the issue is still present.

Thanks @nmeyerhans! Is the problem somehow specific to AWS? I can't reproduce it from a local VM with libnss-resolve being enabled.

I don't see any reason to think it'd be specific to AWS, but it's possible. As the problem is intermittent, it could be that something about the AWS DNS infrastructure triggers it more frequently, but that's just speculation.

I can confirm that I'm unable reproduce the issue with the latest AMIs at AWS anymore 🥳
I tested debian-12-amd64-20240429-1732 (for example AMI ID ami-02175b0058d3ce245 in us-east-1). The file /etc/nsswitch.conf contains the following configuration in the old vs. new image.

Previously (with "resolve"):

hosts: files resolve [!UNAVAIL=return] dns

New(er) images (without "resolve"):

hosts: files dns myhostname

(As @nmeyerhans pointed out).

I haven't had the chance to test the Debian sid images yet.