记录AWS EC2：Temporary failure in name resolution问题

背景

内核团队在群里@我说我司云上AWS有一台EC2突然出现apt update报错，问是不是source源设置错了： 我这一看就不是啊，明显是DNS出问题了。赶紧上机器排查。

排查相关服务

$ cat /etc/systemd/resolved.conf
[Resolve]
DNS=127.0.0.53 223.5.5.5

没毛病，在查看另一台EC2对比之下没有区别，为什么出现这种情况？ DNS服务器正常

$ ping 127.0.0.53 -c 3
PING 127.0.0.53 (127.0.0.53) 56(84) bytes of data.
64 bytes from 127.0.0.53: icmp_seq=1 ttl=64 time=0.030 ms
64 bytes from 127.0.0.53: icmp_seq=2 ttl=64 time=0.029 ms
64 bytes from 127.0.0.53: icmp_seq=3 ttl=64 time=0.023 ms

--- 127.0.0.53 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2026ms
rtt min/avg/max/mdev = 0.023/0.027/0.030/0.005 ms

DNS服务也正常

$ netstat -tnpl| grep systemd-resol
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      20844/systemd-resol

 

暂时没思路，先恢复业务

添加对应解析尝试

$ vim /etc/hosts

18.166.27.145 cn-north-1.ec2.archive.ubuntu.com
185.125.190.36 security.ubuntu.com

再次尝试，ok

$ apt update
Hit:1 http://cn-north-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://cn-north-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:3 http://cn-north-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease
Get:4 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Fetched 88.7 kB in 2s (49.1 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
54 packages can be upgraded. Run 'apt list --upgradable' to see them.

原因排查

大概过了几分钟发现了问题，只顾着看/etc/systemd/resolved.conf文件是否有误了，/etc/resolv.conf文件不知道被谁删了。。。 Ubuntu在域名解析时，最直接使用的是/etc/resolve.conf文件，它是/run/systemd/resolve/resolve.conf的软链接，在另一台EC2中查看到初始文件内容如下：

$ vim /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0
search cn-north-1.compute.internal

Ubuntu如果在/etc/resolv.conf修改DNS会被覆盖，因为注释中即标明了：This file is managed by man:systemd-resolved(8). Do not edit.，系统在往/run/systemd/resolve/resolv.conf里面写域名解析服务器地址的时候，会从/etc/systemd/resolved.conf中取得DNS相关的配置。需要在/etc/systemd/resolved.conf中修改合适的DNS。

测试

删除前面添加的hosts文件，测试。

$ apt update
Hit:1 http://cn-north-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://cn-north-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:4 http://cn-north-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
54 packages can be upgraded. Run 'apt list --upgradable' to see them.