Skip to content

Bug: systemd-networkd does not properly connect to network after suspend #28358

Open
#28361
Open
Bug: systemd-networkd does not properly connect to network after suspend#28358
#28361
Labels
bug 🐛Programming errors, that need preferential fixingnetworkregression ⚠️A bug in something that used to work correctly and broke through some recent commit
Milestone
v262

Description

@Robotic-Brain
Robotic-Brain
opened on Jul 11, 2023

systemd version the issue has been seen with

systemd 253 (253.5-2-arch)

Used distribution

ArchLinux

Linux kernel version used

6.1.38-1-lts

CPU architectures issue was seen on

x86_64

Component

systemd-networkd

Expected behaviour you didn't see

Network interfaces should be configured as described in the .network files on the system and like they are after a fresh system boot.
Specifically:

  • 1 statically configured private IPv4 address
  • 2 dynamically configured IPv6 addresses (via SLAAC): one with the default euid64 token, one with a static token.
  • 1 statically configured ULA-only-IPv6 address

Unexpected behaviour you saw

Randomly missing interface or route configurations.
Seen miss-configurations:

  • (Private) IPv4, ULA based IPv6 and Link Local IPv6 correctly set, GUA based IPv6 addresses missing
  • IPv4 correctly set, no IPv6 connectivity whatsoever
  • No network connectivity at all (this is a headless system without management interfaces, so diagnosing this case is pretty much impossible and requires a hardware reboot)
  • ULA based IPv6 and LLIPv6 correctly set, no GUA based IPv6 and missing gateway for IPv4 and IPv6, so only local connectivity

Steps to reproduce the problem

Setup

The system is a headless, local Home-Server/NAS in a consumer grade dual-stack private network.
The Router advertises the dynamic global IPv6 prefix and a custom static ULA prefix.
DHCP and DHCPv6 are both enabled and used to distribute gateway and DNS settings.
Docker is setup on the host but in the log output below messages related to the many veth interfaces are removed. (tell me if they are relevant)
Wake-On-Lan is enabled in the UEFI.

Configuration files

/etc/systemd/network/50-wired.link

[Match]
MACAddress=<REDACTED MAC>

[Link]
NamePolicy=keep kernel database onboard slot path
AlternativeNamesPolicy=database onboard slot path
MACAddressPolicy=persistent
WakeOnLan=magic

Please note that the Static address configurations are an attempt to work around this bug and have no impact in it at all.

/etc/systemd/network/20-wired.network

[Match]
Name=enp1s0

[Network]
Address=192.168.0.4/24
Gateway=192.168.0.1
DNS=192.168.0.1
DNS=<READACTED IPv6 of DNS>
IPv6PrivacyExtensions=yes
Address=<REDACTED ULA Prefix+STATIC1>/64
Address=<REDACTED ULA Prefix+STATIC_TOKEN>/64
Address=<REDACTED ULA Prefix+EUI64>/64
IPv6AcceptRA=1
DHCP=yes

[IPv6AcceptRA]
Token=eui64
Token=static:::<REDACTED STATIC_TOKEN>
UseRoutePrefix=true
UseGateway=yes

[DHCPv4]
UseDNS=yes
UseNTP=yes
UseGateway=yes

[DHCPv6]
UseDNS=yes
UseNTP=yes

[Route]
Gateway=<READACTED LLIPv6 of Gateway>

How to reproduce

  1. login via ssh
  2. put the system to sleep with:
systemctl suspend && logout
  1. wait some time (usually overnight)
  2. send a magic packet to wakeup the system with:
    wol <MAC_ADDRESS>
  3. wait a few seconds
  4. try to ssh into the system again
  5. If it doesn't work, try some of the other addresses (ULA, GUA, LLv6, v4, v6) - (otherwise the bug did not occur this time)
    8.1 If Unable to connect => use hardware button to reboot - (bug occurred! system works perfectly fine if you could connect to it)
    8.2 If able to connect => verify with ip addr that not all IP-addresses have been configured. ping and DNS queries might fail because of missing routes, "unreachable networks" or unreachable DNS resolvers.
  6. restarting the systemd-networkd service fixes the issue

Additional program output to the terminal or log subsystem illustrating the issue

Jul 09 01:19:23 Mnemosyne systemd[1]: Starting Network Configuration...
Jul 09 01:19:26 Mnemosyne systemd-networkd[1543]: lo: Link UP
Jul 09 01:19:26 Mnemosyne systemd-networkd[1543]: lo: Gained carrier
Jul 09 01:19:26 Mnemosyne systemd-networkd[1543]: Enumeration completed
Jul 09 01:19:26 Mnemosyne systemd[1]: Started Network Configuration.
Jul 09 01:19:26 Mnemosyne systemd-networkd[1543]: enp1s0: Configuring with /etc/systemd/network/20-wired.network.
Jul 09 01:19:27 Mnemosyne systemd-networkd[1543]: enp1s0: Link UP
Jul 09 01:19:30 Mnemosyne systemd-networkd[1543]: enp1s0: Gained carrier
Jul 09 01:19:33 Mnemosyne systemd-networkd[1543]: enp1s0: Gained IPv6LL
Jul 09 01:19:33 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv4 address 192.168.0.4/24, gateway 192.168.0.1 acquired from 192.168.0.1
Jul 09 01:20:35 Mnemosyne systemd-networkd[1543]: docker0: Link UP
Jul 09 02:29:54 Mnemosyne systemd-networkd[1543]: enp1s0: Lost carrier
Jul 09 02:29:54 Mnemosyne systemd-networkd[1543]: enp1s0: DHCP lease lost
Jul 09 02:29:54 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv6 lease lost
Jul 09 02:29:57 Mnemosyne systemd-networkd[1543]: enp1s0: Gained carrier
Jul 09 02:30:00 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv4 address 192.168.0.4/24, gateway 192.168.0.1 acquired from 192.168.0.1
Jul 09 04:00:03 Mnemosyne systemd-networkd[1543]: enp1s0: Configuring with /etc/systemd/network/20-wired.network.
Jul 09 04:00:03 Mnemosyne systemd-networkd[1543]: enp1s0: DHCP lease lost
Jul 09 04:00:03 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv6 lease lost
Jul 09 04:00:03 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv4 address 192.168.0.4/24, gateway 192.168.0.1 acquired from 192.168.0.1
Jul 09 18:08:02 Mnemosyne systemd-networkd[1543]: enp1s0: Lost carrier
Jul 09 18:08:02 Mnemosyne systemd-networkd[1543]: enp1s0: DHCP lease lost
Jul 09 18:08:02 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv6 lease lost
Jul 09 18:08:02 Mnemosyne systemd-networkd[1543]: enp1s0: Gained carrier
Jul 09 18:08:02 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv4 address 192.168.0.4/24, gateway 192.168.0.1 acquired from 192.168.0.1
Jul 09 18:08:02 Mnemosyne systemd-networkd[1543]: enp1s0: Could not set route: Nexthop has invalid gateway. Network is unreachable
Jul 09 18:08:02 Mnemosyne systemd-networkd[1543]: enp1s0: Failed
Jul 09 18:08:02 Mnemosyne systemd-networkd[1543]: lo: Unmanaging interface.
Jul 09 18:08:02 Mnemosyne systemd-networkd[1543]: docker0: Unmanaging interface.
Jul 10 04:00:31 Mnemosyne systemd-networkd[1543]: enp1s0: Configuring with /etc/systemd/network/20-wired.network.
Jul 10 04:00:31 Mnemosyne systemd-networkd[1543]: enp1s0: Failed to send DHCP RELEASE, ignoring: Stale file handle
Jul 10 04:00:31 Mnemosyne systemd-networkd[1543]: enp1s0: DHCP lease lost
Jul 10 04:00:31 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv6 lease lost
Jul 10 04:00:31 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv4 address 192.168.0.4/24, gateway 192.168.0.1 acquired from 192.168.0.1
Jul 10 15:02:14 Mnemosyne systemd-networkd[1543]: enp1s0: Lost carrier
Jul 10 15:02:14 Mnemosyne systemd-networkd[1543]: enp1s0: DHCP lease lost
Jul 10 15:02:14 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv6 lease lost
Jul 10 15:02:14 Mnemosyne systemd-networkd[1543]: enp1s0: Gained carrier
Jul 10 15:02:14 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv4 address 192.168.0.4/24, gateway 192.168.0.1 acquired from 192.168.0.1
Jul 10 15:02:14 Mnemosyne systemd-networkd[1543]: enp1s0: Could not set route: Nexthop has invalid gateway. Network is unreachable
Jul 10 15:02:14 Mnemosyne systemd-networkd[1543]: enp1s0: Failed
Jul 10 15:02:14 Mnemosyne systemd-networkd[1543]: lo: Unmanaging interface.
Jul 10 15:02:14 Mnemosyne systemd-networkd[1543]: docker0: Unmanaging interface.
Jul 10 15:49:58 Mnemosyne systemd-networkd[1543]: enp1s0: Lost carrier
Jul 10 15:50:01 Mnemosyne systemd-networkd[1543]: enp1s0: Gained carrier
Jul 11 03:27:14 Mnemosyne systemd-networkd[1543]: enp1s0: Lost carrier
Jul 11 03:27:14 Mnemosyne systemd-networkd[1543]: enp1s0: Gained carrier
Jul 11 03:27:14 Mnemosyne systemd-networkd[1543]: lo: Unmanaging interface.
Jul 11 03:27:14 Mnemosyne systemd-networkd[1543]: docker0: Unmanaging interface.
Jul 11 04:00:00 Mnemosyne systemd-networkd[1543]: enp1s0: Configuring with /etc/systemd/network/20-wired.network.
Jul 11 04:00:00 Mnemosyne systemd-networkd[1543]: enp1s0: Failed to send DHCP RELEASE, ignoring: Stale file handle
Jul 11 04:00:00 Mnemosyne systemd-networkd[1543]: enp1s0: DHCP lease lost
Jul 11 04:00:00 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv6 lease lost
Jul 11 04:00:00 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv4 address 192.168.0.4/24, gateway 192.168.0.1 acquired from 192.168.0.1
Jul 11 13:26:55 Mnemosyne systemd-networkd[1543]: enp1s0: Lost carrier
Jul 11 13:26:55 Mnemosyne systemd-networkd[1543]: enp1s0: DHCP lease lost
Jul 11 13:26:55 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv6 lease lost
Jul 11 13:26:55 Mnemosyne systemd-networkd[1543]: enp1s0: Gained carrier
Jul 11 13:26:55 Mnemosyne systemd-networkd[1543]: enp1s0: DHCPv4 address 192.168.0.4/24, gateway 192.168.0.1 acquired from 192.168.0.1
Jul 11 13:26:55 Mnemosyne systemd-networkd[1543]: enp1s0: Could not set route: Nexthop has invalid gateway. Network is unreachable
Jul 11 13:26:55 Mnemosyne systemd-networkd[1543]: enp1s0: Failed
Jul 11 13:26:56 Mnemosyne systemd-networkd[1543]: lo: Unmanaging interface.
Jul 11 13:26:56 Mnemosyne systemd-networkd[1543]: docker0: Unmanaging interface.


...log after wake-on-lan wakeup and bug occurence...


Jul 11 21:07:44 Mnemosyne systemd[1]: Stopping Network Configuration...
Jul 11 21:07:44 Mnemosyne systemd[1]: systemd-networkd.service: Deactivated successfully.
Jul 11 21:07:44 Mnemosyne systemd[1]: Stopped Network Configuration.
Jul 11 21:07:44 Mnemosyne systemd[1]: systemd-networkd.service: Consumed 1.099s CPU time.
Jul 11 21:07:44 Mnemosyne systemd[1]: Starting Network Configuration...
Jul 11 21:07:45 Mnemosyne systemd-networkd[3743804]: lo: Link UP
Jul 11 21:07:45 Mnemosyne systemd-networkd[3743804]: lo: Gained carrier
Jul 11 21:07:45 Mnemosyne systemd-networkd[3743804]: enp1s0: Link UP
Jul 11 21:07:45 Mnemosyne systemd-networkd[3743804]: enp1s0: Gained carrier
Jul 11 21:07:45 Mnemosyne systemd-networkd[3743804]: docker0: Link UP
Jul 11 21:07:45 Mnemosyne systemd-networkd[3743804]: enp1s0: Gained IPv6LL
Jul 11 21:07:45 Mnemosyne systemd-networkd[3743804]: Enumeration completed
Jul 11 21:07:45 Mnemosyne systemd[1]: Started Network Configuration.
Jul 11 21:07:45 Mnemosyne systemd-networkd[3743804]: enp1s0: Configuring with /etc/systemd/network/20-wired.network.
Jul 11 21:07:45 Mnemosyne systemd-networkd[3743804]: enp1s0: DHCPv4 address 192.168.0.4/24, gateway 192.168.0.1 acquired from 192.168.0.1

Related issue #24997

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug 🐛Programming errors, that need preferential fixingnetworkregression ⚠️A bug in something that used to work correctly and broke through some recent commit

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions