macvlan, ipvlan-l2: only configure a default route when a gateway address is supplied #50929
Conversation
robmry
added
area/networking
robmry
force-pushed
the
mac_ip_vlan_gateway_config
branch
2 times, most recently
from
8ccbe38 to
93ae863
Compare
robmry
changed the title
| // be treated as having external connectivity. (This preserves old behavior, | ||
| // where a gateway address was assigned from IPAM that did not necessarily | ||
| // correspond with a working gateway.) | ||
| jinfo.ForceGw4() |
There was a problem hiding this comment.
If we find out that this doesn't work that well for some use cases, we could also consider using netlink.LinkSubscribeAt to watch changes made to the container's routing table and update libnet state accordingly.
There was a problem hiding this comment.
Yes, it could get quite fiddly because it takes some time for the autoconfig to work, so it'd be difficult to do during the join because we wouldn't know how long to wait. We'd end up reconfiguring gateways asynchronously, so debug might get trickier ("something worked for a couple of seconds, then broke").
If a container with a macvlan/ipvlan-l2 interface has gained a default route via autoconfig, moby doesn't know about it. So, if another network is connected and becomes the gateway, that connection fails because a second default route can't be set up. With #50945, it's at-least cleaned up properly - so the connection can be tried again with a lower gateway priority, so that the macvlan/ipvlan endpoint keeps the gateway. Or, sysctls can be used to prevent the autoconfig. But those remedies aren't obvious. And, actually handing over gateway-ness between an autoconfgured endpoint (if someone has a reason to want that) isn't easily do-able.
So there is more we could do here if folks hit issues.
When macvlan is given no '--gateway' option, an address is allocated from IPAM and a default route is set up via that gateway. But, the gateway address is not assigned to anything in the Docker macvlan network - it must be external, and IPAM shouldn't try to guess it. When IPv6 auto-configuration is enabled in the network the macvlan is connected to, the macvlan driver races against it to set up the gateway. When autoconfig wins, container creation fails because the default route already exists. So ... - disable IPAM gateway address allocation for macvlan - update the Join code to allow for no configured gateway - always disable 'docker_gwbridge' connection for macvlan networks, so it's not hooked up when there is no gateway address. Libnet assumes an endpoint with no statically configured default gateway or route does not provide external connectivity. So, it disables external DNS access, and will not select the endpoint as gateway for containers. So, where an IPAM allocated gateway address would have been assigned before, tell libnet to assume there will be an auto-configured gateway. Signed-off-by: Rob Murray <[email protected]>
When ipvlan in "l2" mode is given no '--gateway' option, an address is allocated from IPAM and a default route is set up via that gateway. But, the gateway address is not assigned to anything in the Docker ipvlan network - it must be external, and IPAM shouldn't try to guess it. So ... - always disable IPAM gateway address allocation for ipvlan-l2 - tell libnet to assume the endpoint has a gateway instead - update the Join code to allow for no configured gateway - always disable 'docker_gwbridge' connection for ipvlan networks, so it's not hooked up when there is no gateway address. Signed-off-by: Rob Murray <[email protected]>
robmry
force-pushed
the
mac_ip_vlan_gateway_config
branch
from
93ae863 to
aa78f19
Compare
thaJeztah
left a comment
thaJeztah
left a comment
There was a problem hiding this comment.
LGTM, but letting Cory have a peek
4 participants
- What I did
When no
--gatewayis supplied for a macvlan or "l2" ipvlan, an address is allocated from IPAM and configured as the default route.When IPv6 autoconfiguration is enabled on the parent interface's network, it may also set up a default route. If the kernel wins the race, container creation fails with an error because a default route already exists when Docker tries to set it up.
So, don't try to guess the gateway address by picking one from IPAM (it's not like the bridge driver, where the gateway address is assigned to the bridge - all the drivers do with the address is configure the default route).
Also, when no gateway is configured, don't let the libnet code assume the
docker_gwbridgeis needed.This will be a breaking change if there are users who haven't been configuring a gateway address, where the gateway is using the first host address in the Docker network's ip-range (the address that'll have been picked for IPAM). Those users will now need to explicitly configure the gateway.
- How I did it
- How to verify it
Updated integration tests.
The commands noted in the linked issue no longer fail.
- Human readable description for the release notes