that's a... surprisingly obvious and a good idea, thanks.

a sudo netstat -lnp | grep 53 reveals:

tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      1763/systemd-resolv
udp        0      0 127.0.0.53:53           0.0.0.0:*                           1763/systemd-resolv

which is... part of ubuntu.

This page https://manpages.ubuntu.com/manpages/focal/man8/systemd-resolved.service.8.html details the service.

Doing a systemctl stop systemd-resolved.service and then re-deploying with the values above allows the container to start.

Thanks for your help!

Hey,

No problem, but if you are using network manager you may need to check the config for it according to the link below NM will try to reactivate it even if the service is disabled.
Here's a link to that. The solution should be what are you looking for. https://askubuntu.com/questions/907246/how-to-disable-systemd-resolved-in-ubuntu

nice one :) i just came from reading that page.

Just a warning for anyone in the future reading this:

I'm using Ubuntu 20.04 LTS and have microk8s installed for my kubernetes cluster. When I disabled systemd-resolved, it was fine until the system restarted - at which point, none of the pods on the node started properly. I re-enabled systemd-resolved and it worked. Obviously microk8s on ubuntu requires systemd-resolved to work somehow.

I haven't fixed this problem. I believe the only way forward is to use Multus (or similar) and configure a macvlan network for the pihole pod, and expose DHCP on that interface instead - but I don't know how to do that.

Instead, I've just used my wifi router's DHCP server instead of my broadband router - there, at least, I can specify DNS servers myself.

So I can't really advise much on the microk8s, but I myself am using k3s. k3s uses klipper-lb https://github.com/k3s-io/klipper-lb which This works by using a host port for each service load balancer and setting up iptables to forward the request to the cluster IP

So the service for Pihole (tcp and udp port 53) get a nodeport assigned. And then klipper will just create an iptables dnat rule for it.

+ iptables -t nat -I PREROUTING '!' -s 10.43.127.10/32 -p TCP --dport 53 -j DNAT --to 10.43.127.10:53
+ iptables -t nat -I POSTROUTING -d 10.43.127.10/32 -p TCP -j MASQUERADE

So then you can keep your systemd-resolved service up and running, but once Pihole is up and running, then that should be taking the requests as iptables is sending the packets to the nodeport before Mr. Systemd-resolved.

Also k8s name resolution usually works by sending any DNS requests outside of the kubernetes cluster (*.cluster.local) to the local system resolv.conf or nameserver. Usually or historically its been popular to use SkyDNS before CoreDNS, but essentially same problem solved. And there was quite a bit of DNSmasq magic to forward requests depending on the hostname to the correct dns server. (thats why the microk8s requires host dns if it was trying to fetch resources outside your k8s cluster)