machine: dns improvements

[Luap99]

When user wants to publish port 53 they run into EADDRINUSE errors in the VM. First we use the default systemd-resolved and second when a network with dns is running aardvark-dns uses port 53 too. The work around generally is to only bind specific ips but this cannot work with machine because a) inside the machine we remove the host ip and always bind everything as the host ip is only used on the mac side to make port forwarding work and b) for some idiotic reason macos only allows binding "priviledged" ports only for 0.0.0.0 not a specific host ip which means gvproxy would fail to bind as we run it rootless.

$ nc -l 0.0.0.0 53
<works>
$ nc -l 127.0.0.1 53    
nc: Permission denied

As such I think we should 1) disable systemd-resolved by default, it doesn't do anything useful for us here. In fact if we count aardvark-dns -> systemd-resolved -> gvproxy dns resolver it means each dns requests currently is read by three resolvers on the local host which adds unnecessary overhead. And there is also the fact that we know systemd-resolved is broken with how it resolves quay.io as we had to disable it on all our CI VMs and as it just kept randomly failing to resolve the name correctly.

And 2) configure aardvark-dns dns_bind_port in containers.conf to something else than 53. Not sure what port to choose any port could cause a conflict theoretically.

Running a dns server in a container seems like a rather common use case we should support out of the box.

see containers/podman#24099 for a user impacted by these problems

cc @mheon @baude

[mheon]

I think 1 is definitely sensible - we shouldn't be exposing users to this. 2 seems like something for a troubleshooting guide instead, IMO - let the user decide what port to use. Maybe we can improve our error messages to mention that port 53 blocks could be Aardvark?

[Luap99]

Maybe we can improve our error messages to mention that port 53 blocks could be Aardvark?

Overall that might help a bit but at the end there is still no nice way to edit files inside the VM so users would need to remember this config changes each time they recreate the VM.
But I guess this isn't necessarily limted to machine VMs, long term it would be nice to have aardvark-dns pick a random free ports and not using 53

[baude]

is the aardvark-dns set by port? if so, we could consider setting a random port as part of init. the downside is that too may be unpredictable for users, but it might stop conflicts.

[Luap99]

Setting a random port can still cause conflicts and in that case would not be predicable for users so I think that would be worse over hard coding a specific one.
In particular we should never choose a port from /proc/sys/net/ipv4/ip_local_port_range range in the config file as the kernel will assign these ports to outgoing connects so it is much more likely to get a conflict in this range.

[gaufde]

mheon:

I think 1 is definitely sensible - we shouldn't be exposing users to this. 2 seems like something for a troubleshooting guide instead, IMO - let the user decide what port to use. Maybe we can improve our error messages to mention that port 53 blocks could be Aardvark?

Luap99:

Overall that might help a bit but at the end there is still no nice way to edit files inside the VM so users would need to remember this config changes each time they recreate the VM.

If dns_bind_port= is not set to something other than port 53 in containers.conf, what would be the recommended way for end users to do this?

Maybe users could supply their own ignition file to podman machine init, but I think that would have to be documented well with some example butane files since it seems like Podman machine needs some custom systemd units per environment and it appears to use it's own ssh key. Otherwise, I guess users could script a sequence of podman machine ssh commands for themselves.

I'm pretty new to Linux in general, but figuring this out is has not been very intuitive and I relied heavily on @Luap99's help. I think both of these approaches are asking the user to learn/understand Podman's inner workings. I'm happy to have this new knowledge, but I think for many people the accessibility of documentation will matter a lot if containers.conf changes need to be done by the end user.

[baude]

Both the suggested 1 and 2 are semi-trivial "fixes" or enhancements. LMK and I can cobble something up