Introduction
If you have seen our article about running kubernetes Ingress-less, you are aware that we are pushing IPv6 only kubernetes clusters at ungleich.
Today, we are looking at making the "internal" kube-dns service world reachable using IPv6 and global DNS servers.
The kubernetes DNS service
If you have a look at your typical k8s cluster, you will notice that you usually have two coredns pods running:
% kubectl -n kube-system get pods -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-558bd4d5db-gz5c7 1/1 Running 0 6d
coredns-558bd4d5db-hrzhz 1/1 Running 0 6d
These pods are usually served by the kube-dns service:
% kubectl -n kube-system get svc -l k8s-app=kube-dns
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 2a0a:e5c0:13:e2::a <none> 53/UDP,53/TCP,9153/TCP 6d1h
As you can see, the kube-dns service is running on a publicly reachable IPv6 address.
IPv6 only DNS
IPv6 only DNS servers have one drawback: they cannot be reached via DNS recursions, if the resolver is IPv4 only.
At ungleich we run a variety of services to make IPv6 only services usable in the real world. In case of DNS, we are using DNS forwarders. They are acting similar to HTTP proxies, but for DNS.
So in our main DNS servers, dns1.ungleich.ch, dns2.ungleich.ch and dns3.ungleich.ch we have added the following configuration:
zone "k8s.place7.ungleich.ch" {
type forward;
forward only;
forwarders { 2a0a:e5c0:13:e2::a; };
};
This tells the DNS servers to forward DNS queries that come in for k8s.place7.ungleich.ch to 2a0a:e5c0:13:e2::a.
Additionally we have added DNS delegation in the place7.ungleich.ch zone:
k8s NS dns1.ungleich.ch.
k8s NS dns2.ungleich.ch.
k8s NS dns3.ungleich.ch.
Using the kubernetes DNS service in the wild
With this configuration, we can now access IPv6 only kubernetes services directly from the Internet. Let's first discover the kube-dns service itself:
% dig kube-dns.kube-system.svc.k8s.place7.ungleich.ch. aaaa
; <<>> DiG 9.16.16 <<>> kube-dns.kube-system.svc.k8s.place7.ungleich.ch. aaaa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23274
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: f61925944f5218c9ac21e43960c64f254792e60f2b10f3f5 (good)
;; QUESTION SECTION:
;kube-dns.kube-system.svc.k8s.place7.ungleich.ch. IN AAAA
;; ANSWER SECTION:
kube-dns.kube-system.svc.k8s.place7.ungleich.ch. 27 IN AAAA 2a0a:e5c0:13:e2::a
;; AUTHORITY SECTION:
k8s.place7.ungleich.ch. 13 IN NS kube-dns.kube-system.svc.k8s.place7.ungleich.ch.
As you can see, the kube-dns service in the kube-system namespace resolves to 2a0a:e5c0:13:e2::a, which is exactly what we have configured.
At the moment, there is also an etherpad test service named "ungleich-etherpad" running:
% kubectl get svc -l app=ungleichetherpad
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ungleich-etherpad ClusterIP 2a0a:e5c0:13:e2::b7db <none> 9001/TCP 3d19h
Let's first verify that it resolves:
% dig +short ungleich-etherpad.default.svc.k8s.place7.ungleich.ch aaaa
2a0a:e5c0:13:e2::b7db
And if that works, well, then we should also be able to access the service itself!
% curl -I http://ungleich-etherpad.default.svc.k8s.place7.ungleich.ch:9001/
HTTP/1.1 200 OK
X-Powered-By: Express
X-UA-Compatible: IE=Edge,chrome=1
Referrer-Policy: same-origin
Content-Type: text/html; charset=utf-8
Content-Length: 6039
ETag: W/"1797-Dq3+mr7XP0PQshikMNRpm5RSkGA"
Set-Cookie: express_sid=s%3AZGKdDe3FN1v5UPcS-7rsZW7CeloPrQ7p.VaL1V0M4780TBm8bT9hPVQMWPX5Lcte%2BzotO9Lsejlk; Path=/; HttpOnly; SameSite=Lax
Date: Sun, 13 Jun 2021 18:36:23 GMT
Connection: keep-alive
Keep-Alive: timeout=5
(attention, this is a test service and might not be running when you read this article at a later time)
IPv6 vs. IPv4
Could we have achived the same with IPv4? The answere here is "maybe": If the kubernetes service is reachable from globally reachable nameservers via IPv4, then the answer is yes. This could be done via public IPv4 addresses in the kubernetes cluster, via tunnels, VPNs, etc.
However, generally speaking, the DNS service of a kubernetes cluster running on RFC1918 IP addresses, is probably not reachable from globally reachable DNS servers by default.
For IPv6 the case is a bit different: we are using globally reachable IPv6 addresses in our k8s clusters, so they can potentially be reachable without the need of any tunnel or whatsoever. Firewalling and network policies can obviously prevent access, but if the IP addresses are properly routed, they will be accessible from the public Internet.
And this makes things much easier for DNS servers, which are also having IPv6 connectivity.
The following pictures shows the practical difference between the two approaches:
Does this make sense?
That clearly depends on your use-case. If you want your service DNS records to be publicly accessible, then the clear answer is yes.
If your cluster services are intended to be internal only (see previous blog post, then exposing the DNS service to the world might not be the best option.
Note on security
CoreDNS inside kubernetes is by default configured to allow resolving for any client that can reach it. Thus if you make your kube-dns service world reachable, you also turn it into an open resolver.
The following coredns configuration does correctly block requests, IF your coredns version is new enough:
Corefile: |
.:53 {
acl k8s.place7.ungleich.ch {
allow net ::/0
}
acl . {
allow net 2a0a:e5c0:13::/48
block
}
forward . /etc/resolv.conf {
max_concurrent 1000
}
...
We tested this with coredns-1.8.4 in which the ACL behaviour is fixed.
More of this
We are discussing kubernetes and IPv6 related topics in the #hacking:ungleich.ch Matrix channel (you can signup here if you don't have an account) and will post more about our k8s journey in this blog. Stay tuned!