seeing issue with coredns

[pandeybk]

I created a 5 nodes kubeadm cluster in AWS using kubeadm using following,

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

And apply bridget manifest by simply downloading the configuration,

curl -O https://raw.githubusercontent.com/kvaps/bridget/master/bridget.yaml

Bridget pods are running,

 kubectl -n kube-system get pod -l app=bridget -o wide
NAME            READY   STATUS    RESTARTS   AGE     IP            NODE             NOMINATED NODE   READINESS GATES
bridget-49l64   1/1     Running   0          4m37s   10.20.1.62    ip-10-20-1-62    <none>           <none>
bridget-4prxh   1/1     Running   0          4m37s   10.20.1.172   ip-10-20-1-172   <none>           <none>
bridget-gvz57   1/1     Running   0          4m37s   10.20.1.95    ip-10-20-1-95    <none>           <none>
bridget-qz59n   1/1     Running   0          4m37s   10.20.1.157   ip-10-20-1-157   <none>           <none>
bridget-z5j2d   1/1     Running   0          4m37s   10.20.1.15    ip-10-20-1-15    <none>           <none>

But coredns pods are not passing healthcheck,

coredns-66bff467f8-fmc4f                 0/1     Running   0          5m10s   10.244.1.2    ip-10-20-1-95    <none>           <none>
coredns-66bff467f8-jn4xj                 0/1     Running   0          9m39s   10.244.2.2    ip-10-20-1-172   <none>           <none>

Looks like its having problem while trying to reach k8s endpoint (service object)

kubectl logs coredns-66bff467f8-fmc4f  -n kube-system

Output:

I0625 22:08:45.801746       1 trace.go:116] Trace[60780408]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-06-25 22:08:15.801233016 +0000 UTC m=+310.027901299) (total time: 30.000480721s):
Trace[60780408]: [30.000480721s] [30.000480721s] END
E0625 22:08:45.801770       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0625 22:08:45.802757       1 trace.go:116] Trace[340007387]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-06-25 22:08:15.802351838 +0000 UTC m=+310.029020177) (total time: 30.000388737s):
Trace[340007387]: [30.000388737s] [30.000388737s] END
E0625 22:08:45.802772       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0625 22:08:45.803735       1 trace.go:116] Trace[1304066831]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-06-25 22:08:15.803378498 +0000 UTC m=+310.030046784) (total time: 30.000337496s):
Trace[1304066831]: [30.000337496s] [30.000337496s] END
E0625 22:08:45.803753       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

kubectl version

kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-17T11:41:22Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-17T11:33:59Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• ifconfig on one of the node

ifconfig

       cbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.244.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether d6:60:82:58:3b:0c  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 270 (270.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

describe coredns pod

kubectl describe po coredns-66bff467f8-fmc4f  -n kube-system 
Name:                 coredns-66bff467f8-fmc4f
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 ip-10-20-1-95/10.20.1.95
Start Time:           Thu, 25 Jun 2020 22:03:02 +0000
Labels:               k8s-app=kube-dns
                      pod-template-hash=66bff467f8
Annotations:          <none>
Status:               Running
IP:                   10.244.1.2
IPs:
  IP:           10.244.1.2
Controlled By:  ReplicaSet/coredns-66bff467f8
Containers:
  coredns:
    Container ID:  docker://9b763857bff24d1820d23d4e31c42944d93ab4059c647ef4ea4d02a767a586f3
    Image:         k8s.gcr.io/coredns:1.6.7
    Image ID:      docker-pullable://k8s.gcr.io/coredns@sha256:2c8d61c46f484d881db43b34d13ca47a269336e576c81cf007ca740fa9ec0800
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Thu, 25 Jun 2020 22:03:05 +0000
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-88rpd (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-88rpd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-88rpd
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                     From                    Message
  ----     ------     ----                    ----                    -------
  Normal   Scheduled  8m35s                   default-scheduler       Successfully assigned kube-system/coredns-66bff467f8-fmc4f to ip-10-20-1-95
  Normal   Pulling    8m34s                   kubelet, ip-10-20-1-95  Pulling image "k8s.gcr.io/coredns:1.6.7"
  Normal   Pulled     8m32s                   kubelet, ip-10-20-1-95  Successfully pulled image "k8s.gcr.io/coredns:1.6.7"
  Normal   Created    8m32s                   kubelet, ip-10-20-1-95  Created container coredns
  Normal   Started    8m32s                   kubelet, ip-10-20-1-95  Started container coredns
  Warning  Unhealthy  3m28s (x31 over 8m28s)  kubelet, ip-10-20-1-95  Readiness probe failed: HTTP probe failed with statuscode: 503