Kubernetes · Intermediate · 15 min
Fix a Kubernetes Service with no endpoints
Fix a Kubernetes Service with no endpoints: compare the selector with Pod labels, check targetPort, and find Pods that are not ready.
A Service only forwards to ready Pods whose labels match its selector, on its targetPort. This runbook reads the endpoints and EndpointSlices to tell those three causes apart, fixes each in its source, and covers what to check when the endpoints look right. Commands were run on a kind cluster with Kubernetes v1.37.0.
On this page
See it run
A short replay of real output from commands in this guide, captured on a WSL2 Ubuntu test machine (Docker 29.1.3, kind v0.33.0, Kubernetes v1.37.0, single-node kind cluster). The typing is replayed, and the host name, account and MAC addresses are redacted. Your output will differ, and a test machine is not a production server.
Text version of this demo
ops@web-01:~$ kubectl -n demo-svc get endpoints web 2>/dev/null
NAME ENDPOINTS AGE
web <none> 0s
ops@web-01:~$ kubectl -n demo-svc describe service web | grep -E 'Selector|Endpoints'
Selector: app=frontend
Endpoints:
ops@web-01:~$ kubectl -n demo-svc get pods --show-labels
NAME READY STATUS RESTARTS AGE LABELS
web-6bd858c6f9-qqjht 1/1 Running 0 10s app=web,pod-template-hash=6bd858c6f9
ops@web-01:~$ kubectl -n demo-svc patch service web --type=json -p '[{"op":"replace","path":"/spec/selector","value":{"app":"web"}}]' && sleep 2 && kubectl -n demo-svc get endpoints web 2>/dev/null
service/web patched
NAME ENDPOINTS AGE
web 10.244.0.5:80 3s
ops@web-01:~$ kubectl -n demo-svc run curl --image=busybox:1.37 --restart=Never --rm -i --quiet -- wget -qO- -T 5 http://web | grep -o '<title>.*</title>'
<title>Welcome to nginx!</title>A Service is a stable name and IP address in front of a set of Pods. It does not know about Pods directly: its selector picks them by label, only Pods that pass their readiness probe receive traffic, and targetPort says which container port to send it to. When any of those three is wrong, the Service exists and resolves in DNS, but requests are refused or time out. This runbook finds which of the three it is in about a minute.
Commands and output were captured on a single-node kind cluster (kind v0.33.0, Kubernetes v1.37.0, kubectl v1.37.1) with an nginx:1.27-alpine Deployment.
0. Rules of engagement
- Read before you change. Sections 1 to 3 change nothing.
- Fix the Service or the Pod template in its source (manifest, Helm values, Kustomize overlay), not only with
kubectl patch; the next deploy would undo a live patch. - Do not remove a readiness probe to make endpoints appear. It would send traffic to Pods that are not ready.
1. Look at the endpoints
Since Kubernetes 1.33 the Endpoints API is deprecated and kubectl get endpoints prints a warning; it still works. The current source of truth is EndpointSlices, which also show whether each address is ready.
# Endpoints for the Service (deprecated API, still the quickest view)
kubectl -n NAMESPACE get endpoints SERVICE# Real output from the test cluster: no Pod matched the selector
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME ENDPOINTS AGE
web <none> 0s# EndpointSlices with each address and its ready condition
kubectl -n NAMESPACE get endpointslices -l kubernetes.io/service-name=SERVICE \
-o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]} ready={.conditions.ready}{"\n"}{end}'| What you see | Cause | Go to |
|---|---|---|
<none>, no addresses at all | The selector matches no Pod | Section 2 |
Addresses with an unexpected port, for example 10.244.0.5:8080 | targetPort does not match the container port | Section 3 |
Addresses with ready=false | Pods fail their readiness probe | Section 4 |
Correct addresses, ready=true, requests still fail | Not a Service problem: NetworkPolicy, the client, or DNS | Section 5 |
From inside the cluster, each of the first two cases looked the same on the test cluster: the request was refused.
# Real output from a busybox Pod in the cluster, both for a selector mismatch and for a wrong targetPort
wget: can't connect to remote host (10.96.88.7): Connection refused2. The selector matches no Pod
# The Service selector, then the labels the Pods really have
kubectl -n NAMESPACE describe service SERVICE | grep -E 'Selector|TargetPort|Endpoints'
kubectl -n NAMESPACE get pods --show-labels# Real output from the test cluster: the Service wants app=frontend, the Pod is labeled app=web
Selector: app=frontend
TargetPort: 80/TCP
Endpoints:
NAME READY STATUS RESTARTS AGE LABELS
web-6bd858c6f9-6g4vf 1/1 Running 0 11s app=web,pod-template-hash=6bd858c6f9Every key and value in the selector must be on the Pod, letter for letter. Check:
- Typos and case:
app: Webis notapp: web. - Extra selector keys: a selector with
app: web, tier: frontendneeds both labels on the Pod. - The namespace: a Service selects Pods in its own namespace only.
- Helm and Kustomize often add labels such as
app.kubernetes.io/instance; make the Service template use the same helper as the Deployment.
Test a selector directly: kubectl -n NAMESPACE get pods -l app=frontend must list the Pods you expect.
3. targetPort does not match the container
# Real output from the test cluster: the selector is fixed, but targetPort is 8080 and nginx listens on 80
NAME ENDPOINTS AGE
web 10.244.0.5:8080 12sThe endpoint exists, so this looks healthy at a glance, but nothing listens on that port inside the Pod. Compare:
# The port the Service sends to, and the ports the container declares
kubectl -n NAMESPACE get service SERVICE -o jsonpath='{.spec.ports[*].targetPort}{"\n"}'
kubectl -n NAMESPACE get pods -l app=web -o jsonpath='{.items[0].spec.containers[*].ports[*].containerPort}{"\n"}'portis what clients use (http://web:80);targetPortis where the application listens in the container. They often differ (port: 80,targetPort: 8080).- A named
targetPort(for examplehttp) must match aports[].namein the container spec. containerPortis informational: an application that listens on another port than it declares still breaks the Service.kubectl -n NAMESPACE exec POD -- netstat -ltn(orss -ltn) shows what it really listens on, if the image has the tool.
After setting targetPort: 80, the same request returned the nginx welcome page on the test cluster.
4. Pods are not ready
# Real output from the test cluster during a rollout with a readiness probe on a path that does not exist
NAME READY STATUS RESTARTS AGE
web-5f46cfd98f-8gjvv 0/1 Running 0 26s
web-6bd858c6f9-6g4vf 1/1 Running 0 55s
10.244.0.5 ready=true
10.244.0.9 ready=falseA Pod that is Running but 0/1 ready is listed in the EndpointSlice with ready=false and gets no traffic. Here the old Pod is still ready, so the Service keeps working while the rollout is stuck; when every Pod is unready, the Service has no usable endpoint at all.
# Why the probe fails: the events at the end of describe
kubectl -n NAMESPACE describe pod POD | sed -n '/Readiness/p;/Events:/,$p'Fix the probe (path, port, timing) or the application, as described in Troubleshoot Kubernetes probes and resource limits.
5. Endpoints look right, requests still fail
- NetworkPolicy:
kubectl -n NAMESPACE get networkpolicy. A default-deny policy blocks traffic until an allow rule matches the client. - DNS: from a test Pod,
nslookup SERVICE.NAMESPACE.svc.cluster.local. Across namespaces, the short name does not resolve. - Headless Services (
clusterIP: None) have no virtual IP; clients get the Pod addresses directly from DNS. - The client itself may use the wrong port, or
httpsagainst a plain-HTTP application.
6. Verify
# A request from inside the cluster, through the Service name
kubectl -n NAMESPACE run curl --image=busybox:1.37 --restart=Never --rm -i --quiet -- wget -qO- -T 3 http://SERVICE# Real output from the test cluster after the fix (trimmed)
<title>Welcome to nginx!</title>7. Rollback and prevention
- Rollback: reapply the previous Service manifest, or
kubectl rollout undo deployment/NAMEif a Pod template change broke the labels or ports. - Generate labels and selectors from one place (a Helm helper or Kustomize
labels), so the Service and the Deployment cannot drift apart. - Name your ports (
name: http) and use the name astargetPort, so a port change in the container follows through. - Alert on Services with no ready endpoints, for example with kube-state-metrics or a synthetic check through the Service name.
References
Official reference
Technical steps are original explanations; consult the upstream documentation for version-specific details.
Behavior differs by distribution and tool version. Commands are examples for authorized systems; review output for secrets before sharing.
Found this useful?
Follow us on YouTube for video explainers, and find us on Facebook to keep up with what is new.