Mastering Kubernetes Service: Types, Use Cases & Best Practices

Table of Contents

Kubernetes Services: Stop Treating Your Network Like a Magic Black Box

It was 3:14 AM on a Tuesday in 2019. I was staring at a Grafana dashboard that looked like a heart attack. Our checkout service latency had spiked from 40ms to 12 seconds. We weren’t under a DDoS attack. We hadn’t even deployed new code. I eventually found the culprit: a junior dev had created a kubernetes service with a selector that accidentally matched 4,000 unrelated batch-processing pods. The kube-proxy on every node in the cluster was trying to sync an iptables rule-set the size of a Tolstoy novel. The kernel spent more time traversing chains than routing packets. The nodes locked up. The cluster died.

I’ve spent a decade in the trenches of site reliability. If there is one thing I’ve learned, it’s that the kubernetes service is the most misunderstood abstraction in the entire stack. People think it’s a load balancer. It isn’t. People think it’s a stable IP address. It’s actually a collection of lies told by the Linux kernel. If you don’t understand how those lies are constructed, you’re just waiting for your own 3:00 AM wake-up call.

The Documentation Is Lying To You

The official Kubernetes documentation is great for passing a CKA exam, but it’s terrible for running a production environment. It treats a Service as a clean, logical object. In reality, a Service is a distributed configuration state that relies on three or four different subsystems working in perfect harmony. When one fails, the error messages are cryptic or, worse, non-existent.

Most tutorials tell you to just “apply the YAML” and move on. They don’t mention that kube-proxy might be running in iptables mode, which is essentially a giant list of if/else statements for your network packets. They don’t tell you that CoreDNS is the fragile glue holding your service discovery together. They certainly don’t tell you that ExternalTrafficPolicy: Cluster will butcher your source IPs and add unnecessary hops across your VPC.

Pro-tip: Stop using iptables mode for kube-proxy if your cluster has more than 500 services. Switch to IPVS. It uses a hash table instead of a linear list, turning an O(n) lookup into an O(1) lookup. Your CPU will thank you.

ClusterIP: The Virtual IP That Doesn’t Exist

The ClusterIP is the default service type. It’s an internal-only IP. But here’s the kicker: if you try to ping a ClusterIP, it often won’t respond. Why? Because there is no network interface with that IP address. It’s a ghost.

When a pod tries to talk to 10.96.14.55 (a ClusterIP), the packet leaves the pod and hits the node’s network namespace. The kernel looks at the packet, sees the destination, and checks its iptables rules. It says, “Oh, you’re looking for 10.96.14.55? I’m going to pretend that IP is actually 192.168.1.12 (a real pod IP) and send you there instead.” This is Destination Network Address Translation (DNAT).


# This is what the lie looks like inside your node
iptables -t nat -L KUBE-SERVICES | grep my-api-service
# Output:
# KUBE-SVC-XXXX  tcp  --  anywhere  10.96.14.55  /* default/my-api-service:http */ tcp dpt:http

If you have three replicas of your pod, kube-proxy sets up a probability-based rule. The first rule has a 33% chance to jump to Pod A. If it doesn’t jump, the next rule has a 50% chance to jump to Pod B. If that fails, it defaults to Pod C. This isn’t “intelligent” load balancing. It’s just math. It doesn’t care if Pod A is currently OOM-killed or if Pod B is struggling with a 100% CPU load. If the readiness probe hasn’t failed yet, the traffic goes there to die.

The NodePort Security Hole

I hate NodePort. It’s the “I give up” of Kubernetes networking. It opens a port (usually in the 30000-32767 range) on every single node in your cluster. Every. Single. One.

It bypasses your ingress controllers, making it harder to manage TLS and headers.
It exposes your internal node IPs to whatever is calling the service.
It creates a massive security surface area. If you have 100 nodes, you now have 100 entry points for that specific service.
It’s a nightmare for firewall management in locked-down environments like api.stripe.com or banking backends.

If you find yourself using NodePort because “it’s easier,” you’re doing it wrong. Use an Ingress or a LoadBalancer type. The only valid reason for a NodePort in 2024 is if you are building your own Load Balancer controller or dealing with some truly cursed legacy UDP protocol that Ingress controllers can’t handle.

LoadBalancer: The Cloud Tax

When you set type: LoadBalancer, Kubernetes talks to the cloud provider’s API (AWS, GCP, Azure) and asks for a real, external load balancer. This is where things get expensive and slow.

On AWS, this usually spins up a Classic Load Balancer (CLB) or a Network Load Balancer (NLB). The problem is the default behavior: externalTrafficPolicy: Cluster. When traffic hits the NLB, it picks a random node. If that node doesn’t have a pod for your service, it forwards the traffic to a node that *does*. This adds a hop. It also obscures the client’s source IP. Your logs will show the internal IP of the node, not the IP of the customer. This makes rate-limiting and fraud detection impossible.


apiVersion: v1
kind: Service
metadata:
  name: checkout-api
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local # This is the magic sauce
  selector:
    app: checkout
  ports:
    - port: 443
      targetPort: 8443

Setting externalTrafficPolicy: Local forces the load balancer to only send traffic to nodes that are actually running the pod. If a node doesn’t have the pod, the health check fails, and the LB ignores that node. You preserve the source IP and you save a network hop. Why isn’t this the default? Because it can lead to uneven load distribution if your pods aren’t spread perfectly across nodes. Trade-offs everywhere.

Headless Services: For When You Actually Care

Sometimes you don’t want a virtual IP. You want the real IPs of all the pods. This is common for databases like PostgreSQL or Cassandra, or service discovery for Akka clusters. You do this by setting clusterIP: None.

When you query the DNS for a headless service, instead of getting one ClusterIP, you get an A record for every pod IP currently matching the selector. It’s raw. It’s honest. It puts the burden of load balancing on the client, which is exactly where it should be for high-performance stateful systems.


apiVersion: v1
kind: Service
metadata:
  name: redis-cluster
spec:
  clusterIP: None # Headless!
  selector:
    app: redis
  ports:
    - port: 6379

Note to self: When using headless services with StatefulSets, ensure your serviceName in the StatefulSet spec matches the Service name. If you mess this up, your pods won’t get their predictable network identities (e.g., redis-0.redis-cluster.default.svc.cluster.local).

The DNS Trap: ndots:5

This is the “expert level” gotcha that kills performance in large-scale Kubernetes clusters. By default, the /etc/resolv.conf in a Kubernetes pod has ndots:5. This means if you try to resolve a name like api.stripe.com, the DNS resolver will try to look it up inside the cluster first.

It will try:

api.stripe.com.production.svc.cluster.local (Fail)
api.stripe.com.svc.cluster.local (Fail)
api.stripe.com.cluster.local (Fail)
api.stripe.com.us-east-1.compute.internal (Fail)
api.stripe.com (Success!)

That is five DNS queries for every single connection. If your app isn’t using persistent connections, you are hammering CoreDNS for no reason. I’ve seen CoreDNS pods OOM-killed because of this exact behavior during a traffic surge. The fix? Use fully qualified domain names (FQDNs) by adding a dot at the end: api.stripe.com. or tune your dnsConfig.

The Endpoints vs. EndpointSlice Evolution

In the old days (pre-v1.16), Kubernetes used an Endpoints object. It was a single object containing every IP and port for a service. If you had a service with 5,000 pods, and one pod restarted, the entire 5,000-item list had to be sent to every single node in the cluster. It was a massive amount of data being pushed over the wire for a tiny change.

EndpointSlices fixed this by breaking the list into smaller chunks. It’s more scalable, but it adds another layer of abstraction to debug. If a pod isn’t receiving traffic, don’t just check kubectl get svc. Check kubectl get endpointslices. If the pod IP isn’t there, your readiness probe is likely failing, or your labels are mismatched. Don’t guess. Verify the slice.


# Debugging the actual endpoints
kubectl get endpointslices -l kubernetes.io/service-name=my-api-service
# Check if "Ready" is true for your pod IP

The “Hairpin NAT” Problem

Here is a fun one. Imagine Pod A is trying to talk to Service X. Service X is backed by Pod A. The packet goes out to the Service IP, gets NAT’d by the node, and tries to come back to the same pod. In many network configurations, the kernel sees a packet coming from itself and going to itself but with different IPs, and it just drops it. This is “Hairpin NAT.”

Most CNI plugins (like Calico or Cilium) handle this now, but if you’re rolling your own networking on bare metal or using an outdated version of Flannel, you’ll see intermittent connection resets that only happen when a pod tries to “talk to itself” via a Service. It’s a nightmare to debug because it only happens to a fraction of your traffic.

Why I Prefer Debian-Slim over Alpine for Networking

This is a hill I will die on. Everyone loves Alpine because it’s 5MB. But Alpine uses musl instead of glibc. When it comes to networking and DNS resolution, musl handles things differently—specifically, it doesn’t support search domains in the same way glibc does, and it handles concurrent DNS requests over the same socket in a way that can lead to race conditions (the “5-second DNS delay” bug).

I use debian-slim. It’s 30MB bigger, but it uses glibc. I haven’t had a mysterious 5-second DNS timeout since I made the switch. In a world where we spend thousands on cloud bills, 30MB of storage is a rounding error. Your sanity is not.

Troubleshooting the Service Chain

When a service isn’t working, don’t start by deleting the service. Follow the path of the packet:

DNS: Can the pod resolve the service name? nslookup my-svc. If not, check CoreDNS logs.
Endpoints: Does kubectl get endpoints show the pod IPs? If not, check your selector labels and readinessProbes.
Kube-Proxy: Is kube-proxy running on the node? Check kubectl logs -n kube-system -l k8s-app=kube-proxy.
Iptables/IPVS: Are the rules actually there? iptables-save | grep MY-SVC-NAME.
The Pod: Is the application actually listening on 0.0.0.0? If it’s listening on localhost, the Service will never reach it.
MTU Mismatch: If small packets (like headers) get through but large packets (like JSON payloads) hang, you have an MTU mismatch. Your overlay network (VXLAN) adds overhead, and if your physical network can’t handle the jumbo frames, packets get dropped silently.

I once spent two days debugging a “Service failure” that turned out to be an MTU mismatch on a GRE tunnel. The curl -I (headers only) worked perfectly. The curl (full body) timed out. I felt like an idiot, but I never forgot to check the MTU again.

The Future: Service Mesh and eBPF

We are moving away from iptables. Tools like Cilium are replacing kube-proxy with eBPF. Instead of the kernel traversing a list of rules, eBPF allows us to hook into the socket layer and route the packet directly to the destination pod. It’s faster, more observable, and avoids the “iptables rule storm” I mentioned at the start.

But even with a Service Mesh like Istio or Linkerd, the underlying kubernetes service object remains the source of truth for identity. You can’t escape it. You can only layer more complexity on top of it. If you don’t understand the base layer, the mesh will just give you more sophisticated ways to fail.

Final Advice

Treat your Kubernetes Services like code, not like infrastructure. Version them. Monitor their endpoint counts. And for the love of everything holy, stop using NodePort in production. The network isn’t magic; it’s just a series of very fast, very specific lies. Learn the lies, and you’ll survive the on-call shift.

Stop over-engineering your selectors and start monitoring your conntrack tables. If you don’t know how full your conntrack table is right now, you don’t actually know if your cluster is healthy.

Explore more insights and best practices: