Advanced Pod Scheduling

Kubernetes Advanced Pod Scheduling Techniques

In Kubernetes, scheduling is the process of assigning pods to nodes. By default, the scheduler chooses a node based on resource availability and requirements. But in real world environments, teams often need more control, whether to isolate workloads, spread them across zones, or co locate specific types of pods.

Before we dive into advanced pod scheduling in Kubernetes, let’s quickly set the context. Each one addresses a different type of scheduling need.

Taints and Tolerations help you prevent certain pods from running on a node unless they are explicitly allowed. Think of it as a node saying "not for everyone" and only specific pods are tolerated.

NodeSelector is a straightforward way for a pod to say "run me only on nodes with this label." It is simple matching based on exact key value labels.

Node Affinity builds on NodeSelector but allows for more flexible rules. You can specify preferred or required conditions and even match on a range of values or expressions.

Pod Affinity and Anti-Affinity are used when a pod needs to be scheduled based on the presence or absence of other pods. You can choose to place pods together or ensure they run on different nodes for availability or performance reasons.

1. Taints and Tolerations

Taints are applied to nodes using kubectl taint and tell the scheduler not to place pods on them unless those pods explicitly tolerate the taint.

Example: Applying a taint to a node

kubectl taint nodes node2 app=blue:NoSchedule

This means node2 will repel all pods unless they tolerate the app=blue taint. The effect NoSchedule ensures that pods without a matching toleration will not be scheduled on this node at all.

Taints are a node level repelling mechanism. If applied incorrectly, they can result in unschedulable pods.

The NoExecute taint needs special attention, as it evicts running pods.

Common Taint Effects

NoSchedule: Pod will not be scheduled unless it tolerates the taint.

PreferNoSchedule: Tries to avoid placing a pod, but not strictly enforced.

NoExecute: Existing pods without a toleration are evicted from the node.

Tolerations are added to pods to let them run on nodes with matching taints. They don’t guarantee placement, but they remove the restriction caused by the taint.

tolerations:

- key: "app"

value: "blue"

effect: "NoSchedule"

This tells the scheduler that the pod can tolerate the app=blue:NoSchedule taint.

As you can observe in the above illustration, Node 2 is tainted with app=blue. Only the blue-labeled pods include a matching toleration.

So, the scheduler places those pods on Node 2. All other pods, which do not tolerate the taint, are placed on Node 1.

Having established how taints and tolerations work together to control pod placement on restricted nodes, let’s now dive into the next set of techniques that guide pod scheduling based on node labels and pod relationships.

2. NodeSelector

NodeSelector is the simplest way to constrain a pod to run only on specific nodes. It works by matching a pod’s nodeSelector field with a node’s label. If no match is found, the pod will remain unscheduled.

This method is hard matching only. There’s no fallback or preference logic. It is either a match or not. Assume Node 1 is labeled as:

kubectl label nodes node1 app=green

The pod manifest would include:

spec:

nodeSelector:

app: green

This means the pod will only be scheduled on nodes where app=green label is present. If no such node exists or is available, the pod will stay in Pending state.

The pod with nodeSelector: app=green can only be scheduled onto Node 1, which has the matching label.

Node 2, labeled app=blue, is skipped by the scheduler, even if it has capacity. This is why the scheduler in the visual below selects only nodes that carry the exact label.

3. Node Affinity

Node Affinity works like NodeSelector but offers more flexibility and control.

While NodeSelector only allows exact match on single labels, Node Affinity supports:

Multiple match expressions (In, NotIn, Exists, etc.)

Required (hard rules) and preferred (soft preferences) conditions

Topology awareness using topologyKey (e.g. zones or regions)

Example difference

NodeSelector:

nodeSelector:

app: green

Node Affinity:

affinity:

nodeAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

nodeSelectorTerms:

- matchExpressions:

- key: app

operator: In

values:

- green

- blue

This allows the pod to be scheduled on nodes labeled app=green or app=blue, giving more options to the scheduler.

4. Pod Affinity and Anti-Affinity

Unlike node level techniques, Pod Affinity and Anti-Affinity schedule pods based on the presence or absence of other pods. They are useful for grouping or separating pods based on application logic, performance needs, or fault tolerance.

Pod Affinity

To place a pod on the same node (or group of nodes) where app=green is already running:

affinity:

podAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

- labelSelector:

matchLabels:

app: green

topologyKey: "topology.techopsexamples.com/rack"

Pod Anti-Affinity

To ensure a pod is not placed on nodes where app=blue pods already exist:

affinity:

podAntiAffinity:

requiredDuringSchedulingIgnoredDuringExecution:

- labelSelector:

matchLabels:

app: blue

topologyKey: "topology.techopsexamples.com/rack"

I hope this was an insightful edition. See you in the next edition on Saturday.

Take care and have a nice day.