Kubernetes Labels vs Annotations

There is quite a lot of confusion about labels and annotations in Kubernetes. Both are key-value pairs attached to objects, so what exactly is the difference, and why did the creators of Kubernetes introduce both of them? This article explores this question in detail.

It is interesting to note that both labels and annotations were present in Kubernetes right from the start. Looking at the official Kubernetes documentation can leave one perplexed; here is what it says about labels:

Labels are key/value pairs attached to objects such as Pods. They are intended to specify identifying attributes of objects that are meaningful and relevant to users but do not directly imply semantics to the core system. Labels can be used to organize and select subsets of objects. They can be attached to objects at creation time and subsequently added and modified at any time. Each object can have a defined set of key/value labels. Each Key must be unique for a given object.

"metadata": {
  "labels": {
    "key1" : "value1",
    "key2" : "value2"
  }
}

Labels allow for efficient queries and watches and are ideal for use in UIs and CLIs. Non-identifying information should be recorded using annotations.

And here is what it says about annotations:

You can use Kubernetes annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata.

This is all a bit cryptic, but in a nutshell, it means that, broadly speaking, labels are used to identify objects, and annotations are used to attach properties to objects. Of course, there are nuances and overlaps, and we will go over the details in this article.

Summary of Key Differences Between Kubernetes Labels and Annotations

Consideration	Labels	Annotations
Communicate with built-in Kubernetes controllers?	Yes	No
Identify objects and allow filtering and selecting?	Yes	No
Communicate with non-built-in Kubernetes controllers?	Possible, but not the primary purpose, and essentially not used for that purpose in practice	Yes
Provide information to humans and automated processes about Kubernetes objects?	Possible, but not the primary purpose	Yes

Note that the official Kubernetes documentation recommends well-known labels and annotations. The best practice when deploying resources on Kubernetes is to follow these recommendations.

Practical Usage Examples of Kubernetes Labels vs Annotations

To illustrate the various use cases we will explore in this article, we’ll use a minikube cluster to run some commands. To create a local minikube cluster, you just need to install the latest versions of minikube, VirtualBox (or some other minikube backend), and kubectl. Once those tools are installed, run the following to create a minikube cluster:

$ minikube start

Once minikube has started, you can then run the following command to check that everything is running well:

$ kubectl get nodes
NAME       STATUS   ROLES           AGE    VERSION
minikube   Ready    control-plane   2m8s   v1.27.4

Communicating with Built-in Kubernetes Controllers

The primary purpose of labels is to communicate with built-in Kubernetes controllers, which is why they were introduced in the first place. Labels are used to identify resources, especially from built-in Kubernetes controllers, based on the presence and values of labels.

A bit of clarification might be needed here. Built-in controllers are developed and released as part of Kubernetes; they are essential to the basic functions of Kubernetes. Other types of controllers are optional and installed as add-ons to a given Kubernetes cluster.

Let’s have a look at a simple example: a deployment. A deployment is a Kubernetes object that manages a number of identical pods, called replicas, using another Kubernetes object called a ReplicaSet. A Deployment object is managed by the Deployment controller, which creates a ReplicaSet object, which in turn is managed by the ReplicaSet controller. In both cases, both controllers will try their best to ensure that the state of the objects match the expectations (in this case, the number of pods).

Let’s create a simple deployment, like this:

$ kubectl create deployment apache --image=httpd --replicas=2
deployment.apps/apache created
$ kubectl get deployment
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
apache   2/2     2            2           32s

The Kubernetes Deployment controller creates a ReplicaSet:

$ kubectl get replicaset
NAME                DESIRED   CURRENT   READY   AGE
apache-55785dd485   2         2         2       59s

The ReplicaSet controller then creates and manages pods based on their label value. Additionally, the label app=apache was applied automatically upon deployment:

$ kubectl get pod --label-columns=app
NAME                      READY   STATUS    RESTARTS   AGE    APP
apache-55785dd485-7dlxx   1/1     Running   0          114s   apache
apache-55785dd485-dbxgp   1/1     Running   0          114s   apache

The ReplicaSet controller uses the pods’ labels to identify which pods are part of the replica set. For example, let’s change the app label of one of these pods from ‘apache’ to ‘another’:

$ kubectl label --overwrite pod apache-55785dd485-7dlxx app=another
pod/apache-55785dd485-7dlxx labeled
$ kubectl get pod --label-columns=app
NAME                      READY   STATUS    RESTARTS   AGE     APP
apache-55785dd485-7dlxx   1/1     Running   0          2m53s   another
apache-55785dd485-9c7sh   1/1     Running   0          20s     apache
apache-55785dd485-dbxgp   1/1     Running   0          2m53s   apache

The ReplicaSet controller expects to find two pods matching the requirements (in this case, the label app with the corresponding value ’apache’), but after our change, it sees that there is only one pod with the correct label. This causes it to spin a new replica with the app=apache label to maintain the desired state.  This illustrates how Kubernetes built-in controllers use labels to identify resources. However, built-in controllers do not look at annotations at all. In the example above, you could add as many annotations as you want to the pods—nothing would change for the ReplicaSet controller and the pods it controls.

Filtering and Selecting Objects

Another use for labels is closely related to the one we just saw: it instructs the Kubernetes API to filter and select Kubernetes objects based on the presence and value of certain labels.

To continue with the example started in the previous section, we can select the pods with a label app with a value of  another, like so:

$ kubectl get pod -l app=another
NAME                      READY   STATUS    RESTARTS   AGE
apache-55785dd485-7dlxx   1/1     Running   0          3m33s

We can also use labels to perform other operations, such as deleting an unwanted pod:

$ kubectl delete pod -l app=another
pod "apache-55785dd485-7dlxx" deleted

Finally, using labels can be a convenient way to view the combined logs of an application’s replicas without having to reference each pod’s full name in separate commands:

$ kubectl logs -l app=apache
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.0.25. Set the 'ServerName' directive globally to suppress this message
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.0.25. Set the 'ServerName' directive globally to suppress this message
[Sun Jul 28 08:50:52.754519 2024] [mpm_event:notice] [pid 1:tid 1] AH00489: Apache/2.4.62 (Unix) configured -- resuming normal operations
[Sun Jul 28 08:50:52.754608 2024] [core:notice] [pid 1:tid 1] AH00094: Command line: 'httpd -D FOREGROUND'
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.0.23. Set the 'ServerName' directive globally to suppress this message
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.0.23. Set the 'ServerName' directive globally to suppress this message
[Sun Jul 28 08:41:32.372255 2024] [mpm_event:notice] [pid 1:tid 1] AH00489: Apache/2.4.62 (Unix) configured -- resuming normal operations
[Sun Jul 28 08:41:32.372361 2024] [core:notice] [pid 1:tid 1] AH00094: Command line: 'httpd -D FOREGROUND'

This illustrates how we can instruct the Kubernetes API to select and filter based on the value of labels. Note that the presence of labels, regardless of value, can also be used for filtering:

$ kubectl get pod -l app
NAME                      READY   STATUS    RESTARTS   AGE
apache-55785dd485-9c7sh   1/1     Running   0          4m20s
apache-55785dd485-dbxgp   1/1     Running   0          7m13s

Communicating with Non-built-in Controllers

Non-built-in Kubernetes controllers are controllers that do not come as part of Kubernetes and are installed on top of an existing cluster. Ingress controllers are a typical example of non-built-in controllers because Kubernetes does not provide controllers for ingresses by default. Examples of such ingress controllers are the NGINX Ingress Controller and the AWS Load Balancer Controller. 

Non-built-in controllers make use of annotations to pick up the configurations they need to apply to the corresponding Kubernetes objects. In this way, you can see that annotations are carrying configuration information. 

As an example, let’s install the NGINX Ingress Controller and see how annotations are used to communicate with it. The NGINX Ingress Controller can be installed in several ways. Since we are using minikube, the simplest way is just to enable it using the command line (when using other types of Kubernetes clusters, you might need to install it using the official manifest file or Helm chart):

$ minikube addons enable ingress

After a few minutes, minikube should tell you that the NGINX Ingress Controller has been installed successfully. You can verify that it has been installed properly like this:

$ kubectl -n ingress-nginx get pod
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-blt2q        0/1     Completed   0          3m8s
ingress-nginx-admission-patch-qs8x2         0/1     Completed   0          3m8s
ingress-nginx-controller-7799c6795f-gcg52   1/1     Running     0          3m8s

Next, let’s create a Service for our previous Deployment, like this:

$ kubectl expose deployment/apache --port=80
service/apache exposed
$ kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
apache       ClusterIP   10.108.167.187   <none>        80/TCP    3s
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   13m

You can quickly test the apache deployment for connectivity by using a port forward, which tells the service to listen locally on port 8080 for http traffic, and forwards that traffic to port 80 on the pod.

$ kubectl port-forward svc/apache 8080:80
Forwarding from 127.0.0.1:8080 -> 80

Now, in another terminal, use curl to request a response from apache:

$ curl http://localhost:8080
<html><body><h1>It works!</h1></body></html>

Now that we’ve tested connectivity to Apache, let’s create an Ingress to map incoming traffic from outside of the cluster to internal cluster services.  In the following example, we are mapping requests for the root path “/” to port 80 on the pod: :

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: apache
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      return 200 "testing";
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: apache
            port:
              number: 80

Before deploying it, we’ll need to start a minikube tunnel():

minikube tunnel

Note: Run the above command in a separate terminal window as it needs to keep running.

Then, in your original terminal, continue with the following steps:

First, enable snippet annotations in the NGINX controller

kubectl patch configmap nginx-load-balancer-conf -n kube-system --patch '{"data":{"allow-snippet-annotations": "true"}}'

Restart the ingress controller to apply changes

kubectl -n kube-system rollout restart deployment ingress-nginx-controller

Wait for the controller to be ready

kubectl -n kube-system rollout status deployment ingress-nginx-controller

Now apply our ingress resource and verify it is ready

kubectl apply -f ingress.yaml

Now apply our ingress resource

$ kubectl get ingresses
NAME     CLASS   HOSTS   ADDRESS   PORTS   AGE
apache   nginx   *                 80      3s

Test the ingress to see our custom response from the snippet

$ curl http://localhost
testing

Please note the use of annotations in the ingress manifest file. The NGINX Ingress Controller reads these annotations to configure the ingress. Unlike labels, they are not used for identification, and the NGINX Ingress Controller does not use labels to decide how to configure the ingress.

Attaching Information and Metadata to Kubernetes Objects

Labels can provide information about Kubernetes objects, but typically, they do not include anything beyond the metadata used to identify such objects. Additional information must be attached in the form of annotations. In other words, using labels as a source of information is incidental to their primary purpose, which is to allow built-in Kubernetes controllers to identify objects. Let’s have a look at how to do that with Helm (you will need to install it if you have not done so already).

Let’s create a basic chart:

$ helm create tst

Create a values.yaml file with the following content:

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "10254"

Now let’s install this Helm chart and then have a look at the annotations attached to the created pod:

$ helm install tst ./tst -f values.yaml
...
$ kubectl get pod -l app.kubernetes.io/name=tst 
  -o jsonpath='{.items[*].metadata.annotations}'
{"prometheus.io/port":"10254","prometheus.io/scrape":"true"}

As you can see, annotations containing information/metadata about the pod have been attached to it. In this case, an appropriately configured Prometheus server can use these annotations to scrape metrics from this pod on port 10254. This example illustrates how metadata attached to a Kubernetes object can be used by an automated process to perform certain actions.

Similarly, annotations could be added with a human audience in mind. For example, annotations could show the owner of the Kubernetes object or which department to contact for support about this object.

Well-known Labels and Annotations

The official Kubernetes documentation lists many “well-known” labels and annotations. “Well-known,” in this case, means that Kubernetes and many tools across the Kubernetes ecosystem will recognize these labels and annotations as having a certain meaning.

Here are some examples of well-known labels:

• app.kubernetes.io/instance: A unique name to identify Kubernetes objects belonging to an application. The default Helm chart uses the release name to populate this label.
• app.kubernetes.io/name: The name of the application the object belongs to. The default Helm chart uses the chart name to populate this label. The same chart could be deployed multiple times with different release names, so it is possible to have objects with the same app.kubernetes.io/name labels but different values for the app.kubernetes.io/instance labels.
• app.kubernetes.io/managed-by: The name of the tool that manages this object. Once again, the default Helm chart sets this label to “Helm” to indicate that Helm manages this object.
• app.kubernetes.io/version: The version of the deployed application.

Some examples of well-known annotations:

• cluster-autoscaler.kubernetes.io/safe-to-evict: Set to “true” to inform the Cluster Autoscaler that it is safe to terminate this pod if it runs on a node that needs to be terminated, even if other rules would normally prevent this action.
• kubernetes.io/description: A free-form text description of the object.
• kubernetes.io/enforce-mountable-secrets: When set to “true”, Kubernetes will check that data mounted by the ServiceAccount is secret. If not, Kubernetes will refuse to create the object.
• service.kubernetes.io/topology-mode: This annotation informs the service to consider certain elements of the network topology when deciding how to route traffic.

As you can see once again, labels identify objects and are used by built-in controllers, while annotations contain informational data and configuration.

Best practices for using Labels and Annotations

When deciding which labels and annotations to add to a certain Kubernetes object, the list of “well-known” labels and annotations found in the official Kubernetes documentation should definitely be your first port of call.

Regarding labels specifically, you should at least consider the recommended ones. Usually, it is a good idea to keep things simple and purposeful by deliberately selecting which labels to add and not adding too many. In addition, you should anticipate how some labels could be useful in the future, such as a label indicating the version of the software being deployed.

You typically won’t have to guess what annotations to add—this choice will be dictated by which tools you are using. These tools will have documentation telling you which annotations to add and with which values in order to achieve certain goals. If you have some tools or scripts that your organization is developing internally, again, the required annotations will typically be documented by the team that developed the tool. 

It will be a very rare occurrence to be confused as to whether you need to add a label or an annotation. Typically, when using a tool that is not part of Kubernetes, its documentation will tell you what you need to do. And you won’t use annotations for Kubernetes objects controlled by Kubernetes’ built-in controllers.

Conclusion

It is very common to be confused by labels and annotations and to be puzzled at what the difference actually is between the two. The official Kubernetes documentation is quite mysterious when it comes to explaining the difference between labels and annotations.

The essence of this difference is that labels identify objects, while annotations carry non-identifying information, such as configurations. Labels are typically used by Kubernetes built-in controllers, and annotations are used by pretty much everything else. The use of labels by components other than built-in controllers is typically incidental to their primary use, which is to help built-in controllers identify and select Kubernetes objects.

A practical knowledge of the differences between labels and annotations can help engineers adhere to Kubernetes best practices. Using the well-known labels and annotations and applying instructions found in various documentation will eliminate any doubt about whether a certain key/value pair must be added as a label or annotation.