I was first introduced to Kubernetes by a coworker about seven years ago when it was still a new thing. Since then, every company I have gone to has had the same difficulties onboarding their teams onto Kubernetes. Since I've been asked the same several questions, I decided to write a series of blog post answering them, which one could go through instead of attending the seminar that I often perform for engineers. Some things are going to be a bit oversimplified, but bear with me as we go through these step-by-step. This is by no means documentation for Kubernetes. My only aim is to give you enough general information about the platform to get you started on your journey.
First of all, I assume that you already have some idea of what Kubernetes is and what it's used for. Or at the very least, you understand the idea behind container orchestration and Docker etc.
Quick note before we start: Kubernetes is a platform software that allows you to deploy and manage containered software while giving you APIs and tools to run, scale, monitor, modify, and build access policies around these containered applications.
Concepts & Terminology of Basic Components
Okay, so this is a bit oversimplified, but there are a few important concepts that you need to understand for us to understand Kubernetes.
ETCD: etcd is a distributed key-value database that Kubernetes uses to store the configuration needed to run your containers.
API Server: Kubernetes has a JSON over HTTP server that is detailed in the OpenAPI format. You can learn more about the specific endpoints and structures in their detailed documentation. Kubernetes works by taking any configuration that it needs from you regarding your applications and storing them in ETCD. By interacting with this server, you are essentially modifying the objects in the distributed ETCD database. There is a bit more to it, but we'll get to it later.
Node: A Kubernetes node is essentially just a computer that is connected to the same API server/etcd that is capable of running a container instance.
Scheduler: As the name suggests, this is the component that decides to launch a given container on a given node based on various parameters, some of which can be tuned.
Controller: A Kubernetes controller is a runnable service that listens for changes on the objects stored in the etcd, and performs an action such as creating or modifying other objects, making API calls, or pretty much anything you might want to do. One such controller is the replication controller, which, based on the number of instance you want of a service, will create that many replicas that a scheduler will assign to run on a node. It will create or remove instances if you modify the value of the instance count for your application.
So how does it work?
The very short answer is this: one can call the Kubernetes API server with an object that defines certain configuration. This object is then stored in etcd which is being observed by a controller, which may or may not do something based on the kind and contents of the object. If it's a runnable container (or a Pod—we'll describe what that is later), the scheduler will assign it to a node, which will run it.
Types of Objects
There are several types of objects that you can submit to the API server. Some of the common ones are Pods, Deployments, Services, Configuration Maps, Secrets, Ingresses, etc. There are more, but let's stick to these for now. We can now spend some time discussing each one of these.
Pod: A pod is the smallest runnable component in Kubernetes. A pod is essentially a set of containers that run in the same network space. For example, one pod may include a container that runs an HTTP API and another container that contains a reverse proxy that authenticates requests against this HTTP API. Kubernetes will run these containers in the same network space so that the proxy can access the service at "localhost." However, since this requires the two containers to share the same virtual IP address, the two services cannot use the same port. A pod object contains the information about the containers, as well as information about their environment variables, volume mounts (topic for later), and secret or shared configuration injection (another topic for later).
Deployment: While a pod defines the configuration for one or more containers, a deployment defines what we call a pod template. This means that instead of creating an individual pod, we describe what a pod can look like. This allows us to specify, for example, the number of pods we would like to run, and a deployment controller will create or remove a number of pods based on this information. Actually, a deployment creates a "Replica Set," which creates pods, but let's talk about it later.
Service: Many other container orchestration systems use the term "service" to mean a running container (such as a running service on an operating system), however that is not the case for Kubernetes. When we run a pod, we assign it a virtual IP address. However, since we may be running multiple instances of a pod, we don't want other applications to call them using their IP address since there can be multiple of those. A service is an object where we define labels, which when they match assign the virtual IPs of the pod to a name resolution service. Think of this like creating several DNS entries for a website that runs on several servers with various IPs. A caller can then access an instance of your application that matches the label by calling the name of the service.
Configuration Maps: A configuration map is just that—a map of key-value pairs containing shared configuration. Often we have several applications that share configuration (for example, a pub-sub service might have the name of a Kafka stream as a shared configuration). When creating a Pod (or a Deployment), one could then refer to these maps and mount them either as files, or as environment variables in these pods. Note however that if you mount these maps as files, they will be read-only files, since pods don't own configuration maps.
Secrets: Secrets are very similar to configuration maps in the sense that they are also key-value pairs containing information that can be mounted as files or environment variables. The only difference is that a Kubernetes distribution will encrypt the contents inside these and ensure that not everybody can see their content.
Ingress: To understand an ingress, we have to go back to a service. While a service will create a nameserver entry for a set of pods, they can only be accessed internal to a Kubernetes cluster (there are multiple types of services, but we'll get to that in a bit). An ingress specifies to a server (i.e., an ingress controller) which hostname and [HTTP] path should be routed to which service. For example,
www.example.com could point to a set of pods running Nginx that hosts static files for the website, while
www.example.com/api could point to a dynamic HTTP server providing a REST API for the single-page-application running on the website. Nginx, in fact, is one of the supported reverse proxies that can be used as an Ingress controller. Another popular ingress controller is the Traefik reverse proxy.
Anatomy Of An Object
So far, we have been discussing what objects are available in the API server. Let's now talk about what an object looks like. Let's start with the Deployment object as an example.
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80
There are four things that I would like you to notice in the above example—the API version, the kind of the object, the metadata, and the specification of the object.
API Version: Kubernetes supports several APIs that are maintained separately in the code base. With every new version of Kubernetes, we have new APIs introduced or the older ones removed. However, generally speaking, we also maintain backwards compatibility with some of the older APIs, giving engineers enough time to migrate to the newer ones across versions.
Kind: Kind is exactly what is sounds like—it defines the type definition for the object specification for which will follow.
Metadata: Metadata field looks similar for all fields, it provides very basic information regarding the object like the name, some labels with which one could later look these up, etc. As we progress through the series, we can go in details regarding what each of these metadata fields mean.
spec is the main content of the object. The kind field above specifies the data-type for the contents that in the specification field. This is basically what tells a controller what to do with an object. For example, our deployment object defines the replica count of the pods, as well as a template for what individual instances of these pods will contain.
What about the status?
The keen reader will notice that we have not mentioned the status field of an object. That's because when an object is created, we do not specify the status field. But when we get an object, a status field will appear, the contents of which differ based on the object itself—just like in specification.
It is also important to note that, generally speaking, Kubernetes can modify the metadata field, a controller can modify the status field, but the specification field is always set by the user and unless it's an objected that was created by the controller, it is not modified unless manually done so. The service object is an exception here, where an IP address, once created, can be injected in the specification field by the controller.
If you are already using Kubernetes, you might have noticed that another common sub-field in metadata field is a namespace. A namespace is a logical isolation of resources in Kubernetes that allows us to split our configuration for easy management and cataloging.
We will later discuss objects that do live outside of namespaces and as such do not have this field. However, you should note that for all other objects, in abscence of a namespace field, they are assigned to the
FAQs & Gotchas!
What are the different kinds of services?
There are three main kinds of services —
NodePort. Each one respectively can be accessed over a virtual IP address, a load balancer (depending on the Kubernetes distribution), or a port of the node on which the Pod may be running.
Why go through API server and not directly to etcd?
That's a very good question! There are many reasons. One, for example, is that we often have webhooks that modify the objects before they are stored in the databse. These webhooks are often used for validating the objects as well as modifying their content across version (think of it like a migration script). The API server can also be configured to work with some kind of authentication and authorization system, which we will discuss later.
Can Kubernetes do anything other than run containers?
Yes, it can. As you've noticed, the basic concept is to have objects and react to changes on those objects. There are built-in controllers that run a Pod, however, one could write their own controller that can do something completely different.
On that note, you might be interested in finding out that one can also create their own object type. They are called CRDs or Custom Object Definitions. Then a custom controller (sometimes referred to as an operator) can listen on changes of these objects and do as it pleases with their content.
This post was the first part. I will try to get into more details about Kubernetes and include topics such as CRDs, custom controllers, role based access policies and and networking.