> For the complete documentation index, see [llms.txt](https://docs.ramsdenj.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ramsdenj.com/introduction-3/01-intro/06-cluster-maintenance.md).

# Cluster Maintenance

"drain" node and move pods:

```shell
kubectl drain node-1
```

This "cordons" a node, to uncordon:

```shell
kubectl uncordon node-1
```

cordon marks unschedulable but leaves existing nodes:

```shell
kubectl cordon node-1
```

## Cluster Upgrade Introduction

Components should be somewhat in synch.

kube-apiserver is main component, the controller manager and the kube scheduler should be less than or equal to the version, and be a maximum of one lower inversion. The kubelet and kube proxy should be a maximum of two versions lower than the API server and should not be greater than the version of the API server.

`kubectl` should be +-1

k8s supports last 3 minor versions.

Upgrades do master first (pods stay up meanwhile)

Nex we do workers, can do all at once or one node at a time.

Alternatively create new nodes with higher version and remove old

We need to upgrade `kubeadm` first with `apt`.

Then `kubelet` with `apt`

Upg master:

```shell
kubeadm upgrade plan
apt upgrade -y kubeadm=VERSION
kubectl get nodes
apt upgrade -y kubelet=VERSION
systemctl restart kubelet
kubectl get nodes
```

Upg workers:

```shell
kubectl drain NODE
apt upgrade -y kubeadm=VERSION
kubectl get nodes
apt upgrade -y kubelet=VERSION
systemctl restart kubelet
kubeadm upgrade node config --kubelet-version VERSION
kubectl uncordon NODE
```

[Upgrading kubeadm clusters](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/)

## Backup and Restore

Can save all yaml for cluster via:

```shell
kubectl get all --all-namespaces -o yaml > all-deploy-services.yaml
```

Can backup `etcd` via:

```shell
ETCDCTL_API=3 etcdctl snapshot save snapshot.db
```

To restore:

```shell
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --data-dir=NEW_ETCD_DIR
```

[Operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)

Usually etcd is a static pod, so if we want to edit, edit manifests.

Look at pod:

```shell
kubectl describe ETCD_POD
```

Find ip, `trusted-ca-file`, `key-file` and `cert-file`, test via:

```shell
ETCDCTL_API=3 etcdctl --endpoints IP_ADDR:2379 \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  member list
```

Snapshot to `/opt/snapshot-pre-boot.db`:

```shell
ETCDCTL_API=3 etcdctl --endpoints IP_ADDR:2379 \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  snapshot save /opt/snapshot-pre-boot.db
```

Restore to `/etcd-backup`:

```shell
ETCDCTL_API=3 etcdctl --endpoints IP_ADDR:2379 \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --data-dir=/etcd-backup \
  snapshot restore /opt/snapshot-pre-boot.db
```

We will edit static pod. And point the etcd-data hostpath to new data directory.

## Multi-Cluster

List all:

```shell
kubectl config get-clusters
```

Swap:

```shell
kubectl config use-context CLUSTER
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.ramsdenj.com/introduction-3/01-intro/06-cluster-maintenance.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.