kubernetes-platform

kubernetes-platform

Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.

Use When

Standing up a self-managed Kubernetes cluster on Debian/Ubuntu VPS hosts (kubeadm).
Designing namespace, RBAC, ResourceQuota, LimitRange, and Pod Security boundaries for a multi-team or multi-tenant cluster.
Owning cluster lifecycle: upgrades, certificate rotation, etcd backup and restore.
Installing the minimum platform layer above the cluster: ingress controller, cert-manager, metrics-server.
Choosing between self-managed kubeadm and cloud-managed control planes (EKS, GKE, AKS).

Do Not Use When

Authoring the application architecture inside the cluster — use microservices-architecture (decomposition models and inter-service communication).
Provisioning the cluster's underlying VMs, VPC, or DNS through code — use infrastructure-as-code and cloud-architecture.
Building CI/CD pipelines that deploy into Kubernetes — use cicd-pipelines (pipeline shape and stage-gate design live there).
Cluster-level observability deep-dives — use observability-monitoring (SigNoz-led observability platform).
Service mesh design (Istio, Linkerd, Cilium service mesh) — out of scope.

Sibling Kubernetes Skills

This skill is the platform-team angle. Cross-reference rather than duplicate:

references/kubernetes-fundamentals.md — first cluster, mental model of core objects, kubectl workflow, debug triage.
references/kubernetes-production.md — Helm release patterns, HPA/VPA, observability stack, troubleshooting playbook.
references/kubernetes-saas-delivery.md — multi-tenant isolation models, GitOps with ArgoCD, progressive delivery.

Load the sibling that owns a specific topic. This skill owns the bootstrap, governance, security, and lifecycle layer that those skills assume is in place.

Required Inputs

Target host inventory (control-plane and worker IPs, OS, RAM, CPU).
Pod and service CIDR plan that does not collide with the host network.
Tenant or team list for namespace and RBAC design.
Compliance and data-residency constraints that drive self-managed vs managed.

Prerequisite Skills

cloud-architecture — for the network substrate.
cicd-pipelines — for the Linux operations baseline on Debian/Ubuntu hosts (Jenkins-on-Debian guidance lives there).
web-app-security-audit — for baseline application security posture.

Workflow

Confirm scope: bootstrap, governance, security, lifecycle, or platform-component install. Route topics owned by sibling skills (references/kubernetes-fundamentals.md, references/kubernetes-production.md, references/kubernetes-saas-delivery.md) to those skills before starting here.
Capture inputs: host inventory, CIDR plan, tenant list, compliance constraints.
Apply §1–§10 of this skill in order for a green-field cluster, or jump to the relevant section for a targeted change.
Load the matching references/<topic>.md file only when the depth is needed.
Produce the runbooks listed in Outputs and check them into the platform repo before handing the cluster to tenants.

Quality Standards

Self-managed kubeadm on Debian/Ubuntu is the default; cloud-managed is the alternative when constraints in §10 flip.
Every namespace ships with RBAC, ResourceQuota, LimitRange, and a restricted Pod Security label before workloads land.
Every cluster has an etcd snapshot cron and a documented restore drill. Untested backups do not count.
Every platform component (ingress, cert-manager, metrics-server, CNI) is version-pinned in platform/versions.yaml.
Cross-reference sibling Kubernetes skills rather than duplicating their guidance.

§1 Cluster Architecture

Control-plane components (kubernetes.io/docs/concepts/overview/components/, fetched 2026-05-01):

Component	Role
kube-apiserver	"The core component server that exposes the Kubernetes HTTP API."
etcd	"Consistent and highly-available key value store for all API server data."
kube-scheduler	"Looks for Pods not yet bound to a node, and assigns each Pod to a suitable node."
kube-controller-manager	"Runs controllers to implement Kubernetes API behavior."
cloud-controller-manager	"Integrates with underlying cloud provider(s)." Optional.

Node components:

Component	Role
kubelet	"Ensures that Pods are running, including their containers."
kube-proxy	"Maintains network rules on nodes to implement Services." Optional.
Container runtime	"Software responsible for running containers."

For self-managed Debian/Ubuntu clusters, containerd is the default runtime. Verify against the current kubernetes.io container-runtimes page before pinning a version.

§2 Self-Managed Bootstrap on Debian/Ubuntu

The full kubeadm bootstrap flow, kubelet flags, HA control-plane variant, and CNI selection rubric live in references/kubeadm-bootstrap.md. Minimum viable sequence:

# Control-plane node (Debian/Ubuntu, kubeadm + kubelet + containerd installed)
sudo kubeadm init \
  --apiserver-advertise-address=<control-plane-ip> \
  --pod-network-cidr=10.244.0.0/16   # match the chosen CNI

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Install a CNI (Cilium, Calico, or Flannel) — required for Pod networking
# kubectl apply -f <CNI-manifest>

sudo kubeadm token create --print-join-command

Each worker:

sudo kubeadm join <control-plane-address>:<port> \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

Post-init: install the Pod network add-on, label nodes by role, and copy admin.conf to any management workstation. For HA control planes, use InitConfiguration and JoinConfiguration files; pass kubelet overrides through nodeRegistration.kubeletExtraArgs.

CNI choice (Cilium, Calico, Flannel) is left to the implementing team — pick one and pin it; do not mix.

§3 Workloads — Minimum-Viable Manifests

Authoring core workloads is owned by references/kubernetes-fundamentals.md. The platform team still needs a reference for what "compliant by default" looks like, because every tenant manifest is reviewed against it. A Deployment + Service + Ingress that satisfies the restricted Pod Security profile:

apiVersion: apps/v1
kind: Deployment
metadata: { name: web, namespace: app }
spec:
  replicas: 3
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      serviceAccountName: web
      securityContext:
        runAsNonRoot: true
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: web
          image: ghcr.io/example/web:sha-abc1234
          ports: [{ containerPort: 3000 }]
          resources:
            requests: { cpu: "100m", memory: "128Mi" }
            limits:   { cpu: "500m", memory: "512Mi" }
          readinessProbe: { httpGet: { path: /healthz, port: 3000 }, periodSeconds: 5 }
          livenessProbe:  { httpGet: { path: /healthz, port: 3000 }, periodSeconds: 15 }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
---
apiVersion: v1
kind: Service
metadata: { name: web, namespace: app }
spec:
  selector: { app: web }
  ports: [{ port: 80, targetPort: 3000 }]
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  namespace: app
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls: [{ hosts: [web.example.com], secretName: web-tls }]
  rules:
    - host: web.example.com
      http: { paths: [{ path: /, pathType: Prefix, backend: { service: { name: web, port: { number: 80 } } } }] }

Probe and lifecycle detail → see references/kubernetes-fundamentals.md §"Probes — Get Them Right". Mental model of Deployments, StatefulSets, Services → references/kubernetes-fundamentals.md §"Core Objects".

§4 Helm

Chart structure and reserved directories (helm.sh/docs/topics/charts, fetched 2026-05-01):

Path	Purpose
`Chart.yaml`	"A YAML file containing information about the chart."
`values.yaml`	"The default configuration values for this chart."
`templates/`	"A directory of templates that, when combined with values, will generate valid Kubernetes manifest files."
`charts/`	"A directory containing any charts upon which this chart depends."
`crds/`	Custom Resource Definitions installed before templates.
`values.schema.json`	Optional JSON Schema used to validate `values.yaml`.
`NOTES.txt` (inside the chart's `templates` directory)	Post-install message printed to the user.

Helm reserves use of the charts/, crds/, and templates/ directories. (helm.sh, fetched 2026-05-01.)

Release lifecycle:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install web ./charts/web -n app --create-namespace -f values.prod.yaml
helm upgrade web ./charts/web -n app -f values.prod.yaml
helm rollback web 1 -n app
helm uninstall web -n app

Chart authoring patterns (named templates, subchart values overrides, chart testing, schema-validated values) → see references/helm-charts.md. Helm-vs-Kustomize tradeoff → references/kubernetes-production.md §"Helm vs Kustomize".

§5 RBAC

Four object kinds: Role (namespace-scoped), RoleBinding (namespace-scoped), ClusterRole (cluster-scoped), ClusterRoleBinding (cluster-scoped). ServiceAccount is the workload identity. (kubernetes.io/docs/reference/access-authn-authz/rbac/.)

Namespace-admin Role + RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { namespace: team-a, name: ns-admin }
rules:
  - apiGroups: [""]
    resources: ["pods", "services", "configmaps", "secrets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { namespace: team-a, name: team-a-admins }
subjects:
  - kind: Group
    name: team-a-admins
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: ns-admin
  apiGroup: rbac.authorization.k8s.io

Decision rubric:

Need	Use
Permissions for users/groups inside one namespace	Role + RoleBinding
Cross-cluster permissions for platform admins	ClusterRole + ClusterRoleBinding
Permissions for a workload (pod)	ServiceAccount + Role/RoleBinding
Read-only "view" across all namespaces	Bind built-in `view` ClusterRole via ClusterRoleBinding

Aggregated ClusterRoles, OIDC group mapping, audit patterns, and breakglass identities → references/rbac-patterns.md.

§6 Resource Governance

ResourceQuota enforces aggregate limits per namespace (kubernetes.io/docs/concepts/policy/resource-quotas/, fetched 2026-05-01):

Category	Example resource names
Compute	`requests.cpu`, `limits.cpu`, `requests.memory`, `limits.memory`, `hugepages-<size>`
Storage	`requests.storage`, `persistentvolumeclaims`, `<storage-class>.storageclass.storage.k8s.io/requests.storage`
Object counts	`pods`, `services`, `configmaps`, `deployments.apps`, etc.
Extended / DRA	`requests.nvidia.com/gpu`, DRA device-class quotas

Key behaviour: namespace-scoped, non-retroactive, and once a CPU or memory quota is in force every pod in the namespace must declare matching requests and limits. Over-quota submissions are rejected with HTTP 403.

LimitRange sets default requests/limits and min/max values so authors get sane defaults when they omit them. PriorityClass lets the scheduler preempt low-priority pods to make room for critical workloads.

Quota + LimitRange for a tenant namespace:

apiVersion: v1
kind: ResourceQuota
metadata: { name: team-a-quota, namespace: team-a }
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "100"
    persistentvolumeclaims: "20"
---
apiVersion: v1
kind: LimitRange
metadata: { name: team-a-defaults, namespace: team-a }
spec:
  limits:
    - type: Container
      default:        { cpu: "500m", memory: "512Mi" }
      defaultRequest: { cpu: "100m", memory: "128Mi" }
      max:            { cpu: "2",    memory: "4Gi" }

Per-tenant quota sizing, PriorityClass design for system vs workload tiers, and HPA interaction with quotas → references/kubernetes-production.md §"Resource Management".

§7 Pod Security

Three Pod Security Standards profiles, cumulative (kubernetes.io/docs/concepts/security/pod-security-standards/, fetched 2026-05-01):

Profile	Stance	Notable restrictions
Privileged	Unrestricted.	None — known privilege escalations allowed. For system/infrastructure workloads only.
Baseline	Minimally restrictive.	No host namespaces (`hostNetwork`, `hostPID`, `hostIPC`); no privileged containers; HostPath volumes restricted; capabilities limited; host ports restricted; seccomp not Unconfined; AppArmor/SELinux overrides restricted.
Restricted	Hardened best-practice.	All Baseline restrictions plus: run as non-root, read-only root filesystem, drop all capabilities, additional capability restrictions.

Enforce by labelling the namespace: pod-security.kubernetes.io/enforce: restricted (verify the exact label key against the current Pod Security Admission docs before applying in production).

Compliant securityContext for restricted:

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  seccompProfile: { type: RuntimeDefault }
containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities: { drop: ["ALL"] }

NetworkPolicy is the namespace firewall. Default-deny ingress + egress, then allow only what is needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: team-a }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

Migrating an existing namespace from baseline to restricted, exempting infrastructure namespaces, and warn-vs-enforce strategy → references/pod-security.md.

§8 Cluster Lifecycle

Upgrade flow:

kubeadm upgrade plan to see available targets and skew rules.
kubeadm upgrade apply v<minor>.<patch> on the first control-plane node.
kubeadm upgrade node on the remaining control-plane nodes and on every worker, one at a time, draining before and uncordoning after.

Verify the exact command sequence against the current kubeadm upgrade docs for the target minor version before running in production.

Certificate rotation: kubeadm certs check-expiration and kubeadm certs renew all rotate the kubeadm-issued control-plane certificates. After renewal, restart the static pods (kube-apiserver, kube-controller-manager, kube-scheduler) by removing and re-adding their manifests in /etc/kubernetes/manifests/.

etcd snapshot:

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-$(date +%F).db

Schedule the snapshot with cron and ship it to off-host storage. Restore with etcdctl snapshot restore per the upstream etcd docs.

Drain ordering, PodDisruptionBudgets, version-skew rules between kubelet and apiserver, and add-on upgrade order → references/cluster-lifecycle.md. Production-side rolling upgrade playbook → references/kubernetes-production.md §"Cluster and Node Upgrades".

§9 Cluster-Managed Components

Minimum platform layer above a kubeadm cluster (CNI plus the three below):

Component	Purpose	Reference
Ingress controller (ingress-nginx, Traefik)	Implements `Ingress` resources, terminates HTTPS, routes by host/path.	https://kubernetes.github.io/ingress-nginx/
cert-manager	Issues TLS certs from Let's Encrypt or other ACME issuers and internal CAs; renews them automatically.	https://cert-manager.io/docs/
metrics-server	Provides resource metrics for HPA and `kubectl top`.	https://github.com/kubernetes-sigs/metrics-server

Pin a version per component, install with Helm or static manifest, and bake the chosen versions into the cluster bootstrap script. Detail on ClusterIssuer patterns, wildcard certs, ingress-class isolation per tenant, and metrics-server TLS flags → references/cluster-managed-components.md.

§10 Self-Managed vs Managed

Lead with self-managed kubeadm on Debian/Ubuntu. Treat managed control planes as the alternative when the constraints below flip:

Constraint	Choice	Why
Sovereignty / data residency on-prem or own VPS	Self-managed kubeadm on Debian/Ubuntu	Full control over node OS, no cloud lock-in.
Small platform team, predictable workload	Self-managed	Lower per-cluster cost; predictable ops surface.
Need elastic scale, multi-region failover, integrated cloud IAM	Cloud-managed (EKS/GKE/AKS)	Control-plane HA and upgrades handled by the provider.
Need GPU pools, integrated cloud services (S3, IAM)	Cloud-managed	Cluster-native cloud auth and storage classes.

If two or more "cloud-managed" rows apply, do not self-manage — the operational tax outweighs the sovereignty benefit.

Anti-Patterns

Running tenant workloads in the same namespace as platform components.
Granting cluster-admin to a team because RBAC design felt slow.
Setting CPU limits equal to requests everywhere by reflex — causes throttling under spikes; reserve that pattern for latency-sensitive workloads.
Enforcing restricted cluster-wide on day one without first running it as warn and fixing offenders.
Skipping the etcd snapshot cron because "we never had to restore" — the first restore drill is not the day of the outage.
Mixing two CNIs in one cluster.
Copy-pasting EKS/GKE manifests into a kubeadm cluster without checking that the storage class, load-balancer class, and IAM assumptions still hold.

Outputs

A bootstrap runbook for the chosen control-plane and worker layout.
A namespace, RBAC, ResourceQuota, and LimitRange template per tenant tier.
A Pod Security enforcement plan with warn → enforce migration steps.
An upgrade and etcd-backup runbook with rotation and restore drills.
A pinned-version manifest for ingress controller, cert-manager, and metrics-server.

Companion Skills

references/kubernetes-fundamentals.md — core objects, kubectl, debug triage.
references/kubernetes-production.md — Helm release patterns, scaling, observability stack.
references/kubernetes-saas-delivery.md — multi-tenant isolation, GitOps, progressive delivery.
cloud-architecture — VPC, load balancers, DNS substrate.
infrastructure-as-code — Terraform modules that provision the VMs and network.
cicd-pipelines — pipelines that build and deploy into the cluster (pipeline shape and Jenkins-on-Debian guidance live there).
observability-monitoring — SigNoz-led observability above the cluster.
web-app-security-audit — application-layer security for workloads inside the cluster.

References

Tier 1 — Canonical:

Kubernetes in Action, Marko Luksa, Manning.
Production Kubernetes, Josh Rosso, Rich Lander, Alex Brand, John Harris, O'Reilly.
kubernetes.io documentation — https://kubernetes.io/docs/.
Components page — https://kubernetes.io/docs/concepts/overview/components/.
kubeadm cluster guide — https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/.
Pod Security Standards — https://kubernetes.io/docs/concepts/security/pod-security-standards/.
Resource Quotas — https://kubernetes.io/docs/concepts/policy/resource-quotas/.
Helm chart docs — https://helm.sh/docs/topics/charts/.

Tier 2 — Supporting:

ingress-nginx — https://kubernetes.github.io/ingress-nginx/.
cert-manager — https://cert-manager.io/docs/.
metrics-server — https://github.com/kubernetes-sigs/metrics-server.

Tier 3 — Supplementary:

kubeadm upgrade — https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/.

Deep-dives in references/:

kubeadm-bootstrap.md — full bootstrap runbook, HA control plane, CNI selection.
helm-charts.md — chart authoring patterns, schema validation, subchart overrides.
rbac-patterns.md — aggregated roles, OIDC group mapping, audit, breakglass.
pod-security.md — warn to enforce migration, exemptions, NetworkPolicy patterns.
cluster-lifecycle.md — upgrade ordering, version skew, etcd restore drill.
cluster-managed-components.md — ingress, cert-manager, metrics-server install patterns.