SuperAPI AI Directory

k8s-launch-kit-config

General & other

View repository

Updated 10-16 02:34

About this skill

l8k: Configuration Files

PREREQUISITE: Read ../k8s-launch-kit-shared/SKILL.md for install paths, global flags, and output modes.

Understand, create, or edit l8k configuration files.

File Types

File Source Purpose
cluster-config.yaml Generated by l8k discover Hardware inventory plus the resolved deployment profile
l8k-config.yaml User-created or copied from cluster-config Full config with both hardware + deployment settings

Usage

# Use a config file for generation
l8k generate --user-config my-config.yaml \
  --save-deployment-files ./output
Combine user config with live discovery (hardware refreshed, settings kept)

l8k discover --user-config my-config.yaml
--kubeconfig ~/.kube/config
--save-cluster-config ./updated-config.yaml


## Profile Resolution

Fresh discovery and file-backed generation resolve settings with this
precedence:

1. Canonical built-in defaults from the repository-root `cluster-config.yaml`.
2. Hardware-derived profile defaults.
3. Existing config values.
4. Explicit CLI flags.

Generation never rewrites the source config. It stores the exact effective
configuration at `<output>/.l8k/resolved-config.yaml`; deploy and validate use
that sidecar automatically when no explicit config is passed. Explicit false,
zero, and empty values override defaults. YAML `null` means unset.

Discovery with `--user-config` follows a stricter refresh contract: only
`clusterConfig` is replaced. Every other section remains as loaded unless an
explicit CLI flag overrides its corresponding field. Generation still fills
missing profile fields when it consumes that config.

## Config Sections Quick Reference

| Section | What It Controls |
|---------|-----------------|
| `networkOperator` | Operator namespace, version, image repository, Helm repository, and `skipHelmChart` ownership switch |
| `docaDriver` | OFED/DOCA driver image, version, blacklist settings |
| `maintenance` | Maintenance Operator, SR-IOV drain, and legacy OFED upgrade concurrency |
| `nvIpam` | NV-IPAM per-node block size, IP pool ranges, and subnet generation |
| `sriov` | VF count, resource prefix, MTU, link type |
| `hostdev` | Host device resource name |
| `rdmaShared` | RDMA shared device resource name |
| `ipoib` | IPoIB master interface, resource name |
| `macvlan` | MacVLAN master interface, mode |
| `nicConfigurationOperator` | NIC firmware template settings |
| `spectrumX` | OVS bridge config, multiplane mode, RDMA settings |
| `profile` | Profile selection criteria (fabric, deployment, multirail) |
| `clusterConfig[]` | Per-group hardware: NICs, nodes, capabilities, selectors |

Each `clusterConfig[]` entry has these key fields:

- `identifier` — group name (used for `NicNodePolicy` naming). For groups with both `machineType` and `gpuType` resolved, this is the lowercased machine/GPU identity with complete `NVIDIA` segments removed and common machine segments shortened (`ThinkSystem` → `ts`, `PowerEdge` → `pe`), bounded to 30 bytes with balanced component prefixes and a 6-character deterministic hash when needed; otherwise a fallback `group-N`. The Launch Kit machine node label uses the same value.
- `machineType` — server model (e.g. `PowerEdge-XE9680`); populated from `nvidia.com/gpu.machine` label or DMI fallback.
- `gpuType` — GPU SKU (e.g. `NVIDIA-H200`); populated from `nvidia.com/gpu.product` label or `nvidia-smi` fallback. **Note:** this field used to be called `productType` — the rename happened to disambiguate it from the server model. Old `productType:` keys in hand-authored configs must be renamed to `gpuType:`.
- `netplanManaged` — true when any worker has an NVIDIA PF whose current MAC
  is selected by a host netplan `match.macaddress` stanza with a non-empty
  `set-name`. The flag identifies a potential conflict with NCO udev naming;
  host-specific MACs are not persisted. If generation would emit a
  `NicInterfaceNameTemplate` for this group, clean up the affected `set-name`
  stanzas and re-run discovery instead of editing this flag by hand.
- `capabilities.nodes.{sriov,rdma,ib}` — what the underlying hardware supports.
- `pfs[]` — physical function list with PCI address, device ID, RDMA device,
  network interface, traffic class, rail, NUMA, GPU affinity, and `model` (the
  VPD model/description string read from `NicDevice.Status.modelName`).
  - **One rail per NIC (default).** Discovery advertises one rail per physical NIC: a NIC's multi-plane east-west PFs (planes of one port, e.g. Spectrum-X ConnectX-8/9) collapse to the master PF, so an 8-PF node lists 4 east-west PFs / 4 rails. A NIC whose `model` is genuinely dual-port (`2-port`/`Dual-port`) keeps a rail per port. Run `l8k discover --collapse-nic-rails=false` to emit one rail per PF (legacy/dev behaviour).
- `nodeSelector` — Kubernetes node selector for this group. Source groups key on the machine label written by `l8k discover`: `nvidia.kubernetes-launch-kit.machine: <identifier>`. Auto-merged groups (different machineTypes sharing a GPU type) key on `nvidia.kubernetes-launch-kit.gpu: <gpuType>` instead — the GPU label retains its discovered value, including `NVIDIA`, so the merged selector binds correctly across source machineTypes.
- `workerNodes` — explicit hostnames (populated by discovery).

For the full field-by-field reference with types, defaults, and descriptions,
read `references/config-reference.md`.

## Topology Presets

The `presets/` directory contains pre-recorded topologies for known hardware combinations. A preset is a `topology.yaml` file with the following shape:

```yaml
machineType: PowerEdge-XE9680   # required
gpuType: NVIDIA-H200            # required — both keys are matched as a pair
nicModel: BlueField-3 SuperNIC (ConnectX-7)
gpuInterconnect: NV18
numaNodes: 2

# Required if the preset will be used with `l8k generate --for`.
# Discovery-time overlay does not need this block.
capabilities:
  nodes:
    sriov: true
    rdma: true
    ib: false

pfs:
  - deviceID: a2dc
    pciAddress: 0000:1a:00.0
    traffic: east-west
    rail: 0
    numaNode: 0
    connectedGPU: GPU0
    gpuProximity: PIX

Lookup is exact-match on (machineType, gpuType). No any-GPU fallback — a preset that doesn't declare gpuType: is rejected at load time. Multi-variant presets for the same machine (different GPU SKUs) live in separate directories with composite names like PowerEdge-XE9680-H200 / PowerEdge-XE9680-B200. The directory name is shown by l8k preset list and is what l8k generate --for <name> accepts.

Validation deviations. When the matched preset's PF count, PCI addresses, or device IDs don't exactly match discovered hardware, the preset is NOT applied — discovery keeps the live-discovered topology (traffic/rail/NUMA), because overlaying a preset onto a different PCI layout would corrupt the classification (a coincidentally-overlapping PCI address would inherit the preset's unrelated traffic/rail/GPU fields). The discrepancies are recorded under clusterConfig[*].presetDeviation and every subsequent config load re-emits a warning listing each deviation. The preset's authoritative topology is overlaid only on an exact match (zero deviations); presetApplied: true appears only in that case. Part-number and PSID differences are expected (firmware/SKU variants) and never block application.

Common Edits

# Change VF count per PF and fabric-specific MTUs
sriov:
  numVfs: 16
  ethernetMtu: 9000
  infinibandMtu: 4000

# Set the DOCA driver version. Advanced users can also forward literal
# environment variables through the generated NicClusterPolicy or
# NicNodePolicy; custom values override any value produced by another
# docaDriver field with the same name.
docaDriver:
  version: "doca3.3.0-26.01-1.0.0.0-6"
  env:
    - name: THIRD_PARTY_RDMA_MODULES
      value: "nvidia_peermem"

# Allow four simultaneous maintenance operations. Network Operator 26.1+
# uses the global Maintenance Operator limits; older releases use the legacy
# SR-IOV/OFED limits where applicable.
maintenance:
  maxParallelOperations: 4
  maxUnavailable: 4
  maxNodeMaintenanceTimeSeconds: 3600
  maxParallelUpgrades: 4

# The selected release supplies the Helm repository URL.
networkOperator:
  # Keep generating/applying Network Operator CRs but let another system own
  # the Helm release. Generate/deploy/validate have the equivalent CLI flag
  # --skip-network-operator-helm; clean reads this persistent ownership setting
  # and retains the release while deleting Network Operator CRs.
  skipHelmChart: true

# Configure NV-IPAM subnets manually
nvIpam:
  perNodeBlockSize: 10
  subnets:
    - subnet: "10.10.0.0/16"
      gateway: "10.10.0.1"

# Namespaces for the secondary-network CRs + example test DaemonSets.
# One independent copy is rendered per namespace (shared resources like
# IPPools and NodePolicies are NOT duplicated). Defaults to ["default"].
networkNamespaces: ["my-namespace"]

Tips

  • Start by running discovery (l8k discover) to generate a baseline, then edit it.
  • A discovered config can be passed directly to l8k generate without repeating profile flags; use flags only for overrides.
  • Use l8k schema to discover the Network Operator release keys supported by the installed l8k version.
  • Treat a GA catalog entry as sticky during nightly synchronization: an upstream beta or release candidate for the next patch must not replace its public Network Operator artifact set. Update the entry again when the patch is GA.
  • nvIpam subnets are auto-generated if not specified — one per rail using non-routable ranges.
  • nvIpam.perNodeBlockSize defaults to 10; generation warns when it is lower than sriov.numVfs because the per-node allocation may be too small for all VFs.
  • docaDriver.unloadThirdPartyRDMAModules: true auto-populates UNLOAD_THIRD_PARTY_RDMA_MODULES from discovered OFED-dependent modules.
  • docaDriver.env is an advanced escape hatch. Values override generated MOFED environment entries by name, duplicate custom names use the last value, and bad driver options can disrupt node networking.
  • For release 26.1+, SR-IOV requestor mode requires both the Network Operator drain requestor and the SR-IOV external drainer. l8k renders both; applying only CRs cannot enable their Deployment environment variables.
  • Updating an existing release to the generated requestor-mode Helm values requires --overwrite-existing.

See Also

OpenShift

OpenShift configuration uses top-level flavor: ocp, sriov.operatorNamespace, nfd.operatorNamespace, nfd.configurationName, and maintenance.operatorNamespace. Install operator packages externally; l8k configures existing installations.

Optional standard NIC tuning

nicConfigurationOperator.deployNicConfigurationTemplate defaults to false. Root, generate and discover accept --deploy-nic-configuration-template[=false]. It enables NCO independently of interface naming and renders per-source, east-west PCI-scoped templates for standard SR-IOV, RDMA-shared and host-device profiles on k8s/OCP. VF count comes from sriov.numVfs; link type comes from profile.fabric. Ethernet gets DSCP trust and priority-3 PFC; InfiniBand omits RoCE tuning. Spectrum-X rejects this option. Only SR-IOV disables the Mellanox plugin (Helm values on k8s, additive disablePlugins merge on OCP). Kubernetes external Helm owners must configure that plugin themselves.

Deploy and validate consume the generated template and require all matched NicDevices to report current ConfigUpdateInProgress=False with reason UpdateSuccessful; stale/partial status waits and failures surface. Opting out does not restore applied NIC settings. On k8s, omitted templates (also for unselected groups) block deploy preflight unless --overwrite-existing is used, which deletes them. OCP preserves existing templates and disabled plugins. Generation rejects inventory with selected and excluded ports on the same PCI device. Review whole-NIC scope, including ports absent from inventory, and qualify runtime RoCE support for the installed NCO/DOCA/RHCOS versions before live use. See docs/reference/configuration.md#nic-configuration-and-naming.