kOps Sets DigitalOcean ProviderID And Bumps Cloud Addons

kOps is the Kubernetes installer that still owns the AWS, GCP, Azure, DigitalOcean, and Hetzner paths many platform teams run. Master took 29 commits in the last week, most of them lining the unreleased 1.37 series up with Kubernetes 1.37 cloud provider addons. The operator visible work is a DigitalOcean providerID fix that unblocks those addons, a CCM and CSI version train, and the last of the old PKI keyset aliases.

The headline diffstat is 1893 files, 113387 insertions, 39413 deletions. Most of that is vendor and generated clients after the k8s.io libraries moved to v0.37.0. The rest is addon templates, test fixtures, and a few behavior changes that will show up on kops update cluster.

kOps names DigitalOcean nodes after their private IP. The DigitalOcean cloud controller manager used to resolve those nodes by scanning droplet addresses. That stopped working once the controller moved to InstancesV2.

Ciprian Hacman first pinned the addon at v0.1.45 after a bump to v0.1.69 left every node stuck with the node.cloudprovider.kubernetes.io/uninitialized taint. v0.1.62 restored a name lookup, but that lookup calls Droplets.ListByName, which never matches a node named after its IP. Clusters failed to validate. The pin is in the digitalocean-cloud-controller addon template.

The real fix sets providerID on the kubelet. nodeup reads the droplet ID from the metadata service through dometadata.GetDropletID() and registers digitalocean://<id>. The helper already served the bootstrap authenticator, so this adds no extra HTTP path.

With that in place, the addon moves to v0.1.69 again. Existing clusters can take the image without a coordinated node roll. v0.1.45 already wrote digitalocean://<id> into spec.providerID for nodes it initialized, and that field is immutable. New nodes get it from the kubelet.

The same bump changes the default DigitalOcean load balancer type. A new LoadBalancer Service with no type annotation now gets a network load balancer, not a regional one. Network load balancers forward TCP and UDP only. They do not terminate TLS and they do not speak HTTP or PROXY protocol. Existing Services keep the type the controller already recorded. Set the annotation if the old behavior is required:

metadata:
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-type: "REGIONAL"

The same day, every supported cloud got a CCM or CSI bump, recorded in the 1.37 release notes. kops update cluster applies the addon templates. The ecr-credential-provider binary lives in the node image, so it lands only when nodes are replaced.

  • AWS: cloud controller manager v1.37.0, EBS CSI driver v1.65.0, ecr-credential-provider v1.37.0
  • GCP: cloud controller manager v36.2.4, PD CSI driver v1.26.0
  • Azure: cloud controller manager and cloud node manager v1.36.5, azuredisk CSI driver v1.34.5
  • DigitalOcean: CSI driver v4.18.0
  • Hetzner: cloud controller manager v1.36.0, CSI driver v2.22.2
  • Akamai (Linode): cloud controller manager v0.9.8

AWS also stops rendering the unused ebs-csi-controller-volumemodifier Service. kOps never shipped the volume-modifier-for-k8s sidecar, because that image is published only to a registry kOps does not mirror. The Service has had no endpoints since kOps 1.36. Volume modification through ebs.csi.aws.com annotations has been unavailable for the same period. The EBS CSI addon template does not prune, so an existing cluster keeps the Service until you delete it with kubectl delete service -n kube-system ebs-csi-controller-volumemodifier.

Hetzner support for Kubernetes 1.32 is deprecated in kOps 1.37. Hetzner dropped 1.32 from its support matrix in CSI driver v2.21.0. A 1.32 cluster on Hetzner now runs untested component versions. Plan the 1.33 jump before picking up this train.

Two commits close a conversion path that has been around since the Keyset API rewrite.

vfs_keystorereader.go no longer maps service-account to master or kubernetes-ca to ca. Those aliases existed because kOps before 1.19 stored the service account TLS key under master, and 1.22 renamed kubernetes-ca to ca. Dropping the aliases is a hard stop: FindKeyset looks up the requested name only.

The follow up removes LegacyFormat from Keyset and from the Keypair task. oldFormat disappears from the serialized task YAML. The render path that fetched a keyset and rewrote it into v1alpha2 is gone. CA tasks no longer reissue a certificate just because the stored object was in the old format.

This is mostly internal cleanup. Clusters that have been through a kOps 1.22 or later apply already store keysets in the current format. A state store that still has only the pre 1.19 master keyset name will not resolve service-account. Rename those objects before running a master build against that store.

Peter Rifel added Distribution.IsAmazonLinux() so Amazon Linux is a family, not a one off amazonlinux2023 project string. The new predicate also silences the HasDNF warning that logged unknown project for HasDNF ("amazonlinux2023") on every Amazon Linux node.

The next commit adds experimental Amazon Linux 2027. Detection matches ID=amzn and VERSION_ID=2027 in distributions.go. Preview images set VARIANT_ID=preview, which kOps ignores, so the same match should work at GA. ForceNftables stays false. PackagesBuilder installs iptables-nft, same as Amazon Linux 2023. AWS does not recommend the preview for production. AL2027 ships kernel 7.1, systemd 260, glibc 2.44, DNF 5, SELinux in enforcing mode, and cgroup v2 only.

On the toolchain side, Go moves to 1.26.8 in go.mod and the other modules, and the k8s.io libraries move to v0.37.0. Regenerated clients now emit discovery.DiscoveryInterfaces. That vendor and CRD churn is most of the insertion count.

DigitalOcean LoadBalancer Services without service.beta.kubernetes.io/do-loadbalancer-type get a network load balancer after the CCM bump. Set REGIONAL before kops update cluster if HTTP or TLS rules are in use.

Delete ebs-csi-controller-volumemodifier by hand. The addon will not remove it, and it has had no endpoints since 1.36.

Do not run Amazon Linux 2027 in production. Treat it as a preview image for nodeup testing. Hetzner plus Kubernetes 1.32 is untested; move those clusters to 1.33 before kOps 1.37. If a state store still has only private/master or private/ca keyset files, rename them before this master build.