Is there a way to enable nested virtualization in GKE cluster node?

Question

I am trying to use KubeVirt with GKE cluster.

I found I am able to create a nested virtualization enabled GCP VM, but I didn't find a way to achieve the same thing for GKE cluster node.

If I cannot enable nested virtualization for GKE cluster node, I can only use the kubevirt with debug.useEmulation which is not what I want.

Thanks

Answer 1

Yes you can -- it isn't even hard to do, it just isn't very intuitive.

1. Start a GKE cluster with ubuntu/containerd, n1-standard nodes and minimum cpu of Haswell. I think you also need to enable "Basic Authorization" to get virtctl working (sorry).

2. Find the template used for your new cluster, then to determine the proper source image:

   gcloud compute instance-templates describe --format=json | jq ".properties.disks[0].initializeParams.sourceImage"

3. Create a copy of the source disk with nested virtualization enabled:

   gcloud compute images --project $PROJECT create $NEW_IMAGE_NAME --source-image $SOURCE_IMAGE --source-image-project=$SOURCE_PROJECT --licenses "https://www.googleapis.com/compute/v1/projects/vm-options/global/licenses/enable-vmx"

4. Use "Create Similar" on the template for your GKE cluster. Change the boot disk to $NEW_IMAGE_NAME. You will also need to drill down to networking/alias and change the default subnet to your pod network.

5. Trigger a rolling update on the group for your GKE nodes to move them to the new template.

6. You can now install kubevirt (I had to use 0.38.1 instead of the current)

Caveats: I don't know how to use google disk images for kubevirt which would be an obvious match. I haven't even figured out how to get private GCR working with CDI. Oh, and console doesn't work due to websocket problems. But... you can shell to a gke node and see /dev/kvm, you can also kubevirt a VM then ssh into it, so yes, it does work.

Anyone know how to make any of this better?

Answer 2

Currently nested virtualization is available only on GCE as per this docs.

There is already question regarding supporting Nested Virtualization on GKE and it can be found here. I'd say it's not introduced yet, thats why you cannot find proper documentation about GKE and nested virtualization.

Also please consider that GCP and GKE are quite different.

Google Compute Engine VM instance is unmanaged by google. So besides ready base image, you can do whatever you need, like it would be normal VM.

However, Google Kubernetes Engine was created especially for containers. Thoses VMs are managed by google. GKE already creates Cluster for you and all VMs are automatically part of the cluster. In GKE you are unable to run Minikube or Kubeadm.

Here you have some characteristics of GKE