Advanced Stack

Appearance

Running an unattended Ubuntu Server VM on a Talos hypervisor node: the 2 GiB cdrom limit and the netboot workaround ​

Talos Linux v1.15.0-alpha.2, a pre-release published on 2026-10-27, adds a hypervisor that runs virtual machines declared as VirtualMachineConfig documents in the host's machine configuration, on top of the siderolabs/libvirtd system extension. I run that build on a Mac mini Late 2012 (Intel i5-3210M, 16 GiB RAM) as a K8s-less node (cluster: {}), where the extension provides libvirtd 12.7.0 and QEMU 11.1. On 2026-11-10 I installed Ubuntu Server 26.04.1 LTS on that node as a VM named ubuntu-vm2, fully unattended, from a ~117 MB netboot mini-ISO plus a NoCloud CIDATA seed. The VM booted, took 192.168.1.131 from DHCP, and accepted ssh pas@192.168.1.131 with the ed25519 key from the seed, with password login disabled. This note records the procedure that worked and the gotchas that actually cost time, all verified live on that date.

The task, precisely ​

The goal was to declare the VM in the node's machine configuration, install Ubuntu Server end to end with zero interactive input, and end up with a guest that answers SSH by key, reports its IP to Talos, and offers a serial login. The constraints on this host: 16 GiB of host RAM (house rule: at most 2 VMs at a time), guest firmware limited to BIOS by the extension, K8s-less mode, one Ethernet link (enp1s0f0, up at 1 Gbps) for macvtap networking, and about 450 GiB free on the EPHEMERAL volume. All node commands below go through talosctl --nodes 192.168.1.117; the talosconfig with the admin credentials stays local and is never published.

Host prerequisites, all verified live on the node:

  • Hardware virtualization: vmx and ept CPU flags, kvm/kvm_intel modules loaded, /dev/kvm present. The host is amd64 and guests need x86-64-v2 (sse4_2 and popcnt are present on the i5-3210M).
  • Talos v1.15+ with the siderolabs/libvirtd extension baked into the image (schematic 84feec1f502dbb1340c6a832821754ae5b4386d7349e86e8ae0b862bc9148197 here). After boot, ext-virtqemud, ext-virtstoraged, ext-virtlockd and ext-virtlogd all report Running.
  • talosctl v1.15+ on the Mac, because the talosctl hypervisor ... subcommands (console, vnc, content-library, start/stop/reboot) only exist in v1.15.
  • Node configuration documents: EtcFileConfig entries for /etc/passwd and /etc/group (qemu uid/gid 107, kvm gid 36), because Talos does not create those files and the libvirt extension services need the qemu user; a UserVolumeConfig + ContentLibraryConfig pair named images for ISOs and seeds; and a UserVolumeConfig + StoragePoolConfig pair named pool1 for VM disks.

Finding 1: the hypervisor works without Kubernetes ​

With cluster: {} in the v1alpha1 machine config (container mode), the whole VM machinery still reconciles: the content library, the storage pool and the VM documents all work without any Kubernetes API. The hidden requirement is the /etc/passwd and /etc/group entries above, because without them the ext-virt* services cannot resolve the qemu user and the VMs never start.

Finding 2: content-library files larger than 2 GiB cannot be attached as cdroms ​

The live-server ISO (ubuntu-26.04.1-live-server-amd64.iso, 2.7G as listed on releases.ubuntu.com) attached as a cdrom makes DomainCreateXML fail with a bare invalid argument on this host. I bisected the threshold on 2026-11-10 with a healthy host: a 3 GB zero-filled dummy file failed the same way, while a 922 kB seed ISO and disk-only VMs started fine. The failure tracks the file size at 2^31 bytes instead of the content, the labeling, or the file locks (SELinux labels and locks were ruled out), which points at a 32-bit overflow somewhere in the extension (libvirt 12.7.0 / QEMU 11.1). The workaround: never attach the big ISO directly. Boot a small netboot mini-ISO that downloads the full ISO over HTTP at boot.

Finding 3: the netboot mini-ISO that fetches the installer over HTTP ​

I built it on the Mac from the Ubuntu netboot tarball and syslinux:

sh
curl -LO https://releases.ubuntu.com/resolute/ubuntu-26.04.1-netboot-amd64.tar.gz
tar xzf ubuntu-26.04.1-netboot-amd64.tar.gz        # -> netboot-amd64/{linux,initrd,...}
curl -LO https://mirrors.edge.kernel.org/pub/linux/utils/boot/syslinux/syslinux-6.03.tar.gz
tar xzf syslinux-6.03.tar.gz
mkdir -p mini-iso
cp netboot-amd64/linux netboot-amd64/initrd mini-iso/
cp syslinux-6.03/bios/core/isolinux.bin mini-iso/
cp syslinux-6.03/bios/com32/elflink/ldlinux/ldlinux.c32 mini-iso/

mini-iso/isolinux.cfg:

text
DEFAULT install
LABEL install
  KERNEL /linux
  INITRD /initrd
  APPEND root=/dev/ram0 ramdisk_size=7340032 ip=dhcp iso-url=https://releases.ubuntu.com/26.04.1/ubuntu-26.04.1-live-server-amd64.iso autoinstall console=ttyS0,115200n8 ---

Then build the ISO with xorriso: xorriso -as mkisofs -o ubuntu-26.04.1-mini3.iso -b isolinux.bin -c boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -J -R -V UBUNTU-NETBOOT mini-iso/. The result lands at about 117 MB. The iso-url= parameter makes the netboot initrd download the full live-server ISO over HTTP into RAM and mount it as the install source.

Finding 4: the ramdisk and the VM RAM must fit the downloaded ISO ​

With the default ~1.5 GiB ramdisk, the kernel panics with deadlocked on memory during the netboot, because the ~2.7 GB ISO does not fit in RAM. Two fixes work together: ramdisk_size=7340032 (KiB, about 7 GiB) in the APPEND line above, and memory.size: 8GiB on the VM during the install. On a 16 GiB host that still leaves room for one more small VM, consistent with the 2-VM rule. Shrink the VM memory after the install, and only while the VM is stopped.

Finding 5: unattended means the autoinstall kernel parameter plus a CIDATA seed ​

Subiquity stops at a "Continue with autoinstall?" prompt unless the kernel command line carries autoinstall (see the APPEND line above). The seed is a second tiny cdrom whose volume label is exactly CIDATA and whose root holds user-data and meta-data:

sh
hdiutil makehybrid -iso -joliet -o ubuntu-vm2-seed.iso -volid CIDATA seed-dir/

The sanitized user-data:

yaml
#cloud-config
autoinstall:
  version: 1
  identity:
    hostname: ubuntu-vm2
    username: pas
    realname: pas
    password: "$6$<your-own-sha512-crypt-hash>"   # generate with: openssl passwd -6
  ssh:
    install-server: true
    authorized-keys:
      - ssh-ed25519 AAAA...<your-ed25519-public-key>...
    allow-pw: false
  storage:
    layout:
      name: direct
  network:
    version: 2
    ethernets:
      any:
        match:
          name: "en*"
        dhcp4: true
  packages:
    - qemu-guest-agent
  late-commands:
    - curtin in-target -- sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="[^"]*"/GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200n8"/' /etc/default/grub
    - curtin in-target -- update-grub
    - curtin in-target -- systemctl enable serial-getty@ttyS0.service
    - curtin in-target -- systemctl enable qemu-guest-agent.service
    - curtin in-target -- sh -c "echo 'pas ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/99-pas-nopasswd && chmod 0440 /etc/sudoers.d/99-pas-nopasswd"

Three details each cost an iteration:

  • identity.password is required by subiquity since 24.04 even when SSH keys are set; the autoinstall reference states the encrypted password "is required for use with sudo, even if SSH access is configured". Generate your own hash with openssl passwd -6 and never publish it.
  • packages: [qemu-guest-agent] is what makes talosctl get virtualmachinestatuses show the guest IP in ADDRESSES, because Talos reads it through the guest agent channel. Without it you fall back to scanning the LAN and matching the VM MAC.
  • The late-commands enable the serial console and passwordless sudo in the installed system, so talosctl hypervisor console ubuntu-vm2 gives a login prompt after the install.

Finding 6: firmware is BIOS-only and cdroms go on the scsi bus ​

The libvirtd extension ships no OVMF firmware, so firmware.type: uefi fails with invalid argument; use firmware.type: bios. A cdrom on bus: sata produced EINVAL on this host; use bus: scsi. And provision.fromImage is only valid on cdrom disks, while a writable disk needs pool, size and provision.blank. The working disk layout:

yaml
disks:
  - name: sys
    pool: pool1
    size: 40GiB
    format: qcow2
    bootOrder: 1
    provision:
      blank: {}
  - name: install
    type: cdrom
    bus: scsi
    bootOrder: 2
    provision:
      fromImage:
        library: images
        file: ubuntu-26.04.1-mini3.iso
        digest: sha256:1cf59c34dfdc1a3e32909d90d9868c94a88dd75ebf43d4a75057946bd571fdbf
  - name: seed
    type: cdrom
    bus: scsi
    provision:
      fromImage:
        library: images
        file: ubuntu-vm2-seed.iso
        digest: sha256:59dd506f2738770efa1f1532a5fb43650746235d43ad06c5fc601891dadc9e1f

plus cpu.count: 2, memory.size: 8GiB, firmware.type: bios, console.serial.enabled: true, console.vnc.enabled: true, one interface net0 on link: enp1s0f0 (macvtap, DHCP on the LAN), and guest.agent.enabled: true.

Finding 7: digests must match between the upload and the VM document ​

Upload both files with the sha256 from shasum -a 256, and reference the same digest in the VirtualMachineConfig:

sh
talosctl hypervisor content-library upload images ubuntu-26.04.1-mini3.iso \
  --name ubuntu-26.04.1-mini3.iso \
  --digest sha256:1cf59c34dfdc1a3e32909d90d9868c94a88dd75ebf43d4a75057946bd571fdbf
talosctl get contentlibrarystatuses        # images -> ready
talosctl patch machineconfig --patch @vm-ubuntu2.yaml
talosctl get virtualmachinestatuses        # ubuntu-vm2 -> running / ready

The node verifies integrity on upload. A stale digest kept by a strategic-merge patch on a list field surfaces later as digest mismatch at VM start; the fix is to edit the YAML file and re-apply the whole document instead of patching piecemeal.

Finding 8: never patch or destroy a running VM, and how I recovered a poisoned host ​

On 2026-11-09, destroying a running domain crashed ext-virtqemud (alpha bug), and the crash left stale /var/lib/libvirt/qemu/domain-N-* directories on EPHEMERAL. After that, every DomainCreateXML failed with a bare invalid argument, for any VM and any pool, surviving node reboots and talosctl service ext-virtqemud restart. Talos has no shell to remove those directories. The recovery that worked, verified on 2026-11-10: talosctl reset --reboot --system-labels-to-wipe EPHEMERAL, then re-create the pools and library and re-upload the files. The machine config, the PKI and the talosconfig live in the STATE partition and survive the wipe, so the node stays manageable, and a fresh disk-only VM ran immediately after. The rule of thumb: stop gracefully first (talosctl hypervisor stop <vm>), or edit the file, apply, and reboot the node.

Finding 9: consoles, and the VNC quirks ​

The serial console is the reliable one: talosctl hypervisor console ubuntu-vm2, ideally inside tmux so it survives disconnects. For VNC, run talosctl hypervisor vnc ubuntu-vm2 --port 5901 and point a viewer at 127.0.0.1:5901. Port 5900 is macOS Screen Sharing ("cannot control own machine" means you hit it). QEMU allows one viewer only, and a canceled stream can leave a ghost client on the node's vnc.sock that silently drops new viewers; check with talosctl read /proc/net/unix | grep vnc and restart the domain to clear it. No VNC password exists, and the tunnel listener lives until Ctrl-C. Lifecycle commands are talosctl hypervisor start|stop|reboot ubuntu-vm2. Removing the VM later is a delete patch ($patch: delete on the VirtualMachineConfig), and library files can only be deleted once no VM references them.

Key takeaways ​

  • The Talos v1.15 hypervisor runs VMs from machine-config documents and works in K8s-less mode, provided the node has the qemu passwd/group entries plus an images content library and a pool1 storage pool.
  • On the siderolabs/libvirtd extension build tested (libvirt 12.7.0, QEMU 11.1), any content-library file over 2^31 bytes fails DomainCreateXML with a bare invalid argument when attached as a cdrom, so large installer ISOs must be fetched at boot with iso-url= instead of being attached.
  • A fully unattended Ubuntu Server install needs the autoinstall kernel parameter plus a CIDATA NoCloud seed; the seed also installs qemu-guest-agent (for IP reporting in virtualmachinestatuses) and enables the serial console and passwordless sudo via late-commands.
  • The netboot path loads the whole ISO into guest RAM, so set ramdisk_size=7340032 and give the VM 8 GiB during the install on a 16 GiB host.
  • Guest firmware is BIOS-only with this extension (no OVMF), cdroms attach on bus: scsi, and the digest must match between the upload and the VM document.
  • Stop a VM before patching or destroying it. On this alpha, a destroy crash left stale libvirt domain dirs that made every VM create fail until an EPHEMERAL wipe; STATE (machine config, PKI, talosconfig) survives that wipe.
  • Two upstream issues worth reporting on siderolabs/talos: the over-2^31-byte cdrom failure and the destroy-crash/stale-dir EINVAL cascade. I searched the tracker on 2026-11-10 and found no matching issue; related merged PRs around the VM work are #14518, #14495 and #14541.

FAQ ​

Why not attach the full live-server ISO as a cdrom? ​

Files over 2^31 bytes fail with a bare invalid argument in this extension build. I confirmed the threshold with a 3 GB zero-filled dummy file, which fails the same way, so the failure tracks the size instead of the content or the labeling.

Can I boot the guest with UEFI? ​

The libvirtd extension bundles no OVMF, so firmware.type: uefi fails with invalid argument. Use firmware.type: bios.

How do I get the VM's IP without scanning the LAN? ​

Install qemu-guest-agent through the seed's packages; Talos reads the guest IPs through the agent channel and shows them in the ADDRESSES column of talosctl get virtualmachinestatuses. Fallback: nmap -sn 192.168.1.0/24 and match the VM's MAC.

Can I change the VM's RAM after the install? ​

Only while the VM is stopped, or by editing the file, applying, and rebooting the node. Patching a running VM's document is what crashed the host on 2026-11-09.

What survives talosctl reset --system-labels-to-wipe EPHEMERAL? ​

The machine config, the PKI and the talosconfig (STATE partition). Pools, libraries and VM disks live under /var on EPHEMERAL and must be re-created and re-uploaded.

Is the install really zero-touch? ​

Yes: with autoinstall on the kernel command line, the "Continue with autoinstall?" prompt is skipped and subiquity runs the CIDATA seed to completion. Watch it anyway over VNC or the serial console, because a wrong digest or a missing autoinstall flag fails at boot without an obvious error.

Read more ​