8 November 2023

Day3 creating a simulacrul

by mpagot

This day is spent in creating a playground to test different setups.

DONE

Today I played with:

KVM and libvirt
MicroOS
ignition and combustion (mam say “Do not play with fire, but …”)
BTRFS cli tools: mkfs.btrfs/mount/btrfs

The general idea is to play with different FS configuration in a virtualized environment. I’m not sure how this one is similar or not in term of performance but at list it is super flexible and hopefully fast.

At some point I also made some decisions about how to proceed. I’d like to give a try to MicroOS as it is immutable and should be relatively minimal. I’m mostly interested in the immutable part of that: I think it can improve system robustness (rollback).

Ignition and combustion

Some good reading:

HowTo: Ignition_Quick_Start
How to use ignition with virt-install install-opensuse-microos-kvm-ignition
How to use combustion installing-opensuse-microos-and-tumbleweed-in-qemu

Ignition ‘n combustion

They are one alternative to the other. Ignition is a json file and combustion is a bash script. So ignition is maybe a little bit simpler but less flexible. Let’s use it to:

configure the password for root
allow me to ssh to the machine using my ssh key

Before to start I need to do something about the root password. The ignition file needs its hash. It can be created using:

% openssl passwd -6

Password:
Verifying - Password:
$6$hJ2svTtl37mM/tRL$lWEdUzMOpsqLrJLtTqp2D/WEmZ9O0Jg4gs6RGClnc82c0Ady8iGdjCe1u6TtUFutKlIZc2wB0EB4f9fiRVTOD.

A good entry point to create an ignition file is fuel-ignition: is is a web base ignition file editor (again thanks Mr Lu).

Here as reference my config.ign file

{
  "ignition": {
    "version": "3.2.0"
  },
  "passwd": {
    "users": [
      {
        "name": "root",
        "passwordHash": "**********",
        "sshAuthorizedKeys": [
          "ssh-rsa AA*********\n"
        ]
      }
   ]
  },
  "storage": {
    "filesystems": [
      {
        "device": "/dev/disk/by-label/ROOT",
        "format": "btrfs",
        "mountOptions": [
          "subvol=/@/home"
        ],
        "path": "/home",
        "wipeFilesystem": false
      }
    ]
  },
  "systemd": {
    "units": [
      {
        "name": "sshd.service",
        "enabled": true
      }
    ]
  }
}

A small note about the storage section: it seems only to be needed due to the MicroOS configuration. There’s this note in the fuel-ignition webpage

This toggle is about adding a section in the ignition json like:

  "storage": {
    "filesystems": [
      {
        "device": "/dev/disk/by-label/ROOT",
        "format": "btrfs",
        "mountOptions": [
          "subvol=/@/home"
        ],
        "path": "/home",
        "wipeFilesystem": false
      }
    ]
  },

Not selecting it result in ssh as root not to work even if the deployment looks like ok. Digging in the log of the first boot:

How this problem has to be addressed using combustion?

virt-install

First of all I need an image. I have many possibility here:

I can try Leap Micro or MicroOS. They are more or less like Leap vs Tumbleweed but for immutable OS. I decided for the rolling release one
I can get an ISO and connect it to the VM as a cdrom and perform the installation on an empty qcow2 disk. Or I can directly get the image qcow2 and attach it like a disk to the VM.

Install from ISO

The ISO is here

Install from qcow2

Ignition and combustion in KVM

Provision as disk

In this article there’s a section about doing it in a physical machine with a USB driver

But I’m more interested in doing it for a VM

Create a small disk image

truncate --size=10M ignition_disk.img

Create a filesystem in it

mkfs.vfat -n ignition ignition_disk.img

Mount to fill the disk with the ignition json file

mount ignition_disk.img mnt/
mkdir mnt/ignition
cp config.ign mnt/ignition/
umount mnt

or for the combustion method:

mkdir mnt/combustion
cp script mnt/combustion/

At this point umount and convert the .img disk image to a qcow2

qemu-img convert -O qcow2 ./ignition_disk.img ./ignition_disk.qcow2

(credit Mr.Lu)

Provision as virt-install argument

It is an alternative method and probably a simpler one. This method needs a different syntax:

if using ignition or combustion
if booting from .iso or .qcow2

Install from ISO with ignition is done with something like:

% virt-install ... \
    --boot cdrom --cdrom openSUSE-MicroOS.x86_64-ContainerHost-SelfInstall.iso\
    --sysinfo type=fwcfg,entry0.name="opt/com.coreos/config",entry0.file="/var/lib/libvirt/images/config.ing"

Or using the qcow2 and ignition

% virt-install ... \
    --boot hd --disk size=20,backing_store=openSUSE-MicroOS.x86_64-kvm-and-xen.qcow2,bus=virtio,serial=os \
    --qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=/var/lib/libvirt/images/config.ign"

Or using the qcow2 and combustion

% virt-install ... \
    --boot hd --disk size=20,backing_store=openSUSE-MicroOS.x86_64-kvm-and-xen.qcow2,bus=virtio,serial=os \
    --qemu-commandline="-fw_cfg name=opt/org.opensuse.combustion/script,file=/var/lib/libvirt/images/script"

Play with BTRFS

Create a raid pool

RAID in BTRFS start at the formatting command, where multiple devices can be provided to it to create a pool.

BTRFS support different kind of RAID mode with different level of stability.

BTRFS allow to create a RAID pool for the data or for the metadata separately.

% mkfs.btrfs -f -L "Arca" -d raid1 -m raid1 /dev/sda /dev/sdb /dev/sdc

btrfs-progs v6.5.1
See https://btrfs.readthedocs.io for more information.

Performing full device TRIM /dev/sdc (30.00GiB) ...
Performing full device TRIM /dev/sda (30.00GiB) ...
Performing full device TRIM /dev/sdb (30.00GiB) ...
NOTE: several default settings have changed in version 5.15, please make sure
      this does not affect your deployments:
      - DUP for metadata (-m dup)
      - enabled no-holes (-O no-holes)
      - enabled free-space-tree (-R free-space-tree)

Label:              Arca
UUID:               5ce9855b-9680-469a-b790-4297e6f73648
Node size:          16384
Sector size:        4096
Filesystem size:    90.00GiB
Block group profiles:
  Data:             RAID1             1.00GiB
  Metadata:         RAID1             1.00GiB
  System:           RAID1             8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, skinny-metadata, no-holes, free-space-tree
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  3
Devices:
   ID        SIZE  PATH
    1    30.00GiB  /dev/sda
    2    30.00GiB  /dev/sdb
    3    30.00GiB  /dev/sdc

Now it is time to mount it. It works by just mounting one of the devices

% mount /dev/sda /mnt/

At this point

% lsblk -f
NAME   FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda    btrfs        Arca  5ce9855b-9680-469a-b790-4297e6f73648     29G     2% /mnt
sdb    btrfs        Arca  5ce9855b-9680-469a-b790-4297e6f73648
sdc    btrfs        Arca  5ce9855b-9680-469a-b790-4297e6f73648

notice as UUID is the same across all devices in the RAID array

Play with some btrfs commands

device scan

% btrfs device scan

Scanning for Btrfs filesystems
registered: /dev/vda3
registered: /dev/sdb
registered: /dev/sdc
registered: /dev/sda

device stats

% btrfs device stats /dev/sda

[/dev/sda].write_io_errs    0
[/dev/sda].read_io_errs     0
[/dev/sda].flush_io_errs    0
[/dev/sda].corruption_errs  0
[/dev/sda].generation_errs  0

filesystem show

% btrfs filesystem show /mnt/
Label: none  uuid: a0c0cb3b-6eed-48cd-ad13-6658dd7818d2
        Total devices 3 FS bytes used 144.00KiB
        devid    1 size 30.00GiB used 0.00B path /dev/sda
        devid    2 size 30.00GiB used 2.01GiB path /dev/sdb
        devid    3 size 30.00GiB used 2.01GiB path /dev/sdc

filesystem usage

Keep in mind that results from df and du are not so important when using BTRFS. The native BTRFS alternative is:

% btrfs filesystem usage /mnt/

Overall:
    Device size:                  19.98GiB
    Device allocated:              3.52GiB
    Device unallocated:           16.45GiB
    Device missing:                  0.00B
    Device slack:                  3.50KiB
    Used:                          1.21GiB
    Free (estimated):             18.31GiB      (min: 10.08GiB)
    Free (statfs, df):            18.31GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:                5.50MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:3.01GiB, Used:1.15GiB (38.25%)
   /dev/vda3       3.01GiB

Metadata,DUP: Size:256.00MiB, Used:31.27MiB (12.21%)
   /dev/vda3     512.00MiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
   /dev/vda3      16.00MiB

Unallocated:
   /dev/vda3      16.45GiB

Scrub

% btrfs scrub start /mnt
scrub started on /mnt, fsid 56c0cabd-349e-4ad1-8bfe-cb25fcf5ce89 (pid=1986)


% btrfs scrub status /mnt
UUID:             56c0cabd-349e-4ad1-8bfe-cb25fcf5ce89
Scrub started:    Wed Nov  8 15:41:01 2023
Status:           finished
Duration:         0:00:00
Total to scrub:   304.00KiB
Rate:             0.00B/s
Error summary:    no errors found

You can also scrub per device

Balance

% btrfs balance start --full-balance /mnt/
Done, had to relocate 3 out of 3 chunks

tags: