Customize a Raspberry Pi image without any hardware

Customizing SD card images is the worst—having to boot the device over and over again, and then grabbing the SD card to copy it back to your system, only to repeat it all over again the next time you need to make a change. Bleh.

Beyond that, this causes all kinds of problems in your operations story:

How would you reproduce your card image if you lost it?
How would you patch, upgrade, or otherwise modify the base image?
How do you know what is actually being delivered to customers?

I'm here to tell you that there's a better way! In this article, I'll show you a method for making all the changes you want to your embedded Linux operating system image without any actual hardware, even if the target architecture is different.

Yes, I am serious.

You'll learn how to prepare a chroot environment that will allow you to run commands inside your Raspberry Pi image as if you had the hardware running at your desk. It'll be possible to:

Install packages,
Customize the boot config,
Set up SSH keys,
And who knows what else!

The only limit is your imagination, and you can do it all right from the comfort of your computer.

This recipe targets Raspberry Pi because of its ubiquity, but can be adapted to almost any SD card image.

Prerequisites

A VirtualBox environment.
Vagrant installed.

Set up a Vagrant box

We'll complete the steps in this article inside a Vagrant virtual machine, because when you make mistakes while dealing with filesystem images, chances are, you'll end up with hung devices and needing to reboot often to recover.

Using Vagrant allows us to reboot a virtual machine instead and not disrupt our flow. It also means that you can complete this tutorial on any host machine, running Linux or not.

Create the following Vagrantfile in an empty directory, modifiying the values for vb.cpus and vb.memory as needed. Be sure to give the box as much oomph as you can spare, as your computer will get hungry when compressing and uncompressing the image:

# Vagrantfile
Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/jammy64"
  config.vm.provider "virtualbox" do |vb|
    vb.cpus = 3
    vb.memory = "4096"
  end
end

With the Vagrantfile in place, start your Vagrant box and log in to it:

vagrant up
vagrant ssh

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ubuntu/jammy64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'ubuntu/jammy64' version '20230302.0.0' is up to date...

...
...
    default: /vagrant => /home/brett/Projects/examples/card-image-ci
$ vagrant ssh
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-67-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
...
vagrant@ubuntu-jammy:~$

Download the image

Go to the Raspberry Pi download page and find one you like. I chose Raspberry Pi OS Lite because the download is much smaller and I can add whatever I like to it. I also chose 64-bit because I have a newer Raspberry Pi.

You can download the exact image I chose by doing the following:

wget --progress=bar:noscroll https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2023-05-03/2023-05-03-raspios-bullseye-armhf-lite.img.xz

vagrant@ubuntu-jammy:~$ wget --progress=bar:noscroll https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2023-05-03/2023-05-03-raspios-bullseye-armhf-lite.img.xz
--2023-06-24 02:57:30--  https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2023-05-03/2023-05-03-raspios-bullseye-armhf-lite.img.xz
Resolving downloads.raspberrypi.org (downloads.raspberrypi.org)... 176.126.240.86, 46.235.230.122, 93.93.135.117, ...
Connecting to downloads.raspberrypi.org (downloads.raspberrypi.org)|176.126.240.86|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 381558864 (364M) [application/x-xz]
Saving to: ‘2023-05-03-raspios-bullseye-armhf-lite.img.xz’

2023-05-03-raspios-bullseye-armhf-lite. 100%[=============================================================================>] 363.88M  9.43MB/s    in 43s

2023-06-24 02:58:14 (8.44 MB/s) - ‘2023-05-03-raspios-bullseye-armhf-lite.img.xz’ saved [381558864/381558864]

Set up the filesystem

With the image downloaded, you're ready to prepare the image for access.

Uncompress the image

The first step is to uncompress the image if needed. The Raspberry Pi image downloaded above is compressed in the .xz format, so you'll need the xz and unxz commands. These should be pre-installed on the Vagrant box, but if they are not, they're easy to install:

sudo apt-get install -y xz-utils

Uncompress the image. You can add the -v flag to print the progress in real-time:

unxz -v 2023-05-03-raspios-bullseye-armhf-lite.img.xz

vagrant@ubuntu-jammy:~$ unxz -v 2023-05-03-raspios-bullseye-armhf-lite.img.xz
2023-05-03-raspios-bullseye-armhf-lite.img.xz (1/1)
  100 %      363.9 MiB / 1876.0 MiB = 0.194    44 MiB/s       0:42

You can tell it's uncompressed because it's now enormous, lol:

du -hs 2023-05-03-raspios-bullseye-armhf-lite.img

vagrant@ubuntu-jammy:~$ du -hs 2023-05-03-raspios-bullseye-armhf-lite.img
1.3G    2023-05-03-raspios-bullseye-armhf-lite.img

Resize the image (optional)

If you're going to customize an OS image, you'll probably hit the existing disk size limit pretty quickly.

Install the amazing qemu-img utility:

sudo apt-get install -y qemu-utils

Inspect the image to find out what you're working with:

qemu-img info 2023-05-03-raspios-bullseye-armhf-lite.img

vagrant@ubuntu-jammy:~$ qemu-img info 2023-05-03-raspios-bullseye-armhf-lite.img
image: 2023-05-03-raspios-bullseye-armhf-lite.img
file format: raw
virtual size: 1.83 GiB (1967128576 bytes)
disk size: 1.37 GiB

If you wanted to install, let's say, LibreOffice or something, it would need to be quite a bit larger. Let's resize the image file itself:

qemu-img resize 2023-05-03-raspios-bullseye-armhf-lite.img +2G

vagrant@ubuntu-jammy:~$ qemu-img resize -f raw 2023-05-03-raspios-bullseye-armhf-lite.img +2G
Image resized.

But that only resizes the image. You still need to grow the root partition that has all the stuff, and then expand the filesystem to fill the partition.

Let's find out which partition is which:

fdisk -l 2023-05-03-raspios-bullseye-armhf-lite.img

vagrant@ubuntu-jammy:~$ fdisk -l 2023-05-03-raspios-bullseye-armhf-lite.img
Disk 2023-05-03-raspios-bullseye-armhf-lite.img: 3.83 GiB, 4114612224 bytes, 8036352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x4c4e106f

Device                                      Boot  Start     End Sectors  Size Id Type
2023-05-03-raspios-bullseye-armhf-lite.img1        8192  532479  524288  256M  c W95 FAT32 (LBA)
2023-05-03-raspios-bullseye-armhf-lite.img2      532480 3842047 3309568  1.6G 83 Linux

Looks like the second partition is what we want. Let's use the growpart command to expand it:

growpart 2023-05-03-raspios-bullseye-armhf-lite.img 2

vagrant@ubuntu-jammy:~$ growpart 2023-05-03-raspios-bullseye-armhf-lite.img 2
CHANGED: partition=2 start=532480 old: size=3309568 end=3842048 new: size=7503839 end=8036319

fdisk will now report the expanded size, which grew from 1.6G to 3.6G, as expected:

vagrant@ubuntu-jammy:~$ fdisk -l 2023-05-03-raspios-bullseye-armhf-lite.img
Disk 2023-05-03-raspios-bullseye-armhf-lite.img: 3.83 GiB, 4114612224 bytes, 8036352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x4c4e106f

Device                                      Boot  Start     End Sectors  Size Id Type
2023-05-03-raspios-bullseye-armhf-lite.img1        8192  532479  524288  256M  c W95 FAT32 (LBA)
2023-05-03-raspios-bullseye-armhf-lite.img2      532480 8036318 7503839  3.6G 83 Linux

Later on, after we mount the loopback devices, we'll be able to grow the filesystem to use the newly available space.

Add loopback devices

The next tool you're going to need is the losetup command, which allows you to manage loopback devices. We'll use it to create new loopback devices for the card image partitions and mount them to directories.

In my case, the first available device was /dev/loop6, but your specific device number may differ whenever this command is called. The losetup command returns the loopback device path, so we'll assign it to a variable to avoid referring to it directly:

DEVICE=$(sudo losetup -f --show -P 2023-05-03-raspios-bullseye-armhf-lite.img)
echo $DEVICE

vagrant@ubuntu-jammy:~$ DEVICE=$(sudo losetup -f --show -P 2023-05-03-raspios-bullseye-armhf-lite.img)
vagrant@ubuntu-jammy:~$ echo $DEVICE
/dev/loop6

You can use lsblk to inspect the newly available loopback devices:

sudo lsblk -o name,label,size $DEVICE

vagrant@ubuntu-jammy:~$ sudo lsblk -o name,label,size $DEVICE
NAME      LABEL   SIZE
loop6             3.8G
├─loop6p1 bootfs  256M
└─loop6p2 rootfs  3.6G

bootfs and rootfs are classic Raspberry Pi device labels, so it's looking like we're in good shape.

If you forget which device you had, run losetup -l to find it again:

losetup -l

vagrant@ubuntu-jammy:~$ losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                                                DIO LOG-SEC
/dev/loop1         0      0         1  1 /var/lib/snapd/snaps/lxd_24322.snap                        0     512
/dev/loop4         0      0         1  1 /var/lib/snapd/snaps/snapd_19457.snap                      0     512
/dev/loop2         0      0         1  1 /var/lib/snapd/snaps/core20_1950.snap                      0     512
/dev/loop0         0      0         1  1 /var/lib/snapd/snaps/core20_1822.snap                      0     512
/dev/loop6         0      0         0  0 /home/vagrant/2023-05-03-raspios-bullseye-armhf-lite.img   0     512
/dev/loop3         0      0         1  1 /var/lib/snapd/snaps/snapd_18357.snap                      0     512

In this case, it's /dev/loop6. Set the $DEVICE variable manually to continue:

DEVICE=/dev/loop6

If you opted to resize the image, you can grow the filesystem to its new size by running e2fsck and then resize2fs:

sudo e2fsck -f ${DEVICE}p2
sudo resize2fs ${DEVICE}p2

vagrant@ubuntu-jammy:~$ sudo e2fsck -f ${DEVICE}p2
e2fsck 1.46.5 (30-Dec-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
rootfs: 49948/103584 files (0.1% non-contiguous), 341565/413696 blocks
vagrant@ubuntu-jammy:~$ sudo resize2fs ${DEVICE}p2
resize2fs 1.46.5 (30-Dec-2021)
Resizing the filesystem on /dev/loop5p2 to 937979 (4k) blocks.
The filesystem on /dev/loop5p2 is now 937979 (4k) blocks long.

Your filesystem is now its full size.

Mount partitions

The disk image is ready to be mounted to your local filesystem so that it appears as any other directory would, except that it will be the root filesystem of your Raspberry Pi image. Let's go!

You'll want to mirror the final filesystem layout as much as possible. That means opening up the root filesystem, finding /etc/fstab, and seeing what it has first.

Create the rootfs directory:

mkdir -p rootfs

Mount the rootfs partition onto the rootfs/ directory:

sudo mount ${DEVICE}p2 rootfs/

You should now be able to inspect the filesystem:

ls rootfs/

vagrant@ubuntu-jammy:~$ ls rootfs/
bin  boot  dev  etc  home  lib  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

From there, you can discover the /etc/fstab file:

cat rootfs/etc/fstab

vagrant@ubuntu-jammy:~$ cat rootfs/etc/fstab
proc            /proc           proc    defaults          0       0
PARTUUID=4c4e106f-01  /boot           vfat    defaults          0       2
PARTUUID=4c4e106f-02  /               ext4    defaults,noatime  0       1

This is promising! All we're missing are the contents of the rootfs/boot/ directory, which we might have guessed because the bootfs partition hasn't been mounted yet. Easy peasy—there's even already an empty rootfs/boot/ directory waiting for us, which we can make sure by doing the following:

ls rootfs/boot/

vagrant@ubuntu-jammy:~$ ls rootfs/boot/
vagrant@ubuntu-jammy:~$

So let's mount the bootfs partition there:

sudo mount ${DEVICE}p1 rootfs/boot/

All the fun Raspberry Pi stuff is there now:

vagrant@ubuntu-jammy:~$ ls rootfs/boot/
COPYING.linux           bcm2708-rpi-zero.dtb      bcm2710-rpi-zero-2-w.dtb  bootcode.bin  fixup4x.dat   kernel7l.img  start4x.elf
LICENCE.broadcom        bcm2709-rpi-2-b.dtb       bcm2710-rpi-zero-2.dtb    cmdline.txt   fixup_cd.dat  kernel8.img   start_cd.elf
bcm2708-rpi-b-plus.dtb  bcm2709-rpi-cm2.dtb       bcm2711-rpi-4-b.dtb       config.txt    fixup_db.dat  overlays      start_db.elf
bcm2708-rpi-b-rev1.dtb  bcm2710-rpi-2-b.dtb       bcm2711-rpi-400.dtb       fixup.dat     fixup_x.dat   start.elf     start_x.elf
bcm2708-rpi-b.dtb       bcm2710-rpi-3-b-plus.dtb  bcm2711-rpi-cm4-io.dtb    fixup4.dat    issue.txt     start4.elf
bcm2708-rpi-cm.dtb      bcm2710-rpi-3-b.dtb       bcm2711-rpi-cm4.dtb       fixup4cd.dat  kernel.img    start4cd.elf
bcm2708-rpi-zero-w.dtb  bcm2710-rpi-cm3.dtb       bcm2711-rpi-cm4s.dtb      fixup4db.dat  kernel7.img   start4db.elf

Mount special host filesystems

Some commands you'll want to run, like those for installing packages, enabling services, or even connecting to the Internet, may require /dev, /proc, and /sys filesystems to be mounted inside the rootfs/ directory. They should currently be empty, except for rootfs/dev/:

ls rootfs/sys/
ls rootfs/proc/
ls rootfs/dev/

vagrant@ubuntu-jammy:~$ ls rootfs/sys/
vagrant@ubuntu-jammy:~$ ls rootfs/proc/
vagrant@ubuntu-jammy:~$ ls rootfs/dev/
console  fd  full  null  ptmx  pts  random  shm  stderr  stdin  stdout  tty  urandom  zero

Once you've confirmed those directories exist and are empty, mount the filesystems by doing the following:

sudo mount -t proc /proc rootfs/proc/
sudo mount --bind /sys rootfs/sys/
sudo mount --bind /dev rootfs/dev/

Now those directories will be populated with a ton of magic stuff:

vagrant@ubuntu-jammy:~$ ls rootfs/sys/
block  bus  class  dev  devices  firmware  fs  hypervisor  kernel  module  power
vagrant@ubuntu-jammy:~$ ls rootfs/proc/
1     112   131   175  221  32   408  645  852  98          diskstats      kallsyms     misc          slabinfo       version_signature
10    1131  14    18   24   33   409  646  853  99          dma            kcore        modules       softirqs       vmallocinfo
102   1132  15    187  25   34   410  649  86   acpi        driver         key-users    mounts        stat           vmstat
...
vagrant@ubuntu-jammy:~$ ls rootfs/dev/
autofs           fd            loop3    port    shm       tty15  tty28  tty40  tty53  tty9    ttyS20  ttyS5      vcs3   vcsu3
block            full          loop3p1  ppp     snapshot  tty16  tty29  tty41  tty54  ttyS0   ttyS21  ttyS6      vcs4   vcsu4
bsg              fuse          loop3p2  psaux   snd       tty17  tty3   tty42  tty55  ttyS1   ttyS22  ttyS7      vcs5   vcsu5
...

Enable ARM virtualization

At this point, we're almost ready to chroot into the environment. If we tried to use the Pi filesystem right now, we'd discover that none of the commands work:

./rootfs/bin/bash

vagrant@ubuntu-jammy:~$ ./rootfs/bin/bash
-bash: ./rootfs/bin/bash: cannot execute binary file: Exec format error

That's because the Raspberry Pi is an ARM platform, and our machine is x86-64.

That's okay though. We can add qemu-user-static to the filesystem so that ARM binaries run via QEMU when executed on our system:

sudo apt-get install -y qemu-user-static

If we run the ARM bash command again, we'll get... a different error!

vagrant@ubuntu-jammy:~$ ./rootfs/bin/bash
arm-binfmt-P: Could not open '/lib/ld-linux-armhf.so.3': No such file or directory

That's good. This error is different and expected because the bash binary is dynamically linked and has no way of linking to its nice ARM friends right now. We'll fix that next, when we finally chroot into our system.

Access the filesystem

It's the moment you've been waiting for. Let's run our fake Raspberry Pi!

You can "log in" to your filesystem with chroot, which will start an instance of the default shell:

sudo chroot rootfs/

$ sudo chroot rootfs/
root@ubuntu-jammy:/#

Wow, so cool! I can go home sweet home!

root@ubuntu-jammy:/# cd
root@ubuntu-jammy:~# ls
root@ubuntu-jammy:~# pwd
/root

I can more or less use it as though it were my machine. Give it a try.

You can run raspi-config, because why not:

raspi-config

Running `raspi-config` on my "Raspberry Pi".

You can install vim, because it's everyone's favorite code editor, right?

apt-get update -y
apt-get install -y vim

You can edit the hostname:

echo "my-cool-computer" > /etc/hostname
cat /etc/hostname

root@ubuntu-jammy:/# cat /etc/hostname
old-hostname
root@ubuntu-jammy:/# echo "my-cool-computer" > /etc/hostname
root@ubuntu-jammy:/# cat /etc/hostname
my-cool-computer

We can exit the chroot environment with exit:

exit

root@ubuntu-jammy:/# exit
exit
vagrant@ubuntu-jammy:~$

Pretty much anything that you could configure on the actual device can now be configured beforehand without the need to set up any hardware, which is pretty great.

It doesn't stop at hand edits either. You can pipe scripts into the chroot environment, if you have a set of tasks that you need to run often:

echo "ls /etc" | sudo chroot rootfs/ bash -

vagrant@ubuntu-jammy:~$ echo "ls -1 /etc" | sudo chroot rootfs/ bash -
NetworkManager
X11
adduser.conf
alternatives
apparmor.d
...

Or you can get even fancier. Ansible, for example, has a chroot connection plugin, so you can run Ansible playbooks to configure your Pi filesystem—again, without the need for actual hardware.

Anyway, let's assume we've set everything up the way we want. Yay! But we're not done yet. This filesystem won't do us any good until we repack it to be actually copied onto a real SD card. That comes next.

Tear down the filesystem

We now need to repeat everything we just did, but in the reverse order to get back to a packed-up, ready-to-ship card image.

Unmount filesystems

Unmount all the special filesystems we mounted:

sudo umount rootfs/dev/
sudo umount rootfs/sys/
sudo umount rootfs/proc/

Unmount the loopback devices for the bootfs and rootfs partitions:

sudo umount rootfs/boot/
sudo umount rootfs/

Delete loopback devices

Use losetup to detach the loopback devices that you created earlier to sever the final link between your disk image and your host operating system:

sudo losetup -d $DEVICE

Compress the image

Now that everything is disconnected, you can repack the SD card image with xz. This might take a significant amount of time, during which time your computer will purr like an angry kitten. Be patient, and add the -v flag if you like having literally any feedback on what's happening:

xz -v 2023-05-03-raspios-bullseye-armhf-lite.img

vagrant@ubuntu-jammy:~$ xz -v 2023-05-03-raspios-bullseye-armhf-lite.img
2023-05-03-raspios-bullseye-armhf-lite.img (1/1)
  0.8 %         26.3 MiB / 32.3 MiB = 0.815   3.2 MiB/s       0:10

Eventually, it should finish. And then, huzzah! Look, our 3.6G image is compressed to 421MB:

du -hs 2023-05-03-raspios-bullseye-armhf-lite.img.xz

vagrant@ubuntu-jammy:~$ du -hs 2023-05-03-raspios-bullseye-armhf-lite.img.xz
421M    2023-05-03-raspios-bullseye-armhf-lite.img.xz

Ship it!

Your disk image is now in the exact same state that it was in when you downloaded it, except that it's actually not. The filesystem is now much larger and has all your super cool modifications ready to deploy to devices anywhere in the world.

Upload it to your FTP server, post it on USENET, or even put it on an actual physical SD card. Whatever you do, don't keep it to yourself, because the world needs your custom distribution of Sonic Pi, or whatever it was that you needed to customize your Pi image for.

Cleanup

If you want to save the card image you created, move it into the /vagrant/ directory. It will then be in the same directory as your Vagrantfile:

mv *.xz /vagrant/

Then exit the vagrant ssh session:

exit

vagrant@ubuntu-jammy:~$ exit
exit
$

Finally, destroy the box you created:

vagrant destroy

Conclusion

The tools and techniques discussed in this article are generally applicable to all kinds of embedded Linux disk images, not just Raspberry Pi. Once you know the filesystem layout, anything is possible!

You can even take all these instructions and dump them into a CI pipeline so that you never have to do this ever again!

Happy coding!

Tags