How to Convert a Debian 11 x86 Install to ARM on AWS

by Ryan Thoryk -
8/28/21 version

This document details how to switch an existing Debian installation to a different processor architecture (crossgrading, this document inspired this). This howto uses Amazon AWS, the source architecture AMD64 and the destination architecture ARM64, making it an x86 to ARM conversion. I wrote this document because I wanted to see if I could convert a complex amd64 VM directly to arm64 without having to do a reinstall/migration. This should work for other architectures too, I'd love to do it for RISC-V. The conversion process is fairly straightforward, the only real trouble is wrestling with the package manager.

As for me, I've been using Linux since August 1998 (the original RedHat 5.1), and Debian since July 2002 (right when Debian 3.0 Woody came out). I use Debian exclusively as my main OS, and have done so for a decade now. I used to work with many other architectures, such as PowerPC, SPARC, MIPS, Alpha, and PA-RISC, and also commercial UNIX for those such as Sun Solaris and SGI IRIX.

This requires the VM instance to have an EFI partition, verify with the df command. If your system doesn't have one, you'll need to rebuild your root volume to include one.
This was written for Debian Linux, but should work fine on Debian derivatives such as Ubuntu or Mint. This won't work with RedHat/CentOS/Fedora/Rocky.

First, log into the x86/amd64 VM that you want to convert. This was done for a newly created Debian 11 VM. Make sure it's backed up first. Be very careful when doing this conversion, mistakes could result in an unbootable/broken system.
Fix /etc/apt/sources.list and comment out backports entries.
Before you begin, upgrade the system. I also usually install aptitude, which I use for advanced package management tasks.

If you don't believe this can work, basically it's possible both due to the fact that Debian supports having multiple architectures of packages installed simulaneously (multiarch, introduced in Debian 7), and also qemu-user-static allows the Linux kernel to run binaries of other architectures as if they were native executables. If you think about it, it's really not that complicated. The theory behind it is that if you switch all binaries including the kernel over to the other architecture and get grub working, it should work, and it does.

Do this at your own risk, since you can really mess up your Linux installation if things go wrong.

First, make sure you're root.

Add arm64 as an architecture:

# dpkg --print-architecture
# dpkg --add-architecture arm64
# apt update
# dpkg --print-foreign-architectures

In order to run ARM binaries on the system, you need to install qemu-user-static for emulation:

# apt install qemu-user-static
The system can now execute arm64 binaries under system-level emulation. This also installs binfmt-support, which adds support for executing foreign binaries.

Test arm execution by doing a (I decided to use bzip for this):

# apt install bzip2:arm64
# apt clean
# file /usr/bin/bzip2
/usr/bin/bzip2: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV)...
# bzip2
bzip2: I won't write compressed data to a terminal.
bzip2: For help, type: `bzip2 --help'.
This should report as an arm binary, and should appear to run natively. It will uninstall the amd64 bzip2 and install the arm64 version.

Switch package manager to arm, this causes arm64 to become the default architecture for the dpkg package manager:

# apt --download-only install dpkg:arm64 tar:arm64 apt:arm64 apt-utils:arm64
# dpkg --install /var/cache/apt/archives/*.deb
Usually you need to run dpkg command twice.
# apt clean
# dpkg --print-architecture


# apt install aptitude
the package manager should report as broken and suggest a remedy.

This step was more simple with the Debian 10 installation, but with 11, Perl and Python break. The workaround I found was to force-install the related environments.
Use the download command to grab any extra packages that might be needed, for a basic VM install, this is what I found as being needed:
# dpkg --purge python3-pyrsistent:amd64
# apt-get download libfile-fcntllock-perl liblocale-gettext-perl perl libperl5.32 python3 python3-cffi-backend python3-minimal
# apt-get download perl-base python3.9-minimal libgdbm-compat4 libgdbm6 libexpat1 libpython3.9-minimal python3.9
# apt-get download libpython3-stdlib libpython3.9-stdlib python3-cffi-backend python3-pyrsistent
# dpkg -i *.deb
Run the dpkg command twice, and it'll result in a broken state. Have the package manager clean up the mess:
# apt --fix-broken install
That appears to fix it for me. Be careful with that command, and make sure it doesn't uninstall any important packages.
I did a test-run of a MySQL/MariaDB server install, and it requested to remove those packages. The only workaround I found was to let it remove them, and reinstall them afterwards.
After the package reinstallation, MariaDB came up on ARM as if nothing had changed.
# apt clean
# rm *.deb

Switch system base over to arm:

# arch
# apt install coreutils base-files
# arch

Make list of packages that need switching to arm:

# dpkg -l | grep amd64 | cut -d ' ' -f 3 | cut -d ':' -f 1 > out
# vi out

Remove qemu-user-static, and binfmt-support from the list, those will kill the OS otherwise due to the removal of arm emulation.
Remove grub packages from the list, this will be done later.
Remove linux-image packages from the list, this will be done later.
Remove shim packages from the list, these are part of grub.

Install the list of packages that you just made.

# apt --download-only install `cat out`
# dpkg -i /var/cache/apt/archives/*.deb
Run the dpkg command twice. You should get an error with udev.
System will be running fairly slow now due to most of it being under emulation.

Clean packages:

# apt clean
# rm out

System still runs due to /lib being a multiarch design (it contains both x86 and arm libraries side-by-side), while almost all binaries on system are arm.

Doing a:

# file /usr/bin/* | grep x86-64 | less
should show grub, systemd and qemu as the only non-arm binaries.

Install kernel for arm:

# apt install linux-image-arm64

Verify that a kernel for each architecture exists:

# ls -l /boot

Switch grub to arm:

# dpkg --purge grub-cloud-amd64 grub-pc-bin grub-efi-amd64-bin shim-signed shim-unsigned shim-helpers-amd64-signed
# apt install grub-efi-arm64 grub-common:arm64 grub2-common:arm64
# apt clean
optionally remove grub-efi-arm64-signed package.

To match Debian arm setup:

-create a file /etc/default/grub.d/20_console.cfg with contents:

# Add a serial console


To fix a problem with Grub, first reconfigure Grub to make sure it installs the correct EFI boot loader:

# dpkg-reconfigure grub-efi-arm64

That command will show some menus. On the screen that asks about "Force extra installation to the EFI removable media path?" choose Yes, otherwise you might end up with an unbootable system.

Regenerate grub config:

# /usr/sbin/update-grub

The binfmt-support package is no longer needed, but make sure you keep it until after the system boots onto arm hardware, otherwise if you remove it now, it'll kill the running os.

Remove the running amd64 kernel, this needs to be done because grub uses the amd64 one as default instead of arm64:

# dpkg --purge linux-image-cloud-amd64 linux-image-5.10.0-8-cloud-amd64

If you do a package analysis of the system, you will find that binfmt-support, qemu-user-static, and systemd are the only things left on amd64:

# dpkg -l | grep amd64 | cut -d ' ' -f 3 | cut -d ':' -f 1 > amd64
# dpkg -l | grep arm64 | cut -d ' ' -f 3 | cut -d ':' -f 1 > arm64
# diff -u amd64 arm64 | less
# rm amd64 arm64
(look for lines starting with minus signs)

Verify that systemd is now running under arm emulation:
# ps -ef | grep systemd
systemd's init process will look like this:
root         1     0  0 10:52 ?        00:00:12 /usr/libexec/qemu-binfmt/aarch64-binfmt-P /lib/systemd/systemd --system --deserialize 32

On the system, if you do a ps -ef, you'll notice that every running service is prepended with qemu for arm emulation, such as:

# ps -ef | grep sshd
root     11089     1  0 15:27 ?        00:00:00 /usr/libexec/qemu-binfmt/aarch64-binfmt-P /usr/sbin/sshd -D

Hopefully all goes well now. Time to shut down the system:

# halt -p

Congratulations, you just successfully bricked your amd64 Debian system.
Since the system is now unbootable, don't try to start it up before switching architectures, if you do, it won't do anything at all.

From the AWS console, detach the disk volume from the instance.
Then, start up a new Debian 11 arm instance and shut it down immediately. Note the device name of the disk, mine is "/dev/xvda".
Detach and delete the disk volume from that instance, since it's not needed.
Attach the original disk you worked on before to the arm instance as /dev/xvda, then start the instance.

Another congratulations, your previously-amd64 system should now be running natively on ARM hardware. SSH should work for you now, otherwise check the serial console for errors.

Finally, force-purge the amd64 packages from your system. Make sure you've migrated all the packages over to arm first, and keep a backup of the /etc folder just in case, before doing this.

# dpkg --purge --force-all `dpkg -l | grep amd64 | cut -d ' ' -f 3`

The conversion to ARM is now complete.

Final step, remove amd64, skip this step if you want to still run x86 applications on the system:
# dpkg --remove-architecture amd64
# apt update

After your system is running, if you want to run x86 applications on it (amd64 on arm64, the opposite of before), install the qemu-user-static application again:

# apt install qemu-user-static

If you're interested in knowing, for the original Debian 10 guide, I had 2 failed attempts at this architecture conversion (disposable VMs are great), the first was when I accidentally switched the binfmt-support package which killed the emulation and the OS with it, the second one was when I switched systemd over without making a root login for the serial console, and lost ssh access and the ability to log in. On the third try, grub wouldn't boot, but I found that it was due to a bug, copied over the older modules, and that got it working.

The Debian 11 guide resulted in more broken VMs, mostly due to the Perl/Python packages getting really screwed up, until I came up with the method of force-installing most of them.