How to Convert a Debian 10 x86 Install to ARM on AWS

This document details how to switch an existing Debian installation to a different processor architecture (crossgrading, this document inspired this). This howto uses Amazon AWS, the source architecture AMD64 and the destination architecture ARM64, making it an x86 to ARM conversion. I wrote this document because I wanted to see if I could convert a complex amd64 VM directly to arm64 without having to do a reinstall/migration. This should work for other architectures too, I'd love to do it for RISC-V. The conversion process is fairly straightforward, the only real trouble is wrestling with the package manager.

As for me, I've been using Linux since August 1998 (the original RedHat 5.1), and Debian since July 2002 (right when Debian 3.0 Woody came out). I use Debian exclusively as my main OS, and have done so for a decade now. I used to work with many other architectures, such as PowerPC, SPARC, MIPS, Alpha, and PA-RISC, and also commercial UNIX for those such as Sun Solaris and SGI IRIX.

This requires the VM instance to have an EFI partition, verify with the df command. If your system doesn't have one, you'll need to rebuild your root volume to include one.
This was written for Debian Linux, but should work fine on Debian derivatives such as Ubuntu or Mint. This won't work with RedHat/CentOS/Fedora/Rocky.

First, log into the x86/amd64 VM that you want to convert. This was done for a newly created Debian 10 VM. Make sure it's backed up first. Be very careful when doing this conversion, mistakes could result in an unbootable/broken system.
Fix /etc/apt/sources.list and comment out backports entries.
Before you begin, upgrade the system. I also usually install aptitude, which I use for advanced package management tasks.

If you don't believe this can work, basically it's possible both due to the fact that Debian supports having multiple architectures of packages installed simulaneously (multiarch, introduced in Debian 7), and also qemu-user-static allows the Linux kernel to run binaries of other architectures as if they were native executables. If you think about it, it's really not that complicated. The theory behind it is that if you switch all binaries including the kernel over to the other architecture and get grub working, it should work, and it does.

Do this at your own risk, since you can really mess up your Linux installation if things go wrong.

First, make sure you're root.

Add arm64 as an architecture:

# dpkg --print-architecture
amd64
# dpkg --add-architecture arm64
# apt update
# dpkg --print-foreign-architectures
arm64

In order to run ARM binaries on the system, you need to install qemu-user-static for emulation:

# apt install qemu-user-static

Test arm execution by doing a (I decided to use bzip for this):

# apt install bzip2:arm64
# apt clean
# file /usr/bin/bzip2
/usr/bin/bzip2: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV)...
# bzip2
bzip2: I won't write compressed data to a terminal.
bzip2: For help, type: `bzip2 --help'.

Switch package manager to arm, this causes arm64 to become the default architecture for the dpkg package manager:

# apt --download-only install dpkg:arm64 tar:arm64 apt:arm64
# dpkg --install /var/cache/apt/archives/*.deb

# apt clean
# dpkg --print-architecture
arm64

Try:

# apt install aptitude

# apt --download-only --fix-broken install
# dpkg --install /var/cache/apt/archives/*.deb

# apt clean

Install apt-utils package manually:

# apt-get download apt-utils
# dpkg -i apt-utils*.deb
# rm apt-utils*.deb

Switch system base over to arm:

# arch
x86_64
# apt install coreutils base-files
# arch
aarch64

Make list of packages that need switching to arm:

# dpkg -l | grep amd64 | cut -d ' ' -f 3 | cut -d ':' -f 1 > out
# vi out

Remove the systemd package from the list, that is sensitive and should be done at a later step.
Remove qemu-user-static, and binfmt-support from the list, those will kill the OS otherwise due to the removal of arm emulation.
Remove grub packages from the list, this will be done later.
Remove linux-image packages from the list, this will be done later.

Set password for root, which will allow root logins on the serial console, used later.

# passwd 

Install the list of packages that you just made.

# apt --download-only install `cat out`
# dpkg -i /var/cache/apt/archives/*.deb

Clean packages:

# apt clean
# rm out

System still runs due to /lib being a multiarch design (it contains both x86 and arm libraries side-by-side), while almost all binaries on system are arm.

Doing a:

# file /usr/bin/* | grep x86-64 | less

Install kernel for arm:

# apt install linux-image-arm64

Verify that a kernel for each architecture exists:

# ls -l /boot

Switch grub to arm:

# dpkg --purge grub-cloud-amd64 grub-pc-bin grub-efi-amd64-bin
# apt install grub-efi-arm64 grub-common:arm64 grub2-common:arm64
# apt clean

To match Debian arm setup:
-add elevator=noop to /etc/default/grub.d/10_cloud_disk_scheduler.cfg command-line options, like this:

# Set disk scheduler to noop and request block multiqueue usage

GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX elevator=noop scsi_mod.use_blk_mq=Y"

# Add a serial console

GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX console=ttyS0,115200"

To fix a problem with Grub, first reconfigure Grub to make sure it installs the correct EFI boot loader:

# dpkg-reconfigure grub-efi-arm64

Regenerate grub config:

# /usr/sbin/update-grub

The binfmt-support package is no longer needed, but make sure you keep it until after the system boots onto arm hardware, otherwise if you remove it now, it'll kill the running os.

Remove the running amd64 kernel, this needs to be done because grub uses the amd64 one as default instead of arm64:

# dpkg --purge linux-image-4.19.0-17-cloud-amd64 linux-image-cloud-amd64

If you do a package analysis of the system, you will find that binfmt-support, qemu-user-static, and systemd are the only things left on amd64:

# dpkg -l | grep amd64 | cut -d ' ' -f 3 | cut -d ':' -f 1 > amd64
# dpkg -l | grep arm64 | cut -d ' ' -f 3 | cut -d ':' -f 1 > arm64
# diff -u amd64 arm64 | less
# rm amd64 arm64

For this next step, make sure you're on a native text console, not over ssh. On AWS, switch to a serial console. You will lose networking from this which includes ssh, so be careful.

# apt-get install systemd:arm64

# ps -ef | grep systemd

root         1     0  0 Jun29 ?        00:00:12 /usr/bin/qemu-aarch64-static /lib/systemd/systemd --system --deserialize 32

Since that last step killed networking, do a network restart to bring it back including ssh:

# systemctl restart networking.service

On the system, if you do a ps -ef, you'll notice that every running service is prepended with qemu for arm emulation, such as:

# ps -ef | grep sshd
root     11089     1  0 15:27 ?        00:00:00 /usr/bin/qemu-aarch64-static /usr/sbin/sshd -D

Hopefully all goes well now. Time to shut down the system:

# halt -p

Congratulations, you just successfully bricked your amd64 Debian system.
Since the system is now unbootable, don't try to start it up before switching architectures, if you do, it won't do anything at all.

From the AWS console, detach the disk volume from the instance.
Then, start up a new Debian 10 arm instance and shut it down immediately. Note the device name of the disk, mine is "/dev/xvda".
Detach and delete the disk volume from that instance, since it's not needed.
Attach the original disk you worked on before to the arm instance as /dev/xvda, then start the instance.

Another congratulations, your previously-amd64 system should now be running natively on ARM hardware. SSH should work for you now, otherwise check the serial console for errors.

Finally, force-purge the amd64 packages from your system. Make sure you've migrated all the packages over to arm first, and keep a backup of the /etc folder just in case, before doing this.

# dpkg --purge --force-all `dpkg -l | grep amd64 | cut -d ' ' -f 3`

The conversion to ARM is now complete.

# dpkg --remove-architecture amd64
# apt update

After your system is running, if you want to run x86 applications on it (amd64 on arm64, the opposite of before), install the qemu-user-static application again:

# apt install qemu-user-static

If you're interested in knowing, I had 2 failed attempts at this architecture conversion (disposable VMs are great), the first was when I accidentally switched the binfmt-support package which killed the emulation and the OS with it, the second one was when I switched systemd over without making a root login for the serial console, and lost ssh access and the ability to log in. On the third try, grub wouldn't boot, but I found that it was due to a bug, copied over the older modules, and that got it working.