Replace A Broken Kernel On Odroid XU4 Arch Linux Using UBoot And TFtp

Overview

In case you own one or more Odroid XU4 boards from Hardkernel, you already know they are very good low-cost, all-round performing devices. Their longevity in the context of the increasing ARM market competition is proof enough of that.

Despite Hardkernel officially supporting just the Ubuntu Linux distribution, I decided to use the Arch Linux one, due to the fact that it arrives more barebone by default, thus, marginally, more performant. Since I did not intend to make use of the graphic facilities of the board, the Arch distribution was a natural choice.

The kernel updates are well maintained, and the emphasis is on sticking to LTS variants with point updates. The Pacman package ecosystem is equally healthy. Unfortunately, every once in a while, a kernel version update triggers a fatal mishap, where the system wouldn't boot. The pain gets particularly compounded if you keep your Odroid(s) detached from any monitor and keyboard, and rely exclusively on the network connection for communicating with it.

This article describes the steps you can take to boot your Odroid board using a network-downloaded kernel image and then to chroot into your damaged setup and downgrade your kernel package to the last working version. This is specific to XU4, but I believe that, with minor modifications, you can adapt this to work with other ARM boards as well.

The aim is to download the kernel, the initramfs pseudo-root file system file and the device tree binary definitions file via TFTP, load them into the board's memory, and initiate the boot sequence.

Prerequisites

There are a few required items for this procedure to have any chance of success.

First, you need a (wired) network, and a regular desktop or laptop computer with a terminal software installed on it: minicom (on Linux systems) or Putty (on Windows). Both the computer and the Odroid board must be connected to the same network; the Odroid must be wired through a cable, since there is no wireless support coming with the boot loader.

Secondly, I hope you purchased an USB UART Kit alongside your Odroid board. If you didn't, this article will not be able to help, since you will be unable to communicate in any way with the board. The UART must be plugged into the offender board and its USB end must be connected to the desktop or laptop.

Thirdly, you need a working Arch image. You can generally find the latest here, as per the installation section. Always keep a good version or two of these images on your desktop or laptop, since the one currently online could, unfortunately, contain the broken kernel.

Unpack the Odroid image somewhere in the temp folder structure, such as /tmp/arch

$ cd /tmp
$ mkdir arch
$ wget wget http://os.archlinuxarm.org/os/ArchLinuxARM-odroid-xu3-latest.tar.gz
$ bsdtar -xpf ArchLinuxARM-odroid-xu3-latest.tar.gz -C arch/

Install TFTP (One Time Only)

$ sudo apt-get install xinetd tftpd tftp
$ sudo vim /etc/xinetd.d/tftp

Inside the newly created tftp file, enter the minimally required settings:

service tftp
{
protocol        = udp
port            = 69
socket_type     = dgram
wait            = yes
user            = nobody
server          = /usr/sbin/in.tftpd
server_args     = /tmp/arch/boot
disable         = no
}

Note how the server_args entry contains the "boot" subdirectory of the location where we unzipped the image file. Now we are ready to start or restart the xinet wrapper daemon in order to have tftp running:

$ sudo service xinetd restart
$ sudo service xinetd status (make sure it is running)

Connect to Odroid Using minicom (on Linux)

If you are using Linux, make sure that the UART is registered as a device. When you run $ ls /dev/ttyUSB* -l, you should get a valid entry listed, and not this:

ls: cannot access '/dev/ttyUSB*': No such file or directory

Unplug and re-plug the UART a few times, it may help.

Once you successfully set up the terminal application, power on the Ordoid and, most likely, you will notice that it freezes after the message: Kernel is starting

uboot$ usb start

Next, we need to inform the boot loader which ip addresses to use, for the board itself and for connecting to the tftp server. Among its system variables, which you can list using printenv, the serverip variable holds the IP address for the TFTP server host, while ipaddr holds the address for the board.

You will have to replace 192.168.1.20 and 192.168.1.50 with your real addresses. Ideally, you had assigned a fixed IP for the board using your router's MAC to IP association. If you are not sure of the IP, you can use a tool such as fping to ping for an IP range and have it list which ones appear to be alive.

uboot$ setenv serverip 192.168.1.20
uboot$ setenv ipaddr 192.168.1.50

Next, we are going to download three essential components of the Linux boot process: the kernel image (zImage), the device tree binary file (dtbs/exynos5422-odroidxu4.dtb), and the initramfs minimal root file system, with busybox on it.

uboot$ tftpboot ${kernel_addr_r} zImage
uboot$ tftpboot ${fdt_addr_r} dtbs/exynos5422-odroidxu4.dtb
uboot$ tftpboot ${ramdisk_addr_r} initramfs-linux.img

Great! We have the images in memory. We are ready to start the boot process. If you peek inside the "boot" subdirectory of the expanded Arch image, you will notice a boot.txt file. That is the "uncompiled" form of the configuration that u-boot eventually expects, in order to be able to boot.

The most important variable to control the boot process is bootargs. A typical bootargs contains several arguments, such as root, macaddr, video and many others. We are going to use a stripped down version, that gets right down to business: we need the system to boot from the emergency initramfs that we downloaded over tftp.

Once that is set up, we can tell u-boot to start the boot process from the in-memory components: we start by passing bootz the address of the compressed kernel image, followed by the address and the size of the initramfs image, and - finally - the address of the device tree image:

$ setenv bootargs "console=tty1 ${console} initrd=${ramdisk_addr_r}"
$ bootz ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}

After running the bootz command above, the kernel should successfully boot (hit "Enter" once to obtain the console prompt if a kernel log message obscured it). Now we are ready to downgrade our kernel package.

Mount the Storage Devices

My own setup is to have the eMMC card as the initial boot device, and the root file system being loaded from an USB harddrive afterwards. Because of this, I need to keep the eMMC and the hard disk in sync, so I must mount them and downgrade the kernel package on both. You don't have to do this if you only use the eMMC or the SD card. Just make sure you correctly identify the device in the /dev section.

First, we mount the eMMC and the hard drive (assuming they are identified as /dev/mmcblk0p1 and /dev/sda1):

$ mkdir -p /mnt/hdd
$ mkdir -p /mnt/mmc
$ mount /dev/sda1 /mnt/hdd
$ mount /dev/mmcblk0p1  /mnt/mmc/

Once the devices are mounted, we need to be able to chroot inside their file systems to make use of the Arch Linux package management facilities.

Downgrade the Kernel

Usually, with Arch Linux, we have the arch-chroot utility that handles the preliminary steps to chroot into an Arch system. Since we do not have that, we must chroot into the eMMC root using regular chroot. Before executing chroot proper, we must bind the temporary filesystems; please read this article as well:

$ cd /mnt/mmc
$ mount -t proc proc proc/
$ mount --rbind /sys sys/
$ mount --rbind /dev dev/
$ mount --rbind /run run/
$ chroot /mnt/mmc /bin/bash

Now that we are inside ArchLinux, I am identifying it with the arch$ prefix. Go to the Pacman cache location and identify a working previous kernel version:

arch$ cd /var/cache/pacman/pkg
arch$ ls linux-odroid-*

Let's assume the broken kernel version is 4.14.15, and one previous version we have in our local cache that we know worked without any issue is 4.14.13. To choose that version, and downgrade to it, we are going to use pacman -U:

arch$ pacman -U linux-odroid-xu3-4.14.13-1-armv7h.pkg.tar.xz

Once the downgrade is complete, we exit from the eMMC chroot:

arch$ exit

Similarly, we chroot into the HDD root, using regular chroot. This is an optional step, not necessary to perform, unless you use the hard drive as a root file system, as I do:

$ cd /mnt/hdd
$ mount -t proc proc proc/
$ mount --rbind /sys sys/
$ mount --rbind /dev dev/
$ mount --rbind /run run/
$ chroot /mnt/hdd /bin/bash

The downgrade steps are identical to the ones above. Once we exit from the chroot, the operation should be complete. We exit from the initramfs with exit again, then we can power off Odroid, re-plug the power supply, and hope that we have a working Linux again. Good luck !