Lolbins? Where we’re going, we don’t need lolbins.

At NCC Group, as a consultant in our hardware and embedded systems practice¹, I often get to play with various devices, which is always fun, but getting your own software to run on them can be a bit of a pain.
This article documents a few realisations and tricks that make my life easier. There is nothing new about anything mentioned here, but there is also hardly anything written about these (ab)use cases.

The challenges we are looking to solve are:

• Running standard Linux tools on your embedded device
• Compiling your own tools to run on the embedded device
• Running binaries from the embedded device on your PC

This can often be achieved by cross-compiling and/or statically compiling the target binary. It is very much a valid approach, but it can also be time-consuming, even when you’re seasoned at this sort of thing. So, the approach described here does not do that.

Taking the dependency libraries with you

The realisation is that while dynamically linked binaries need some of the environment, it is actually not that hard to figure out what that environment is and to copy it over to the system where you want to run the binary.

Consider the example of running strace from an arm64 Raspberry Pi on an arm64 Android phone.

Just copying won’t work, since Android differs too much from common Linux distributions:

pi@rpi:~ $ adb push `which strace` /data/local/tmp
/usr/bin/strace: 1 file pushed, 0 skipped. 16.0 MB/s (1640712 bytes in 0.098s)
pi@rpi:~ $ adb exec-out /data/local/tmp/strace -ttewrite /bin/echo X
/system/bin/sh: /data/local/tmp/strace: No such file or directory
pi@rpi:~ $ adb exec-out ldd /data/local/tmp/strace 
    linux-vdso.so.1 => [vdso] (0x73edb38000)
CANNOT LINK EXECUTABLE "linker64": library "libc.so.6" not found: needed by main executable

We can list the dependencies. strace depends on the dynamic linker, libraries and often on some special bits like VDSO.

pi@rpi:~ $ ldd `which strace`
    linux-vdso.so.1 (0x0000007faf464000)
    libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007faf0b0000)
    /lib/ld-linux-aarch64.so.1 (0x0000007faf427000)

We can copy the dependencies and use the appropriate dynamic linker to load them (the first highlighted bit enumerates the dependencies):

pi@rpi:~ $ bin=`which strace`; adb push $bin $(ldd $bin | sed -nre 's/^[^/]*(\/.*) \(0x.*\)$/\1/p') /data/local/tmp/
/usr/bin/strace: 1 file pushed, 0 skipped. 19.4 MB/s (1640712 bytes in 0.081s)
/lib/aarch64-linux-gnu/libc.so.6: 1 file pushed, 0 skipped. 23.7 MB/s (1651472 bytes in 0.067s)
/lib/ld-linux-aarch64.so.1: 1 file pushed, 0 skipped. 18.6 MB/s (202904 bytes in 0.010s)
3 files pushed, 0 skipped. 19.0 MB/s (3495088 bytes in 0.176s)
pi@rpi:~ $ adb exec-out /data/local/tmp/ld-linux-aarch64.so.1 --library-path /data/local/tmp/ /data/local/tmp/strace -ttewrite /bin/echo X
10:36:27.842717 write(1, "X\n", 2X
)      = 2
10:36:27.845895 +++ exited with 0 +++

There, perfect, we have strace on our device now, and it only took two ugly one-liners.

Installing that odd architecture on your PC

My preferred way of setting up a cross architecture Linux chroot is using debootstrap and schroot. This assumes you are using a distribution from a Debian family (I do see there’s debootstrap for Fedora as well, haven’t tried it though).

Logan Chien posted this nice and short guide², which basically boils down to following three sections:

Setting up the tools

kali@kali:~$ apt install debootstrap qemu-user-static schroot

Installing a base system

kali@kali:~$ sudo debootstrap --arch=arm64 bookworm ~/chroots/arm64-test
...
I: Base system installed successfully.

This takes a minute or two and installs a base Debian Bookworm system for arm64. The distribution names come from Debian (http://ftp.debian.org/debian/dists/) or Ubuntu (http://archive.ubuntu.com/ubuntu/dists/). For the architecture names navigate into subfolders (for example http://ftp.debian.org/debian/dists/bookworm/main/). If you need a less common architecture try the testing channel, which supports riscv64 for example.

Setting up schroot

kali@kali:~$ echo "[arm64-test] 
directory=$HOME/chroots/arm64-test
users=$(whoami)
root-users=$(whoami)
type=directory" | sudo tee /etc/schroot/chroot.d/arm64-test

Now you can enter the chroot:

kali@kali:~$ schroot -c arm64-test
(arm64-test)kali@kali:~$ uname -a
Linux kali 6.5.0-kali3-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.6-1kali1 (2023-10-09) aarch64 GNU/Linux
(arm64-test)kali@kali:~$ logout
kali@kali:~$ schroot -c arm64-test -u root
(arm64-test)root@kali:/home/kali# id
uid=0(root) gid=0(root) groups=0(root),4(adm),20(dialout),119(wireshark),142(kaboxer)

Finish

So, here you have it. Install the wanted binary, and copy it to your target device like we did before from Raspberry Pi.

(arm64-test)kali@kali:~$ # sudo apt install strace adb
...
(arm64-test)kali@kali:~$ bindeps() { echo "$1" $(ldd "$1" | sed -nre "s/^[^/]*(\/.*) \(0x.*\)$/\1/p"); }
(arm64-test)kali@kali:~$ for i in $(bindeps `which strace`); do adb push $i /data/local/tmp/; done
/usr/bin/strace: 1 file pushed, 0 skipped. 13.3 MB/s (1640712 bytes in 0.117s)
/lib/aarch64-linux-gnu/libc.so.6: 1 file pushed, 0 skipped. 14.0 MB/s (1651472 bytes in 0.113s)
/lib/ld-linux-aarch64.so.1: 1 file pushed, 0 skipped. 7.4 MB/s (202904 bytes in 0.026s)
(arm64-test)kali@kali:~$ adb shell
sargo:/ $ xrun() { k=/data/local/tmp; bin=$1; shift; $k/ld* --library-path $k $k/$bin "$@"; }
sargo:/ $ xrun strace -tte write /bin/echo X
16:31:59.159266 write(1, "X\n", 2X ) = 2 16:31:59.163322 +++ exited with 0 +++

Cross-compiling without cross-compiling

Well, now you have a full Linux distribution of your chosen architecture running, so you can just compile any special tools. Sure, it is emulated through QEMU under the hood, but for anything smallish one does not even notice the performance hit.

And you have avoided dealing with a toolchain to cross-compile for that one-off task.

Running the device’s binaries on your PC

In the above examples the binary we wished to run was copied onto the target device. Occasionally, one wants to do the reverse, run the binary from the device locally on your PC.

The exact same approach should work, and for anything non-trivial I would recommend a custom setup chroot, so you can easily place the required files in the correct locations (and it is also easy to later delete it all).

For a simple tool though, one can get away by using QEMU:

kali@kali:~/android_test$ adb exec-out 'bindeps() { echo "$1" $(ldd "$1" | sed -nre "s/^[^/]*(\/.*) \(0x.*\)$/\1/p"); }; bindeps `which dexdump`' | xargs -n1 adb pull
/apex/com.android.art/bin/dexdump: 1 file pulled, 0 skipped. 11.6 MB/s (108744 bytes in 0.009s)
/apex/com.android.art/lib64/libdexfile.so: 1 file pulled, 0 skipped. 4.7 MB/s (347040 bytes in 0.070s)
/apex/com.android.art/lib64/libartpalette.so: 1 file pulled, 0 skipped. 2.0 MB/s (14896 bytes in 0.007s)
/apex/com.android.art/lib64/libbase.so: 1 file pulled, 0 skipped. 13.3 MB/s (251152 bytes in 0.018s)
/apex/com.android.art/lib64/libartbase.so: 1 file pulled, 0 skipped. 20.8 MB/s (497272 bytes in 0.023s)
/apex/com.android.art/lib64/libc++.so: 1 file pulled, 0 skipped. 8.1 MB/s (671496 bytes in 0.079s)
/apex/com.android.art/lib64/libziparchive.so: 1 file pulled, 0 skipped. 1.2 MB/s (79752 bytes in 0.066s)
/apex/com.android.runtime/lib64/bionic/libc.so: 1 file pulled, 0 skipped. 26.1 MB/s (1013048 bytes in 0.037s)
/apex/com.android.runtime/lib64/bionic/libdl.so: 1 file pulled, 0 skipped. 2.0 MB/s (13728 bytes in 0.006s)
/apex/com.android.runtime/lib64/bionic/libm.so: 1 file pulled, 0 skipped. 12.7 MB/s (221072 bytes in 0.017s)
/system/lib64/libz.so: 1 file pulled, 0 skipped. 6.5 MB/s (98016 bytes in 0.014s)
/system/lib64/liblog.so: 1 file pulled, 0 skipped. 5.8 MB/s (62176 bytes in 0.010s)
/system/lib64/libc++.so: 1 file pulled, 0 skipped. 25.7 MB/s (700400 bytes in 0.026s)
kali@kali:~/android_test$ adb pull /system/bin/linker64
/system/bin/linker64: 1 file pulled, 0 skipped. 13.1 MB/s (1802728 bytes in 0.131s)
kali@kali:~/android_test$ qemu-arm64 -E LD_LIBRARY_PATH=$PWD ./linker64 $PWD/dexdump
linker: Warning: failed to find generated linker configuration from "/linkerconfig/ld.config.txt"
WARNING: linker: Warning: failed to find generated linker configuration from "/linkerconfig/ld.config.txt"
dexdump E 05-02 13:14:39 1728592 1728592 dexdump_main.cc:126] No file specified
dexdump E 05-02 13:14:39 1728592 1728592 dexdump_main.cc:41] Copyright (C) 2007 The Android Open Source Project
dexdump E 05-02 13:14:39 1728592 1728592 dexdump_main.cc:41] 
dexdump E 05-02 13:14:39 1728592 1728592 dexdump_main.cc:42] dexdump: [-a] [-c] [-d] [-e] [-f] [-h] [-i] [-j] [-l layout] [-n]  [-o outfile] dexfile...
...

With the dynamic linker, dependency libraries and the target binary, we can use qemu-user to run our binary.

The observant reader will notice this differs slightly from the way ld was invoked before. It appears Android’s dynamic linker doesn’t support an argument to specify library path, so we have used LD_LIBRARY_PATH (in the first example above, we could have invoked strace this way as well: LD_LIBRRAY_PATH=/data/local/tmp /data/local/tmp/ld-linux-aarch64.so.1 /data/local/tmp/strace).

Epilogue

I hope you found this useful.

Helper functions

The list of dependencies that are to be copied along with the binary can be generated with:

$ bindeps() { echo "$1" $(ldd "$1" | sed -nre "s/^[^/]*(\/.*) \(0x.*\)$/\1/p"); }
$ bindeps `which strace`

The binary can then be run on the target device with (Note the path will need adjusting and possibly linker arguments as well):

$ xrun() { k=/data/local/tmp; bin=$1; shift; $k/ld* --library-path $k $k/$bin "$@"; }
$ xrun strace -tte write /bin/echo X