How to prepare and boot a XIP Linux kernel on LPC43xx

Home / blog / How to prepare and boot a XIP Linux kernel on LPC43xx

On memory-constrained platforms, it’s often desirable to run an eXecute-In-Place Linux kernel to save RAM. Linux supports XIP since ages, but configuring the kernel image, and setting a bootloader to boot properly can be painful. Nowadays eXecute-In-Place is not that popular so this particular configuration is not too tested, and it might break from time to time.

That said, the configuration is not all that complex once you know what you are doing. To illustrate the steps, we have picked an MMU-less platform. These platforms have a small amount of RAM, and it makes a lot of sense to run a XIP kernel on them. You will probably want a flash-based RootFS as well, but that’s not covered in this tutorial.

We will use a Hitex LPC4350 Eval board, which has a NXP LPC4350 MCU (ARM Cortex-M4). This board has a 4 MiB NOR flash which is large enough to store U-Boot and a XIP kernel.

Emcraft offers a full BSP that supports this board, which includes U-Boot, Linux 2.6.x and uCLinux RootFS. We will use Emcraft’s U-Boot, since it provides network and NOR support. You can find more information about Emcraft’s BSP here:

http://www.emcraft.com/products/232

This tutorial will assume you have a nice bootloader, or some JTAG tool to program the NOR flash.

The first step: defining a flash layout

First of all, we need to decide where the kernel will be running from. The flash memory should support execute-in-place, so this means we’ll need a NOR (some SoCs have support to execute from SPI flashes).

On the LPC4350 the NOR flash is memory-mapped at address 0x1c00_0000.

flash_layout_xip

About the U-Boot XIP kernel image

We will use Emcraft’s U-Boot and so we need to craft an U-Boot XIP image. This image is composed by the concatenation of an U-Boot 64-byte header, and the kernel xipImage that Linux produces as a result of the normal XIP build.

This U-Boot version is not fully devicetree-aware, yet the LPC43xx is a full devicetree platform. This means we need a bit of magic to pass the devicetree blob. It would be ideal to upgrade our bootloader, but hacking the kernel is surprisingly simpler, so let’s give it a try…

Patching the kernel

Just modify setup_arch in arch/arm/kernel/setup.c like this:

Where CONFIG_ARM_DTB_ADDR is the physical address of your DTB.

Configuring the Linux kernel

As we have seen above, the NOR address starts at 0x1c00_0000, and we will put the kernel at 0x1c04_0000. The kernel needs to know its own location so, because the image needs a 64-bytes U-Boot header, the kernel config will be:

Building the U-Boot XIP Linux image

Once the kernel is configured, the next step is to build it and prepare the image. The build target is xipImage, and we also want to build the devicetree blob:

To prepare the image, let’s first create a small stub file for the U-Boot header:

And append it to the kernel image:

And now, we can run the mkuboot.sh script:

Notice how the entry point is 0x1c040041 and not 0x1c040040! This is because the entry point is a branch target. The least significant bit of the branch address is used to specify the branch target instruction mode. A cleared LSB means ARM mode, while a set LSB means Thumb mode. However, a cortex-M CPU will throw an exception in ARM mode, because it only supports Thumb mode… and so we need the entry point address to be odd (LSB bit set).

Writing to flash

Once the image is ready, we need to write it to the NOR flash. We can use U-Boot to upload the image from a tftp server, and then write it to 0x1c040000.

The devicetree blob must be written to flash too:

Let’s boot it!

Booting from SPIFI

This tutorial not only applies to NOR memory, but also to SPIFI. However, there are some subtle pitfalls to consider. First, a kernel fix is required for eXecute-In-Place to work properly when booting from SPIFI (in fact, this bug was found while doing the research for this tutorial!).

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=a4124e7296000242243996e1ae2601cfadf276a5

As you can see, it’s already accepted in upstream, so the issue is fixed in Linux v4.5+. If you are using an older version, don’t be afraid to backport it. The fix is small and straightforward.

There’s another pitfall to consider. If you also store the rootfs in SPIFI, this procedure might not work. The reason for this is that the SPIFI MTD driver will mess up with the SPIFI memory, preventing the kernel to be executed from there at the same time.

In particular, the SPIFI MTD driver performs a reset operation when the driver is initialized. Write operations to the flash will operate in command mode, so the SPIFI MTD driver enters command mode during a write to flash. However, code execution requires memory mode, and so the CPU will fault at this point.

(Although we haven’t tested it, the reset operation could be avoided. However, this is only a workaround for read-only rootfs operation).

Conclusion

As we have just seen, it’s not all that hard to run a XIP kernel, which is an attractive option if you want to save precious RAM. If you have questions about this, or you have a project on a MMU-less platform and you need an expert hand, don’t hesitate to drop us a line!

Showing 4 comments
  • Bo Mellberg
    Reply

    Excellent article, Ezequiel.

    I run into problems when the kernel is about to load:

    Physio-Control 200130-00> bootm 0x14000000
    ## Booting kernel from Legacy Image at 14000000 …
    Image Name: J2929
    Image Type: ARM Linux Kernel Image (uncompressed)
    Data Size: 6021316 Bytes = 5.7 MB
    Load Address: 14000000
    Entry Point: 14000041
    Verifying Checksum … OK
    XIP Kernel Image … OK
    OK

    Starting kernel …

    UNHANDLED EXCEPTION: HARD FAULT
    R0 = 00000000 R1 = 00001038
    R2 = 30000000 R3 = 00000003
    R12 = 00000004 LR = 1a00becd
    PC = 14000040 PSR = 61000000

    I find it a bit strange that PC = 14000040 and not 14000041, which is the entry point.

    /Bo

    • Ezequiel Garcia
      Reply

      Hello Bo,

      Yes, the PC seems off. Let me think about it…

  • Aurélien HENRY
    Reply

    Hello, I would like to do the same thing with a stm32f7-som of emcraft. I use their Linux and projects files. I have modified rules.make file to generate xip image, xip and mtd support in kernel config but I don’t know if address are good. Nor flash is at 0x6000000.
    I have loaded xip image with u-boot but I have an error: Wrong Image Format for bootm command ERROR: can’t get kernel image!
    If you have an issue to my problem, I will happy.

    • Ezequiel Garcia
      Reply

      Hi Henry,
      I’m sorry for the delay… not entirely sure what the problem might be. These low level issues usually require to tinker around the board itself.

Leave a Reply to Aurélien HENRY Cancel reply