C Index

System Development

System development requires knowing how to debug kernel know how, in this example will be used Qemu and GDB. Qemu creates the virtual machine that kernel will run on and GDB will connect to it to help us understand how things tick.

Build Kernel

        $ tar xf linux-4.9.258.tar.xz
        $ cd linux-4.9.258

Default configuration disable some security configurations that allow us to debug (random memory layout KALSR), CONFIG_COMPAT_BRK don't randomize position of the programs (randomize_va_space) useful when debugging a program (tracing). Configuration flags to enable;

CONFIG_64BIT
CONFIG_DEBUG_KERNEL
CONFIG_HAVE_ARCH_KGDB
CONFIG_FTRACE
CONFIG_PRINTK
CONFIG_BLK_DEV_INITRD
CONFIG_BINFMT_ELF
CONFIG_TTY
CONFIG_DEBUG_INFO
CONFIG_DEBUG_INFO_DWARF4
CONFIG_GDB_SCRIPTS
CONFIG_READABLE_ASM
CONFIG_FRAME_POINTER
CONFIG_KGDB
CONFIG_KGDB_LOW_LEVEL_TRAP
CONFIG_EARLY_PRINTK
CONFIG_COMPAT_BRK

And to disable;

CONFIG_CC_OPTIMIZE_FOR_SIZE

This changes can be achieved by creating a config-fragment and then merge it with the configuration.

        $ cat <<EOF >.config-fragment
        CONFIG_64BIT=y
        CONFIG_DEBUG_KERNEL=y
        CONFIG_HAVE_ARCH_KGDB=y
        CONFIG_COMPAT_BRK=y
        CONFIG_FTRACE=y
        CONFIG_PRINTK=y
        CONFIG_BLK_DEV_INITRD=y
        CONFIG_BINFMT_ELF=y
        CONFIG_TTY=y
        CONFIG_DEBUG_INFO=y
        CONFIG_DEBUG_INFO_DWARF4=y
        CONFIG_GDB_SCRIPTS=y
        CONFIG_READABLE_ASM=y
        CONFIG_FRAME_POINTER=y
        CONFIG_KGDB=y
        CONFIG_KGDB_LOW_LEVEL_TRAP=y
        CONFIG_EARLY_PRINTK=y
        CONFIG_CC_OPTIMIZE_FOR_SIZE=n
        EOF

Create a tiny config;

        $ make ARCH=x86_64 tinyconfig

Merge config with the following script;

        $ ./scripts/kconfig/merge_config.sh .config .config-fragment

Check or change the configuration according to your needs;

        $ make nconfig

Build Kernel and modules;

        $ make -j $(nproc)

Simple Init

Now that you have the kernel compiled you can create a simple init program, this program is called when kernel finish to load and setup its internals and is ready to launch first process. Init program should not exit or kernel will panic. Create init.S;

        .global _start
        _start:
            mov $1, %rax
            mov $1, %rdi
            mov $message, %rsi
            mov $message_len, %rdx
            syscall
            jmp .
            message: .ascii "FOOBAR FOOBAR FOOBAR FOOBAR FOOBAR FOOBAR FOOBAR\n"
            .equ message_len, . - message

Assemble, link and create simple initial ram disk;

        mkdir d
        as --64 -o init.o init.S # assemble
        ld -o d/init init.o      # link
        cd d
        find . | cpio -o -H newc | gzip > ../rootfs.cpio.gz

Clean temporary directory;

        cd ..
        rm -r d/

Can be used C to create init program;

        #include <stdio.h>
        #include <unistd.h>

        int main() {
            printf("FOOBAR FOOBAR FOOBAR FOOBAR FOOBAR FOOBAR FOOBAR\n");
            sleep(0xFFFFFFFF);
            return 0;
        }

        $ gcc -static init.c -o init

Start Debugging

Test qemu, kernel and simple init program, you should see "FOOBAR FOOBAR FOOBAR FOOBAR FOOBAR FOOBAR";

        $ qemu-system-x86_64 -enable-kvm --kernel arch/x86_64/boot/bzImage \
        --initrd rootfs.cpio.gz

If everything goes well you can start qemu without starting the CPU (-S) and with gdb server on TCP port 1234 (-s).

        $ qemu-system-x86_64 -enable-kvm --kernel arch/x86/boot/bzImage \
                --initrd rootfs.cpio.gz \
                -S -s

On another terminal start gdb;

        gdb \
            -ex "add-auto-load-safe-path $(pwd)" \
            -ex "file vmlinux" \
            -ex 'set arch i386:x86-64:intel' \
            -ex 'target remote localhost:1234' \
            -ex 'hbreak start_kernel' \
            -ex 'continue' \
            -ex 'disconnect' \
            -ex 'set arch i386:x86-64' \
            -ex 'target remote localhost:1234'

	(gdb) info thread
	  Id   Target Id         Frame
	* 1    Thread 1 (CPU#0 [running]) start_kernel () at init/main.c:480

	(gdb) info frame
	Stack level 0, frame at 0xffffffff81e03f90:
	 rip = 0xffffffff81f4db2d in start_kernel (init/main.c:480); saved rip = 0xffffffff81f4d28e
	 called by frame at 0xffffffff81e03fa0
	 source language c.
	 Arglist at 0xffffffff81e03f80, args:
	 Locals at 0xffffffff81e03f80, Previous frame's sp is 0xffffffff81e03f90
	 Saved registers:
	  rip at 0xffffffff81e03f88

	(gdb) print $rip
	$2 = (void (*)()) 0xffffffff81f4db2d <start_kernel>
	(gdb)

lx-symbols allows to debug kernel modules, after starting the vm and loading the module use lx-symbols to load the symbols from all the modules loaded in the kernel.

        (gdb) apropos lx
        (gdb) lx-symbols

It's useful to set conditional breakpoints or a break point can be trigger by unrelated tasks, example of a break point on do_exit function but only by the process with pid 1;

        (gdb) br do_exit if $lx_current()->pid == 1

C Index