2. Linux Makefiles

Before perusing Linux code, we should get some basic idea about how Linux is composed, compiled and linked. A straightforward way to achieve this goal is to understand Linux makefiles. Check Cross-Referencing Linux if you prefer online source browsing.

2.1. linux/Makefile

Here are some well-known targets in this top-level makefile:

Overview of linux/Makefile is outlined below:
include .depend
include .config
include arch/i386/Makefile

vmlinux: generate linux/vmlinux
        /* entry point "stext" defined in arch/i386/kernel/head.S */
        $(LD) -T $(TOPDIR)/arch/i386/vmlinux.lds -e stext
        /* $(HEAD) */
        + from arch/i386/Makefile
                arch/i386/kernel/head.o
                arch/i386/kernel/init_task.o
        init/main.o
        init/version.o
        init/do_mounts.o
        --start-group
        /* $(CORE_FILES) */
        + from arch/i386/Makefile
                arch/i386/kernel/kernel.o
                arch/i386/mm/mm.o
        kernel/kernel.o
        mm/mm.o
        fs/fs.o
        ipc/ipc.o
        /* $(DRIVERS) */
        drivers/...
                char/char.o
                block/block.o
                misc/misc.o
                net/net.o
                media/media.o
                cdrom/driver.o
                and other static linked drivers
                + from arch/i386/Makefile
                        arch/i386/math-emu/math.o (ifdef CONFIG_MATH_EMULATION)
        /* $(NETWORKS) */
        net/network.o
        /* $(LIBS) */
        + from arch/i386/Makefile
                arch/i386/lib/lib.a
        lib/lib.a
        --end-group
        -o vmlinux
        $(NM) vmlinux | grep ... | sort > System.map
tags: generate linux/tags for vim
modules: generate modules
modules_install: install modules
clean mrproper distclean: clean up build directory
psdocs pdfdocs htmldocs mandocs: generate kernel documents

include Rules.make

rpm: generate an rpm
"--start-group" and "--end-group" are ld command line options to resolve symbol reference problem. Refer to Using LD, the GNU linker: Command Line Options for details.

Rules.make contains rules which are shared between multiple Makefiles.

2.2. linux/arch/i386/vmlinux.lds

After compilation, ld combines a number of object and archive files, relocates their data and ties up symbol references. linux/arch/i386/vmlinux.lds is designated by linux/Makefile as the linker script used in linking the resident kernel image linux/vmlinux.

/* ld script to make i386 Linux kernel
 * Written by Martin Mares <[email protected]>;
 */
OUTPUT_FORMAT("elf32-i386", "elf32-i386", "elf32-i386")
OUTPUT_ARCH(i386)
/* "ENTRY" is overridden by command line option "-e stext" in linux/Makefile */
ENTRY(_start)
/* Output file (linux/vmlinux) layout.
 * Refer to Using LD, the GNU linker: Specifying Output Sections */
SECTIONS
{
/* Output section .text starts at address 3G+1M.
 * Refer to Using LD, the GNU linker: The Location Counter */
  . = 0xC0000000 + 0x100000;
  _text = .;                    /* Text and read-only data */
  .text : {
        *(.text)
        *(.fixup)
        *(.gnu.warning)
        } = 0x9090
/* Unallocated holes filled with 0x9090, i.e. opcode for "NOP NOP".
 * Refer to Using LD, the GNU linker: Optional Section Attributes */

  _etext = .;                   /* End of text section */

  .rodata : { *(.rodata) *(.rodata.*) }
  .kstrtab : { *(.kstrtab) }

/* Aligned to next 16-bytes boundary.
 * Refer to Using LD, the GNU linker: Arithmetic Functions */
  . = ALIGN(16);                /* Exception table */
  __start___ex_table = .;
  __ex_table : { *(__ex_table) }
  __stop___ex_table = .;

  __start___ksymtab = .;        /* Kernel symbol table */
  __ksymtab : { *(__ksymtab) }
  __stop___ksymtab = .;

  .data : {                     /* Data */
        *(.data)
        CONSTRUCTORS
        }
/* For "CONSTRUCTORS", refer to
 * Using LD, the GNU linker: Option Commands */

  _edata = .;                   /* End of data section */

  . = ALIGN(8192);              /* init_task */
  .data.init_task : { *(.data.init_task) }

  . = ALIGN(4096);              /* Init code and data */
  __init_begin = .;
  .text.init : { *(.text.init) }
  .data.init : { *(.data.init) }
  . = ALIGN(16);
  __setup_start = .;
  .setup.init : { *(.setup.init) }
  __setup_end = .;
  __initcall_start = .;
  .initcall.init : { *(.initcall.init) }
  __initcall_end = .;
  . = ALIGN(4096);
  __init_end = .;

  . = ALIGN(4096);
  .data.page_aligned : { *(.data.idt) }

  . = ALIGN(32);
  .data.cacheline_aligned : { *(.data.cacheline_aligned) }

  __bss_start = .;              /* BSS */
  .bss : {
        *(.bss)
        }
  _end = . ;

/* Output section /DISCARD/ will not be included in the final link output.
 * Refer to Using LD, the GNU linker: Section Definitions */
  /* Sections to be discarded */
  /DISCARD/ : {
        *(.text.exit)
        *(.data.exit)
        *(.exitcall.exit)
        }

/* The following output sections are addressed at memory location 0.
 * Refer to Using LD, the GNU linker: Optional Section Attributes */
  /* Stabs debugging sections.  */
  .stab 0 : { *(.stab) }
  .stabstr 0 : { *(.stabstr) }
  .stab.excl 0 : { *(.stab.excl) }
  .stab.exclstr 0 : { *(.stab.exclstr) }
  .stab.index 0 : { *(.stab.index) }
  .stab.indexstr 0 : { *(.stab.indexstr) }
  .comment 0 : { *(.comment) }
}

2.3. linux/arch/i386/Makefile

linux/arch/i386/Makefile is included by linux/Makefile to provide i386 specific items and terms.

All the following targets depend on target vmlinux of linux/Makefile. They are accomplished by making corresponding targets in linux/arch/i386/boot/Makefile with some options.

Table 1. Targets in linux/arch/i386/Makefile

TargetCommand
zImage [a] @$(MAKE) -C arch/i386/boot zImage [b]
bzImage@$(MAKE) -C arch/i386/boot bzImage
zlilo @$(MAKE) -C arch/i386/boot BOOTIMAGE=zImage zlilo
bzlilo @$(MAKE) -C arch/i386/boot BOOTIMAGE=bzImage zlilo
zdisk @$(MAKE) -C arch/i386/boot BOOTIMAGE=zImage zdisk
bzdisk @$(MAKE) -C arch/i386/boot BOOTIMAGE=bzImage zdisk
install @$(MAKE) -C arch/i386/boot BOOTIMAGE=bzImage install
Notes:
a. zImage alias: compressed;
b. "-C" is a MAKE command line option to change directory before reading makefiles;
Refer to GNU make: Summary of Options and GNU make: Recursive Use of make.

It is worth noticing that this makefile redefines some environment variables which are exported by linux/Makefile, specifically:
OBJCOPY=$(CROSS_COMPILE)objcopy -O binary -R .note -R .comment -S
The effect will be passed to subdirectory makefiles and will change the tool's behavior. Refer to GNU Binary Utilities: objcopy for objcopy command line option details.

Not sure why $(LIBS) includes "$(TOPDIR)/arch/i386/lib/lib.a" twice:
LIBS := $(TOPDIR)/arch/i386/lib/lib.a $(LIBS) $(TOPDIR)/arch/i386/lib/lib.a
It may be employed to work around linking problems with some toolchains.

2.4. linux/arch/i386/boot/Makefile

linux/arch/i386/boot/Makefile is somehow independent as it is not included by either linux/arch/i386/Makefile or linux/Makefile.

However, they do have some relationship:

$(BOOTIMAGE) value, which is for target zdisk, zlilo or zdisk, comes from linux/arch/i386/Makefile.

Table 2. Targets in linux/arch/i386/boot/Makefile

TargetCommand
zImage
$(OBJCOPY) compressed/vmlinux compressed/vmlinux.out
tools/build bootsect setup compressed/vmlinux.out $(ROOT_DEV) > zImage
bzImage
$(OBJCOPY) compressed/bvmlinux compressed/bvmlinux.out
tools/build -b bbootsect bsetup compressed/bvmlinux.out $(ROOT_DEV) \
        > bzImage
zdisk
dd bs=8192 if=$(BOOTIMAGE) of=/dev/fd0
zlilo
if [ -f $(INSTALL_PATH)/vmlinuz ]; then mv $(INSTALL_PATH)/vmlinuz
        $(INSTALL_PATH)/vmlinuz.old; fi
if [ -f $(INSTALL_PATH)/System.map ]; then mv $(INSTALL_PATH)/System.map
        $(INSTALL_PATH)/System.old; fi
cat $(BOOTIMAGE) > $(INSTALL_PATH)/vmlinuz
cp $(TOPDIR)/System.map $(INSTALL_PATH)/
if [ -x /sbin/lilo ]; then /sbin/lilo; else /etc/lilo/install; fi
install
sh -x ./install.sh $(KERNELRELEASE) $(BOOTIMAGE) $(TOPDIR)/System.map
        "$(INSTALL_PATH)"
tools/build builds boot image zImage from {bootsect, setup, compressed/vmlinux.out}, or bzImage from {bbootsect, bsetup, compressed/bvmlinux,out}. linux/Makefile "export ROOT_DEV = CURRENT". Note that $(OBJCOPY) has been redefined by linux/arch/i386/Makefile in Section 2.3.

Table 3. Supporting targets in linux/arch/i386/boot/Makefile

Target: PrerequisitesCommand
compressed/vmlinux: linux/vmlinux@$(MAKE) -C compressed vmlinux
compressed/bvmlinux: linux/vmlinux@$(MAKE) -C compressed bvmlinux
tools/build: tools/build.c $(HOSTCC) $(HOSTCFLAGS) -o $@ $< -I$(TOPDIR)/include [a]
bootsect: bootsect.o $(LD) -Ttext 0x0 -s --oformat binary bootsect.o [b]
bootsect.o: bootsect.s$(AS) -o $@ $<
bootsect.s: bootsect.S ... $(CPP) $(CPPFLAGS) -traditional $(SVGA_MODE) $(RAMDISK) $< -o $@
bbootsect: bbootsect.o $(LD) -Ttext 0x0 -s --oformat binary $< -o $@
bbootsect.o: bbootsect.s$(AS) -o $@ $<
bbootsect.s: bootsect.S ... $(CPP) $(CPPFLAGS) -D__BIG_KERNEL__ -traditional $(SVGA_MODE) $(RAMDISK) $< -o $@
setup: setup.o $(LD) -Ttext 0x0 -s --oformat binary -e begtext -o $@ $<
setup.o: setup.s$(AS) -o $@ $<
setup.s: setup.S video.S ... $(CPP) $(CPPFLAGS) -D__ASSEMBLY__ -traditional $(SVGA_MODE) $(RAMDISK) $< -o $@
bsetup: bsetup.o $(LD) -Ttext 0x0 -s --oformat binary -e begtext -o $@ $<
bsetup.o: bsetup.s$(AS) -o $@ $<
bsetup.s: setup.S video.S ... $(CPP) $(CPPFLAGS) -D__BIG_KERNEL__ -D__ASSEMBLY__ -traditional $(SVGA_MODE) $(RAMDISK) $< -o $@
Notes:
a. "$@" means target, "$<" means first prerequisite; Refer to GNU make: Automatic Variables;
b. "--oformat binary" asks for raw binary output, which is identical to the memory dump of the executable; Refer to Using LD, the GNU linker: Command Line Options.
Note that it has "-D__BIG_KERNEL__" when compile bootsect.S to bbootsect.s, and setup.S to bsetup.s. They must be Place Independent Code (PIC), thus what "-Ttext" option is doesn't matter.

2.5. linux/arch/i386/boot/compressed/Makefile

This makefile handles image (de)compression mechanism.

It is good to separate (de)compression from bootstrap. This divide-and-conquer solution allows us to easily improve (de)compression mechanism or to adopt a new bootstrap method.

Directory linux/arch/i386/boot/compressed/ contains two source files: head.S and misc.c.

Table 4. Targets in linux/arch/i386/boot/compressed/Makefile

TargetCommand
vmlinux[a] $(LD) -Ttext 0x1000 -e startup_32 -o vmlinux head.o misc.o piggy.o
bvmlinux $(LD) -Ttext 0x100000 -e startup_32 -o bvmlinux head.o misc.o piggy.o
head.o $(CC) $(AFLAGS) -traditional -c head.S
misc.o
$(CC) $(CFLAGS) -DKBUILD_BASENAME=$(subst $(comma),_,$(subst -,_,$(*F)))
        -c misc.c[b]
piggy.o
tmppiggy=_tmp_$$$$piggy; \
rm -f $$tmppiggy $$tmppiggy.gz $$tmppiggy.lnk; \
$(OBJCOPY) $(SYSTEM) $$tmppiggy; \
gzip -f -9 < $$tmppiggy > $$tmppiggy.gz; \
echo "SECTIONS { .data : { input_len = .; \
        LONG(input_data_end - input_data) input_data = .; \
        *(.data) input_data_end = .; }}" > $$tmppiggy.lnk; \
$(LD) -r -o piggy.o -b binary $$tmppiggy.gz -b elf32-i386 \
        -T $$tmppiggy.lnk; \
rm -f $$tmppiggy $$tmppiggy.gz $$tmppiggy.lnk
Notes:
a. Target vmlinux here is different from that defined in linux/Makefile;
b. "subst" is a MAKE function; Refer to GNU make: Functions for String Substitution and Analysis.

piggy.o contains variable input_len and gzipped linux/vmlinux. input_len is at the beginning of piggy.o, and it is equal to the size of piggy.o excluding input_len itself. Refer to Using LD, the GNU linker: Section Data Expressions for "LONG(expression)" in piggy.o linker script.

To be exact, it is not linux/vmlinux itself (in ELF format) that is gzipped but its binary image, which is generated by objcopy command. Note that $(OBJCOPY) has been redefined by linux/arch/i386/Makefile in Section 2.3 to output raw binary using "-O binary" option.

When linking {bootsect, setup} or {bbootsect, bsetup}, $(LD) specifies "--oformat binary" option to output them in binary format. When making zImage (or bzImage), $(OBJCOPY) generates an intermediate binary output from compressed/vmlinux (or compressed/bvmlinux) too. It is vital that all components in zImage or bzImage are in raw binary format, so that the image can run by itself without asking a loader to load and relocate it.

Both vmlinux and bvmlinux prepend head.o and misc.o before piggy.o, but they are linked against different start addresses (0x1000 vs 0x100000).

2.6. linux/arch/i386/tools/build.c

linux/arch/i386/tools/build.c is a host utility to generate zImage or bzImage.

In linux/arch/i386/boot/Makefile:
tools/build bootsect setup compressed/vmlinux.out $(ROOT_DEV) > zImage

tools/build -b bbootsect bsetup compressed/bvmlinux.out $(ROOT_DEV) > bzImage
"-b" means is_big_kernel, used to check whether system image is too big.

tools/build outputs the following components to stdout, which is redirected to zImage or bzImage:

  1. bootsect or bbootsect: from linux/arch/i386/boot/bootsect.S, 512 bytes;

  2. setup or bsetup: from linux/arch/i386/boot/setup.S, 4 sectors or more, sector aligned;

  3. compressed/vmlinux.out or compressed/bvmlinux.out, including:

    1. head.o: from linux/arch/i386/boot/compressed/head.S;

    2. misc.o: from linux/arch/i386/boot/compressed/misc.c;

    3. piggy.o: from input_len and gzipped linux/vmlinux.

tools/build will change some contents of bootsect or bbootsect when outputting to stdout:

Table 5. Modification made by tools/build

OffsetByteVariableComment
1F1 (497)1setup_sectorsnumber of setup sectors, >=4
1F4 (500)2sys_sizesystem size in 16-bytes, little-endian
1FC (508)1minor_rootroot dev minor
1FD (509)1major_rootroot dev major

In the following chapters, compressed/vmlinux will be referred as vmlinux and compressed/bvmlinux as bvmlinux, if not confusing.

2.7. Reference