All the top-level test configurations are contained in a YAML file:
test-configs.yaml
.
This defines everything that can be run on test platforms. The primary
use-case is to generate and submit test job definitions as done by
kci_test
.
There are several sections in this file:
The test configurations are the main entries as they define which tests
actually have to be run. They refer to entries in other sections of the file
in order to provide some full combinations, for example to define that an
igt-kms-exynos
test plan using a buster-igt
file system should be run on an
odroid-xu3
device.
In addition to those sections, there are some filters (passlist, blocklist) to fine tune which tests should be run. For example, some platforms are not supported in older stable kernel branches so they’ll typically have a blocklist to only run tests with newer kernels.
Each device type has an entry in the device_types
dictionary. Here’s an
example:
device_types:
beagle_xm:
name: 'beagle-xm'
mach: omap2
class: arm-dtb
boot_method: uboot
dtb: 'omap3-beagle-xm.dtb'
filters:
- blocklist: *allmodconfig_filter
- blocklist: {kernel: ['v3.14']}
The attributes are:
name
needs to match what test labs use to identify the platform.mach
is to define a family of SoCs, originally from the arch/arm/mach-*
board file names.class
is used here to define a particular class of devices such as
arm-dtb
or arm64-dtb
.arch
is to define the CPU architecture following the Linux kernel names
(arm64
, riscv
, x86_64
…). It is not required with the arm-dtb
class
as it is already specific to the arm
architecture.boot_method
is to define how to boot the device (uboot
, grub
…).dtb
is an optional attribute to specify the name of the device tree. By
default, device types of the arm-dtb
and arm64-dtb
class will use name
if there is no explicit dtb
attribute.filters
is an arbitrary list of filters to only run tests with certain
configuration combinations. See the filters section for more
details.flags
is an arbitrary list of strings with properties of the device type,
to also filter out some job configurations. See the Flags section
below for more details.Note: In this example, the class is architecture-specific so it also defines
the arch
value which is why it does not appear here.
Test configuration filters may use any of the following options:
arch
is the CPU architecture namedefconfig
is the full defconfig namekernel
is the full kernel version namebuild_environment
is basically the compiler nametree
is the name of the kernel git tree (e.g. mainline
, next
…)branch
is the name of the kernel git branchlab
is the lab nameIn order to avoid duplicating or referencing the same filters repeatedly, there are some default filters which apply to all test configurations. Any filter explicitly defined will take precedence over the default ones.
test_plan_default_filters
This filter definition acts as the default for all test plans. It’s essentially there to only test the relevant defconfig for each arch (i.e. multi_v7_defconfig on arm, defconfig on arm64 etc…).
device_default_filters
Similarly, this defines the default filters for all the devices. Typically it
disables things that can’t be run on most devices, such as the allmodconfig
kernel builds as they are too large to boot from a ramdisk with all the
modules. With some architectures, they are known to not really boot anyway.
Device types can also have a list of flags, for example:
device_types:
meson_gxbb_p200:
name: 'meson-gxbb-p200'
mach: amlogic
class: arm64-dtb
boot_method: uboot
flags: ['lpae', 'big_endian']
filters:
- blocklist: {defconfig: ['allnoconfig', 'allmodconfig']}
This can then be used to filter out some jobs, for example kci_test
will pass
the big_endian
flag when the kernel build was for big-endian.
Flags currently in use are:
big_endian
to tell whether the device can boot big-endian kernelslpae
to tell whether the device can boot kernels built with LPAE enabled
(Large Physical Address Extension for ARMv7)fastboot
to tell whether the device can boot with fastboot (stored in jobs
meta-data but not actively used)Each test plan has a set of template files, and an entry in the test_plans
dictionary. Typically, tests are for LAVA and the templates are stored in the
config/lava
.
Test plans rely on rootfs definitions, so here’s a simplified example:
file_systems:
debian_buster_ramdisk:
type: debian
ramdisk: 'buster/20210730.6/{arch}/rootfs.cpio.gz'
test_plans:
sleep:
rootfs: debian_buster_ramdisk
params:
sleep_params: mem freeze
It’s required to specify a rootfs
attribute which points to an entry in the
file_systems
dictionary. The root file system should contain all the tools
and test suites required to run the test plan.
When generating test job definitions, the path to the test template file is created using this default pattern:
'{plan}/{category}-{method}-{protocol}-{rootfs}-{plan}-template.jinja2'
The plan
and rootfs
values are coming from the test plan definition. The
other values are coming from the test platform (determined later at runtime)
and file system type definitions: method
is the boot method, protocol
is
how to download the kernel etc.
So when adding a test plan, typically there will be one template with only the
test steps and other templates inheriting it to add configuration specific
steps. For example, still with the sleep
test
plan:
generic-barebox-tftp-ramdisk-sleep-template.jinja2
generic-depthcharge-tftp-ramdisk-sleep-template.jinja2
generic-uboot-tftp-ramdisk-sleep-template.jinja2
sleep.jinja2
There are 3 templates to run this test plan on devices that can boot either
with U-Boot, Barebox or Depthcharge. They all include the test steps defined
in sleep.jinja2
.
Defining device types, file systems and test plans is necessary but not
sufficient to run tests. There also need to be an entry in test_configs
to
bind together a device with a list of test plans. For example, this device
type is configured to run several test plans:
test_configs:
- device_type: bcm2836-rpi-2-b
test_plans:
- baseline
- ltp-crypto
- sleep
- usb
It’s possible to have several entries with the same device_type
to define
special filters. For example, if some tests need to only be run on that device
in a specific lab, or with a specific defconfig or tree etc. Here’s an example
to run all tests on any kernel except igt
which should only be on kernels
more recent than v4.14
(i.e. v4.19
LTS onwards):
test_configs:
- device_type: rk3399-gru-kevin
test_plans:
- baseline
- baseline-nfs
- cros-ec
- ltp-fcntl-locktests
- ltp-pty
- ltp-timers
- sleep
- v4l2-compliance-uvc
- device_type: rk3399-gru-kevin
test_plans:
- igt-gpu-panfrost
- igt-kms-rockchip
filters:
- blocklist: {kernel: ['v3.', 'v4.4', 'v4.9', 'v4.14']}
File systems contain all the user-space files. They are required to include the necessary dependencies in order to run the test suites associated with them.
Each file system has a type to define the base URL and architecture names. For
example, here’s the buildroot
type:
file_system_types:
buildroot:
url: 'http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05-6-g8983f3b738df'
arch_map:
arm64be: [{arch: arm64, endian: big}]
armeb: [{arch: arm, endian: big}]
armel: [{arch: arm}]
x86: [{arch: i386}, {arch: x86_64}]
mipsel: [{arch: mips}]
url
is the base URL where the file systems can be downloaded.arch_map
is a dictionary to translate kernel CPU architecture names into
file system specific names.Then file systems can be defined for each type, with some additional information to work out the full URL of each variant. For example:
file_systems:
buildroot_ramdisk:
type: buildroot
ramdisk: '{arch}/base/rootfs.cpio.gz'
debian_buster_nfs:
type: debian
ramdisk: 'buster/20210730.6/{arch}/initrd.cpio.gz'
nfs: 'buster/20210730.6/{arch}/full.rootfs.tar.xz'
root_type: nfs
These file systems use different types: buildroot
and debian
, so they have
different base URLs and different arch_map
attributes as each distro has its
own architecture naming convention. The file system configs each provide a
different URL for their different variants (ramdisk and NFS in this case). The
URL names such as ramdisk
or nfs
are arbitrary and used by kci_test
to
render templates into test job definitions. The {arch}
template value will
be replaced by one of the entries in the arch_map
for the file system type or
the regular kernel one.