Automatically Simple Since 2002
01 March 2019
Lately I have been working on setting up a kernelCI lab (board farm) in my home using Linaro Automated Validation Architecture (LAVA). This lab will have a number of ARM-based boards attached to it which kernelCI will use to test the Linux kernel.
Linaro Automated Validation Architecture (LAVA) is a software project that is used to automate running jobs on physical hosts in a board farm, often for the purposes of testing and continuous integration (CI). It has a server/dispatcher architecture that allows a large amount of boards to be managed by a single LAVA installation.
Docker is a widely used software project that makes packaging, managing, and deploying software easier.
kernelCI is a project that is used to build, boot, and test the Linux kernel across a wide number of boards in a distributed set of LAVA labs.
LAVA Federation (lavafed) is a project within LAVA that is used to continuously integrate (CI) LAVA itself. It does this by running LAVA jobs in participating LAVA labs, in order to test LAVA itself.
I started off with a few goals in mind:
The traditional way of installing LAVA is to install Debian (based on whichever version LAVA is supporting), and then follow the LAVA documentation to install using apt.
A more modern approach is to deploy using Docker containers. This is an approach that is under active development, as it was only recently added as a supported way to run LAVA.
One more digression before getting into the details: LAVA has a steep learning curve. It has a lot of components, and the things it automates (embedded boards) are also complicated. The standard advice is to start slowly with the most common and easiest use-cases before moving to more advanced tasks. Heed this advice!
To that end, the first recommended use-case is to install lava-server and lava-dispatcher to the same host, and add a QEMU virtual device. This is a nice first goal because it does not require hardware - the details about physically attaching and dealing with hardware can be deferred until the basic LAVA stuff is understood and working. Besides, QEMU devices are super handy to use in testing.
This is where Docker starts to come in handy. If you want to install LAVA using the traditional method, you will need a host running Debian Stretch. However, if you are using Docker - you just need a host with Docker installed. Let’s get started.
LAVA distributes two basic containers: lavasoftware/lava-server and lavasoftware/lava-dispatcher. These are multi-arch containers that support x86_64 and aarch64.
It is easy enough to start them and poke around to see how they work. For example, run:
drue@xps:~$ docker run --rm -it lavasoftware/lava-server
Starting postgresql
[ ok ] Starting PostgreSQL 9.6 database server: main.
done
Waiting for postgresql
[done]
...
ctrl-c out of that and then likewise, lava-dispatcher can be run similarly:
drue@xps:~$ docker run --rm -it lavasoftware/lava-dispatcher
2019-03-01 20:21:38,538 INFO [INIT] LAVA slave has started.
2019-03-01 20:21:38,538 INFO [INIT] Using protocol version 3
2019-03-01 20:21:38,540 DEBUG [INIT] Connection is not encrypted
2019-03-01 20:21:38,551 INFO [BTSP] Connecting to master [tcp://localhost:5556] as <324b989947b4>
2019-03-01 20:21:38,552 INFO [BTSP] Greeting the master [tcp://localhost:5556] => 'HELLO'
2019-03-01 20:21:43,557 DEBUG [BTSP] Checking master [localhost:5556] to create socket for 324b989947b4
2019-03-01 20:21:43,558 DEBUG [BTSP] socket IPv4 address: 127.0.0.1
2019-03-01 20:21:43,558 INFO [BTSP] Greeting master => 'HELLO_RETRY' (using the same version?)
Without additional configuration, not much will happen. We need to deal with all of the state of the containers; things like the database and the job output directory. We also need to handle the configuration of the containers, in a way that is persistent and in source control so that changes can be tracked over time.
For configuration, the general strategy is to set and pass environmental variables into the containers at runtime to control their behavior. When environmental variables are insufficient, configuration files can be mounted in using volume mounts.
For state, docker volumes should be used to save the contents of the database and job output on the docker host, rather than inside the docker container.
Docker Compose is a python tool that allows us to put our docker configuration into a yaml file - making it easier to manage, and more portable, than things like shell scripts.
An example docker-compose.yml can be found at github.com/danrue/lava-docker-compose. This repository is designed to serve as a reference implementation for a simple but complete docker deployment of LAVA with a single QEMU device.
Follow the instructions in the README to get started.
Additionally, the following video is a (long) walk-through on how to go from scratch to the finished docker-compose environment.
The docker-compose.yml seemingly has a lot going on, so it’s important to understand how everything fits together. Below is the compose file, followed by a line-by-line explanation. This file may be outdated as compared to the one in github, but the explanation of how everything fits together should be useful even as the specific details change over time.
version: '3.4'
services:
database:
image: postgres:9.6
container_name: lava_postgres
environment:
POSTGRES_USER: lavaserver
POSTGRES_PASSWORD: mysecretpassword
volumes:
- pgdata:/var/lib/postgresql/data
squid:
image: datadog/squid
container_name: lava_squid
volumes:
- squid:/var/spool/squid
- ./squid/squid.conf:/etc/squid/squid.conf
ports:
- 3128:3128
server:
#image: lavasoftware/amd64-lava-server:2019.01
# Use an updated entrypoint; this is necessary to run provision.sh at boot time
# See https://git.lavasoftware.org/lava/pkg/docker/merge_requests/10
build:
context: ./server-docker
container_name: lava_server
ports:
- 80:80
volumes:
# Job artifact storage
- joboutput:/var/lib/lava-server/default/media/job-output
# server configuration files and directories
- ./server-overlay/etc/lava-server/settings.conf:/etc/lava-server/settings.conf
- ./server-overlay/etc/lava-server/instance.conf:/etc/lava-server/instance.conf
- ./server-overlay/etc/lava-server/dispatcher-config/health-checks:/etc/lava-server/dispatcher-config/health-checks
- ./server-overlay/etc/lava-server/dispatcher-config/devices:/etc/lava-server/dispatcher-config/devices
- ./server-overlay/etc/lava-server/env.yaml:/etc/lava-server/env.yaml
# provisioning script to add users and boards
- ./server-overlay/root/provision.sh:/root/provision.sh
depends_on:
- database
dispatcher:
image: lavasoftware/amd64-lava-dispatcher:2019.01
container_name: lava_dispatcher
devices:
- /dev/kvm # needed for QEMU
- /dev/net/tun # needed for QEMU
cap_add:
- NET_ADMIN # needed for QEMU
environment:
- "DISPATCHER_HOSTNAME=--hostname=dispatcher"
- "LOGGER_URL=tcp://server:5555" # url to send logs
- "MASTER_URL=tcp://server:5556" # url of lava master
volumes:
- '/boot:/boot:ro'
- '/lib/modules:/lib/modules:ro'
volumes:
# postgres data volume
pgdata:
name: lava-server-pgdata
# squid cache volume
squid:
name: lava-squid-cache
# lava-server job artifact volume
joboutput:
name: lava-server-job-output
For reference, all of the docker-compose file format documentation can be found at docs.docker.com.
version: '3.4'
docker-compose implements file versioning for backward compatibility purposes. Version 3.4 is specified here because it was necessary for the volume name parameter, for example.
services:
Each container that will run will be listed under the services: heading.
database:
image: postgres:9.6
container_name: lava_postgres
environment:
POSTGRES_USER: lavaserver
POSTGRES_PASSWORD: mysecretpassword
volumes:
- pgdata:/var/lib/postgresql/data
The database container. Using postgresql’s official container makes it easy to choose our database version and manage our data outside the container using a docker volume. In this definition, we’re using docker image postgres version 9.6. The container’s runtime name will be ‘lava_postgres’ for convenience. Two environment variables are being passed in - one for username and one for password. These same values will have to be given to LAVA server. Finally, the ‘pgdata’ named volume is mounted into the container at the default database path, so that our actual data remains outside the container.
squid:
image: datadog/squid
container_name: lava_squid
volumes:
- squid:/var/spool/squid
- ./squid/squid.conf:/etc/squid/squid.conf
ports:
- 3128:3128
This is a nice-to-have. Squid is commonly used to cache downloads. In this case, it’s using datadog’s squid container, and it’s using a docker volume for the cache path. Squid requires a config file, so that is managed locally and mounted into the container at runtime. Finally, port 3128 is exposed so that squid can be used outside of docker.
server:
#image: lavasoftware/lava-server:2019.01
# Use an updated entrypoint; this is necessary to run provision.sh at boot time
# See https://git.lavasoftware.org/lava/pkg/docker/merge_requests/10
build:
context: ./server-docker
container_name: lava_server
ports:
- 80:80
volumes:
# Job artifact storage
- joboutput:/var/lib/lava-server/default/media/job-output
# server configuration files and directories
- ./server-overlay/etc/lava-server/settings.conf:/etc/lava-server/settings.conf
- ./server-overlay/etc/lava-server/instance.conf:/etc/lava-server/instance.conf
- ./server-overlay/etc/lava-server/dispatcher-config/health-checks:/etc/lava-server/dispatcher-config/health-checks
- ./server-overlay/etc/lava-server/dispatcher-config/devices:/etc/lava-server/dispatcher-config/devices
- ./server-overlay/etc/lava-server/env.yaml:/etc/lava-server/env.yaml
# provisioning script to add users and boards
- ./server-overlay/root/provision.sh:/root/provision.sh
depends_on:
- database
The lava-server container. This one needs quite a bit of configuration. Hopefully over time this will become more simple. In this case we have a build context - this means that we are modifying the upstream container. In this case, it is done so that the entrypoint can be modified to run our provisioning script. This need will go away with the release of LAVA 2019.02.
Port 80 is exposed to the host. This is the main port that LAVA users will use to talk to LAVA server via port 80. The volume mounts are for configuration files that require modification, the job output volume, and the provision script that runs on startup by the entrypoint.
Finally, there’s a depends_on database clause, which just makes it so that server doesn’t start until the database container has started.
dispatcher:
image: lavasoftware/lava-dispatcher:2019.01
container_name: lava_dispatcher
devices:
- /dev/kvm # needed for QEMU
- /dev/net/tun # needed for QEMU
cap_add:
- NET_ADMIN # needed for QEMU
environment:
- "DISPATCHER_HOSTNAME=--hostname=dispatcher"
- "LOGGER_URL=tcp://server:5555" # url to send logs
- "MASTER_URL=tcp://server:5556" # url of lava master
volumes:
- '/boot:/boot:ro'
- '/lib/modules:/lib/modules:ro'
The dispatcher is used to actually communicate with boards and devices under test. It needs additional host level privileges for QEMU - /dev/kvm and /dev/net/tun, in addition to NET_ADMIN capability. There are a couple required environment variables, which tell the dispatcher how to connect to the lava server. Finally, /boot and /lib/modules are mounted in because the dispatcher requires them.
volumes:
# postgres data volume
pgdata:
name: lava-server-pgdata
# squid cache volume
squid:
name: lava-squid-cache
# lava-server job artifact volume
joboutput:
name: lava-server-job-output
Docker volumes are used to store persistent data. Use docker volume help to
manage and view docker volumes. Ultimately, they are just directories on the
docker host.