The Rub

Automatically Simple Since 2002

Running LAVA with Docker Compose

01 March 2019

Lately I have been working on setting up a kernelCI lab (board farm) in my home using Linaro Automated Validation Architecture (LAVA). This lab will have a number of ARM-based boards attached to it which kernelCI will use to test the Linux kernel.

Background

Linaro Automated Validation Architecture (LAVA) is a software project that is used to automate running jobs on physical hosts in a board farm, often for the purposes of testing and continuous integration (CI). It has a server/dispatcher architecture that allows a large amount of boards to be managed by a single LAVA installation.

Docker is a widely used software project that makes packaging, managing, and deploying software easier.

kernelCI is a project that is used to build, boot, and test the Linux kernel across a wide number of boards in a distributed set of LAVA labs.

LAVA Federation (lavafed) is a project within LAVA that is used to continuously integrate (CI) LAVA itself. It does this by running LAVA jobs in participating LAVA labs, in order to test LAVA itself.

Goals

I started off with a few goals in mind:

  • Learn LAVA and kernelCI
  • Deploy my own board farm and add it to kernelCI
  • Help improve LAVA and kernelCI
  • Make it easier for future contributors and users

Installing LAVA

The traditional way of installing LAVA is to install Debian (based on whichever version LAVA is supporting), and then follow the LAVA documentation to install using apt.

A more modern approach is to deploy using Docker containers. This is an approach that is under active development, as it was only recently added as a supported way to run LAVA.

One more digression before getting into the details: LAVA has a steep learning curve. It has a lot of components, and the things it automates (embedded boards) are also complicated. The standard advice is to start slowly with the most common and easiest use-cases before moving to more advanced tasks. Heed this advice!

To that end, the first recommended use-case is to install lava-server and lava-dispatcher to the same host, and add a QEMU virtual device. This is a nice first goal because it does not require hardware - the details about physically attaching and dealing with hardware can be deferred until the basic LAVA stuff is understood and working. Besides, QEMU devices are super handy to use in testing.

Deploying LAVA with QEMU on Docker

This is where Docker starts to come in handy. If you want to install LAVA using the traditional method, you will need a host running Debian Stretch. However, if you are using Docker - you just need a host with Docker installed. Let’s get started.

Containers

LAVA distributes two basic containers: lavasoftware/lava-server and lavasoftware/lava-dispatcher. These are multi-arch containers that support x86_64 and aarch64.

It is easy enough to start them and poke around to see how they work. For example, run:

[email protected]:~$ docker run --rm -it lavasoftware/lava-server
Starting postgresql
[ ok ] Starting PostgreSQL 9.6 database server: main.
done

Waiting for postgresql
[done]
...

ctrl-c out of that and then likewise, lava-dispatcher can be run similarly:

[email protected]:~$ docker run --rm -it lavasoftware/lava-dispatcher
2019-03-01 20:21:38,538    INFO [INIT] LAVA slave has started.
2019-03-01 20:21:38,538    INFO [INIT] Using protocol version 3
2019-03-01 20:21:38,540   DEBUG [INIT] Connection is not encrypted
2019-03-01 20:21:38,551    INFO [BTSP] Connecting to master [tcp://localhost:5556] as <324b989947b4>
2019-03-01 20:21:38,552    INFO [BTSP] Greeting the master [tcp://localhost:5556] => 'HELLO'
2019-03-01 20:21:43,557   DEBUG [BTSP] Checking master [localhost:5556] to create socket for 324b989947b4
2019-03-01 20:21:43,558   DEBUG [BTSP] socket IPv4 address: 127.0.0.1
2019-03-01 20:21:43,558    INFO [BTSP] Greeting master => 'HELLO_RETRY' (using the same version?)

Without additional configuration, not much will happen. We need to deal with all of the state of the containers; things like the database and the job output directory. We also need to handle the configuration of the containers, in a way that is persistent and in source control so that changes can be tracked over time.

For configuration, the general strategy is to set and pass environmental variables into the containers at runtime to control their behavior. When environmental variables are insufficient, configuration files can be mounted in using volume mounts.

For state, docker volumes should be used to save the contents of the database and job output on the docker host, rather than inside the docker container.

Using Docker Compose

Docker Compose is a python tool that allows us to put our docker configuration into a yaml file - making it easier to manage, and more portable, than things like shell scripts.

An example docker-compose.yml can be found at github.com/danrue/lava-docker-compose. This repository is designed to serve as a reference implementation for a simple but complete docker deployment of LAVA with a single QEMU device.

Follow the instructions in the README to get started.

Additionally, the following video is a (long) walk-through on how to go from scratch to the finished docker-compose environment.

Detailed docker-compose walk through

The docker-compose.yml seemingly has a lot going on, so it’s important to understand how everything fits together. Below is the compose file, followed by a line-by-line explanation. This file may be outdated as compared to the one in github, but the explanation of how everything fits together should be useful even as the specific details change over time.

version: '3.4'
services:

  database:
    image: postgres:9.6
    container_name: lava_postgres
    environment:
      POSTGRES_USER: lavaserver
      POSTGRES_PASSWORD: mysecretpassword
    volumes:
      - pgdata:/var/lib/postgresql/data

  squid:
    image: datadog/squid
    container_name: lava_squid
    volumes:
      - squid:/var/spool/squid
      - ./squid/squid.conf:/etc/squid/squid.conf
    ports:
      - 3128:3128

  server:
    #image: lavasoftware/amd64-lava-server:2019.01
    # Use an updated entrypoint; this is necessary to run provision.sh at boot time
    # See https://git.lavasoftware.org/lava/pkg/docker/merge_requests/10
    build:
      context: ./server-docker

    container_name: lava_server
    ports:
      - 80:80
    volumes:
      # Job artifact storage
      - joboutput:/var/lib/lava-server/default/media/job-output

      # server configuration files and directories
      - ./server-overlay/etc/lava-server/settings.conf:/etc/lava-server/settings.conf
      - ./server-overlay/etc/lava-server/instance.conf:/etc/lava-server/instance.conf
      - ./server-overlay/etc/lava-server/dispatcher-config/health-checks:/etc/lava-server/dispatcher-config/health-checks
      - ./server-overlay/etc/lava-server/dispatcher-config/devices:/etc/lava-server/dispatcher-config/devices
      - ./server-overlay/etc/lava-server/env.yaml:/etc/lava-server/env.yaml

      # provisioning script to add users and boards
      - ./server-overlay/root/provision.sh:/root/provision.sh

    depends_on:
      - database

  dispatcher:
    image: lavasoftware/amd64-lava-dispatcher:2019.01
    container_name: lava_dispatcher
    devices:
      - /dev/kvm # needed for QEMU
      - /dev/net/tun # needed for QEMU
    cap_add:
      - NET_ADMIN # needed for QEMU
    environment:
      - "DISPATCHER_HOSTNAME=--hostname=dispatcher"
      - "LOGGER_URL=tcp://server:5555" # url to send logs
      - "MASTER_URL=tcp://server:5556" # url of lava master
    volumes:
      - '/boot:/boot:ro'
      - '/lib/modules:/lib/modules:ro'

volumes:
  # postgres data volume
  pgdata:
    name: lava-server-pgdata

  # squid cache volume
  squid:
    name: lava-squid-cache

  # lava-server job artifact volume
  joboutput:
    name: lava-server-job-output

docker-compose details

For reference, all of the docker-compose file format documentation can be found at docs.docker.com.

version: '3.4'

docker-compose implements file versioning for backward compatibility purposes. Version 3.4 is specified here because it was necessary for the volume name parameter, for example.

services:

Each container that will run will be listed under the services: heading.

  database:
    image: postgres:9.6
    container_name: lava_postgres
    environment:
      POSTGRES_USER: lavaserver
      POSTGRES_PASSWORD: mysecretpassword
    volumes:
      - pgdata:/var/lib/postgresql/data

The database container. Using postgresql’s official container makes it easy to choose our database version and manage our data outside the container using a docker volume. In this definition, we’re using docker image postgres version 9.6. The container’s runtime name will be ‘lava_postgres’ for convenience. Two environment variables are being passed in - one for username and one for password. These same values will have to be given to LAVA server. Finally, the ‘pgdata’ named volume is mounted into the container at the default database path, so that our actual data remains outside the container.

  squid:
    image: datadog/squid
    container_name: lava_squid
    volumes:
      - squid:/var/spool/squid
      - ./squid/squid.conf:/etc/squid/squid.conf
    ports:
      - 3128:3128

This is a nice-to-have. Squid is commonly used to cache downloads. In this case, it’s using datadog’s squid container, and it’s using a docker volume for the cache path. Squid requires a config file, so that is managed locally and mounted into the container at runtime. Finally, port 3128 is exposed so that squid can be used outside of docker.

  server:
    #image: lavasoftware/lava-server:2019.01
    # Use an updated entrypoint; this is necessary to run provision.sh at boot time
    # See https://git.lavasoftware.org/lava/pkg/docker/merge_requests/10
    build:
      context: ./server-docker

    container_name: lava_server
    ports:
      - 80:80
    volumes:
      # Job artifact storage
      - joboutput:/var/lib/lava-server/default/media/job-output

      # server configuration files and directories
      - ./server-overlay/etc/lava-server/settings.conf:/etc/lava-server/settings.conf
      - ./server-overlay/etc/lava-server/instance.conf:/etc/lava-server/instance.conf
      - ./server-overlay/etc/lava-server/dispatcher-config/health-checks:/etc/lava-server/dispatcher-config/health-checks
      - ./server-overlay/etc/lava-server/dispatcher-config/devices:/etc/lava-server/dispatcher-config/devices
      - ./server-overlay/etc/lava-server/env.yaml:/etc/lava-server/env.yaml

      # provisioning script to add users and boards
      - ./server-overlay/root/provision.sh:/root/provision.sh

    depends_on:
      - database

The lava-server container. This one needs quite a bit of configuration. Hopefully over time this will become more simple. In this case we have a build context - this means that we are modifying the upstream container. In this case, it is done so that the entrypoint can be modified to run our provisioning script. This need will go away with the release of LAVA 2019.02.

Port 80 is exposed to the host. This is the main port that LAVA users will use to talk to LAVA server via port 80. The volume mounts are for configuration files that require modification, the job output volume, and the provision script that runs on startup by the entrypoint.

Finally, there’s a depends_on database clause, which just makes it so that server doesn’t start until the database container has started.

  dispatcher:
    image: lavasoftware/lava-dispatcher:2019.01
    container_name: lava_dispatcher
    devices:
      - /dev/kvm # needed for QEMU
      - /dev/net/tun # needed for QEMU
    cap_add:
      - NET_ADMIN # needed for QEMU
    environment:
      - "DISPATCHER_HOSTNAME=--hostname=dispatcher"
      - "LOGGER_URL=tcp://server:5555" # url to send logs
      - "MASTER_URL=tcp://server:5556" # url of lava master
    volumes:
      - '/boot:/boot:ro'
      - '/lib/modules:/lib/modules:ro'

The dispatcher is used to actually communicate with boards and devices under test. It needs additional host level privileges for QEMU - /dev/kvm and /dev/net/tun, in addition to NET_ADMIN capability. There are a couple required environment variables, which tell the dispatcher how to connect to the lava server. Finally, /boot and /lib/modules are mounted in because the dispatcher requires them.

volumes:
  # postgres data volume
  pgdata:
    name: lava-server-pgdata

  # squid cache volume
  squid:
    name: lava-squid-cache

  # lava-server job artifact volume
  joboutput:
    name: lava-server-job-output

Docker volumes are used to store persistent data. Use docker volume help to manage and view docker volumes. Ultimately, they are just directories on the docker host.