The Rub

Automatically Simple Since 2002

Setting up a board farm with LAVA and Docker

05 March 2019

In the previous post, I covered the simple use-case of deploying LAVA using Docker Compose and a single QEMU device. Let’s add some actual physical boards to the configuration to see how it can be extended.

Physical Layout

Here is my fledgling lab!

LAVA board farm with embedded ARM boards mounted in rack

Pictured from left to right is a network switch, an x84_64 host running lava-server and lava-dispatcher, a USB hub, a Beaglebone Black, an ethernet connected relay board, and a Beaglebone X-15. On the lower shelf is an old ethernet-switchable PDU.

Of note is how the small and embedded boards are mounted. It took me a long time to figure out how to do this, because I’m used to large systems with cases that are either rack mountable or large enough to sit on a shelf and not move. With small ARM embedded boards, it didn’t occur to me until recently that cases don’t help! When a board is lighter than the cables its attached to, with or without a case, it’s difficult to maintain order.

It also took me a while to realize that standoffs come in two sizes: M2 and M3. Some of the boards I have require M2, and some require M3. They are not interchangeable in the least; you have to have both available.

To actually mount the boards, I used some 3/4” plywood and a wire shelf, both of which I already had. The plywood is cut into equally sized pieces that fit exactly 3 across. The plywood planks sit unattached on the wire shelf, but the shelf has a bit of a lip around the perimeter to keep the boards in place. Since the plywood boards are interchangeable and removable, they can be removed or swapped out from the rack for maintenance.

Plywood provides nice options for attaching boards. I opted to drill shallow holes to match the standoffs, and then fill them in with epoxy (I used JB Weld because it’s awesome and I had it laying around) to secure the standoffs to the plywood. Then, I used some sticky fasteners for zip-tie cable management. It is nice and solid, and the boards can easily be stacked for higher density.

Beaglebone X15 mounted to plywood

This initial LAVA lab therefore has three devices - the two boards pictures, and a QEMU virtual device.

Using LAVA and Docker with hardware devices

Instead of going through the configuration line-by-line, what follows is a list of references to my and others’ docker-based LAVA implementations, followed by an FAQ of sorts that shows how various issues were solved.


danrue/ - The full docker-compose configuration for

beaglebone-black implementation

  • lava-docker-compose extended to support a single beaglebone-black.


  • Fork of lava-docker-compose which implements fastboot support using an lxc mocker, and SQUAD for reporting.


  • LAVA package docker-compose recommended implementation. This implementation has each lava-server process broken out into its own container.

lava/pkg/docker - LAVA docker container implementation. Useful for investigating the Dockerfile and entrypoints for lava-server and lava-dispatcher.

There is additional ongoing work to make the docker deployment in LAVA easier and cleaner.


Serial Interface

Often, ser2net is used to manage the serial connections to the physical boards, which allows the LAVA dispatcher to telnet to ser2net to view the serial console. The lava-dispatcher container comes with ser2net installed, but it does not start by default. My initial approach was to use it, but that was a mistake for two main reasons.

  1. The version of ser2net in debian stretch is old. There is a killer feature in newer versions, which allows you to set max-connections so that more than one client can telnet to a serial device at a time. When troubleshooting LAVA/board interaction, this is incredibly useful.
  2. Running ser2net in the dispatcher makes the boards coupled with the dispatchers, because only one process can attached to the serial devices at a time. This, for example, precludes running lavafed, and is an added complication.

The solution is to run ser2net in its own container. I’ve created one at danrue/ser2net, and it is simple to use:

    version: '3.4'
        image: danrue/ser2net:3.5
          - ./ser2net/ser2net.conf:/etc/ser2net.conf
          - /dev/serial/by-id/usb-Silicon_Labs_CP2102_USB_to_UART_Bridge_Controller_0001-if00-port0


TFTP is a simple network service that runs on port 69/udp. This can easily be run in a container, the only slight complication being networking.

Docker containers running under docker-compose share a docker network - so name resolution just works. A board, however, would come up on the local network and not be able to access tftp in docker’s internal network. Therefore, the port needs to be exposed to the host system, and the host system’s dns name or IP needs to be used for TFTP.

Specifically, the IP needs to be set in lava-server’s /etc/dispatcher.d/dispatcher.yaml file:

dispatcher_ip: # used for nfs and tftp

Similar to ser2net, I published a docker container that can be used to host tftp. A volume needs to be mounted into the tftp container and shared with the lava-dispatcher container. At runtime, lava-dispatcher writes files to /srv/tftp. Likewise, by default, tftpd-hpa serves files from /src/tftp.

Therefore, danrue/tftpd-hpa can be used like:

    version: '3.4'
        image: danrue/tftpd-hpa
          - tftp:/srv/tftp
          - 69:69/udp
        name: tftp-filesystem

The docker volume should be mounted into the dispatcher container in the same way.


NFS is similar to TFTP. Here is how I am deploying it:

    image: erichough/nfs-server
    container_name: lava_nfsd
      - 2049:2049 #nfsv4
      - nfsd:/var/lib/lava/dispatcher/tmp
      - ./nfsd/exports:/etc/exports
      - SYS_ADMIN
    restart: always
    name: lava-nfsd

The nfsd volume also needs to be mounted into the dispatcher in the same way at /var/lib/lava/dispatcher/tmp.