Setting up a board farm with LAVA and Docker
05 March 2019
In the previous post, I covered the simple use-case of deploying LAVA using Docker Compose and a single QEMU device. Let’s add some actual physical boards to the configuration to see how it can be extended.
Physical Layout
Here is my fledgling lab!
Pictured from left to right is a network switch, an x84_64 host running lava-server and lava-dispatcher, a USB hub, a Beaglebone Black, an ethernet connected relay board, and a Beaglebone X-15. On the lower shelf is an old ethernet-switchable PDU.
Of note is how the small and embedded boards are mounted. It took me a long time to figure out how to do this, because I’m used to large systems with cases that are either rack mountable or large enough to sit on a shelf and not move. With small ARM embedded boards, it didn’t occur to me until recently that cases don’t help! When a board is lighter than the cables its attached to, with or without a case, it’s difficult to maintain order.
It also took me a while to realize that standoffs come in two sizes: M2 and M3. Some of the boards I have require M2, and some require M3. They are not interchangeable in the least; you have to have both available.
To actually mount the boards, I used some 3/4” plywood and a wire shelf, both of which I already had. The plywood is cut into equally sized pieces that fit exactly 3 across. The plywood planks sit unattached on the wire shelf, but the shelf has a bit of a lip around the perimeter to keep the boards in place. Since the plywood boards are interchangeable and removable, they can be removed or swapped out from the rack for maintenance.
Plywood provides nice options for attaching boards. I opted to drill shallow holes to match the standoffs, and then fill them in with epoxy (I used JB Weld because it’s awesome and I had it laying around) to secure the standoffs to the plywood. Then, I used some sticky fasteners for zip-tie cable management. It is nice and solid, and the boards can easily be stacked for higher density.
This initial LAVA lab therefore has three devices - the two boards pictures, and a QEMU virtual device.
Using LAVA and Docker with hardware devices
Instead of going through the configuration line-by-line, what follows is a list of references to my and others’ docker-based LAVA implementations, followed by an FAQ of sorts that shows how various issues were solved.
References
danrue/lava.therub.org - The full docker-compose configuration for lava.therub.org.
beaglebone-black implementation
- lava-docker-compose extended to support a single beaglebone-black.
- Fork of lava-docker-compose which implements fastboot support using an lxc mocker, and SQUAD for reporting.
- LAVA package docker-compose recommended implementation. This implementation has each lava-server process broken out into its own container.
lava/pkg/docker - LAVA docker container implementation. Useful for investigating the Dockerfile and entrypoints for lava-server and lava-dispatcher.
There is additional ongoing work to make the docker deployment in LAVA easier and cleaner.
FAQ
Serial Interface
Often, ser2net is used to manage the serial connections to the physical boards, which allows the LAVA dispatcher to telnet to ser2net to view the serial console. The lava-dispatcher container comes with ser2net installed, but it does not start by default. My initial approach was to use it, but that was a mistake for two main reasons.
- The version of ser2net in debian stretch is old. There is a killer feature in newer versions, which allows you to set max-connections so that more than one client can telnet to a serial device at a time. When troubleshooting LAVA/board interaction, this is incredibly useful.
- Running ser2net in the dispatcher makes the boards coupled with the dispatchers, because only one process can attached to the serial devices at a time. This, for example, precludes running lavafed, and is an added complication.
The solution is to run ser2net in its own container. I’ve created one at danrue/ser2net, and it is simple to use:
version: '3.4'
services:
ser2net:
image: danrue/ser2net:3.5
volumes:
- ./ser2net/ser2net.conf:/etc/ser2net.conf
devices:
- /dev/serial/by-id/usb-Silicon_Labs_CP2102_USB_to_UART_Bridge_Controller_0001-if00-port0
TFTP
TFTP is a simple network service that runs on port 69/udp. This can easily be run in a container, the only slight complication being networking.
Docker containers running under docker-compose share a docker network - so name resolution just works. A board, however, would come up on the local network and not be able to access tftp in docker’s internal network. Therefore, the port needs to be exposed to the host system, and the host system’s dns name or IP needs to be used for TFTP.
Specifically, the IP needs to be set in lava-server’s /etc/dispatcher.d/dispatcher.yaml file:
dispatcher_ip: 10.100.0.60 # used for nfs and tftp
Similar to ser2net, I published a docker container that can be used to host
tftp. A volume needs to be mounted into the tftp container and shared with the
lava-dispatcher container. At runtime, lava-dispatcher writes files to
/srv/tftp. Likewise, by default, tftpd-hpa serves files from /src/tftp.
Therefore, danrue/tftpd-hpa can be used like:
version: '3.4'
services:
tftpd-hpa:
image: danrue/tftpd-hpa
volumes:
- tftp:/srv/tftp
ports:
- 69:69/udp
volumes:
tftp:
name: tftp-filesystem
The docker volume should be mounted into the dispatcher container in the same way.
NFS
NFS is similar to TFTP. Here is how I am deploying it:
services:
nfsd:
image: erichough/nfs-server
container_name: lava_nfsd
ports:
- 2049:2049 #nfsv4
volumes:
- nfsd:/var/lib/lava/dispatcher/tmp
- ./nfsd/exports:/etc/exports
cap_add:
- SYS_ADMIN
restart: always
volumes:
nfsd:
name: lava-nfsd
The nfsd volume also needs to be mounted into the dispatcher in the same way at
/var/lib/lava/dispatcher/tmp.