05 March 2019
In the previous post, I covered the simple use-case of deploying LAVA using Docker Compose and a single QEMU device. Let’s add some actual physical boards to the configuration to see how it can be extended.
Here is my fledgling lab!
Pictured from left to right is a network switch, an x84_64 host running lava-server and lava-dispatcher, a USB hub, a Beaglebone Black, an ethernet connected relay board, and a Beaglebone X-15. On the lower shelf is an old ethernet-switchable PDU.
Of note is how the small and embedded boards are mounted. It took me a long time to figure out how to do this, because I’m used to large systems with cases that are either rack mountable or large enough to sit on a shelf and not move. With small ARM embedded boards, it didn’t occur to me until recently that cases don’t help! When a board is lighter than the cables its attached to, with or without a case, it’s difficult to maintain order.
It also took me a while to realize that standoffs come in two sizes: M2 and M3. Some of the boards I have require M2, and some require M3. They are not interchangeable in the least; you have to have both available.
To actually mount the boards, I used some 3/4” plywood and a wire shelf, both of which I already had. The plywood is cut into equally sized pieces that fit exactly 3 across. The plywood planks sit unattached on the wire shelf, but the shelf has a bit of a lip around the perimeter to keep the boards in place. Since the plywood boards are interchangeable and removable, they can be removed or swapped out from the rack for maintenance.
Plywood provides nice options for attaching boards. I opted to drill shallow holes to match the standoffs, and then fill them in with epoxy (I used JB Weld because it’s awesome and I had it laying around) to secure the standoffs to the plywood. Then, I used some sticky fasteners for zip-tie cable management. It is nice and solid, and the boards can easily be stacked for higher density.
This initial LAVA lab therefore has three devices - the two boards pictures, and a QEMU virtual device.
Instead of going through the configuration line-by-line, what follows is a list of references to my and others’ docker-based LAVA implementations, followed by an FAQ of sorts that shows how various issues were solved.
lava/pkg/docker - LAVA docker container implementation. Useful for investigating the Dockerfile and entrypoints for lava-server and lava-dispatcher.
There is additional ongoing work to make the docker deployment in LAVA easier and cleaner.
Often, ser2net is used to manage the serial connections to the physical boards, which allows the LAVA dispatcher to telnet to ser2net to view the serial console. The lava-dispatcher container comes with ser2net installed, but it does not start by default. My initial approach was to use it, but that was a mistake for two main reasons.
The solution is to run ser2net in its own container. I’ve created one at danrue/ser2net, and it is simple to use:
version: '3.4' services: ser2net: image: danrue/ser2net:3.5 volumes: - ./ser2net/ser2net.conf:/etc/ser2net.conf devices: - /dev/serial/by-id/usb-Silicon_Labs_CP2102_USB_to_UART_Bridge_Controller_0001-if00-port0
TFTP is a simple network service that runs on port 69/udp. This can easily be run in a container, the only slight complication being networking.
Docker containers running under docker-compose share a docker network - so name resolution just works. A board, however, would come up on the local network and not be able to access tftp in docker’s internal network. Therefore, the port needs to be exposed to the host system, and the host system’s dns name or IP needs to be used for TFTP.
Specifically, the IP needs to be set in lava-server’s /etc/dispatcher.d/dispatcher.yaml file:
dispatcher_ip: 10.100.0.60 # used for nfs and tftp
Similar to ser2net, I published a docker container that can be used to host
tftp. A volume needs to be mounted into the tftp container and shared with the
lava-dispatcher container. At runtime, lava-dispatcher writes files to
/srv/tftp. Likewise, by default, tftpd-hpa serves files from
Therefore, danrue/tftpd-hpa can be used like:
version: '3.4' services: tftpd-hpa: image: danrue/tftpd-hpa volumes: - tftp:/srv/tftp ports: - 69:69/udp volumes: tftp: name: tftp-filesystem
The docker volume should be mounted into the dispatcher container in the same way.
NFS is similar to TFTP. Here is how I am deploying it:
services: nfsd: image: erichough/nfs-server container_name: lava_nfsd ports: - 2049:2049 #nfsv4 volumes: - nfsd:/var/lib/lava/dispatcher/tmp - ./nfsd/exports:/etc/exports cap_add: - SYS_ADMIN restart: always volumes: nfsd: name: lava-nfsd
The nfsd volume also needs to be mounted into the dispatcher in the same way at