I’ve got a QNAP NAS and two Linux servers. Whenever the power goes down, the UPS kicks in and shut downs the NAS and the Linux servers, all good. The servers + NAS are automatically started when the power comes back on line using WOL. All good.
The problem is that I have apps running using Docker which heavily rely on connections to the NAS. As the Linux servers boot quicker than the NAS, the mount points are not mounted, and thus everything falls apart. Even when I manually re-mount, it’s not propagated to the Docker instances. All mount points use NFS.
Currently, I just reboot the Linux servers manually, and then all works well.
Probably easiest would be to run a cron job to check the mounts every x minutes, and if they are not mounted, then just reboot. The only issue is that this may cause an infinite loop of reboots if e.g. the NAS has been turned off.
I could also install a monitoring solution, but I’ve seen so many options that I’m not sure which one to do. If it’s easier with a monitoring solution, I’d like the simplest one.
I think that is a good question to write something positive about SystemD.
I start my services with SystemD. I also moved my containers and docker-compose stack to be started by systemd. And it does mounting and bind-mounts, too. So I removed things from /etc/fstab and instead created unit files for systemd to mount the network mounts. And then you can edit the service file that starts the docker-container and say it relies on the mount. SystemD will figure it out and start them in the correct order, wait until the network and the mounts are there.
You have to put some effort in but it’s not that hard. And for me it’s turned out to be pretty reliable and low maintenance.
Just specaluting… is it possible to mount NFS through systemd and make docker service dependent from that mount?
This is the answer. You can straight up make things dependent on .mount units that represent stuff in fstab. To add, you can create any number of systemd services that just check if something is “as you want it” and only then “start”. You simply make the Exec line “/bin/bash -c ‘your script here’”. Then you make whatever else you want dependent on it. For example I have such a unit that monitors for Internet connection by checking some public DNS servers. Then I have services that depend on Internet connection dependent on that. Here’s for example my Plex service which demonstrates how to depend on a mount, docker and shows how to manage a docker container with systemd:
~$ cat /etc/systemd/system/plex-docker.service [Unit] Description=Plex Media Server After=docker.service network-internet.service media-storage\x2dvolume1.mount After=docker.service [Service] TimeoutStartSec=0 Restart=always RestartSec=10 ExecStartPre=-/usr/bin/docker rm -f plex ExecStartPre=/usr/bin/docker pull plexinc/pms-docker:latest ExecStart=/usr/bin/docker run \ --name plex \ --net=host \ -e TZ="US/Eastern" \ -e "PLEX_UID=1000" \ -e "PLEX_GID=1000" \ -v /tmp:/tmp \ -v /var/lib/plex/config:/config \ -v /var/cache/plex/transcode:/transcode \ -v "/media/storage-volume1:/media/storage-volume1" \ plexinc/pms-docker:latest [Install] WantedBy=multi-user.targetBTW you can also do timers in systemd which allows doing what you can do with cron but much more flexibly and utilize dependencies too.
