I have some of my Docker containers bind to Tailscale IPs on the host. Even though I edited the Docker service to depend on tailscaled.service and it starts about 10 seconds after it, every once in a while Tailscale interface won’t be ready by the time the containers try to spin up after a reboot.

Since restart policies do not apply to containers that has never started and ran at least for a while, Docker gives me no choice but to restart either those containers or the Docker daemon manually.

Is there a way to have Docker daemon try to restart containers even though they failed on the first try?

A restart policy only takes effect after a container starts successfully. In this case, starting successfully means that the container is up for at least 10 seconds and Docker has started monitoring it. This prevents a container which doesn’t start at all from going into a restart loop.

https://docs.docker.com/engine/containers/start-containers-automatically

  • Daniel Quinn@lemmy.caEnglish
    12·
    2 months ago

    You probably want to look into Health Checks. I believe you can tell Docker to “start service B when service A is healthy”, so you can define your health check with a script that depends on Tailscale functioning.

    • just_another_person@lemmy.worldEnglish
      4·
      2 months ago

      This is the correct answer, but you need a few things to clarify:

      1. The issue isn’t the Docker system service. Don’t make that depend on Tailscale
      2. Add a healthcheck and restart policy to the container to make it fail when conditions aren’t met, and restart until they are successful
      3. Build in some time tolerance at the service level inside the container to prevent it from flailing if your Tailscale healthchecks don’t pass after they initially start. Don’t rely solely on container health checks to ensure it works properly as that might not always be possible.
      • horse_battery_staple@lemmy.worldEnglish
        2·
        2 months ago

        Considering it’s tailscale, one may want to have the service fail though as tailscale is sometimes not used for convenience but security concerns instead.

    • Bakkoda@sh.itjust.worksEnglish
      2·
      2 months ago

      I need to figure this out on my immich stack. VM reboot only starts two out of the 4 services. Ty for the info.

  • horse_battery_staple@lemmy.worldEnglish
    8·
    2 months ago

    Delay the start of your containers with the tailscale dependency. Are you using required or depends_on in your docker-compose.yml

    https://hatchjs.com/docker-compose-conditionally-start-service/

    If you’re using kubernetes you can make the requirements at the pod level

    https://stackoverflow.com/questions/69423932/container-initialization-order-in-pod-on-k8s

    Edit: If using docker-compose.yml you can set a condition on a healthcheck

    You can also specify a condition that must be met before the service is started. For example, the following configuration will start the web service only if the db service is healthy:

    version: ‘3.7’
services:
web:
image: nginx
depends_on:
– db
condition: service_healthy

    The service_healthy condition checks the health of the db service using the docker-compose ps command. If the db service is healthy, the web service will be started.