Container running state error.

[taylorabranson]

I've been seeing this error since the 2.17.0 release. Nautical-backup backs up scrutiny correctly, but then doesn't seem to know the state of the container and throws this error during every backup. Any idea what's causing this behavior?

2026-05-21 05:35:55.714325 - INFO: Backing up group: scrutiny
2026-05-21 05:35:55.717266 - INFO: Stopping scrutiny...
2026-05-21 05:35:55.827656 - DEBUG: scrutiny-collector - Source directory does not exist, but that's okay
2026-05-21 05:35:55.830139 - INFO: Stopping scrutiny-collector...
2026-05-21 05:35:57.995776 - DEBUG: influxdb - Source directory does not exist, but that's okay
2026-05-21 05:35:57.998208 - INFO: Stopping influxdb...
2026-05-21 05:35:58.300172 - INFO: Backing up scrutiny...
2026-05-21 05:35:58.300272 - DEBUG: RUNNING: 'rsync -raq --delete /app/source/scrutiny/ /app/destination/scrutiny/'
2026-05-21 05:35:58.421519 - INFO: Starting scrutiny...
2026-05-21 05:35:58.742755 - ERROR: Container scrutiny was not in running state. Trying again (Attempt 1/3)
2026-05-21 05:35:58.746429 - DEBUG: Container scrutiny was not stopped. No need to start.
2026-05-21 05:35:58.746582 - INFO: Backup of scrutiny complete!
2026-05-21 05:35:58.749860 - INFO: Starting scrutiny-collector...
2026-05-21 05:35:58.953812 - INFO: Backup of scrutiny-collector complete!
2026-05-21 05:35:58.956731 - INFO: Starting influxdb...
2026-05-21 05:35:59.460520 - INFO: Backup of influxdb complete!

[Minituff]

Fixed in #686

Thanks for the detailed logs!. This definitely was a bug, now fixed.

Fix

The next release corrects both issues:

• The retry now properly propagates its return value (return self._start_container(...)).
• The initial guard now specifically checks for running (never stopped) vs. any other non-exited state (transitional), and waits up to 3 × 2 s for the container to stabilize before giving up.

The backup itself was completing correctly in your case.. the container did come up, but it was just the state-tracking logic that reported it incorrectly.

[Minituff]

This will be included in v2.20.0 https://github.com/Minituff/nautical-backup/releases/tag/v2.20.0

Let me know if this still persists and I can check it out again.

[taylorabranson]

Confirmed working! Thanks for the quick fix!

[taylorabranson]

@Minituff Had it happen again after running a couple backups without any issues (previously it had been giving the error every time). Backup is still completing successfully. There's also a new WARN about the container being in a 'restarting' state. Here are my logs:

2026-05-25 01:30:14.342222 - INFO: Backing up group: scrutiny
2026-05-25 01:30:14.346325 - INFO: Stopping scrutiny...
2026-05-25 01:30:14.638144 - DEBUG: scrutiny-collector - Source directory does not exist, but that's okay
2026-05-25 01:30:14.642426 - INFO: Stopping scrutiny-collector...
2026-05-25 01:30:16.901636 - DEBUG: influxdb - Source directory does not exist, but that's okay
2026-05-25 01:30:16.904775 - INFO: Stopping influxdb...
2026-05-25 01:30:17.381612 - INFO: Backing up scrutiny...
2026-05-25 01:30:17.381725 - DEBUG: RUNNING: 'rsync -raq --delete /app/source/scrutiny/ /app/destination/scrutiny/'
2026-05-25 01:30:17.544267 - INFO: Starting scrutiny...
2026-05-25 01:30:17.957587 - ERROR: Container scrutiny was not in running state. Trying again (Attempt 1/3)
2026-05-25 01:30:17.960206 - WARN: Container scrutiny is in 'restarting' state, waiting for it to stabilize (Attempt 2/3)
2026-05-25 01:30:19.963859 - WARN: Container scrutiny is in 'restarting' state, waiting for it to stabilize (Attempt 3/3)
2026-05-25 01:30:21.967109 - INFO: Backup of scrutiny complete!
2026-05-25 01:30:21.970345 - INFO: Starting scrutiny-collector...
2026-05-25 01:30:22.273585 - INFO: Backup of scrutiny-collector complete!

[Minituff]

Hey @taylorabranson, did Scrutiny successfully startup? I'm thinking that this line might be a little misleading:

2026-05-25 01:30:17.957587 - ERROR: Container scrutiny was not in running state. Trying again (Attempt 1/3)

If the container did startup then I will probably changes this from an ERROR to WARN -- I could also add a variable to allow for longer timeouts for startup for containers.

[taylorabranson]

Yes, it did startup just fine.

[Minituff]

Okay, given that it started up successfully I think the fix here is the make the startup-timeout configurable so we can wait on some slower containers.

I just merged in v2.21.0 that should fix this.

You should be able to add this new Docker Label to Scrutiny https://minituff.github.io/nautical-backup/labels/#start-timeout

Edit: Had to update the GitHub actions pipeline -- it should now be released in v2.21.2

[taylorabranson]

I appreciate the quick response! I did a little testing and I'm still seeing couple issues.

First, even though I don't get an ERROR in the log, the still shows the container backup as having failed at the end.

2026-05-25 15:37:10.560149 - INFO: Backing up group: scrutiny
2026-05-25 15:37:10.562179 - INFO: Stopping scrutiny...
2026-05-25 15:37:10.743135 - DEBUG: scrutiny-collector - Source directory does not exist, but that's okay
2026-05-25 15:37:10.745917 - INFO: Stopping scrutiny-collector...
2026-05-25 15:37:12.954581 - DEBUG: influxdb - Source directory does not exist, but that's okay
2026-05-25 15:37:12.957765 - INFO: Stopping influxdb...
2026-05-25 15:37:13.319212 - INFO: Backing up scrutiny...
2026-05-25 15:37:13.319318 - DEBUG: RUNNING: 'rsync -raq --delete /app/source/scrutiny/ /app/destination/scrutiny/'
2026-05-25 15:37:13.436616 - INFO: Starting scrutiny...
2026-05-25 15:37:13.892083 - WARN: Container scrutiny was not in running state. Trying again (Attempt 1/5)
2026-05-25 15:37:13.894578 - WARN: Container scrutiny is in 'restarting' state, waiting for it to stabilize (Attempt 2/5)
2026-05-25 15:37:15.897330 - WARN: Container scrutiny is in 'restarting' state, waiting for it to stabilize (Attempt 3/5)
2026-05-25 15:37:17.900015 - WARN: Container scrutiny is in 'restarting' state, waiting for it to stabilize (Attempt 4/5)
2026-05-25 15:37:19.903456 - WARN: Container scrutiny is in 'restarting' state, waiting for it to stabilize (Attempt 5/5)
2026-05-25 15:37:21.948170 - INFO: Backup of scrutiny complete!
2026-05-25 15:37:21.951300 - INFO: Starting scrutiny-collector...
2026-05-25 15:37:22.219356 - INFO: Backup of scrutiny-collector complete!
2026-05-25 15:37:22.221851 - INFO: Starting influxdb...
2026-05-25 15:37:22.856490 - INFO: Backup of influxdb complete!
...
2026-05-25 15:37:47.175819 - ERROR: Failed 1 containers: scrutiny

Second, I tried adding the label for start-timeout, but based on the logs it seems to have skipped the scrutiny and scrutiny-collector containers. Here's the relevant log section with the label active:

2026-05-25 15:32:28.176051 - INFO: Backing up group: scrutiny
2026-05-25 15:32:28.176193 - DEBUG: influxdb - Source directory does not exist, but that's okay
2026-05-25 15:32:28.178155 - INFO: Stopping influxdb...
2026-05-25 15:32:28.381075 - INFO: Starting influxdb...
2026-05-25 15:32:28.738856 - INFO: Backup of influxdb complete!

[Minituff]

Interesting, can I see your nautical and scrutiny config? It seems like it's not taking the label.

You could try the global environment variable just to see that the logic is working right and if that helps we can dial in the label.

[taylorabranson]

I don't use any global envs, so all the config is in the yamls. Here's the nautical backup config (tailscale and docker-proxy containers omitted for noise):

  nautical-backup:
    image: minituff/nautical-backup:2.21.2@sha256:024408e4fcb6d434cfc9454f2886f9c2ed1f818ea73b0541f58be5bdeb7fd6e8
    container_name: nautical-backup
    security_opt:
      - no-new-privileges:true
    depends_on:
      ts-nautical-backup:
        condition: service_healthy
    network_mode: service:ts-nautical-backup
    volumes:
      # - /var/run/docker.sock:/var/run/docker.sock
      - ../../../Data/nautical-backup/config:/config
      - ../../../Data:/app/source:ro
      - ../../../Backup/nas-main-backup:/app/destination
    environment:
      - DOCKER_HOST=tcp://docker_proxy_backup:2375
      - TZ=America/Chicago
      - CRON_SCHEDULE=30 1 * * *
      - CRON_SCHEDULE_ENABLED=true
      - REQUIRE_LABEL=true
      # - BACKUP_ON_START=true
      - LOG_LEVEL=DEBUG
      - REPORT_FILE_LOG_LEVEL=DEBUG
    restart: unless-stopped
    logging:
      driver: local

Here's the scrutiny yaml (minus the tailscale container):

  influxdb:
    container_name: influxdb
    image: influxdb:2.9.1@sha256:c9acb506f536915c97ff71c971970ad9c2a38b2b8a0d54200ef0ef430f7a37f0
    security_opt:
      - no-new-privileges:true
    # ports:
    #   - 8086:8086
    environment:
      - TZ=America/Chicago
    volumes:
      - ../../../Data/scrutiny/influxdb:/var/lib/influxdb2
    restart: unless-stopped
    labels:
      - "nautical-backup.enable=true"
      - "nautical-backup.stop-before-backup=true"
      - "nautical-backup.group=scrutiny"
      - "nautical-backup.group.scrutiny.priority=105"
      - "nautical-backup.source-dir-required=false"
    logging:
      driver: local
  scrutiny:
    container_name: scrutiny
    image: ghcr.io/analogj/scrutiny:v0.9.2-web@sha256:71be54e9960826f356fe290eb196762b2e18001e39f6ee09eeb17fe926acbf2c
    # ports:
    #   - 8080:8080
    security_opt:
      - no-new-privileges:true
    volumes:
      - ./config:/opt/scrutiny/config
      - ../../../Data/scrutiny/configdb:/opt/scrutiny/configdb
    environment:
      - TZ=America/Chicago
      - SCRUTINY_WEB_INFLUXDB_HOST=influxdb
      - SCRUTINY_WEB_INFLUXDB_PORT=8086
      - SCRUTINY_WEB_INFLUXDB_TOKEN=${SCRUTINY_DB_TOKEN}
    depends_on:
      ts-scrutiny:
        condition: service_healthy
      influxdb:
        condition: service_started
    restart: unless-stopped
    network_mode: service:ts-scrutiny
    logging:
      driver: local
    labels:
      - "nautical-backup.enable=true"
      - "nautical-backup.stop-before-backup=true"
      - "nautical-backup.group=scrutiny"
      - "nautical-backup.group.scrutiny.priority=115"
      - "nautical-backup.use-default-rsync-args=true"
      - "nautical-backup.rsync-custom-args=--delete"
      - "nautical-backup.start-timeout=30"
  scrutiny-collector:
    container_name: scrutiny-collector
    network_mode: service:ts-scrutiny
    restart: unless-stopped
    logging:
      driver: local
    image: ghcr.io/analogj/scrutiny:v0.9.2-collector@sha256:a0ba60a1f3734ef82871c658a7e5545fd6b70b560935c451149ec102e35bb054
    cap_add:
      - SYS_RAWIO
      - SYS_ADMIN
    security_opt:
      - no-new-privileges:true
    volumes:
      - ./config:/opt/scrutiny/config
      - /run/udev:/run/udev:ro
    environment:
      TZ: America/Chicago
      COLLECTOR_API_ENDPOINT: http://scrutiny:8080
      COLLECTOR_HOST_ID: 'NAS - Main'
      COLLECTOR_RUN_STARTUP: false
    depends_on:
      - scrutiny
    devices:
      - /dev/nvme0n1:/dev/nvme0n1
      - /dev/nvme1n1:/dev/nvme1n1
      - /dev/sata1:/dev/sata1
      - /dev/sata2:/dev/sata2
      - /dev/sata3:/dev/sata3
      # - /dev/sata4:/dev/sata4
    labels:
      - "nautical-backup.enable=true"
      - "nautical-backup.stop-before-backup=true"
      - "nautical-backup.group=scrutiny"
      - "nautical-backup.group.scrutiny.priority=110"
      - "nautical-backup.source-dir-required=false"

[Minituff]

Second, I tried adding the label for start-timeout, but based on the logs it seems to have skipped the scrutiny and scrutiny-collector containers. Here's the relevant log section with the label active:

I'm thinking for this, the containers were restarting so that's likely why they were skipped.

Another thing that I think is going on here is some weirdness with the group and the start-timeout. Are you able to apply "nautical-backup.start-timeout=30" to all the containers scrutiny depends on?

I'm going to update the logs again so its a little more clear whats going on. I think for a container that successfully backs up, but cant startup within it's timeout show have a specific call out -- like its not a fail but not a success either. #690