iocell list recommends a migration to ucl, but fails to execute it.

[lyuts]

Hi,

I've recently hit this strange behavior of iocell that I haven't been able to resolve.

Version: iocell 2.1.2 (2017-06-17)
When I run list it reports this error

% iocell list
  ERROR: please run as root to migrate - to ucl.
JID  UUID                                  BOOT  STATE  TAG                TYPE       IP4             RELEASE
...

Following the recommendation is failing with the following output:

% sudo iocell list
Password:
cannot open 'tank/iocell/jails/': trailing slash in name
cannot open 'tank/iocell/jails/': trailing slash in name
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
Error occured: cannot open file /iocell/jails//config: No such file or directory
cannot open 'tank/iocell/jails/': trailing slash in name
cannot open 'tank/iocell/jails/': trailing slash in name
cannot open 'tank/iocell/jails/': trailing slash in name
cannot open 'tank/iocell/jails/': trailing slash in name
  ERROR: missing property or UUID/TAG!

Additional information:
Prior to getting these errors I had iocell getting removed during one of the port updates. I didn't catch what lead to iocell removal though. iocell was reinstalled, and I ran iocell activate to point it to an already existing pool. After doing that I see all the jails I have, but now with this ucl errors.

Make sure to follow and check these boxes before submitting an issue! Thank you.

• [+] Supply iocell version
• [+] Supply the commands used, along with any steps to recreate it.
• [+] Provide the output from iocell
• [+] Supply what you expected the result or output to be
• [-] Checked that the problem has not already been fixed on develop if using the stable release or master branch. Skipped, due to stability concerns of the develop branch, called out on the main page.
  Thank you.

[rostwald]

Hello,

Can you please update to a recent version (2.2.0 has been in ports/pkg since february) and run iocell as root?
TBH I never use iocell as a regular user, but the error about UCL shouldn't be related to this.

IIRC the switch to UCL was a long time ago - what OS version are you running and/or from which version did you upgrade prior to getting that error?
When (on what OS/iocell version) were those jails created?

Re-running 'activate' on an already activated pool shouldn't cause any issues, IIRC iocell just skips any dataset that's already there, but not 100% sure now. I have a test system at hand where I could try that tomorrow.

Regards,
sko

[lyuts]

IIRC the switch to UCL was a long time ago

This what puzzled me too. I scanned the commit history and saw commits related to UCL from several years ago.

Anyway, I'm on FreeBSD 13.3, and I think things got messed up on my system.

% iocell --version
iocell 2.1.2 (2017-06-17)      

however

% pkg info | grep iocell                                            
iocell-2.2.0_1                 Feature rich Jail containers manager (iocage fork)

[lyuts]

Actually, 2.1.2 is hardcoded, from what I can tell.

https://github.com/bartekrutkowski/iocell/blob/master/iocell#L34

if [ "${1}" = "--version" ] || [ "${1}" = "version" ] ; then
    echo "iocell 2.1.2 (2017-06-17)"
    exit 0
fi

[rostwald]

Anyway, I'm on FreeBSD 13.3, and I think things got messed up on my system.

Did you just recently upgrade to 13.3 (possibly from a much older release + iocell version)?

Also, can you try running iocell as root instead of via sudo - sudo has various implications to what it does (or doesn't do) to the environment variables and shell; e.g. it might be possible that shellouts don't inherit the elevated privileges and hence iocell fails to access some paths/files. What makes this worse is that sudo has become a fast moving (and bloated) target and often changes behavior.
I would highly recommend using doas instead or simply become root for administrative tasks and then drop privileges again...

I really can't reproduce this error. Also what puzzles me is that iocell is looking for '/iocell/jails/config' - that config file however should be within each jails dataset/path... So this looks like iocell gets an empty jail uuid from somewhere and hence all the paths are breaking...
Can you check/show your zfs datasets and if the jail datasets are actually mounted? (zfs list -r tank/iocell/jails and zfs mount | grep iocell/jails)

[rostwald]

Actually, 2.1.2 is hardcoded, from what I can tell.

https://github.com/bartekrutkowski/iocell/blob/master/iocell#L34

if [ "${1}" = "--version" ] || [ "${1}" = "version" ] ; then
    echo "iocell 2.1.2 (2017-06-17)"
    exit 0
fi

True, that's actually bad, at least because it wasn't updated. I'm not sure if a PR against the development branch makes sense, as the release/tag couldn't be updated that way - at least I assume that from my (very limited) understanding how git releases/tags work...

@bartekrutkowski how can/should this be updated for the 2.2.0 tag? Or could some of the PRs be merged and we move on to the next tag and update the version string at the same time?

[lyuts]

Did you just recently upgrade to 13.3 (possibly from a much older release + iocell version)?

No recent world updates or upgrades. But this all started after I upgraded ports. This is when I think iocell upgraded from actual 2.1.2 to 2.2.0. Looks like the port got updated around Feb-Mar time range.

[lyuts]

Also, can you try running iocell as root instead of via sudo ...

Ok, did that, and after that when I run iocell list under regular user I'm no longer seeing
errors like:

Error occured: cannot open file /iocell/jails//config: No such file or directory
cannot open 'tank/iocell/jails/': trailing slash in name

but this error is still there.

ERROR: please run as root to migrate - to ucl.

However, when I run `iocell list' under root now, i'm still seeing all the errors from the 1st post.

[lyuts]

Can you check/show your zfs datasets and if the jail datasets are actually mounted? (zfs list -r tank/iocell/jails and zfs mount | grep iocell/jails)

% zfs list -r tank/iocell/jails
NAME                                                          USED  AVAIL     REFER  MOUNTPOINT
tank/iocell/jails                                             260G  1.05T       96K  /iocell/jails
tank/iocell/jails/24d01cae-c762-11e7-b551-38d547ac277c       7.64G  1.05T     4.02M  /iocell/jails/24d01cae-c762-11e7-b551-38d547ac277c
tank/iocell/jails/24d01cae-c762-11e7-b551-38d547ac277c/root  7.63G  1.05T     5.11G  /iocell/jails/24d01cae-c762-11e7-b551-38d547ac277c/root
tank/iocell/jails/2b8eebdd-a8e8-11ea-8ebe-38d547ac277c       9.18M  1.05T      104K  /iocell/jails/2b8eebdd-a8e8-11ea-8ebe-38d547ac277c
tank/iocell/jails/2b8eebdd-a8e8-11ea-8ebe-38d547ac277c/root  8.54M  1.05T     4.32M  /iocell/jails/2b8eebdd-a8e8-11ea-8ebe-38d547ac277c/root
tank/iocell/jails/634dd529-ca8d-11e8-b666-38d547ac277c        536M  1.05T      104K  /iocell/jails/634dd529-ca8d-11e8-b666-38d547ac277c
tank/iocell/jails/634dd529-ca8d-11e8-b666-38d547ac277c/root   535M  1.05T     22.6G  /iocell/jails/634dd529-ca8d-11e8-b666-38d547ac277c/root
tank/iocell/jails/668c7f34-527d-11e8-bf6f-38d547ac277c        238M  1.05T     4.07M  /iocell/jails/668c7f34-527d-11e8-bf6f-38d547ac277c
tank/iocell/jails/668c7f34-527d-11e8-bf6f-38d547ac277c/root   228M  1.05T      112M  /iocell/jails/668c7f34-527d-11e8-bf6f-38d547ac277c/root
tank/iocell/jails/76eb8d26-eeac-11ec-8190-38d547ac277c       98.0M  1.05T      120K  /iocell/jails/76eb8d26-eeac-11ec-8190-38d547ac277c
tank/iocell/jails/76eb8d26-eeac-11ec-8190-38d547ac277c/root  97.3M  1.05T     4.15G  /iocell/jails/76eb8d26-eeac-11ec-8190-38d547ac277c/root
tank/iocell/jails/ad6ff5c5-8787-11ec-b413-38d547ac277c        832M  1.05T      128K  /iocell/jails/ad6ff5c5-8787-11ec-b413-38d547ac277c
tank/iocell/jails/ad6ff5c5-8787-11ec-b413-38d547ac277c/root   832M  1.05T      719M  /iocell/jails/ad6ff5c5-8787-11ec-b413-38d547ac277c/root
tank/iocell/jails/c7582dad-aecb-11ea-8ebe-38d547ac277c       1.47G  1.05T      104K  /iocell/jails/c7582dad-aecb-11ea-8ebe-38d547ac277c
tank/iocell/jails/c7582dad-aecb-11ea-8ebe-38d547ac277c/root  1.47G  1.05T     1.00G  /iocell/jails/c7582dad-aecb-11ea-8ebe-38d547ac277c/root
tank/iocell/jails/c8e6fcb4-4c12-11e8-bf6f-38d547ac277c        220G  1.05T     4.10M  /iocell/jails/c8e6fcb4-4c12-11e8-bf6f-38d547ac277c
tank/iocell/jails/c8e6fcb4-4c12-11e8-bf6f-38d547ac277c/root   220G  1.05T      207G  /iocell/jails/c8e6fcb4-4c12-11e8-bf6f-38d547ac277c/root
tank/iocell/jails/db124dd1-0377-11ea-ad4b-38d547ac277c       1.69G  1.05T      104K  /iocell/jails/db124dd1-0377-11ea-ad4b-38d547ac277c
tank/iocell/jails/db124dd1-0377-11ea-ad4b-38d547ac277c/root  1.69G  1.05T     1.35G  /iocell/jails/db124dd1-0377-11ea-ad4b-38d547ac277c/root
tank/iocell/jails/e6a36ff2-00ae-11ed-a37a-38d547ac277c       1.20G  1.05T      112K  /iocell/jails/e6a36ff2-00ae-11ed-a37a-38d547ac277c
tank/iocell/jails/e6a36ff2-00ae-11ed-a37a-38d547ac277c/root  1.20G  1.05T      834M  /iocell/jails/e6a36ff2-00ae-11ed-a37a-38d547ac277c/root
tank/iocell/jails/e6ce3e09-c8fc-11e6-87af-38d547ac277c       25.7G  1.05T     3.99M  /iocell/jails/e6ce3e09-c8fc-11e6-87af-38d547ac277c
tank/iocell/jails/e6ce3e09-c8fc-11e6-87af-38d547ac277c/root  25.7G  1.05T     24.4G  /iocell/jails/e6ce3e09-c8fc-11e6-87af-38d547ac277c/root

% zfs mount | grep iocell/jail
tank/iocell/iocell/jails        /tank/iocell/iocell/jails
tank/iocell/jails               /iocell/jails
tank/iocell/jails/2b8eebdd-a8e8-11ea-8ebe-38d547ac277c  /iocell/jails/2b8eebdd-a8e8-11ea-8ebe-38d547ac277c
tank/iocell/jails/ad6ff5c5-8787-11ec-b413-38d547ac277c  /iocell/jails/ad6ff5c5-8787-11ec-b413-38d547ac277c
tank/iocell/jails/db124dd1-0377-11ea-ad4b-38d547ac277c  /iocell/jails/db124dd1-0377-11ea-ad4b-38d547ac277c
tank/iocell/templates/jenkins_slave_node  /iocell/jails/jenkins_slave_node
tank/iocell/jails/c8e6fcb4-4c12-11e8-bf6f-38d547ac277c  /iocell/jails/c8e6fcb4-4c12-11e8-bf6f-38d547ac277c
tank/iocell/jails/24d01cae-c762-11e7-b551-38d547ac277c  /iocell/jails/24d01cae-c762-11e7-b551-38d547ac277c
tank/iocell/jails/c7582dad-aecb-11ea-8ebe-38d547ac277c  /iocell/jails/c7582dad-aecb-11ea-8ebe-38d547ac277c
tank/iocell/jails/634dd529-ca8d-11e8-b666-38d547ac277c  /iocell/jails/634dd529-ca8d-11e8-b666-38d547ac277c
tank/iocell/jails/668c7f34-527d-11e8-bf6f-38d547ac277c  /iocell/jails/668c7f34-527d-11e8-bf6f-38d547ac277c
tank/iocell/jails/e6ce3e09-c8fc-11e6-87af-38d547ac277c  /iocell/jails/e6ce3e09-c8fc-11e6-87af-38d547ac277c
tank/iocell/jails/2b8eebdd-a8e8-11ea-8ebe-38d547ac277c/root  /iocell/jails/2b8eebdd-a8e8-11ea-8ebe-38d547ac277c/root
tank/iocell/jails/76eb8d26-eeac-11ec-8190-38d547ac277c  /iocell/jails/76eb8d26-eeac-11ec-8190-38d547ac277c
tank/iocell/jails/ad6ff5c5-8787-11ec-b413-38d547ac277c/root  /iocell/jails/ad6ff5c5-8787-11ec-b413-38d547ac277c/root
tank/iocell/jails/e6a36ff2-00ae-11ed-a37a-38d547ac277c  /iocell/jails/e6a36ff2-00ae-11ed-a37a-38d547ac277c
tank/iocell/jails/24d01cae-c762-11e7-b551-38d547ac277c/root  /iocell/jails/24d01cae-c762-11e7-b551-38d547ac277c/root
tank/iocell/jails/668c7f34-527d-11e8-bf6f-38d547ac277c/root  /iocell/jails/668c7f34-527d-11e8-bf6f-38d547ac277c/root
tank/iocell/jails/76eb8d26-eeac-11ec-8190-38d547ac277c/root  /iocell/jails/76eb8d26-eeac-11ec-8190-38d547ac277c/root
tank/iocell/templates/jenkins_slave_node/root  /iocell/jails/jenkins_slave_node/root
tank/iocell/jails/c7582dad-aecb-11ea-8ebe-38d547ac277c/root  /iocell/jails/c7582dad-aecb-11ea-8ebe-38d547ac277c/root
tank/iocell/jails/e6ce3e09-c8fc-11e6-87af-38d547ac277c/root  /iocell/jails/e6ce3e09-c8fc-11e6-87af-38d547ac277c/root
tank/iocell/jails/634dd529-ca8d-11e8-b666-38d547ac277c/root  /iocell/jails/634dd529-ca8d-11e8-b666-38d547ac277c/root
tank/iocell/jails/db124dd1-0377-11ea-ad4b-38d547ac277c/root  /iocell/jails/db124dd1-0377-11ea-ad4b-38d547ac277c/root
tank/iocell/jails/c8e6fcb4-4c12-11e8-bf6f-38d547ac277c/root  /iocell/jails/c8e6fcb4-4c12-11e8-bf6f-38d547ac277c/root
tank/iocell/jails/e6a36ff2-00ae-11ed-a37a-38d547ac277c/root  /iocell/jails/e6a36ff2-00ae-11ed-a37a-38d547ac277c/root

[rostwald]

OK, nothing suspicios with the datasets, however where does the mount tank/iocell/iocell/jails /tank/iocell/iocell/jails come from? This might be the reason why iocell is trying to read some empty jail UUIDs...
Cn you also give the full output for zfs list -r | grep iocell, just to be sure there are not more datasets that might throw off iocell.
Does that /tank/iocell/iocell/jails (or even /tank/iocell) have any contents? What happens if you unmount it and run iocell again?

[lyuts]

Good catch, I didn't notice it before. It doesn't seem to have anything in it. I think me trying to fix it on my own lead to me somehow creating it. I removed it with zfs destroy, and I no longer see this error doing iocell list both as regular user and root. I wish I knew what exactly was going on internally, but my immediate issue is now resolved.
Thank you for looking into this, and helping spot the problem.