TMP being filled if not mounted (Apptainer)

[ffelten]

System:
Linux / Apptainer / Airfoil

Error message
Error: [Errno 28] No space left on device: '/home/mdolabuser/.pyenv/versions/3.9.12/lib/python3.9/site-packages/adflow/libadflow.so' -> '/tmp/tmpzo20nluh/libadflow.so'

See #177 for workaround.

I do not agree with the workaround being merged as it will lead to other errors in the future. We should understand what is causing tmp to be filled and not "just mount your own /tmp in it."

Questions to answer

• What kind of files are going to tmp? MPI? Apptainer image conversion?
• Who's writing these files
• Proper ways to fix this.

[g-braeunlich]

This is coming from an apptainer limit. See
https://apptainer.org/docs/user/main/persistent_overlays.html
(the 1st info box about --writable-tmpfs). The default limit is 64MiB 🤭
The limit can be changed in apptainer.conf

[g-braeunlich]

A possible solution would be to add
--workdir /tmp / --workdir /cluster/scratch/<USERNAME> (apptainer only).
But the problem with this approach is that it might break either Euler / other HPC clusters or other workflows on a local machine.

Also note that usually on a local machine, podman or docker are used.

=> Still unsure what to do about this. What do you think @ffelten @markfuge

[ffelten]

A possible solution would be to add --workdir /tmp / --workdir /cluster/scratch/<USERNAME> (apptainer only). But the problem with this approach is that it might break either Euler / other HPC clusters or other workflows on a local machine.

Also note that usually on a local machine, podman or docker are used.

=> Still unsure what to do about this. What do you think @ffelten @markfuge

Still not sure either.

I can see us going back and forth, tweaking these until it works on Euler, local, and Zaratan, but it will be hell to maintain.

There is indeed a much higher chance of seeing Docker being used on local machines, and probably, this issue will not occur often. We could document the current issue and workaround in the Airfoil problem documentation.

[markfuge]

This seems like this doesn't have an easy solution that makes everything happy. In this case, I agree with @ffelten that we could document this current constraint in the documentation pages as well as the workaround. If we can get it to work well with Singularity in environments that would use that 99% of the time (i.e., HPCs), and then it also works for Docker/Podman on local machines, where it would be used 99% of the time, then I am OK with having a special workaround for the 1% of the time that someone wants to use the Singularity+Local Machine combination.

Ultimately, if this becomes a serious blocker for people in the long term, we will get issue requests on it and can decide then whether or not to address and maintain that functionality in the future

[g-braeunlich]

OK. I opened #204

[markfuge]

@g-braeunlich it looks like #204 fixed this issue, right? OK to close it?

[g-braeunlich]

#204 does not fix it but just warns users about this issue. Therefore I also would leave this issue open.
Otherwise people get the expression that the issue is really resolved