Overcommitting storage silently corrupts data

[crackerjam]

Trying to copy a 10GB file to a 2GB Stratis filesystem appears to succeed, but in reality causes the file to be corrupted. To reproduce:

1. Login to a RHEL 9.2 system as root (System has a 20GB root disk, plus a 2GB blank disk)
2. Install/enable/start stratisd and stratis-cli (version 3.4.1)
   yum -y install stratisd stratis-cli; systemctl enable --now stratisd
3. stratis pool create testpool /dev/sdb
4. stratis filesystem create testpool testfilesystem
5. mkdir -p /mnt/test
6. Add this line to /etc/fstab
/dev/stratis/testpool/testfilesystem /mnt/test xfs defaults,x-systemd.requires=stratisd.service 0 0
7. mount -a
8. Download a large file to /root for testing
   wget -P /root http://mirrors.rit.edu/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-Everything-2009.iso
9. Copy the file to the Stratis filesystem
   cp /root/CentOS-7-x86_64-Everything-2009.iso /mnt/test
10. Hash the test file in both locations
    md5sum /root/CentOS-7-x86_64-Everything-2009.iso
md5sum /mnt/test/CentOS-7-x86_64-Everything-2009.iso

These checksums will come back different, indicating data corruption.

[mulkieran]

Thanks for the report. We believe that this problem has been addressed in subsequent stratisd releases, which are available in RHEL 9. Here is some documentation on the design changes that we made in Stratis 3 to make extension of the Stratis filesystems more robust: https://stratis-storage.github.io/thin-provisioning-redesign/ . If you reproduce the same problem in a more recent version of Stratis, please let us know, we will be eager to investigate. Thanks!

[crackerjam]

@mulkieran This is NOT fixed on RHEL 9.2 with stratis 3.4.1. I just re-ran the steps in my original post on a fresh system and see the same behavior.

I've updated my original post to include those versions.

[mulkieran]

@crackerjam Can you confirm that /dev/sdb is the 2 GiB device that you have in your initial statement? I may have initially misread your statement, and thought that the device you were using for the Stratis pool was the 20 GiB one that you mention first. If that is the case, i.e., that the pool itself is only 2 GiB there is not much to do, because the cp command itself does not perform any sync operation or confirm that the write has completed.

The pool itself is supposed to maintain information about itself, and can give some warnings. When you first create the pool, it will assess its state and may issue a warning, which will be seen with the stratis pool list command. After you issue the cp command, the pool will try to accommodate the additional data by expanding the filesystem. If you list the pool after the failed cp command, you may see additional warnings.

[crackerjam]

@mulkieran Correct, /dev/sdb is a 2GiB disk.

If that is the case, i.e., that the pool itself is only 2 GiB there is not much to do, because the cp command itself does not perform any sync operation or confirm that the write has completed.

That doesn't seem to be correct. If I just format the raw disk with XFS directly and try to run the copy, I receive:

cp: error writing '/mnt/test/CentOS-7-x86_64-Everything-2009.iso': No space left on device

If you list the pool after the failed cp command, you may see additional warnings.

stratis pool list does show an alert of "WS0001" next to the pool, which is something, but honestly not very helpful. Googling "stratis WS0001" doesn't return any results, so I'm not even able to identify what this error code means.

Think of a scenario where you have a stratis volume that receives and stores critical business documents. Sure, ideally you would have monitoring set up for disk usage, but let's say you don't. With any other filesystem your application would start receiving errors when the disk is full. Users will see that something's going on, it will be clear that things aren't working properly. Things will grind to a halt, but you won't lose data.

With this issue, everything just appears to silently work. Files show up on disk and report correct sizes. However, when you read the files, they're junk. Depending on the use case you could go weeks or months without noticing that all of your data is just evaporating. This should be an extremely critical priority issue.

[mulkieran]

@crackerjam In a case like this, you would be advised to create a pool with no overprovisioning mode set. In such a pool the size of the XFS filesystem can not be made any larger that the actual space that is available to it. In that case, the filesystem will behave like the one that you placed on the raw device.

For the raw device, check the filesystem size. You'll notice that it is significantly smaller than the default filesystem size, 1 TiB, that Stratis creates when overprovisioning is allowed.

[crackerjam]

@mulkieran I don't think that's really an appropriate stance. What is 'a case like this'? A case where you don't want your files to get corrupted? No filesystem, ever, should report back that a write succeeded when it did not, regardless of options used on creation. That's just a core tenant of how filesystems work. It is especially true here, when I'm using the default options.

I have to be honest, I only discovered Stratis as part of some RHEL training. Personally, and professionally, if I ever get a whiff of anyone thinking about using Stratis I will tell them to steer clear due to how this bug is being handled. In my opinion, this is a massive issue that needs to be resolved if you want this to be taken seriously as a replacement for LVM, and I can only imagine how many other data-destroying bugs there are that have been ignored because "well you should have just used the right command arguments".

If you'd like any more assistance testing scenarios around this, please feel free to let me know. Otherwise, it seems like you have a particular opinion here that I probably won't be able to change.

[mulkieran]

@crackerjam I'm afraid that you are failing to understand how filesystems work. To produce an analogous situation with LVM, simply create a thin pool on your 2 GiB device, construct a thin device of say, 1 TiB, on that thinpool, create an XFS filesystem on it, and do your cp as before. The cp will return success, but the file will not be copied in toto.

It is surprising to some people that that is how filesystems behave, but it is a consequence of the original design choices made for filesystems in a simple world that did not include thinly provisioned devices supplied by kernel modules.

If you want to do a copy that is sure to fail if there is really no place to write the bytes, you will have to explicitly sync your data.