Bug: Downloading E2E file onto android via Firefox yields partial file

[zwimer]

I E2E uploaded a roughly 7GB file to gokapi. On my laptop I can download this as expected without issue. When I go to my android phone and try to download it (my browser is Firefox) it tends to download somewhere between 10-60 MB then tells me the download has completed, showing no error.

It is possible this error is not at all related to gokapi, but I've never seen a partial-download marked as completed without error on Firefox on Android before, so I'm leaning towards this being related to E2E gokapi decryption.

[zwimer]

If it matters, the file size is larger than my total system RAM.

[Forceu]

Thanks for the report! I was able to reproduce the problem and I assume that this issue is related to it: jimmywarting/StreamSaver.js#340

It seems to be a problem with Firefox itself, and I could only find one solutions to it: To save it as a blob, but that would save the whole file in memory and is not feasible for mobile devices. Apparently services like Mega.nz store the file in 5MB chunks and merge it afterwards. But merging would create the same problem.

If anyone know a solution to this problem, please let me know!

[zwimer]

One possible fix, if I understand the problem correctly, is to have the E2E encryption be done in chunks small enough that each chunk can be loaded into the average mobile device's memory. That is, instead of storing an encrypted file as one large encrypted chunk of binary data, store it as a sequence of encrypted chunks. Thus the download function would be a loop that downloads a chunk, decrypts it, then writes the output to the download file.

[zwimer]

Until the underlying issue is fixed, it'd consider the current behavior a bug rather than an unimplemented feature, as currently the download not only yields a partial file, but for me at least, is reported as successfully completed. I think an error condition should be raised instead.

[Forceu]

One possible fix, if I understand the problem correctly, is to have the E2E encryption be done in chunks small enough that each chunk can be loaded into the average mobile device's memory.

Decryption is not the problem, it is downloading the file as a stream. It would be easy to store the file in chunks, but in the end the merged file needs to be downloaded in the browser. As far as I found out, this would only be possible as a blob with Firefox on Android, which would require the file to be completely in memory.

You are right that there should be a warning, I will look into it.

[zwimer]

So the problem is that the output file has to be written all at once? Perhaps something like this might alleviate the issue? https://developer.mozilla.org/en-US/docs/Web/API/FileSystemWritableFileStream

Source: https://stackoverflow.com/questions/39682465/javascript-writing-to-download-stream

[Forceu]

Gokapi is using https://github.com/jimmywarting/StreamSaver.js for that, which uses WriteableStream, or a Ponyfill for it. In the end it might also be a different problem, maybe related to the ServiceWorker, so this issue definitely needs more research. Unfortunately the FileSystemWritableFileStream you linked is not supported on Safari.

[psi-4ward]

Currently there is no way for Firefox and Safari to stream into a "save-as" file: https://caniuse.com/filesystem

[daniel-tp]

Ran up against this myself just now, Firefox on desktop would only download ~200mb out of a 900mb encrypted file. It worked fine in chrome. Hopefully there's a possible solution.