try to backport retries for failed files

[ThomasWaldmann]

see #7351.

[ThomasWaldmann]

I will have a look at this (again).

IIRC, I did not backport it back then, because it is only that easy in master branch (borg2), which works significantly different (simpler) than borg 1.x does:

• borg 1.x archives .partN files when it checkpoints in the middle of a file
• concatenating all .partN files is expected to result in the original, whole file
• once something is commited (like in such a checkpoint), we can't easily undo/redo it
• refcounting for chunks needs rollback on retry

#9812 is a backport that implements retries, except for in-file checkpointed files.

[ThomasWaldmann]

see also #8926.

[quietquark]

One question/concern here - on a somewhat busy filesystem that is being backed up by borg, there may be continuous modifications to (many) files which are not excluded as part of the backups. Currently borg warns and moves on. With this change, it will retry (upto 10s/MAX_RETRIES).

There is currently no control/option to disable this or even tweak it. As a result, backup times could increase by a lot if there are many such "busy files". It would be good to be able to have some control to either limit the number of tries and/or time limit to have a lower upper bound (or even entirely disable this approach and revert to the direct single-try warning case).

[ThomasWaldmann]

Guess an env var to control this would be good.

But, one thing has to be clear: this is a dirty workaround one can use instead of the real solution (which is stopping such services or backing up a snapshot) and no value of such a setting is guaranteed to always work and not being too slow in extreme cases.

The exponential increase in wait time while it retries should be a good compromise in being as fast as possible (doing the first retry after only 1ms (IIRC)) and also waiting long enough to not have to skip files that are busy for longer (waiting up to 10s for the last retry (IIRC)).