Multi-threaded Borg / Scaling performance with CPU cores

[Hammit]

I'm currently on holiday and only have my tablet, but I'm looking to migrate away from Deja Dup / Duplicity due to poor backup performance. It runs on one CPU core only and takes many many hours to do a backup. Based on what I've seen in the docs and the issues on GitHub, Borg will have the same problem.

It seems like this is something that's on the roadmap though so I'm interested in learning what the current state of affairs is. Is this planned for any particular version in the future? Any and all info appreciated

P.s. Borg looks great from what I've read. Really hope I can send up using it

[ThomasWaldmann]

Having the same problem: yes and no (usually no).

Correct is that borg is currently single-threaded and your first backup using borg will take a relatively long time.

But (and this is different from older backup tools), due to the chunk-based deduplication, all following backups of the same data sets will be much much faster, because it will not process/store data chunks again which it already has in the repository (== backup storage).

Additionally, borg uses some clever tricks with the "files cache" and in-memory hash tables, always making full backups super fast (if a lot of files did not change at all, which is often the case).

Thus, while multithreading is on the roadmap, it is not as important and not as pressing as one might think it is. It's often just a "one-time patience issue" (initial backup) and avoiding CPU work is always better than parallelising cpu work.

Of course, multithreading becomes increasingly more important with growing amounts of changed chunks between backups. Folks who produce lots of new data all the time need it.

Thus, next releases (2.0, 2.1) will address mostly improving crypto and getting rid of old code and some other long-term open issues that we could not yet fix without breaking compatibility. The better crypto will also unblock/solve some multi-repo / multi-client issues of the past.

Once we're finished with that, guess it will be the time for adding multithreading.

[Hammit]

Ok. Interesting. Thanks for your comment on the subject. I shouldn't be saying this, but I look forward to getting back from holidays and trying my first backup 🙂

[Hammit]

Just a follow up on this. I got back from holidays and did a backup using lz4 compression and it was actually very fast. I didn't even see it hit 100% of the CPU. Single threaded performance was good. It took me roughly 1hr to backup 650GB. I think that's roughly 185MB/s. Quite good.

[ThomasWaldmann]

How many files per second is that? Maybe use --stats --info --list to check.

[chpio]

Unfortunately, the file cache trick does not work for btrfs snapshots, borg re-scans all the files on each backuped snaphot. I'm using snapborg for that - https://github.com/enzingerm/snapborg. This works similarity to timemachine on mac.
It rescans 100gb each time to actually upload ~100-300mb. Is there a way to make the cache work by using paths relative to the backed-up directory, if that's even the problem?

[ThomasWaldmann]

borg 1.x uses the full absolute path as cache key, so if the path changes, that will cause cache misses.

for borg2 I changed this to use relative paths.

[Hammit]

1074727 files total I ran cancelled and resumed the backup a few time so I'm not sure exactly how long it took. Does the final printout of stats give total runtime or the time of the most recent backup run? from my mobile

…

On Wed, 24 Aug 2022, 7:59 pm TW, ***@***.***> wrote: How many files per second is that? Maybe use --stats --info --list to check. — Reply to this email directly, view it on GitHub <#6958 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABOIXVLHC22HR7EU3XRDNYLV2YFCRANCNFSM56AQPPSQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ThomasWaldmann]

It refers to the current run and archiving start/end refer to the time needed for the actual backup (not including startup/shutdown overhead of borg).

[ShaiMagal]

@ThomasWaldmann @Hammit
Guys, is there any update after more than 2 years? Multithread borg can be great, one core is not enough and it's bottleneck.

[Hammit]

I'm using Ubuntu and the latest version (24.10) is only at 2.0.0b
Given that it was it wasn't scheduled for 2.0.1, I think it's fair to say this is generally not available yet. Like you though, I'd be interested in an update to know if the intention is still to implement it soon after that

[ThomasWaldmann]

There was some progress in master branch by moving away from a transactional, log-like (always append) repository with a single "manifest" chunk having the directory of archives inside it. Also moving away from using reference counting.

With such a structure it would be hard to do any parallel operations and it required locking to avoid that.

Now we have (in master branch / in borg2 betas):

• separate repository objects (currently, each object is stored in a separate file) (*)
• the "directory of archives" is not in a single object anymore, but is a separate directory with separate objects (each pointing to an archive). adding or removing a directory entry is not a r-m-w operation anymore and does not require a lock anymore.
• borg compact now cleans up by determining used vs. unused objects (that is a boolean) instead of the previous reference counter.
• prune and delete are super fast now
• a lot of operations have a much simpler implementation now
• a new and quite flexible hashtable: borghash (in Cython, the old one was quite a bit of C)
• a new very simple store with misc. backends: borgstore

That now enables:

• multiple borg processes can use the repo in parallel, as their operations can not conflict anymore for most operations. the only operations needing an exclusive lock are compact and check. so we have multi-processing now already. each process has one thread, but as there can be multiple processes working on same repo: there you have it: multi-threading! :-)
• repos via file, ssh, sftp, rclone protocols

So while we still don't have single-process multithreading, there was quite a bit work done that was required before it would make sense to even try that.

(*) we likely need another big change there using client-side built pack files - there is another ticket about that, don't discuss this in here.

[Hammit]

Sorry for the delayed response. The issue I was referencing was that I had a very large backup to do and it looked like creation of the backup was CPU bottlenecked during compression (single threaded). Do these enhancements specifically address that problem in any way?

I.e. Can initial backup be parallelized?

[ThomasWaldmann]

Yes, but only if you run multiple "borg create" processes concurrently (they can now access the same repo, in parallel).

[BenasPaulikas]

@ThomasWaldmann which version is that?
I'm using 1.4 and I can't do more than 1 borg create at a time. Due lock file.

[FelixSchwarz]

You need borg 2.0 (not yet released).

[BenasPaulikas]

Ok @FelixSchwarz @ThomasWaldmann thank you.
For me borg is insane tool. I can't wait for 2.0 to be working.
So I could create / extract / mount without waiting for lock.

Thank you and best of luck.

[Zerophase]

With Borg becoming more multi-threading friendly will network writes move more towards the asynchronous end, or is it still going to be mostly synchronous writes? What is the data access model currently? I'm trying to figure out how to spec a back up server to work as fast as possible with Borg.

Also, since multiple borg create processes can now be run concurrently. If backing up an entire system would it be most efficient to run that command per directory structure being traversed?

[RonnyPfannschmidt]

Unless there's extreme churn, I'd strongly recommend keeping the backup setup simple and ensuring restoration is easy and tested

[Zerophase]

Oh for sure. I'm just optimizing my system for a dedicated backup server. Certain files I won't need to backup anymore with borg. Mainly packages, as I can just restore those from my local cache held on the server.

So, I'll run seperate borg creates for the directories needed, and can simplify my borg command for backing up my systems. Mostly, focusing on just backing up the files I modify myself, and letting the package manager handle everything else. Should drastically reduce the time and size of full system backups.

It just helps me spec the machine out if I know if Borg is mostly async or sync writes.