Raise performance and lower storage needs!

[ThomasWaldmann]

The problem

Backups and restores tend to be rather slow, especially when there are a lot of files. Borg tries to speed things up using deduplication and unchanged file detection, but if there are many new or changed files, it will still be slow.

Some operations, like borg check, scale O(N×M), where N is the archive count and M is the archive size.

On top of that, storage prices have been rising steeply, so we need to be mindful of space — and backups can consume a lot of it.

Root cause analysis

These problems are mostly caused by legacy backup and restore procedures: Borg backs up everything you give it (= high M) and restores everything you ask for. It keeps many archives (= high N) until you prune them, and it will accept any prune policy — even a ridiculous one.

The root cause, if we're honest, is you. You keep too many files. You keep too many archives. You never clean up your repositories. This has gone on long enough.

Solution

AI has been making enormous strides recently and we have been working on BorgAI for months (PR coming soon), so it can help us here:

• Not all files are truly important. It's just tedious for a human to make all those decisions — AI can do it much faster.
• Not all backup archives are important. AI can predict what you'll actually need and remove the superfluous ones that slow down certain operations and eat up your backup space.
• Not all repositories may even be worth keeping. AI can identify and eliminate the useless ones.
• With less unnecessary data in your backups, full restores will naturally be faster too.

Won't that be slow or expensive?

Not necessarily! You may already have an NVIDIA or AMD GPU that sits mostly idle when you're not gaming. Or perhaps some Apple Silicon with plenty of unified memory, which is also excellent for AI workloads. And if you don't have any of that yet — be honest: you've been looking for a good reason to buy great new hardware this year anyway!

If great HW is not an option, you will be also able to use cloud AI, like from OpenAI, Microsoft or Anthropic. Your backup metadata will be processed securely and not used for training — probably.

Is it safe?

Don't worry — BorgAI is very confident in its decisions.

Is it worth the effort?

In our tests, BorgAI reduced archive sizes by 23%, archive count by 73% — and successfully restored 99% of the files users actually wanted.

When will it be released?

Likely to be included in borg 1.4. soon.

[StandingPadAnimations]

Don't forget AI-powered compression! Why use regular compression algorithms when you can have AI do it for you?

[ThomasWaldmann]

Yeah, or deduplication - why compute a hash when you can ask the AI if it is the same?

[marcpope]

Just send your data to be trained on the next LLM update then you can ask OpenAI to retrieve it for you. Don't worry if it's public, it will save you lots of money.

[ThomasWaldmann]

OK, guess we can close this now, April 1st is now globally over - relax!

AI won't decide about what's in your backups.