Support additional tools (other than rsync and bup)

[thePanz]

I used this GUI for a while and later discovered Restic (https://github.com/restic/restic) as a backup tool.
Would it be possible to adapt the current GUI to handle Restic too?

[spersson]

Ooh... that loooks like a nice tool. Haven't seen it before. I guess it didn't exist when I started Kup...
I'm thinking support could be added and then kup would just use bup if it is installed, and restic otherwise (if restic is installed, of course). I don't like the idea of complicating the configuration just to provide different backends which are almost exactly the same.
For that to be acceptable it would also have to have a nice way of browsing files and choosing what to restore, and also doing the restore. I put a lot of time into making that nice for bup.
If you or anybody else wants to work on adding such a backed, I would be open to the idea.

[thePanz]

Thanks for your reply! My asking was about having an abstraction layer for the "backends" and having Kup being able to use them without distinction. The exclude/include and the GUI would stay the same and be reused by all backends.
Of course: some backends may provide more options than other (backup verify, mounting, encryption), but that would be the duty of the abstraction layer to manage those differences.
WDYT?

disclaimer: I worked with QT many years ago, but I'd love to (re-learn it and) contribute :)

[spersson]

Adding another back end for just saving data should be pretty easy, just copy the bup one and change which commands it runs. Or even just experiment with changing the bup back end directly. The more tricky part is probably the restoring.

[carlonluca]

@thePanz are you planning to implement this?

[thePanz]

@carlonluca notthing planned for the next months, I am quite busy with other work-related stuff, sorry.
If you want to start any implementation, feel free :)

[carlonluca]

I started a draft. I'll share if I'll reach reasonable results.

[spersson]

Very cool. To avoid wasting your time, please share your plans if researching makes you think that the whole thing could work. Like which command parameters will be used to include and exclude folders, how restoring backups will work, how to use passwords, etc. Thanks

[carlonluca]

I added PM #61 to open the discussion on a first draft of restic support. Any help and comment is appreciated.

[spersson]

Considering the discussion on pull request #61 and newly opened request #63 to add support for borg as backend, here are my thoughts as of now.
Perhaps it could be acceptable to support all three: bup, borg and restic. They are very very similar. I don't like the idea of having to support lots of code for no good reason (more code = more chance of bugs). But I am beginning to think that these are so similar that perhaps we can have a high degree of code sharing.
I don't want to have a complication of only supporting some configuration options in combination with some backend. I already do that between bup and rsync but at least there I think it is motivated by just how different the backups are. So for bup,restic, borg, etc.. I don't think I want to keep "generate recovery info" since only bup supports that, encryption is out since bup doesn't support that (if that is something many people need then we could discuss only keeping borg and restic... future is uncertain). When it comes to pruning old backups the support in bup is not officially stable but seems to work fine and bup crew have a history of being very careful about promises. So I think it could be added to kup for all three backends.
All three backends could support checking backup integrity.
Code sharing on saving should be very easy. It's not much code anyway.
Sharing on recovery could make use of code added in PR #61, for harvesting version data from a fuse mount. That can work on all three backends with only small differences I think.
I still don't want to force the user to get educated about borg vs restic vs kup. That is a very difficult question with no simple answer. The user either just uses what was preinstalled with the OS or installs one of the supported backends and that gets used. So I think the configuration should stay the same where user selects between incrimental or sync type of backup. For incremental we then decide for ourselves which priority order between bup, restic and borg to use (if one of them is not installed then of course it can't be used). First time a backup is saved we also save info about which backend was used so that if later priorities are changed or the user installs another backend then we don't silently switch backend used for the user.

To summarize the user-visible changes proposed here:

1. remove option to generate recovery information
2. add fallbacks restic and borg in case bup is not installed (this may not be user-visible, or maybe we will just show somewhere which of the three is used but no option to change it).

Important consequence of what is proposed here:

1. Any future feature request will be rejected if not all backends support it.

I invite everyone to comment and provide your view on this. @carlonluca , @Sekhemty, @ShalokShalom, @Uatschitchun, @HinzundKunz , @thePanz , @erenoglu, @diegogangl , @sbaragnaus2

[stefanobartoletti]

I'm the one that suggested Borg as a possible backend, I've detailed all here #63 so I'm not going to repeat myself. This is the official site https://www.borgbackup.org/ and this is a wrapper used for automation https://torsion.org/borgmatic/

My main reason for suggesting another backend is that borg can be set to prune old backups if needed, plus other features like encryption and compression of backups.

Like @spersson said borg, restic and bup share many features, they are after all geared to perform the same task; borg and restic seem to be especially similar (bup is different in being based on git).

Some interesting reads form comparison are here https://blog.stickleback.dk/borg-or-restic/ , here https://mail.python.org/pipermail/borgbackup/2017q3/000778.html , here https://forum.restic.net/t/how-did-you-hear-about-restic/115/8 and especially here for a real world test of the three tools https://gist.github.com/alexbarton/6c84f74bc6a4be1b8664c26afe0535b7

I don't really know the full differences between all these systems, by reading around I only think that borg and restic are the most advanced Linux backup solutions right now, they seem to be more actively maintained than bup and seem to have more advanced features.

For what I gathered from the above reads, here are some insights:

bup

• Pros

  • it is already implemented
  • compression

• Cons

  • not actively maintained like the other two ?
  • can't prune old snapshots / remove files
  • no encryption

borg

• Pros

  • pruning old snapshots
  • encryption (optional)
  • compression
  • third party wrapper Borgmatic could also be used as a base?
  • more flexible / configurable

• Cons

  • only supports local or remote SSH for backup repos (no online storage service, if this is an issue)
  • every machine uses its own repo (if used for networks or servers, it could be an issue)
  • slower to load when mounting repos for accessing files

restic

• Pros

  • pruning old snapshots
  • encryption
  • faster when backing up than the other two (due to not compressing files?)
  • supports online serivces like Amazon S3 for backup repos
  • multiple machines can share the same repo (useful when backing up many machines on a network)
  • configuration files are written in the repo only

• Cons

  • No compression
  • heavier on system resources
  • slower to prune old snapshots and to restore files

Please note that this summarization is just what I've gathered by reading other people impressions, I've not thoroughly tested these apps, and it is not a definitive feature list.

As you can see, there are certain differences, but considering only borg and restic, the former seems to be the best choice for local backups for single machines, while restic has the upper hand in a network utilization. To me, bup seem to offer nothing more than the other two solutions, except being the one already used (and so it is a tested working solution)

---

I would keep in mind the KDE principle "Simple by default, powerful when needed", I think that a simple interface like the current one is perfect for basic users and everyday management, and is probably the best solution for anyone that doesn't know / doesn't care about the technicalities of the various backends.

The "base" configuration could be made to adhere to this principle, and use what will be chosen as the default backend (probably bup, since it is already there); but I think that there should be the possibility to activate an "advanced mode" of sorts in the GUI where one can select his preferred backend, if so desired; using what is found in the system without an explicit possibility to select what one wants seems to be a rather great limitation, if for some reason one has both bup and borg installed (testing, or anything else), and the default is bup > borg, one must be able to tell the program that he wants to use borg instead. Also, by not making explicit what backend I'm using, if one wants to restore the backup to a different machine (when you buy a new laptop, i.e.)

Also, not supporting features shared by all three seem to be a limitation.
Probably when "advanced mode" is enabled, different tabs could be added to configure and fine-tune every chosen backend (even now, being able to set different profiles for different things means that one could set the primary backup with bup and a sync profile for a specific directory with rsync). Let's say that one wants to configure the frequency of the pruning (which bup don't have), to enable/disable the encryption (

Of course, all these suggestions are to be taken into account only if they are feasible. I don't know if supporting all three backends is too much work, and of course I don't want to force anyone to do as I suggest, but if these "advanced" features are feasible without too much hassle, I think that leaving them out just for the sake of extreme simplicity kinda leaves out the "powerful when needed" of the KDE motto.

[spersson]

@Sekhemty thank you for the thoughtful reply! You make some good points.
I am still very uncertain about this whole thing and it's very nice to get someone else's perspective.
I don't mind offering advanced options and already do that in kup, at least a few. But the options still need to stay reasonable, meaning that they can't be decided automatically by the software and also have a real impact on end results.

I have now done some more research and testing of the three backends. So far I really can't see any good reason why somebody would need to use borg or restic instead of bup. Sure, they seem like good softwares. But so is bup. The only advantage I can see is that they offer encryption. But nobody has ever requested encryption from kup.

So I am now leaning towards just sticking with bup and rsync. We can discuss removing old backups in issue #10.
Still interested to hear more opinions... You can see my suggestions changing from one extreme to the other quickly... :)

[carlonluca]

The only reason why I added restic support is because bup reports drop of old backups as experimental: "bup only has experimental support for pruning old backups". At this point, if support is confirmed to be stable, I don't think it makes much sense to implement it. I'd improve the current format (bup) and I'd take care that it works perfectly.
As a matter of fact, however, this borg seems to work very very similarly to the others, and could probably be integrated quickly using the changes I already made in #61. It seems, from the descriptions, to be an improvement to both bup and restic: it seems to implement better compression integration, encryption (I personally need it), pruning seems to be "reported" as stable and seems more alive than bup.
I'll stop any more work on the patch as there is no agreement on that. I however think that, even if restic is no more considered useful, a large portion of #61 could still be used. The abstraction it offers makes it pretty simple to integrate anything else that kup decides to support in the future.