Publishing generated csv files

[mereacre]

We can consider the following options to publish the generated csv files from the yaml files:

• As a separated branch of the repo (in the same way as we do with docusaurus doc pages)
• As a release version

Do you have any other options?

[aloisklink]

As a separated branch of the repo (in the same way as we do with docusaurus doc pages)
This is probably the easiest option, since I believe in discussion was that we were going to do the same thing with JSON files generated from the yaml files.

Asking users to do a git clone would be really heavy for this first pull, but for updates, a git pull is much faster than constantly downloading new release versions, since git pull only needs to fetch the changes.

Plus, git has some simple SHA1 hashes, so it's more resistant to tampering/corruption (although still possible for an attack to break with a SHAttered attack). And we can use GPG to sign our git commits too, to make them more authentic.

As a release version

We'd have to write custom version checking code, to know when there's a new release to download, so this is extra work.

In the long term, having everyone use git pull for versioning is probably not really efficient, so it might be worth instead releasing a tar.gz every week, but that's a bunch of extra work, so we should only do it if/when we need to.

This does have the benefit that it doesn't require git though, so it's also ideal for more constrained devices/routers.

[mcr]

The disk space impact of a git clone on an embedded device (even a Turris) may be non-trivial.
But, a git clone --shallow should be pretty cheap. A .tgz with rsync update would be better though.

[aloisklink]

The disk space impact of a git clone on an embedded device (even a Turris) may be non-trivial.

Hmmm, this is definitely a concern long-term. A quick du -h . on my PC showed the repo currently being ~250 MB, mostly from the ./.git and ./manufacturers folder.

alois@nqm-alois-entroware:~/Documents/ManySecured-D3DB$ du -h .
[ Some entries truncated ]
195M	./manufacturers
43M	./.git

Even if we just git fetch a branch with just the CSV files, the repo will be pretty substantial in size.

But, a git clone --shallow should be pretty cheap. A .tgz with rsync update would be better though.

👍

I think we'll probably stick with doing git clone in the short-term, just because it will be faster to implement.

git clone --shallow should only be used if you're never going to update the repo
Instead, we'd probably want to use git clone --filter=blob:none.

In the long-term, we'd probably need to think of a better more efficient way of doing this.

[mereacre]

I would assume for the purpose of the demo, a static link would be the easiest. Moreover we are alreday using requests package to retrieve csv files:

ManySecured-D3DB/d3-scripts/src/d3_scripts/d3_populate.py

Line 23 in 53a1e2b

data = requests.get(url, allow_redirects=True)

[aloisklink]

I would assume for the purpose of the demo, a static link would be the easiest. Moreover we are alreday using requests package to retrieve csv files:

Ideally, part of the demo would be on how easy it is to update the D3DB with your own claim (assuming internet connection works), so we need some kind of update mechanism, so that we can display:

• contributing to the D3DB repo
• some magic GitHub CI that creates the new CSVs
• updating the D3DB repo on the ManySecured router

If it's low priority, we could ignore this. My gut feeling is that it should be pretty easy to implement though.