import/promote approach to database update

[aebrer]

From OSL:

The current database update process is unnecessarily painful and involves the devops team more than is necessary. An alternative has been developed which will replace the monolithic quarterly updates with small pre-staged weekly imports of updated proteins. These updates can be examined by investigators before being promoted.
The biggest downside is the increase in the size of the database, but that should not be too bad.
History

#1 Updated by Dale Tronrud - almost 3 years ago

It takes me at least an afternoon to evaluate an update to the database, and that time is hard to come by. I'm having difficultly now finding the time to work on this quarter's update. There is no way I can devote this amount of time on a weekly basis.

#2 Updated by Jack Twilley almost 3 years ago

I apologize for not being clear.
Database promotions will still take place quarterly after scientific review. The difference is how the database is prepared for the update process. Currently we drop everything and rebuild from scratch, then we tell you when we're done so you can check stuff out. After this software change is made, you will be able to do the following from the website:
get a list of proposed changes (how many added, how many changed, how many removed, and which ones)
get a list of errors (like what I sent out earlier this week)
switch between the known good data and the to-be-promoted data for searches
(possibly) perform the promotion yourself by clicking a button.
That last one is probably the one that will cause the most trouble, since we don't have the right kind of authentication for that setup, but I think we might be able to work something out with the admin site.
The goal is for you to decide when to review the data and approve it for promotion on your own schedule instead of waiting for us. If you want to do it semi-annually, that's fine. Weekly would probably be a little crazy, unless you're waiting for a specific protein to update or something.
Does that help?

#3 Updated by Jack Twilley over 2 years ago

I met with Elijah Voigt and Ian Kronquist today to discuss this proposal. We will be working on this proposal in pairs to minimize the accrual of technical debt. Our goal is to complete the code change in time for the January update cycle. Possible obstacles include procedural issues due to the new team approach as well as using GitHub instead of code.o.o as our primary repository for the first time. Additionally, these changes will have to be integrated with Sarvesh's work from this summer's GSoC project in the near future.