dashboard: archive and display review conversations from previous uploads

[GraemeWatt]

Background

At an IPPP workshop on 18th March 2026, @jonbutterworth made a request for a Reviewer to be able to see old review comments from previous uploads, without keeping the relevant emails. This is not easy to address, as the current behaviour is to delete all DataSubmission objects and their attached DataReview objects from the database whenever a new upload is made:

hepdata/hepdata/modules/records/utils/submission.py

Lines 455 to 459 in 9f7766c

	# Delete all data records associated with this submission.
	# Fixes problems with ordering where the table names are changed between uploads.
	# See https://github.com/HEPData/hepdata/issues/112
	# Side effect that reviews will be deleted between uploads.
	cleanup_submission(recid, hepsubmission.version, added_file_names)

This behaviour was implemented by me in 2018 via commit 441ed75 to address an issue #112 where the order of data tables was not preserved if table names changed between uploads:

hepdata/hepdata/modules/records/utils/submission.py

Lines 507 to 509 in 9f7766c

	# The line below is commented out since it does not preserve the order of tables.
	# Delete all tables above instead: side effect of deleting reviews between uploads.
	#cleanup_submission(recid, hepsubmission.version, added_file_names)

Grouping review comments of successive uploads by table name is generally not meaningful due to possible renaming of tables between uploads. I'm reluctant to change the behaviour that has been in operation since 2018, so an alternative approach is needed to archive review comments of previous uploads.

Proposal

Existing functionality

From a user's Dashboard, an Uploader/Reviewer can click the speech bubble to bring up a "Conversation" widget showing a list of table names for the current upload together with any review messages. Alternatively, a user with Coordinator permissions can click the gears icon to bring up the "Manage Submission" widget with two tabs "Manage Uploaders & Reviewers" and "Conversation". The review messages are loaded into the widget using the JavaScript load_all_review_messages function which calls the Python function get_all_review_messages. The latter function, invoked by the endpoint /record/data/review/message/<int:publication_recid>, returns all review messages for a publication record in a JSON format.

Proposed new functionality

For the case of a new upload being made, where at least one data table has a review comment, the process_submission_directory function could call get_all_review_messages before deleting the old data tables with cleanup_submission. These old review messages obtained via get_all_review_messages could then be stored in a new PostgresSQL database table as json (or preferably jsonb) along with the publication_recid. This database table may then contain multiple conversations with the same publication_recid corresponding to different uploads, so a field like id would be needed to order them by time. When loading the "Conversation" widget from a user's Dashboard, the code could check for entries in the new PostgresSQL database table with the corresponding publication_recid. If any are found, the "Conversation" widget could display left/right arrows to allow the user to navigate to previous conversations. The left/right arrows should be dimmed or hidden if there is not an older/newer conversation relative to the selected conversation. The left/right arrows should be hidden completely if there is no relevant entry in the new PostgresSQL database table, preserving the current display of the "Conversation" widget. If a publication record is deleted via the delete_submission function, the corresponding entries with the same publication_recid in the new PostgresSQL database table should also be deleted.

Note that this proposal does not involve storing messages entered in other text boxes such as the "Notify Participants" button or Uploader/Reviewer invitations, which are generally only included in emails and are never persisted to the database.