Thesis Harvester - feedback and discussion

[kpsherva]

How to test

You've received the first ingestion test email on the 15th of April (this harvest contains all the changes since 2025-05-15 - the date we have migrated the thesis). After this, the test will be running daily at 4am, picking up record modifications from the last run.

Harvester logs (downloadable) contain all the records which failed the harvest with explanations why did it happen
Harvester reports contain all the records which were successfully harvested - and you can see the changes applied under "view changes" button

---

Alternatively, if the email is not working, you can find all the reports in the admin panel:

Admin Panel:

select harvester reports

select specific report (from 15th of April - bulk update all theses since the migration date)

Records which succeeded harvest

you will see here all the modified records:

to see detailed modification of one, click:

you will see a comparison panel:

where green color means addition, yellow means modification, red means deletion

Failed harvest records

To see the reason why other records were not updated (failed to harvest) download the full errors report under download button:

---

Please provide your feedback regarding this feature in this issue.

We are trying to understand:

• which other fields should fail on update (for now we have only publication date mismatches failing).
• how detailed should the harvester logs be (do we have too much information, not enough)
• is the difference view "View changes" useful and understandable
• are the records updated correctly, are we missing any fields?
• any other suggestions

[PaulinaBaranowska]

Hi @kpsherva,

A few first impressions:

1. Link to "View Harvester Reports" and "Download Harvester Logs" from the Emails does not work - but this is already on your radar I believe (permission issues).

2. From what I can see, the harvester currently is only querying Theses from INSPIRE, correct?

3. Example 28 Apr 2026, 15:39:
   I am not sure what to do with the list of records displaying - is that a list of records that have successfully been updated?
   Most of them have just one update - the "revision_id" and "updated" date has changed. Why are they then showing up in this list- or rather why are they being updated? Is there a change in the record that is not highlighted that we are missing? I would expect to see records here that have a change in the metadata that we need to look at. Something like "Records successfully harvested" and then "Updated records" "Records with conflicts" "Failed records" with a list below.

4. Downloading the harvesting logs and then searching the file is cumbersome. It's also not easy to figure out what needs to be done/checked:

• a. What does WARNING mean? Does it mean that whatever is after the warning was not migrated?
  Looking at the log file of the same example:
  [2026-04-28 15:39] WARNING [INSPIRE#1763969] Accelerator 'CERN LHC' not found. but then the record https://sandbox-cds-rdm.web.cern.ch/records/awfsr-wy973 has CERN LHC as the accelerator
  And why is that a WARNING? CERN LHC should be a valid accelerator value?

• b. ERROR
  I would have assumed that this means the record has failed to harvest but for example
  [2026-04-28 15:39] ERROR [INSPIRE#579982] failed transformation. Errors: Thesis publication date missing (thesis_info and imprint). the record is in https://sandbox-cds-rdm.web.cern.ch/records/egnne-w3379
  It would be nice to see this more clearly, so have a section of records in ERROR, and a section for list of records with WARNING
  It would be really nice to have a link to the INSPIRE record rather than having to copy-past the recID into the URL bar.

Concerning specific ERRORs that I found:

• [2026-04-28 15:39] WARNING Skipping 1 transformed entries with errors. - what does this mean? What should we do here?
• [2026-04-28 15:41] ERROR [INSPIRE#1352917] failed transformation. Errors: More than 1 DOI was found. - This will happen as theses might be indexed in multiple places, e.g. https://inspirehep.net/literature/1352917
• [2026-04-28 15:40] ERROR [INSPIRE#1305729 CDS#cyr0f-6nc13] metadata.publication_date: 2016-10-27 -> 2014 [year_mismatch] Incoming publication_date year differs I tried to find the 2016-10-27 date in https://inspirehep.net/api/literature/1305729 but could not find it... where does it come from?

5. Failed harvests, Example 28 Apr 2026:
   Again, it would be nice to see a list here of INSPIRE-IDs as hyperlinks to the failed records with the corresponding error.
   Error writing entry - What do we have to do here to fix this?
   Also, once we correct a record, how can we re-launch the harvest? Or should the update automatically add it to next day's harvest?

Sorry, there ended up being a lot of questions. Not sure if it would be easier to have a meeting to run through all this, let us know.

[kpsherva]

Hi, thank you for checking, please find my answers below:

Hi @kpsherva,

A few first impressions:

1. Link to "View Harvester Reports" and "Download Harvester Logs" from the Emails does not work - but this is already on your radar I believe (permission issues).

Yes, we are already working on addressing this.

3. From what I can see, the harvester currently is only querying Theses from INSPIRE, correct?

yes, correct - it is better to test it in a smaller data set and thesis was a good candidate

4. Example 28 Apr 2026, 15:39:> I am not sure what to do with the list of records displaying - is that a list of records that have successfully been updated?
   Most of them have just one update - the "revision_id" and "updated" date has changed. Why are they then showing up in this list- or rather why are they being updated? Is there a change in the record that is not highlighted that we are missing? I would expect to see records here that have a change in the metadata that we need to look at. Something like "Records successfully harvested" and then "Updated records" "Records with conflicts" "Failed records" with a list below.

it is because I tested another run on a full set of thesis - the actual run which I wanted to show you for testing was this one from 15th of April, like in the instruction. Please let me know if you can see the previous big harvest.
In the run you are checking there was nothing to update - the revision id update in this cases will be removed in the future

4. Downloading the harvesting logs and then searching the file is cumbersome. It's also not easy to figure out what needs to be done/checked:

• a. What does WARNING mean? Does it mean that whatever is after the warning was not migrated?

no, each log line applies only to the record which ID it shows at the beginning of the line. There are no dependencies between the lines (one line does not influence the other). WARNING means record was harvested but might need curators attention for the reason which is specified in the log line

Looking at the log file of the same example:
[2026-04-28 15:39] WARNING [INSPIRE#1763969] Accelerator 'CERN LHC' not found. but then the record https://sandbox-cds-rdm.web.cern.ch/records/awfsr-wy973 has CERN LHC as the accelerator

yes, we saw this as well and this inconsistency will be addressed with another bunch of upgrades.

And why is that a WARNING? CERN LHC should be a valid accelerator value?

same as above

• b. ERROR
  I would have assumed that this means the record has failed to harvest but for example

yes, error is a hard fail - record will not be harvested

[2026-04-28 15:39] ERROR [INSPIRE#579982] failed transformation. Errors: Thesis publication date missing (thesis_info and imprint). the record is in https://sandbox-cds-rdm.web.cern.ch/records/egnne-w3379
It would be nice to see this more clearly, so have a section of records in ERROR, and a section for list of records with WARNING
It would be really nice to have a link to the INSPIRE record rather than having to copy-past the recID into the URL bar.

Concerning specific ERRORs that I found:

• [2026-04-28 15:39] WARNING Skipping 1 transformed entries with errors. - what does this mean? What should we do here?[ ]

this is our script oversharing information :) we will remove it

[2026-04-28 15:41] ERROR [INSPIRE#1352917] failed transformation. Errors: More than 1 DOI was found. - This will happen as theses might be indexed in multiple places, e.g. https://inspirehep.net/literature/1352917[ ] [2026-04-28 15:40] ERROR [INSPIRE#1305729 CDS#cyr0f-6nc13] metadata.publication_date: 2016-10-27 -> 2014 [year_mismatch] Incoming publication_date year differs I tried to find the 2016-10-27 date in https://inspirehep.net/api/literature/1305729 but could not find it... where does it come from?

the date on the right is what we currently have on the record (CDS version) -> means that harvester is trying to change it from 2016-10-27 to 2014 which causes conflict to be manually verified. But when I am looking at this, I am thinking more and more that we should take INSPIRE as source of truth, unless the record has CDS DOI, and only then raise conflicts. This is because you are actively curating INSPIRE so doing curation twice is too costly - so we should have these conflict errors only if external information about CDS originating record conflicts with CDS (since CDS is source of truth if DOI is minted by us) - please let me know if you agree with this and I will change the conflict triggers

5. Failed harvests, Example 28 Apr 2026:
   Again, it would be nice to see a list here of INSPIRE-IDs as hyperlinks to the failed records with the corresponding error.
   Error writing entry - What do we have to do here to fix this?

as above, we will address this

Also, once we correct a record, how can we re-launch the harvest? Or should the update automatically add it to next day's harvest?

you can add it to the next days harvest (to be verified with Micha, since we are using a bit different technology here and I am not 100% sure how it is working in INSPIRE), like it was done priorly

Sorry, there ended up being a lot of questions. Not sure if it would be easier to have a meeting to run through all this, let us know.

No worries about the questions, I am happy you are testing :)