Data Debug Workflow

This page has been migrated here, please make updates to that documentation.

This is the procedure to follow if your data is not available on Code Ocean. Data goes from the acquisition rig to Code Ocean through a series of steps. The first thing to understand is what step failed, then look at any log information available, and finally contact the appropriate person.

Development Dashboard

Click here for SciComp's under-development dashboard

Debug workflow

1. "Missing data"

   • Contact the trainer for that session to confirm if the mouse actually ran.
   • If the mouse did run, check to see if there is saved data on the acquisition hard drive

2. Data on the acquisition hard drive

   • ForagingSettings.json specifies where data is saved. This defaults to "C:\behavior_data"
   • Each session is a folder nested under rig name, then mouse.
   • Standard format:
     • behavior - raw.harp, json file (structure of each trial only related to this task), log file for the session
     • behavior-videos - if there's data
     • fib - if there's data
     • metadata-dir - rig.json (AIND rig definition), session.json (high-level info for the session, AIND format)
   • Is the data saved on the acquisition computer? If not, it may be backed up on an on-prem server - contact Kenta.

3. Data moves from the acquisition hard drive to VAST via aind-watchdog-service

   • Problems at this stage show up on the email report as "stalled" or "lingering manifest"
   • Each session generates an upload manifest after the first trial finishes
   • Manifests don't conflict each other - can run multiple sessions from the same mouse within a day.
   • Waiting to upload manifests are here: C:\Users<user name>\Documents\aind_watchdog_service\manifests
   • Completed manifests are here: C:\Users<user name>\Documents\aind_watchdog_service\manifests
   • You can check the watchdog log here (requires VPN, to on-site): http://eng-logtools:8080/?channel_filter=watchdog&hide=location,count
   • If your manifest failed to upload, talk to Patrick in SIPE: "Upload failed, what should I do?"
   • Most uploads fail because of a bad project name, be sure to check your project name is one of the accepted names: http://aind-metadata-service/project_names

4. Data moves from VAST to S3 bucket via aind-data-transfer-service

   • Problems at this stage show up on the email report as "No data assets"
   • Failures here either mean the upload to S3 failed, or asset registration on Code Ocean failed
   • You can check the data transfer log here: http://aind-data-transfer-service/jobs
   • Contact Jon Young in Sci.Computing "upload and registration failed, what should I do?"

5. Data gets processed into NWB files

   • Problems at this stage show up on the email report as "No processed data" and "Processing errors"
   • You can see logs under the "Traceback" section of this dashboard grafana dashboard
   • You can also see logs under the "Timeline" tab of the pipeline: Code Ocean Pipeline
   • Ask for credentials if you don't have them
   • If failed at this step, talk to Ahad Bawany@SciComp. "My data uploaded, but did not process into NWBs, what should I do?"

6. Training was inadvertently done under a wrong subjectID

   • Modify the existing watchdog manifest: replace subjectID and file locations thoroughly
   • Modify the corresponding data: names, metadata, behavior-json, etc, change all of them using "replace function". (except for the pythonlog)
   • If the data was already once uploaded under a wrong name, ask Jon Young at SciComp to archive data at s3/CO.
   • cf:https://github.com/AllenNeuralDynamics/aind-behavior-blog/issues/1070