Add diagram of modular experiments

docs/source/acquire_upload/acquire_data.md

@@ -2,7 +2,8 @@
 
 During data acquisition you are responsible for running version-controlled acquisition software and ensuring your data files for each modality are organized according to standardized conventions.
 
-Metadata generated during acquisition captures **what data** should appear in the final NWB files after processing, as well as **what manipulations** were performed (both behavioral stimuli and any procedures).
+Metadata generated during acquisition captures how data was acquired. This includes what data streams are being recorded, what stimuli or behaviors (if any) are used, and any manipulations (procedures) that occur during the session.
+
 
 ## Data
 
@@ -47,4 +48,3 @@ Some tasks are being run on a standardized platform using Bonsai and Harp for da
 |------|-----------|
 | VrForaging | https://github.com/AllenNeuralDynamics/Aind.Behavior.VrForaging |
 | IsoForce | https://github.com/AllenNeuralDynamics/Aind.Behavior.IsoForce |
-

docs/source/acquire_upload/calibration.md

@@ -8,6 +8,66 @@ Common calibrations include measuring power output (e.g. for lasers) with [Power
 
 Calibrations have an option to include fit parameters, if your calibration fit is not available in the [FitType](https://aind-data-schema.readthedocs.io/en/latest/components/measurements.html#fittype) options please request that we add it by opening an [issue](https://github.com/AllenNeuralDynamics/aind-data-schema/issues).
 
-## Instrument testing
+## Testing
 
-When collecting a *test data asset* on an instrument using a "calibration object" instead of a subject or specimen, you should set `subject_id = "calibration"` in all metadata files. Please also use the [CalibrationObject](https://aind-data-schema.readthedocs.io/en/latest/components/subjects.html#calibrationobject) in the `Subject.subject_details` to track information about the physical object used during calibration.
+Use `subject_id="calibration"` to mark data assets as test assets. When a "phantom" or other calibration object is used during testing please provide details about that object in a `CalibrationObject`. Before uploading calibration assets you need to ensure that your asset and metadata are compatible with the processing pipelines that will be run downstream.
+
+Note that test assets that are not intended to be kept long-term should be immediately (or as soon as feasible) marked as archived on Code Ocean. Archived assets are deleted when unused for 30 days.
+
+### Manual calibration metadata
+
+If the processing pipeline that will run on your test data asset **requires certain fields in the subject or procedures metadata to be set** you need to create an actual `subject.json` and `procedures.json` and upload these alongside your data asset.
+
+```{code-block} python
+from aind_data_schema.core.subject import Subject
+from aind_data_schema.core.procedures import Procedures
+from aind_data_schema.components.subjects import CalibrationObject
+from aind_data_schema.components.devices import Device
+from aind_data_schema_models.organizations import Organization
+
+subject_id = "calibration"
+
+subject = Subject(
+    subject_id=subject_id,
+    subject_details=CalibrationObject(
+        empty=True,
+        description="FIP calibration",
+    ),
+)
+
+procedures = Procedures(subject_id=subject_id)
+
+subject.write_standard_file()
+procedures.write_standard_file()
+```
+
+### Automated calibration metadata
+
+If the processing pipeline that will run on your data asset **does not read the subject and/or procedures metadata** you can follow the instructions below to create empty subject and procedures files.
+
+Note that this automation is only available if your job_type runs `aind-metadata-mapper>=1.3.0`. Please include as much detail as possible about your [CalibrationObject](https://aind-data-schema.readthedocs.io/en/latest/components/subjects.html#calibrationobject) in the upload settings.
+
+```{code-block} python
+from aind_data_schema.components.subjects import CalibrationObject
+from aind_data_schema_models.modalities import Modality
+from aind_metadata_mapper.gather_metadata import GatherMetadataJob
+from aind_metadata_mapper.models import JobSettings, DataDescriptionSettings, SubjectSettings
+
+job_settings = JobSettings(
+    output_dir="/path/to/output",
+    subject_id="calibration",
+    data_description_settings=DataDescriptionSettings(
+        project_name="<project-name>",
+        modalities=[Modality.ECEPHYS],
+    ),
+    subject_settings=SubjectSettings(
+        calibration_object=CalibrationObject(
+            description="Neuropixels dummy probe",
+            empty=False,
+        )
+    ),
+)
+
+job = GatherMetadataJob(job_settings=job_settings)
+job.run_job()
+```

docs/source/acquire_upload/prepare_before_acquisition.md

@@ -10,7 +10,8 @@ You are ready to generate data when:
 
 ## Project name
 
-Your *project* and *subproject* (if applicable) needs to be accurate. The full project name `<project_name> - <subproject_name>` is tied directly with the funding and investigator metadata. The list of project names can be viewed at the [metadata-service project_names/ endpoint](https://aind-metadata-service/api/v2/project_names). Projects that do not have metadata in the metadata-service must upload their own `data_description.json` -- reach out to Scientific Computing for help.
+Your *project* and *subproject* (if applicable) needs to be accurate. The full project name `<project_name> - <subproject_name>` is tied directly with the funding and investigator metadata. The list of project names can be viewed at the [metadata-service project_names/ endpoint](https://aind-metadata-service/api/v2/project_names).
+Projects that are not listed in the metadata-service must provide their own `data_description.json` at upload, including funding and investigator fields. Reach out to Scientific Computing for help.
 
 If you need a new project name, please request that it be added with the [project name and funding intake form](https://app.smartsheet.com/b/form/9f366857582b4db98d1fe41ef724a613).
 
@@ -20,7 +21,7 @@ The funding endpoint will be used during data upload to populate your data descr
 
 ```{raw} html
 <div style="margin: 20px 0; padding: 15px; border: 1px solid #ccc; border-radius: 5px; background-color: #f9f9f9;">
-  <label for="projectSelect" style="font-weight: bold; display: block; margin-bottom: 10px;">Select a project fetch funding information from the metadata-service:</label>
+  <label for="projectSelect" style="font-weight: bold; display: block; margin-bottom: 10px;">Select a project to fetch funding information from the metadata-service:</label>
   <select id="projectSelect" style="width: 100%; padding: 8px; margin-bottom: 10px; border: 1px solid #ccc; border-radius: 4px;">
     <option value="">-- Loading projects... --</option>
   </select>
@@ -83,7 +84,7 @@ The funding endpoint will be used during data upload to populate your data descr
     fetch('https://aind-metadata-service/api/v2/funding/' + encodeURIComponent(projectName))
       .then(response => {
         if (!response.ok) {
-          throw new Error('HTTP error! status: ' + response.status);
+          return response.text().then(text => { throw new Error(text || 'HTTP error! status: ' + response.status); });
         }
         return response.json();
       })
@@ -174,7 +175,7 @@ The investigators endpoint will be used during data upload to populate your data
     fetch('https://aind-metadata-service/api/v2/investigators/' + encodeURIComponent(projectName))
       .then(response => {
         if (!response.ok) {
-          throw new Error('HTTP error! status: ' + response.status);
+          return response.text().then(text => { throw new Error(text || 'HTTP error! status: ' + response.status); });
         }
         return response.json();
       })
@@ -265,17 +266,19 @@ Subject metadata is populated by lab animal services (LAS) without your involvem
 
     fetch('https://aind-metadata-service/api/v2/subject/' + encodeURIComponent(subjectId))
       .then(response => {
-        if (!response.ok) {
-          throw new Error('HTTP error! status: ' + response.status);
+        if (!response.ok && response.status !== 400) {
+          return response.text().then(text => { throw new Error(text || 'HTTP error! status: ' + response.status); });
         }
-        return response.json();
+        return response.json().then(data => ({ data, status: response.status }));
       })
-      .then(response => {
-        const data = response.data || response;
-        resultDiv.style.backgroundColor = '#d4edda';
-        resultDiv.style.border = '1px solid #28a745';
-        resultDiv.innerHTML = '<strong>Subject Information:</strong><pre style="margin-top: 10px; white-space: pre-wrap; word-wrap: break-word;">' + 
-                              JSON.stringify(data, null, 2) + '</pre>';
+      .then(({ data, status }) => {
+        const subject = data.data || data;
+        const isInvalid = status === 400;
+        resultDiv.style.backgroundColor = isInvalid ? '#fff3cd' : '#d4edda';
+        resultDiv.style.border = isInvalid ? '1px solid #ffc107' : '1px solid #28a745';
+        resultDiv.innerHTML = (isInvalid ? '<strong>Warning: subject data failed schema validation:</strong>' : '<strong>Subject Information:</strong>') +
+                              '<pre style="margin-top: 10px; white-space: pre-wrap; word-wrap: break-word;">' +
+                              JSON.stringify(subject, null, 2) + '</pre>';
       })
       .catch(error => {
         resultDiv.style.backgroundColor = '#f8d7da';
@@ -394,7 +397,7 @@ Currently, only NSB procedures are automatically attached to data assets during
 
 ### Custom procedures
 
-Custom [Procedures](https://aind-data-schema.readthedocs.io/en/latest/procedures.html) require you to generate a `procedures.json` file manually. Note that the `data-transfer-service` will **NOT** merge your procedures with any stored in NSB, you must pull the NSB procedures and manually merge them ahead of time, please reach out to Scientific Computing for help with this process.
+Custom [Procedures](https://aind-data-schema.readthedocs.io/en/latest/procedures.html) require you to generate a `procedures.json` file manually. Please only provide metadata for procedures that are not stored by NSB.
 
 ### NSB procedures
 
@@ -456,17 +459,19 @@ Standardized procedures that are performed by NSB (link?) are uploaded and acces
 
     fetch('https://aind-metadata-service/api/v2/procedures/' + encodeURIComponent(subjectId))
       .then(response => {
-        if (!response.ok) {
-          throw new Error('HTTP error! status: ' + response.status);
+        if (!response.ok && response.status !== 400) {
+          return response.text().then(text => { throw new Error(text || 'HTTP error! status: ' + response.status); });
         }
-        return response.json();
+        return response.json().then(data => ({ data, status: response.status }));
       })
-      .then(response => {
-        const data = response.data || response;
-        resultDiv.style.backgroundColor = '#d4edda';
-        resultDiv.style.border = '1px solid #28a745';
-        resultDiv.innerHTML = '<strong>Procedures Information:</strong><pre style="margin-top: 10px; white-space: pre-wrap; word-wrap: break-word;">' + 
-                              JSON.stringify(data, null, 2) + '</pre>';
+      .then(({ data, status }) => {
+        const procedures = data.data || data;
+        const isInvalid = status === 400;
+        resultDiv.style.backgroundColor = isInvalid ? '#fff3cd' : '#d4edda';
+        resultDiv.style.border = isInvalid ? '1px solid #ffc107' : '1px solid #28a745';
+        resultDiv.innerHTML = (isInvalid ? '<strong>Warning: procedures data failed schema validation:</strong>' : '<strong>Procedures Information:</strong>') +
+                              '<pre style="margin-top: 10px; white-space: pre-wrap; word-wrap: break-word;">' +
+                              JSON.stringify(procedures, null, 2) + '</pre>';
       })
       .catch(error => {
         resultDiv.style.backgroundColor = '#f8d7da';

docs/source/acquire_upload/process_data.md

@@ -2,7 +2,8 @@
 
 Scientific computing is currently re-organizing pipelines to be per-modality, rather than per-project.
 
-Pipeline development requirements are documented in [Pipeline development](../policies_practices/pipeline_development.md).
+Pipeline development requirements are documented in [Pipeline development](../policies_practices/platform_support.md#pipeline-development)
+and the pipeline versioning policy is documented in [Versioning pipelines](../policies_practices/version_pipelines.md).
 
 ## Per-modality physiology pipelines
 

docs/source/aind/core_services.md

@@ -1,5 +1,7 @@
 # Core Services
 
+The interactions between many of these services are illustrated in the [AIND software diagrams](./diagrams.md).
+
 **aind-data-transfer-service**
 
 FastAPI service to run data compression and transfer jobs on the HPC

docs/source/aind/diagrams.md

@@ -0,0 +1,11 @@
+# AIND Software and Systems Diagrams
+
+This page contains diagrams illustrating the interactions between AIND software and systems, including core services, data storage, and compute resources. These diagrams are intended to provide a high-level overview of how different components fit together, and will be updated periodically as our software and systems evolve.
+
+**New diagrams coming soon for future plans and low-level architecture.**
+
+## High-level architecture
+
+![High-level data flow](../diagrams/high_level/general_data_flow.drawio.svg)
+
+![AIND Software Overview](../_static/aind-software-overview.png)

docs/source/conf.py

@@ -33,6 +33,7 @@
     "sphinx.ext.autodoc",
     "sphinx.ext.napoleon",
     "sphinx_tippy",
+    "sphinx_copybutton",
     "myst_parser",
 ]
 templates_path = ["_templates"]