1430 migrate hermes broker to data service

docs/managing_data/stream_pub_sub.rst

@@ -3,6 +3,37 @@ Publish and Subscribe to a Kafka Stream
 
 Publishing data to a stream and subscribing to a stream are handled independently and we describe each below.
 
+.. note::
+
+   For TOMToolkit version 3, all HERMES-specific code has moved from ``tom_base`` into the
+   ``tom_hermes`` package and the old import paths in ``tom_base`` are removed. If you are upgrading from a
+   pre-refactor ``tom_base`` / ``tom_hermes``, see the *Migration summary* at the bottom of this page for the
+   list of dotted paths that need to be updated.
+
+
+SharingBackend integration point
+################################
+
+Data-sharing destinations are discovered by plug-in rather than by hardcoded string match. An app advertises
+its SharingBackend classes by adding a ``sharing_backends()`` method (i.e. integration point) to its AppConfig;
+``tom_common.sharing`` iterates installed apps, imports the listed class paths, and builds a registry keyed by
+each backend's ``name``. Existing backends:
+
+* **``TomToolkitSharingBackend``** (``tom_common.sharing``) — "share with another TOM Toolkit-based TOM."
+  Registered by ``tom_common.apps.TomCommonConfig``. ``name = 'tom'``.
+* **``HermesSharingBackend``** (``tom_hermes.sharing``) — "publish to HERMES." Registered by
+  ``tom_hermes.apps.TomHermesConfig``. ``name = 'hermes'``.
+
+The share-destination form field's value is formatted as ``'<name>:<sub-destination>'`` (for example
+``'hermes:hermes.test'`` or ``'tom:tom_b'``); ``DataShareView.post`` parses the prefix and dispatches to the
+matching backend's ``share()`` method. A future publisher is registered by writing a ``SharingBackend``
+subclass in its app and returning it from that app's AppConfig ``sharing_backends()`` hook — no changes to
+``tom_base`` required.
+
+To write a new SharingBackend, subclass ``tom_common.sharing.SharingBackend`` and implement ``share()`` and
+``get_destination_choices()``. See the ``tom_common.sharing.SharingBackend`` docstring and the two included
+subclasses for reference.
+
 
 Publish Data to a Kafka Topic
 #############################
@@ -11,64 +42,99 @@ TOM Toolkit supports publishing data to a Kafka stream such as `Hermes <https://
 `HOPSKOTCH <https://hop.scimma.org>`_) and `GCNClassicOverKafka <https://gcn.nasa.gov>`_.
 
 When sharing photometry data via Hermes, the TOM publishes the data to be shared to a topic on the HOPSKOTCH
-Kafka stream. At this time, only photometry data is supported by TOM Toolkit. To submit via the Hermes API, you will
-need to copy your Hermes API Key from your Hermes profile page. When hermes sharing is configured, you will also see
-buttons to open your data in hermes with the form pre-filled - this is a good option if you want to make slight changes
-to your message or data before sharing.
+Kafka stream. To submit via the Hermes API, you will need a HERMES API Key. You can store it either per-user
+on the ``HermesProfile`` (see *Per-user HERMES credentials* below) or TOM-wide in ``settings.DATA_SHARING``.
+``HermesProfile`` credentials take precedence when set.
+
+To customize what data is sent to HERMES from your ReducedDatum or Target models, subclass
+``tom_hermes.sharing.HermesDataConverter`` and override the ``get_hermes_*`` methods to pull the data out of
+your TOM's model fields. Provide the class dotpath in ``settings.DATA_SHARING['hermes']['DATA_CONVERTER_CLASS']``.
+For more information on the structure HERMES expects, see the
+`API Schema Registry here <https://hermes.lco.global/about>`_.
 
-To customize what data is sent to hermes from your ReducedDatum or Target models, please re-implement your own
-``tom_dataproducts.alertstreams.hermes.HermesDataConverter`` and customize the `get_hermes_*` methods to pull out
-the proper data you want to share. You then provide the class dotpath to your custom class in your TOM's settings
-for hermes ``DATA_SHARING`` in the `DATA_CONVERTER_CLASS` key. This is especially useful if you store extra target
-or datum information in custom associated models in your TOM or with custom model field keys. For more information on
-the structure of data HERMES expects, check the `API Schema Registry here <https://hermes.lco.global/about>`_. This is
-the structure you should be mapping your ReducedDatum values to in the Data Converter Class.
 
+Configuring your TOM to publish data
+************************************
 
-Configuring your TOM to Publish Data to a stream:
-*************************************************
+``settings.DATA_SHARING`` is a dict keyed by destination name. Multiple TOM-to-TOM destinations are supported
+(add one entry per destination TOM); the HERMES destination is identified by the presence of
+``HERMES_API_KEY``.
 
-You will need to add a ``DATA_SHARING`` configuration dictionary to your ``settings.py`` that gives the credentials
-for the various streams with which you wish to share data.
+Authentication for TOM-to-TOM destinations: prefer a DRF API key via the ``API_KEY`` key (TOM Toolkit
+auto-generates a DRF token per user, and a service-account token can be created on the destination TOM);
+fall back to HTTP Basic via ``USERNAME`` and ``PASSWORD``.
 
 .. code:: python
 
-   # Define the valid data sharing destinations for your TOM.
     DATA_SHARING = {
+        # One or more TOM-to-TOM destinations. This configuration is used by
+        # tom_common.sharing.TomToolkitSharingBackend.
+        'tom_alice': {
+            'DISPLAY_NAME': 'TOM Alice',
+            'BASE_URL':     'https://tom-alice.example.org/',
+            'API_KEY':      os.getenv('TOM_ALICE_API_KEY', ''),   # preferred; Token auth
+        },
+        'tom_bob': {
+            'DISPLAY_NAME': 'TOM Bob',
+            'BASE_URL':     'https://tom-bob.example.org/',
+            'USERNAME':     os.getenv('TOM_BOB_USERNAME', ''),  # fallback: HTTP Basic; not preferred
+            'PASSWORD':     os.getenv('TOM_BOB_PASSWORD', ''),
+        },
+
+        # HERMES destination. This configuration used by tom_hermes.sharing.HermesSharingBackend.
         'hermes': {
-            'DISPLAY_NAME': os.getenv('HERMES_DISPLAY_NAME', 'Hermes'),
-            'BASE_URL': os.getenv('HERMES_BASE_URL', 'https://hermes.lco.global/'),
-            'HERMES_API_KEY': os.getenv('HERMES_API_KEY', 'set HERMES_API_KEY value in environment'),
-            'DEFAULT_AUTHORS': os.getenv('HERMES_DEFAULT_AUTHORS', 'set your default authors here'),
-            'USER_TOPICS': ['hermes.test', 'tomtoolkit.test']  # You must have write permissions on these topics
-            'DATA_CONVERTER_CLASS': 'tom_dataproducts.alertstreams.hermes.HermesDataConverter'
+            'DISPLAY_NAME':         os.getenv('HERMES_DISPLAY_NAME', 'Hermes'),
+            'BASE_URL':             os.getenv('HERMES_BASE_URL', 'https://hermes.lco.global/'),
+            'HERMES_API_KEY':       os.getenv('HERMES_API_KEY', ''),
+            'DEFAULT_AUTHORS':      os.getenv('HERMES_DEFAULT_AUTHORS', ''),
+            'USER_TOPICS':          ['hermes.test', 'tomtoolkit.test'],
+            'DATA_CONVERTER_CLASS': 'tom_hermes.sharing.HermesDataConverter',
         },
     }
 
+
+Per-user HERMES credentials
+***************************
+
+Instead of (or in addition to) a TOM-wide HERMES API key in ``settings.DATA_SHARING``, each TOM user can
+store their own HERMES credentials on the user profile page. Visit the profile page and click the pencil
+icon on the "HERMES Credentials" card to set:
+
+* ``HERMES API Key`` — the user's HERMES submit API key.
+* ``Hopskotch Username`` / ``Hopskotch Password`` — SCRAM credentials for reading from Hopskotch.
+
+Lookup order when publishing: the user's ``HermesProfile`` first; if unset, the TOM-wide
+``settings.DATA_SHARING['hermes']`` fallback.
+
+
 Subscribe to a Kafka Topic
 ##########################
 
 TOM Toolkit allows a TOM to subscribe to a topic on a Kafka stream, ingesting messages from that topic and handling the data.
 This could involve simply logging the message or extracting the data from the message and saving it if it is properly formatted.
 
-Configuring your TOM to subscribe to a stream:
+
+Configuring your TOM to subscribe to a stream
 **********************************************
 
-First you will need to add ``tom_alertstreams`` to your list of ``INSTALLED_APPS`` in your ``settings.py``.
+First add ``tom_alertstreams`` (and ``tom_hermes``, if you plan to subscribe to HERMES topics) to your
+``INSTALLED_APPS``:
 
 .. code:: python
 
     INSTALLED_APPS = [
         ...
         'tom_alertstreams',
+        'tom_hermes',
     ]
 
-Then you will need to add an ``ALERT_STREAMS`` configuration dictionary to your ``settings.py``. This gives the credentials
-for the various streams to which you wish to subscribe. Additionally, the ``TOPIC_HANDLERS`` section of the stream ``OPTIONS``
-will include a list of handlers for each topic.
+Then add an ``ALERT_STREAMS`` configuration dictionary. This gives the credentials for the various streams
+and maps each subscribed topic to the dotted path of its handler callable.
 
-Some alert handlers are included as examples. Below we demonstrate how to connect to a Hermes Topic. You'll want to check
-out the ``tom-alertstreams`` `README <https://github.com/TOMToolkit/tom-alertstreams>`_ for more details.
+A HERMES alert handler lives at ``tom_hermes.alertstreams.ingester.hermes_alert_handler``. If you are
+upgrading from a pre-refactor TOM, update any ``TOPIC_HANDLERS`` dotted paths that pointed at the old
+location (``tom_dataproducts.alertstreams.hermes_ingester.hermes_alert_handler``) — the old path no longer
+exists.
 
 .. code:: python
 
@@ -78,11 +144,58 @@ out the ``tom-alertstreams`` `README <https://github.com/TOMToolkit/tom-alertstr
             'NAME': 'tom_alertstreams.alertstreams.hopskotch.HopskotchAlertStream',
             'OPTIONS': {
                 'URL': 'kafka://kafka.scimma.org/',
-                'USERNAME': os.getenv('SCIMMA_CREDENTIAL_USERNAME', 'set SCIMMA_CREDENTIAL_USERNAME value in environment'),
-                'PASSWORD': os.getenv('SCIMMA_CREDENTIAL_PASSWORD', 'set SCIMMA_CREDENTIAL_USERNAME value in environment'),
+                'USERNAME': os.getenv('SCIMMA_CREDENTIAL_USERNAME', ''),
+                'PASSWORD': os.getenv('SCIMMA_CREDENTIAL_PASSWORD', ''),
                 'TOPIC_HANDLERS': {
-                    'tomtoolkit.test': 'tom_dataproducts.alertstreams.hermes.hermes_alert_handler',
+                    'tomtoolkit.test': 'tom_hermes.alertstreams.ingester.hermes_alert_handler',
                 },
             },
         },
     ]
+
+
+HERMES as a query source (DataService)
+######################################
+
+``tom_hermes`` also registers a ``DataService`` subclass — ``tom_hermes.dataservices.hermes.HermesDataService`` —
+so users can query HERMES from the Data Services nav-bar entry. The DataService queries the LCO-maintained
+``/api/v0/query`` wrapper, which proxies to the SCIMMA archive. Selecting rows from the query results and
+clicking "Create Targets" runs through the same ``ingest_hermes_alert`` function that the live Hopskotch
+stream handler calls. So, archive-ingest (DataService) and stream-ingest (alertstreams) produce identical
+TOM database entries for the same message.
+
+
+Migration summary
+#################
+
+For TOMToolkit v3, all HERMES interfaces have been refactor from ``tom_base`` to ``tom_hermes``.
+When upgrading from TOMToolkit v2, the following paths must be updated:
+j
+* ``ALERT_STREAMS`` ``TOPIC_HANDLERS`` dotted paths:
+  ``tom_dataproducts.alertstreams.hermes_ingester.hermes_alert_handler`` →
+  ``tom_hermes.alertstreams.ingester.hermes_alert_handler``.
+* ``DATA_SHARING['hermes']['DATA_CONVERTER_CLASS']`` dotted path:
+  ``tom_dataproducts.alertstreams.hermes_publisher.HermesDataConverter`` →
+  ``tom_hermes.sharing.HermesDataConverter``.
+* Direct imports: any code importing ``publish_to_hermes``, ``preload_to_hermes``,
+  ``BuildHermesMessage``, ``HermesDataConverter``, ``HermesMessageException``,
+  ``create_hermes_alert``, ``get_hermes_data_converter_class``, or ``get_hermes_topics``
+  from ``tom_dataproducts.alertstreams.hermes_publisher`` must now import from
+  ``tom_hermes.sharing``.
+* Direct imports: any code importing ``hermes_alert_handler``, ``ingest_hermes_alert``,
+  ``get_hermes_phot_value``, ``create_new_hermes_target``,
+  ``get_or_create_uuid_from_metadata``, or ``HERMES_SPECTROSCOPY_FILE_EXTENSIONS``
+  from ``tom_dataproducts.alertstreams.hermes_ingester`` must now import from
+  ``tom_hermes.alertstreams.ingester``.
+* The ``share_data_with_hermes`` / ``share_target_list_with_hermes`` / ``share_data_with_tom``
+  functions in ``tom_dataproducts.sharing`` are removed. Replace call sites with
+  ``tom_common.sharing.get_sharing_backend(name)().share(...)``.
+* The HERMES "preload" views (the ``Open in Hermes 🗗`` button) have moved from
+  ``tom_targets`` to ``tom_hermes``. URL names change accordingly: any custom
+  template referencing ``{% url 'tom_targets:hermes-preload' %}`` /
+  ``{% url 'targets:hermes-preload' %}`` must use
+  ``{% url 'tom_hermes:target-preload' %}``, and ``targets:group-hermes-preload``
+  becomes ``tom_hermes:target-grouping-preload``. The URL paths themselves
+  also change (``/targets/<pk>/hermes-preload/`` → ``/hermes/targets/<pk>/preload/``),
+  but unless a TOM operator hardcoded those paths instead of using ``{% url %}``,
+  no further action is required.