Elasticsearch output plugin - Errors on _bulk api post

[fredx30]

Environment

• How did you install GRR? [e.g. from release DEB, HEAD DEB, PIP package, source]
  Dockerfile - https://hub.docker.com/layers/grr/grrdocker/grr/v3.4.6.0/images/sha256-30f404087124417eb96420ad157c5bac403748c55eafabf1e0abfbf7c20bb0c9?context=explore

• What GRR version are you running?: [e.g. 3.1.2.3]
  3.4.6.0

• What operating system does the GRR server run on? [e.g. Ubuntu 18.04]
  Ubuntu 18.04.
  This is running in aks (Azure kubernetes service) currently using image-name: "AKSUbuntu-1804gen2containerd-2022.06.08"

• What operating system does the affected GRR client run on, if applicable? [e.g. Windows 10]
  Windows server 2022 (Not affected)

• Elastic cluster version (Likely of relevance here)
  Im using a single node elasticsearch cluster deployed as image (container) )version 7.17.0 deployed with the elastic helmchart elastic/elasticsearch.

Describe the issue
The outputplugin elasticsearch throws errors on usage. In this case im creating a flow to request the netstat listenting ports/processes on a windows based client. API request for the flow looks like the following (Where "grr-server-host" is a dummy replacement for a valid url).

CSRFTOKEN=`curl https://grr-server-host -o /dev/null -s -c - | grep csrftoken  | cut -f 7`; \
	curl -X POST -H "Content-Type: application/json" -H "X-CSRFToken: $CSRFTOKEN" \
	https://grr-server-host/api/v2/clients/C.7e53019687aed8b7/flows -d @- << EOF
{
  "flow": {
    "args": {
      "@type": "type.googleapis.com/grr.NetstatArgs",
      "listeningOnly": true
    },
    "name": "Netstat",
    "runnerArgs": {
      "outputPlugins": [
        {
          "pluginName": "ElasticsearchOutputPlugin",
          "pluginArgs": "CgduZXRzdGF0CgNncnI="
        }
      ]
    }
  }
}
EOF

Without the outputplugin it all works fine.

Its when i add the outputplugin i start getting issues in the grr-worker. This is shown in the grr-worker.log. This is visible both within the pod/container running grr_worker and the admin_ui when viewing a specific flow under the navbar item "managed launched flows" -> "results" -> "1 errors" (clicking this opens a modal view of an errors list with the errors below).

!OBS See Debugging process blow.

• "Error while processing 23 replies: 406 Client Error: Not Acceptable for url: https://grr-server-host:9200/_bulk"
  Upon inspection this appears to be due to a default of a "content type" : "url encode" this is resolved by adding contenttype headers for application/json.
• "Error while processing 23 replies: 400 Client Error: Bad Request for url: https://grr-server-host:9200/_bulk"
  Upon inspection this is due to a missing new line at the end of the json request.

Error logs

From grr-worker.log

(grr-server) root@grr-6c6c959d75-vs9w9:/usr/share/grr-server# grr_worker --verbosity 1 --alsologtostderr
I0712 14:20:16.563979 139940397143872 config_lib.py:870] Loading configuration from /usr/share/grr-server/install_data/etc/grr-server.yaml
I0712 14:20:16.601257 139940397143872 config_lib.py:404] Applying filter resource for install_data/etc@grr-response-core.
I0712 14:20:16.602211 139940397143872 config_lib.py:870] Loading configuration from /usr/share/grr-server/install_data/etc/server.local.yaml
I0712 14:20:16.623253 139940397143872 config_lib.py:576] Configuration writeback is set to /usr/share/grr-server/install_data/etc/server.local.yaml
I0712 14:20:16.624943 139940397143872 server_logging.py:205] Initializing Logging subsystem.
I0712 14:20:16.625142 139940397143872 server_logging.py:158] Will use logging engines ['stderr', 'file']
I0712 14:20:16.625748 139940397143872 config_lib.py:404] Applying filter resource for grr_response_core@grr-response-core.
I0712 14:20:16.626402 139940397143872 server_logging.py:187] Writing log file to /usr/share/grr-server/lib/python3.6/site-packages/grr_response_core/var/log//grr-worker.log
ERROR:2022-07-12 12:55:05,629 386 MainProcess 140000134067968 flow_processing_pool-252777379 flow_base:920] Plugin ElasticsearchOutputPlugin <<ElasticsearchOutputPluginArgs("message ElasticsearchOutputPluginArgs {\n tags : [\n   'netstat'\n   'grr'\n  ]\n}")>> failed to process 23 replies.
Traceback (most recent call last):
  File "/usr/share/grr-server/lib/python3.6/site-packages/grr_response_server/flow_base.py", line 900, in _ProcessRepliesWithFlowOutputPlugins
    [r.AsLegacyGrrMessage() for r in replies])
  File "/usr/share/grr-server/lib/python3.6/site-packages/grr_response_server/output_plugins/elasticsearch_plugin.py", line 89, in ProcessResponses
    self._SendEvents(events)
  File "/usr/share/grr-server/lib/python3.6/site-packages/grr_response_server/output_plugins/elasticsearch_plugin.py", line 156, in _SendEvents
    response.raise_for_status()
  File "/usr/share/grr-server/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://grr-server-host:9200/_bulk

Additional context
Debugging Process
In my case while testing and developing a fix for this i was using a small test.jsonl file generated with real data extracted by adding an exception to the plugin to dump the expected body of the request. I added the line raise Exception("CheckJsonFormat: ---%s",data) at lin 150 in the plugin. https://github.com/google/grr/blob/master/grr/server/grr_response_server/output_plugins/elasticsearch_plugin.py#L150

Here is my output file:

{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 445},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 5985},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 47001},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "192.168.1.5","port": 139},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 5985},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 445},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 47001},"state": "LISTEN","pid": 4,"processName": "System"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 49666},"state": "LISTEN","pid": 420,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 49666},"state": "LISTEN","pid": 420,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 49665},"state": "LISTEN","pid": 568,"processName": "wininit.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 49665},"state": "LISTEN","pid": 568,"processName": "wininit.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 49669},"state": "LISTEN","pid": 640,"processName": "services.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 49669},"state": "LISTEN","pid": 640,"processName": "services.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 49664},"state": "LISTEN","pid": 648,"processName": "lsass.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 49664},"state": "LISTEN","pid": 648,"processName": "lsass.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 3389},"state": "LISTEN","pid": 708,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 3389},"state": "LISTEN","pid": 708,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 135},"state": "LISTEN","pid": 836,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 135},"state": "LISTEN","pid": 836,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 49667},"state": "LISTEN","pid": 1016,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 49667},"state": "LISTEN","pid": 1016,"processName": "svchost.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET6_WIN","type": "SOCK_STREAM","localAddress": {"ip": "::","port": 49668},"state": "LISTEN","pid": 1848,"processName": "spoolsv.exe"},"tags": ["netstat","grr"]}\n{"index": {"_index": "grr-flows"}}\n{"client": {"clientUrn": "aff4:/C.7e53019687aed8b7","hostname": "vm-1","os": "Windows","clientAge": "1657194772979139","uname": "Windows-10-10.0.20348","osRelease": "10","osVersion": "10.0.20348","usernames": "user1","macAddress": "0022489e210b","labels": "","hardwareInfo": {"serialNumber": "0000-0006-8365-0908-2667-6664-92","systemManufacturer": "Microsoft Corporation"},"kernelVersion": "10.0.20348","systemLabels": "","userLabels": ""},"flow": {"urn": "aff4:/C.7e53019687aed8b7/73F4A7462EE70DE6","name": "Netstat","args": {"@type": "type.googleapis.com/grr.NetstatArgs","listeningOnly": true},"runnerArgs": {"notifyToUser": true,"clientId": "aff4:/C.7e53019687aed8b7","flowName": "Netstat","outputPlugins": [{"pluginName": "ElasticsearchOutputPlugin","pluginArgs": "CgduZXRzdGF0CgNncnI="}]},"state": "RUNNING","startedAt": "1657284503042716","lastActiveAt": "1657284511176336","creator": "admin","flowId": "73F4A7462EE70DE6","originalFlow": {"flowId": "722094332E37B1BB","clientId": "C.7e53019687aed8b7"},"clientId": "C.7e53019687aed8b7"},"resultType": "NetworkConnection","result": {"family": "INET","type": "SOCK_STREAM","localAddress": {"ip": "0.0.0.0","port": 49668},"state": "LISTEN","pid": 1848,"processName": "spoolsv.exe"},"tags": ["netstat","grr"]}

Then using this file with curl to attempt the connections myself...

1. port-forward the elasticsearch server via kubectl
2. replicate the request in the form used by the elasticsearch plugin
3. get more human friendly errors.

The first curl request looks as follows, where test.jsonl is the json extracted from the request as mentioned above.

curl -H "Authorization: Basic <auth-token-here>" "https://localhost:9200/_bulk" -k -XPOST --data-binary "@test.jsonl"

This results in the error {"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
So to fix this we add content type to the request and try again.

curl -H "Authorization: Basic <auth-token-here>" "https://localhost:9200/_bulk" -k -XPOST --data-binary "@test.jsonl" -H "Content-Type: application/json"

This results in the error {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\\n]"},"status":400}.

To fix this we add a \n at the end of the json body.

[fredx30]

The following code changes work in my environment. I belive the potential danger with this change is reverse and forward compatability with different versions of the elasticsearch api.

master...fredx30:grr:update-elasticsearch-output-plugin

Currently i am using these changes in a fairly volatile way where i have to reimplement them on container restart. Is there any way of getting these into a mainline release? Am i better off building my own releases to a private container registry of my choosing?

Edit: Is a PR here with these changes welcome?

[micrictor]

Original dev of the Elasticsearch output plugin here.

I looked back at docs for old versions, I don't think there's a regression risk as all versions that still have live docs include that header value.

I submitted my changes via PR, so that seems like a decent route. If you want, I can incorporate your changes and add the requisite tests to ensure that my (broken ):) code would have failed.

[max-vogler]

Thanks for reporting and debugging this issue! Please send a pull request with your fix.

[fredx30]

@micrictor Im not sure what these tests would entail. I see no apparent variable to test for in this change.

I could see a test for verifying the headers are in place, however if this worked in previous version without the headers then i see little reason for testing for this as the api could just as well change again.

If you have a good idea for a test i would gladly attempt to write it, this whole editing opensource code thing is new and exciting to me i could use the practice getting the PR across the finish line.

[micrictor]

I think you hit the nail on the head - we’d want to have a test for the Content-Type header value being one of the two valid ones for both 7.17 and 8.X, and for the presence of the trailing newline. You’re correct that the API could change again in the future, especially as Elasticsearch major versions change, but I think that having the test is still valuable as it prevents refactors or other changes from accidentally breaking it.

…

On Wed, Jul 13, 2022 at 6:30 AM Fredrik ***@***.***> wrote: @micrictor <https://github.com/micrictor> Im not sure what these tests would entail. I see no apparent variable to test for in this change. I could see a test for verifying the headers are in place, however if this worked in previous version without the headers then i see little reason for testing for this as the api could just as well change again. If you have a good idea for a test i would gladly attempt to write it, this whole editing opensource code thing is new and exciting to me i could use the practice getting the PR across the finish line. — Reply to this email directly, view it on GitHub <#990 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTCAGMMGFH7JXYDFJAWQZDVT3AGBANCNFSM53LU3JDA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[fredx30]

I ran into some issues while learning how to run the tests. After a bit of trying to get pdb to work on the wsl cli I wound up setting up a pycharm project for local debugging in windows. There were a few awkward moments but i think i got the basics working. I see the tests pass in your pipelines, my windows testing needed some patches to work. You can see them here if they intrest you. I think my changes to the elastic output plugin tests should work as its run in the ubuntu pipeline but cant be sure- working on a run of my own now.

Will cherry pick these commits into the pr linked to this issue.

[micrictor]

The changes to the plugin and the test look good so far. I think what's missing is that the test method _ParseEvents tries to parse an empty string - the last line, now that there's a trailing newline.

https://github.com/fredx30/grr/blob/adf66de6daa63855de31aee98c0846ba800de87a/grr/server/grr_response_server/output_plugins/elasticsearch_plugin_test.py#L72

As for running tests - I, personally, would just run the tests for the ES plugin locally, and rely on the CI jobs to run the full test suite.

[fredx30]

Well its passing the pipeline run tests now.
Edit: Dealing with PR comments now.

[micrictor]

@fredx30 do you plan on finalizing your PR, or would you mind if I created one to fix this?

[fredx30]

@micrictor I believe i finished the PR and am waiting on @max-vogler to review the requested changes that have been made.
Based on your answer im suspecting i may have missed something thats a todo on my end. Would you enlighten me?

I have a (i think signed, never tested that part) build going to my public github artifacts that i have been using since i got my the tests and the review finished. It uses a small change to the build pipeline to push to github instead of docker. If am am correct in waiting i hope it can serve as some inspiration for the time being. If not i would love to get cracking on finishing this PR up, such that i too can be on a mainstream release.

[micrictor]

That PR looks good, only thing I can think of is manually marking Max's requested changes as complete may be needed for it to request review again.

Thanks for following up on this, I know it's been a long time.

[fredx30]

What i have done to that end is

1. Request a re-review from max
2. Mark all the requested changes as resolved
3. Quoted/answered the main change request thread

I choose to believe the team is simply buzy or tasked with other things -as there have been no missed releases for this code. I trust max or someone else will get to this eventually. Latest activity is 6 months ago as far as releases are concerned ref: dockerhub.

With that said i dont think it would hurt to keep bumping this issue once every few months to keep it from getting stale on our end.