diff --git a/.gitignore b/.gitignore index 684ad86..7fa3bd1 100644 --- a/.gitignore +++ b/.gitignore @@ -4,4 +4,44 @@ acars_parser *.db *.db-* TODO.md -*.creds \ No newline at end of file +*.creds# Binaries +*.exe +*.dll +*.so +*.dylib + +# Archives +*.zip +*.7z +*.rar +*.tar.gz + +# Build artifacts +*.db +*.sqlite +*.test + +# IDE +.vscode/ +.idea/ +*.swp +*.swo + +# OS +.DS_Store +Thumbs.db + +# Backups +*_backup/ +*.bak + +# Local capture and dump artefacts +gui/Acars/ +gui/HFDL-*.json +gui/HFDL-*.txt +gui/VDL2-*.jsonl +gui/acars-log-*.json +gui/acars-log-*.txt +gui/iataicao.csv +gui/test_*.jsonl +gui/Waypoints.txt diff --git a/QUICKSTART_GUI.md b/QUICKSTART_GUI.md new file mode 100644 index 0000000..d421d3f --- /dev/null +++ b/QUICKSTART_GUI.md @@ -0,0 +1,35 @@ +# Quickstart (Windows) + +## 1) Install Go +Install a recent Go version and ensure `go version` works in PowerShell. + +## 2) Build the parser +Open PowerShell in the project folder and run: + +```powershell +go mod download +go build -o acars_parser.exe .\cmd\acars_parser +``` + +## 3) Run extract (CLI) +```powershell +.\acars_parser.exe extract -input messages.jsonl -output out.json -pretty -all +``` + +## 4) Run GUI wrapper +```powershell +python .\gui\acars_parser_gui.py +``` + +## 5) HTML viewer waypoint lookup + +The standalone viewer [gui/acars_viewer_fast_kv_map_v8.html](gui/acars_viewer_fast_kv_map_v8.html) can resolve named FPN and PWI route waypoints through [gui/Waypoints.txt](gui/Waypoints.txt), so those points can be drawn on the map even when the JSON only contains waypoint names. + +The viewer also supports filtering rows that have a Flight value and sorting the Summary / Details column by the amount of message content, which is useful for surfacing the densest ACARS messages first. + +If the viewer is opened via a local `file:` URL, some browsers block automatic loading of sibling files. In that case, use the `Load Waypoints` control in the viewer and select `gui/Waypoints.txt` manually. + +Notes: +- Input must be JSONL (one JSON object per line) matching either: + - `internal/acars.Message`, or + - NATS wrapper (`internal/acars.NATSWrapper`) with `message{...}` inside. diff --git a/README.md b/README.md index 5588c77..40e9621 100644 --- a/README.md +++ b/README.md @@ -22,8 +22,11 @@ acars_parser/ │ ├── patterns/ # Shared regex patterns and extractors │ └── parsers/ # Individual parser implementations │ ├── adsc/ # ADS-C (B6) + │ ├── abs/ # ABS0 route hints from H1 │ ├── agfsr/ # AGFSR flight status (4T) +│ ├── atncm/ # ATN CM logon route hints from VDL2 │ ├── cpdlc/ # CPDLC FANS-1/A (AA) + │ ├── dis/ # RA DIS OFP info │ ├── eta/ # ETA/timing (5Z) │ ├── fst/ # FST reports (15) │ ├── h1/ # H1 FPN/POS/PWI @@ -40,23 +43,131 @@ acars_parser/ │ ├── labelb2/ # Oceanic clearances (B2) │ ├── labelb3/ # Gate info (B3) │ ├── pdc/ # Pre-departure clearances +│ ├── rep301/ # REP301 compact position reports │ └── sq/ # ARINC position (SQ) └── README.md ``` ## Commands +## GUI Viewer + +The standalone HTML viewer in [gui/acars_viewer_fast_kv_map_v8.html](gui/acars_viewer_fast_kv_map_v8.html) now supports a waypoint lookup table from [gui/Waypoints.txt](gui/Waypoints.txt). When the parsed JSON contains named waypoints without embedded coordinates, the viewer can resolve those names to latitude/longitude and plot them on the Leaflet map for FPN route waypoints and H1 PWI route-wind waypoint data. + +When `Waypoints.txt` contains multiple fixes with the same name in different regions, the viewer now keeps all candidates and chooses the most plausible coordinate using nearby route anchors instead of blindly taking the last duplicate. For H1 PWI route-wind data, repeated fixes across multiple flight levels are also merged into a single map marker popup that lists all available wind and temperature layers for that waypoint. The PWI statistics panel on the map now also includes a flight-level drop-down so the coldest point, warmest point, and strongest wind can be inspected for one specific FL from the currently loaded log. + +When the viewer is served over HTTP from the `gui/` directory, it will try to load `Waypoints.txt` automatically. When the HTML file is opened directly from disk and the browser blocks sibling-file fetches, use the `Load Waypoints` picker in the viewer to load the same file manually. + +The viewer can also load airport names from [internal/airports/airports.json](internal/airports/airports.json). When that lookup is available, the Flight column shows one airport-name line for each ICAO code in the saved route between the route string and the saved-route date. For example, a saved route such as `GUCY-GOBD-OMDB` renders `Conakry`, `Blaise Diagne International`, and `Dubai International` on separate lines underneath the route. Those display labels omit `Od:` and `Do:` prefixes, and they also strip a trailing `Airport` suffix to keep the table narrower. Parsed-only rows do not show airport names until the route has been saved, which keeps unsaved rows visually simpler. When the viewer is served over HTTP, it tries to auto-load that airport file; when the HTML file is opened directly from disk, use the `Load Airports` picker if the browser blocks auto-load in file mode. + +When JSON output is produced through [gui/acars_parser_gui_dnd_fix2.py](gui/acars_parser_gui_dnd_fix2.py), the GUI can also enrich each JSON row with route lookup data from [gui/flightroute.sqb](gui/flightroute.sqb) when that database is present. The `Route lookup from flightroute.sqb` checkbox controls that behaviour, so route enrichment can be turned off when faster JSON generation is preferred. When enabled, the lookup uses the already normalised ICAO-style flight value, queries the `FlightRoute` table by the `flight` column, and stores the matched `route` plus the latest `updatetime` on the JSON row. In the HTML viewer, those fields are now rendered directly underneath the Flight value instead of using a separate Route column. If no lookup row is found, or the database is unavailable, extraction continues and only the Flight value is shown. + +When that GUI runs in merge mode with JSON output enabled, it now reuses the same merged output filename instead of creating incremented names such as `_1`. This matches the single-file extraction path and allows the existing merged JSON to be overwritten directly. + +The standalone HTML viewer can now also write to that same `FlightRoute` table through the local Go `routeapi` bridge. Run `acars_parser routeapi -db gui/flightroute.sqb -port 8765`, then the viewer shows two small buttons underneath each row's `Prikaži na mapi` action when the row has a usable flight key: `Upiši` writes the parsed ICAO route in `XXXX-XXXX` format with the row's log date as `updatetime` in `YYYY-MM-DD`, and `Reload` refreshes the saved route for that flight from the database. That refresh is applied to all currently loaded rows with the same flight key so manually synced routes do not keep showing as mismatches in other message types for the same flight. + +In that same Flight column, the viewer now labels route lines with `parsed` or `saved` badges so it is obvious whether the displayed route comes from the current parsed row or from the `FlightRoute` lookup database. When a row has only a parsed route candidate and nothing saved yet, the viewer shows that parsed ICAO route directly instead of leaving the route area blank. + +The viewer now also ignores numeric-only values when selecting a Flight display value or a route-save flight key. That prevents unrelated payload numbers from being shown in the Flight column or from being used as a lookup key. + +That route-write path now also accepts the Unix-seconds timestamps emitted by `acarsdec`, so ACARS JSON rows with values such as `1777761826.271997` can be written to `flightroute.sqb` the same way as ISO-timestamped VDL2 rows. + +To make route troubleshooting less ambiguous when more than one `flightroute.sqb` exists, the `routeapi` bridge now resolves the database path to an absolute path and exposes it through `/api/flightroute/config`. The HTML viewer shows that exact database path in the `Route API` status line, so it is immediately obvious which SQLite file the `Upiši` and `Reload` buttons are using. + +When that rendered `route` value is in the ICAO `XXXX-XXXX` format and it does not match the best parsed origin/destination pair already present in the JSON, the viewer now highlights that route line in dark red under the Flight value. The viewer also highlights the `updatetime` line in dark red when it begins with `1`, which helps suspicious Unix-style timestamps stand out during review. + +The viewer now always shows a `raw text` details block for non-empty ACARS payload text, including single-line messages. That block no longer relies on any symbol-limit style truncation in the details panel; instead it wraps long lines to the available panel width. + +The main search box in the HTML viewer now also has a `srch txt` checkbox. When it is ticked, the search is limited to the JSON `text` or `message.text` payload fields instead of the broader combined row summary. When it is unticked, search behaves as before and scans the wider rendered row content. + +The search bar also has an optional `CPDLC/ADSC/MIAM` checkbox. When it is ticked, the query is matched against the expanded decoded detail text that the viewer renders for CPDLC (`AA`/`BA`), ADS-C (`A6`/`B6`), and MIAM (`MA`) messages. If that checkbox is the only narrow-search option enabled, the search stays limited to those decoded protocol messages instead of falling back to the broader row summary. + +The main `Summary / Details` column in the HTML viewer also no longer truncates the summary text to a fixed 500-character limit for normal row rendering paths. Long summaries now rely on the existing word wrap in that column instead of being cut off. + ### extract -Extracts structured data from JSONL files containing ACARS messages. +Extracts structured data from JSONL files and JAERO TXT logs containing ACARS messages. ```bash -./acars_parser extract -input messages.jsonl [-output output.json] [-pretty] [-all] +./acars_parser extract -input messages.jsonl [-output output.json] [-pretty] [-all] [-format json|text] ``` +The `extract` command autodetects JSONL and JAERO TXT input. For JAERO logs, the CLI converts each timestamped block into a normal ACARS message, keeps only the raw ACARS payload in `message.text`, preserves legitimate multiline payload text, strips JAERO line-wrap artefacts such as inserted `- #MD` continuations, and skips empty blocks. + +The extractor handles both the original JAERO L-Band log format and the C-Band JAERO format produced by a different decoder. C-Band headers use the same `HH:MM:SS DD-MM-YY UTC AES: GES: ... !