rtia: reshape iot_data to Telegraf line-protocol form#1
Open
srmadscience wants to merge 2 commits into
Open
Conversation
Restructure rtia.iot_data from flat columns to the Telegraf outputs.cratedb shape (hash_id / timestamp / name / tags / fields), so the same table can be loaded by COPY FROM or by Telegraf's crate/postgres plugin. That plugin can't write a GEO_POINT, so the geo coords ride along as doubles in fields (geo_lon/geo_lat) and geo_location is GENERATED from them; day is GENERATED from timestamp. - sql/rtia_schema_create.sql: new iot_data DDL (tags/fields OBJECTs, generated day + geo_location, PK (hash_id, timestamp, day), partitioned by day). - sql/rtia_first_queries.sql, sql/rtia_advanced_queries.sql: rewrite iot_data column refs to tags['...'] / fields['...']; plants/devices/ maintenance_log columns unchanged. - grafana/rtia.json: remap 22 panel queries to the bracket-notation access pattern. - README.md: document the line-protocol shape and dual ingestion path. The 500k-row iot_demo_dataset.json (240 MB) is not committed; the canonical copy lives on S3 where the COPY FROM statements read it. Verified end-to-end against a CrateDB 6.3.2 cluster: drop + reload of all 6 rtia tables, and all rtia dashboard/SQL queries execute clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
iot_demo_dataset.json is loaded by COPY FROM from its canonical S3 copy, so it doesn't belong in git. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Restructures
rtia.iot_datafrom flat columns to the Telegrafoutputs.cratedbline-protocol shape (hash_id/timestamp/name/tags/fields), so the same table can be populated either byCOPY FROMor by Telegraf's crate/postgres output plugin. That plugin can't write aGEO_POINT, so the geo coordinates ride along as plain doubles infields(geo_lon/geo_lat) andgeo_locationis aGEO_POINTGENERATEDfrom them;dayisGENERATEDfromtimestamp.Changes
sql/rtia_schema_create.sql— newiot_dataDDL:tags/fieldsOBJECTs, generatedday+geo_location, PK(hash_id, "timestamp", day), partitioned byday.sql/rtia_first_queries.sql,sql/rtia_advanced_queries.sql— rewriteiot_datacolumn refs totags['…']/fields['…'].plants/devices/maintenance_logcolumns are unchanged.grafana/rtia.json— remap 22 panel queries to the bracket-notation access pattern (string-level edit; rest of the file byte-identical).README.md— document the line-protocol shape and the dualCOPY FROM/ Telegraf ingestion path..gitignore— exclude the 500k-rowiot_demo_dataset.json(240 MB); its canonical copy lives on S3 where theCOPY FROMstatements read it.Data
The rewritten
iot_demo_dataset.jsonis not in this PR (240 MB). The matching new-shape file is already published to the S3 bucket the schema script loads from, soCOPY FROMworks as-is.Verification
Validated end-to-end against a CrateDB 6.3.2 cluster:
rtiatables;iot_dataloaded 500,000 rows, withgeo_location/daycorrectly materialized fromfields.rtiadashboard queries (Grafana macros substituted) — zero errors. This surfaced and fixed anORDER BYalias-reference regression in two device-type panels.tags['…']/fields['…']access,tags['metadata_*'], thei.tags['plant_id']join,WITHIN(geo_location, …)).🤖 Generated with Claude Code