METstat2json

A Go library to parse MET (Model Evaluation Tools) output files into JSON format, making weather model evaluation data easily consumable by downstream applications.

Overview

METstat2json parses MET "stat" output files into standardized JSON documents. It supports MET versions >10.0 by using autogenerated type definitions derived directly from MET source files.

The library intelligently merges data from different MET stat file lines with similar metadata, creating consolidated documents that can be stored in databases like Couchbase for visualization and analysis tools.

Getting Started

Quick Start

# Convert a MET stats file to JSON
go run examples/sample_parser -path /path/to/met/files -dataset mymodel -outdir /tmp

# View the output
gunzip -c /tmp/mymodel.json.gz | jq .

For Library Users

If you're a Go developer - METstat2json can be integrated into your data processing pipeline to convert MET output files into JSON for databases or analysis.

The entrypoint for the library is intended to be parser and should be the only import required.

import (
    "github.com/dtcenter/METstat2json/pkg/parser"
)

func main() {
    var doc map[string]interface{}

    // You will most likely want to iterate over the lines in a MET stat file and pass them to ParseLine
    // However, as an example, parse a single line from a MET output file
    headerLine := "VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG..."
    dataLine := "V12.0.0 FCST NA 120000 20120409_120000..."
    fileName := "grid_stat_GFS_TMP.stat" // Used to detect what type of MET output we have
    datasetName := "mymodel"

    // Parse the line and update the document
    doc, err := parser.ParseLine(datasetName, headerLine, dataLine, &doc, fileName, getExternalDocForId)
    if err != nil {
        // Handle error
    }

    // Write the document to a compressed JSON file
    err = parser.WriteJsonToCompressedFile(doc, "/tmp/output.json.gz")
    if err != nil {
        // Handle error
    }
}

// Define a function to retrieve external documents (required by the parser)
func getExternalDocForId(id string) (map[string]interface{}, error) {
    // Implement to retrieve documents from your database or return not found
    return nil, fmt.Errorf("%s: %s", parser.DOC_NOT_FOUND, id)
}

For Library Developers

If you're working on METstat2json itself, you'll need to understand how the code generation works and how to test your changes.

Code Generation

The library uses MET's own schema definitions to autogenerate Go types:

go run ./generator -version=v12.0 > pkg/linetypes/v12_0/linetypes.go && golangci-lint fmt ./...

Note that library generation should be idempotent. Rerunning the generator multiple times for the same MET version should result in the same metLineTypeDefinition file.

Running Tests

Sample MET stat file output for this program is in a separate Git repo at: https://github.com/NOAA-GSL/MET-parser-testdata. It's large, so the test suite automatically downloads it to /tmp/testdata.

# Test the parser on sample data
go test -v ./...

# Benchmarking
cd pkg/parser
go test -bench . -benchmem

Key Packages

• generator: Generates code from MET schema definitions
• pkg/parser: Main parsing logic and entry point
• pgk/linetypes/*: Auto-generated type definitions for each MET version
• pkg/util: Common utilities used across the codebase