YAML and Python script for the AHI CSR BUFR by PraveenKumar-NOAA · Pull Request #45 · NOAA-EMC/spoc

PraveenKumar-NOAA · 2025-05-29T19:41:26Z

This PR introduces the following files for the AHI CSR BUFR converter:

~/dump/mapping/bufr_ahicsr.yaml
~/dump/mapping/bufr_ahicsr.py
~/ush/test/config/bufr_bufr4backend_ahicsr.yaml
~/ush/test/config/bufr_script4backend_ahicsr.yaml

Testing and Validation

All four workflows (bufr2netcdf, bufr4backend, script2netcdf, and script4backend) were executed and tested using serial execution.
Validation tests were completed for the following variables:

-- brightnessTemperature
-- cloudAmount

The validation was performed by comparing the input BUFR files and the corresponding output IODA files. Additional details can be found in #41.

Python Coding Standards

The spoc_python_coding_norms test was run to check compliance with Pycodestyle standards.
Result: All checks passed successfully (100% pass, 0 failures).

Variable Naming Validation

The variable naming convention test is currently failing due to the absence of the following variables in the naming convention table. To resolve this, these variables need to be added to the table.

Variable 'MetaData/numberPixelVertical'
Variable 'MetaData/sensorBandWidth'
Variable 'MetaData/reportIndicator'
Variable 'MetaData/radianceComputationalMethod'
Variable 'MetaData/satelliteInstrumentData'
Variable 'MetaData/numberPixelHorizontal'
Variable 'MetaData/radianceType'
Variable 'MetaData/receiptTimeSignificance'
Variable 'ObsValue/clearSkyStandardDeviation'_

All these variables have been removed from the YAML file, except clearSkyStandardDeviation, which still needs to be added to the ObsSpace.yaml file.

A discussion will be scheduled with users and experts to determine if these variables should be included.

…ow pycodestye (#30) **This PR updates two things:** - Update mapping file and Python script for IASI, CrIS , ATMS Update the following to IASI:** 1. Rename mtiasi to iasi 2. Change logging to self.log.warning 3. Add map_path function Add new data type - cris-fsr** Update the following for ATMS:** 1. Change logging to self.log.warning 2. Add map_path function - Add Python coding norm check (pycodestyle) --- this is an initial implementation so we have a coding norm check for Python script. This will be updated to use pytest in another PR To use it: - Load obsforge modules (~eliu/modules/env_obsforge.sh) - go to ./sorc/spoc/build - ctest -VV -R spoc_python_coding_norms - Modify all Python scripts to follow pycodestyle (ctest to be added in another PR soon) (No change in science)

This PR addes two files for ASCAT - mapping YAML - Python script --- pass Python coding norm check Notes: - This PR should be tested with [bufr-query PR #73](NOAA-EMC/bufr-query#73) - A new PR will be opened to modify the standalone AMV converters accordingly after this PR is merged. ASCAT data processing has the following features: - Convert wind speed and direction to u and v wind components - Add obs type (290) Documentation is in Sphinx style Test: script2netcdf validated Validation plots IODA windEastward ![gdas_ObsValue_windEastward_satwnd_ascat_metop-b](https://github.com/user-attachments/assets/3d71ed27-1d94-4df6-a3e1-a230dca6a9d1) GSI windEastward ![gdas_ObsValue_windEastward_ascatw_ascat_metop-b](https://github.com/user-attachments/assets/35473373-4236-4031-985b-0c118529315c) IODA windNorthward ![gdas_ObsValue_windNorthward_satwnd_ascat_metop-b](https://github.com/user-attachments/assets/a753cdfe-2bac-461f-a418-078ac0ceaa41) GSI windNorthward ![gdas_ObsValue_windNorthward_ascatw_ascat_metop-b](https://github.com/user-attachments/assets/5e49b097-007a-46d7-a9c2-0c13274fd819) --------- Co-authored-by: Emily Liu <eliu@hercules-login-4.hpc.msstate.edu> Co-authored-by: Emily Liu <eliu@hercules-login-3.hpc.msstate.edu>

This PR adds two files for SSMIS - mapping YAML - Python script --- pass Python coding norm test Notes: - This PR should be tested with [bufr-query PR #74](NOAA-EMC/bufr-query#74) SSMIS data processing has the following features: - spatial averaging (this is implemented in bufr-query variable transform) and it can be used by activated the feature from YAML - Add new variables via the obs builder Python API - satellite ascending/descending orbit flag (1 for ascending and -1 for descending) - solar zenith and azimuth angles (use pysolar) Notes: - Documentation is in Sphnix style (following discussion in Issue #35 ) - SSMIS data conversion includes spatial averaging. Therefore the processing can only run with one MPI task. - The solar angles calculation is done using pysolar. The functions used do not support vectors. Therefore, the calculation of solar angles are implemented with multiprocessing (e.g., one MPI task and spawned with 12 CPUs) ``` srun -n 1 --cpus_per_tasks=12 python bufr_ssmis.py ``` or ``` export CPUS_PER_TASKS=12 python bufr_ssmis.py`` ``` Test: script2netcdf validated Plots for output netCDF from SSMIS converter ![gdas_solarZenithAngle_ssmis_f17_channel_4](https://github.com/user-attachments/assets/8a90a7c6-9f69-435a-8d15-46b3aa6e698d) ![gdas_satelliteAscendingFlag_ssmis_f17_channel_4](https://github.com/user-attachments/assets/c583697f-7ce7-435c-969b-1ef953fcd090) ![gdas_ObsValue_ssmis_f17_channel_4](https://github.com/user-attachments/assets/74e80396-4865-4ffa-a18f-3bf822ff98d9) --------- Co-authored-by: Emily Liu <eliu@hercules-login-3.hpc.msstate.edu> Co-authored-by: Emily Liu <eliu@hercules-login-4.hpc.msstate.edu>

nicholasesposito · 2025-05-30T15:32:34Z

+
+    - name: "source"
+      type: string
+      value: "U.S. National Weather Service, National Centres for Environmental Prediction (NCEP)"


"Centers" not Centres

@nicholasesposito thanks, fixed.

nicholasesposito · 2025-05-30T15:32:52Z

+
+    - name: "providerFullName"
+      type: string
+      value: "Japan Meteorological Agencyi (JMA)"


@nicholasesposito thanks, fixed.

nicholasesposito · 2025-05-30T15:34:19Z

+        category: ["h8"]
+        cache categories:            # optional
+          - ["h8"]
+          - ["h9"]


Am I correct in assuming we're not doing h9?

@nicholasesposito thanks for pointing this. We need both, h8 and h9. The backend YAMls were updated for this change.

### This PR updates the YAML and Python scripts for SSMIS ATMS, IASI and CrIS For all four sensors: 1. Simplify the handling of missing categories using the `add_dummy_variable` function 2. Use `map_path` from import.obs_builder for ATMS, CrIS, and IASI 3. Remove "type: netcdf" from the encoder section in the YAML 4. Updates `globals` in YAML 5. The output files were validated against the IODA validator with ObsSpace.yaml; new variables were added to obsSpace.yaml. This validator checks if the variables in the file follow IODA convention in terms of name, unit, and dimension. Here is an example of the output from running the ioda validator for SSMIS output ``` Reading YAML from ../obsForge/sorc/ioda/share/ioda/yaml/validation/ObsSpace.yaml Processing data file: ./testoutput/2024021900/script2netcdf/gdas.t00z.ssmis_f17.tm00.nc Verifying that all required groups exist Verifying group MetaData Verifying group ObsValue Verifying group / Verifying dimension names Verifying variable information Variable MetaData/earthSurfaceType Variable MetaData/sensorZenithAngle Variable MetaData/satelliteIdentifier Variable MetaData/sensorScanPosition Variable MetaData/longitude Variable MetaData/latitude Variable MetaData/sensorViewAngle Variable MetaData/scanLineNumber Variable MetaData/solarZenithAngle Variable MetaData/sensorChannelNumber Variable MetaData/satelliteAscendingFlag Variable MetaData/sensorAzimuthAngle Variable MetaData/rainFlag Variable MetaData/dateTime Variable MetaData/solarAzimuthAngle Variable ObsValue/brightnessTemperature Final results: # errors: 0 # warnings: 0 ``` Notes: - SSMIS specific - ssmis conversion uses updated `comupte_solar_angles` which is under review in [bufr-query PR #96](NOAA-EMC/bufr-query#96) - The update of ObsSpace.yaml from IODA is ongoing. We will add new variables in the [ObsSpace.yaml ](https://github.com/JCSDA-internal/ioda/blob/feature/ObsSpace_Validator/share/ioda/yaml/validation/ObsSpace.yaml)in IODA branch feature/ObsSpace_Validator Specific to IASI and CrIS Modifications made to cloud related variables - `cloudHeight`, `cloudCoverTotal`, `fractionOfClearInFOV`: - change group from `MetaData` to `ObsValue` - change variable type from `int` to `float` - scale the variables so they range between 0 and 1 - change `ObsValue/radiance` to `ObsValue/spectralRadiance` and unit accordingly The above changes are to following IODA convention. These changes do not alter the results (scientific) --------- Co-authored-by: Emily Liu <eliu@hercules-login-3.hpc.msstate.edu> Co-authored-by: Emily Liu <eliu@hercules-login-4.hpc.msstate.edu>

emilyhcliu · 2025-05-31T05:25:20Z

@PraveenKumar-NOAA
I tested the ahicsr converter.
The channel dimension is missing in the IODA file (output)

netcdf gdas.t00z.ahi_h9.tm00 {
dimensions:
        Location = 317847 ;
variables:
        int Location(Location) ;
                Location:_FillValue = 2147483647 ;

This is because your YAML file does not include the dimension section in the encoder section.

Here is the example from ATMS:

spoc/dump/mapping/bufr_atms.yaml

Lines 75 to 80 in 369e647

    
           encoder: 
        
             dimensions: 
        
               - name: Channel 
        
                 source: variables/sensorChannelNumber 
        
                 path: "*/ATMSCH"

emilyhcliu · 2025-05-31T05:26:37Z

+encoder:
+


You need to include dimension section

Here is the example from ATMS:

spoc/dump/mapping/bufr_atms.yaml

Lines 75 to 80 in 369e647

encoder:

dimensions:

- name: Channel

source: variables/sensorChannelNumber

path: "*/ATMSCH"

@emilyhcliu thanks, done.

…hannelNumber variable

PraveenKumar-NOAA · 2025-06-11T17:44:52Z

@PraveenKumar-NOAA I tested the ahicsr converter. The channel dimension is missing in the IODA file (output)
netcdf gdas.t00z.ahi_h9.tm00 {
dimensions:
        Location = 317847 ;
variables:
        int Location(Location) ;
                Location:_FillValue = 2147483647 ;
This is because your YAML file does not include the dimension section in the encoder section.

Here is the example from ATMS:

spoc/dump/mapping/bufr_atms.yaml

Lines 75 to 80 in 369e647

encoder:

dimensions:

- name: Channel

source: variables/sensorChannelNumber

path: "*/ATMSCH"

@emilyhcliu thanks. A dimension section is added in the encoder section for the Channel dimension, there are 12 Channels.

… from YAML

PraveenKumar-NOAA · 2025-08-28T13:38:35Z

@emilyhcliu A brightness temperature plot has been created for the HIMAWARI-8 CLEAR SKY RADIANCES channel and added to the following issue: #41. This completes the validation for this converter, as the test input BUFR file contains only HIMAWARI-8 data.

…m Praveen.Kumar (#101) This PR have 2 updates. 1. Enable extraction of redistribution‑restriction metadata by adding restrictionFlag (RSRD) and restrictionExpiration (EXPRSRD) to the BUFR→IODA conversion. This ensures downstream restriction‑filter logic has the required fields available in MetaData. 2. Add modified IODA-Converters that originally from Praveen in PRs (#45, #55, #61, #62, #69). --------- Co-authored-by: hyundeok choi <hyundeok.choi@clogin09.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: hyundeok choi <hyundeok.choi@clogin04.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: hyundeok choi <hyundeok.choi@clogin02.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: hyundeok choi <hyundeok.choi@clogin06.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: hyundeok choi <hyundeok.choi@clogin08.cactus.wcoss2.ncep.noaa.gov>

rmclaren and others added 25 commits February 25, 2025 09:40

got it working with obs_builder. trying to find additional improvements

ce7764a

enhanced logging

d13e328

more ObsBuilder enhancements

a54fed2

reorganized a little

7e5ff98

Merge branch 'develop' into feature/obs_builder

8e3a565

updated satwnd amv files to latest ObsBuilder spec

5be38e5

made way to get map path

80f02b9

updated and fixed up avhrr to actually run

8aa1576

initial implementation of prepbufr

584e2ed

cleaned up some file names

5b81d5e

more updates

c689f3a

improved get ref time

c2fa886

improved get ref time

2e12fbd

improved get ref time

c6d5468

improved get ref time

1eb1aa2

improved get ref time

6886db3

improved get ref time

61b4341

removed obsolete flags from add_main_functions

b23ebab

Merge branch 'develop' into feature/obs_builder

a05492b

YAML and Python script for AHI-CSR

6a638ec

Files for the bufr and script backends

76c86bb

Removed a commented line from bufr_script4backend_ahicsr.yaml file

36b7558

PraveenKumar-NOAA requested review from emilyhcliu, nicholasesposito, rmclaren and xincjin-NOAA May 29, 2025 19:41

xincjin-NOAA reviewed May 29, 2025

View reviewed changes

Comment thread dump/mapping/bufr_ahicsr.py

nicholasesposito reviewed May 30, 2025

View reviewed changes

Praveen.Kumar and others added 5 commits May 30, 2025 16:37

Updated backend YAMLs for category h9

1cb9d7c

Fixes spelling in bufr_ahicsr.yaml

341dcbb

Merge branch 'develop' into feature/obs_builder

369e647

Merge branch 'feature/obs_builder' into feature/obs_builder_ahicsr

1a5e42f

emilyhcliu requested changes May 31, 2025

View reviewed changes

Praveen.Kumar added 3 commits June 2, 2025 15:02

Delete bufr_adpupa_prepbufr.py and bufr_adpupa_prepbufr_mapping.yaml

2745248

Delete bufr_adpupa_prepbufr.py and bufr_adpupa_prepbufr_mapping.yaml

055ce0a

Updated YAML and Python scripts for the Channel dimension and sensorC…

66c311e

…hannelNumber variable

Praveen.Kumar added 2 commits June 13, 2025 14:01

Updated Channel Numbers from 7 to 16 and removed some extra variables…

0055367

… from YAML

Removed white spaces from the Python script

32fd676

Base automatically changed from feature/obs_builder to develop September 18, 2025 21:29

HyundeokChoi-NOAA mentioned this pull request Mar 24, 2026

add restriction variables and modified ioda-converters originally from Praveen.Kumar #101

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YAML and Python script for the AHI CSR BUFR#45

YAML and Python script for the AHI CSR BUFR#45
PraveenKumar-NOAA wants to merge 35 commits intodevelopfrom
feature/obs_builder_ahicsr

PraveenKumar-NOAA commented May 29, 2025 •

edited

Loading

Uh oh!

Uh oh!

nicholasesposito May 30, 2025

Uh oh!

PraveenKumar-NOAA May 30, 2025

Uh oh!

nicholasesposito May 30, 2025

Uh oh!

PraveenKumar-NOAA May 30, 2025

Uh oh!

nicholasesposito May 30, 2025

Uh oh!

PraveenKumar-NOAA May 30, 2025

Uh oh!

emilyhcliu commented May 31, 2025 •

edited

Loading

Uh oh!

emilyhcliu May 31, 2025 •

edited

Loading

Uh oh!

PraveenKumar-NOAA Jun 11, 2025

Uh oh!

PraveenKumar-NOAA commented Jun 11, 2025

Uh oh!

PraveenKumar-NOAA commented Aug 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	encoder:
	dimensions:
	- name: Channel
	source: variables/sensorChannelNumber
	path: "*/ATMSCH"

Conversation

PraveenKumar-NOAA commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

nicholasesposito May 30, 2025

Choose a reason for hiding this comment

Uh oh!

PraveenKumar-NOAA May 30, 2025

Choose a reason for hiding this comment

Uh oh!

nicholasesposito May 30, 2025

Choose a reason for hiding this comment

Uh oh!

PraveenKumar-NOAA May 30, 2025

Choose a reason for hiding this comment

Uh oh!

nicholasesposito May 30, 2025

Choose a reason for hiding this comment

Uh oh!

PraveenKumar-NOAA May 30, 2025

Choose a reason for hiding this comment

Uh oh!

emilyhcliu commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emilyhcliu May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PraveenKumar-NOAA Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

PraveenKumar-NOAA commented Jun 11, 2025

Uh oh!

PraveenKumar-NOAA commented Aug 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

PraveenKumar-NOAA commented May 29, 2025 •

edited

Loading

emilyhcliu commented May 31, 2025 •

edited

Loading

emilyhcliu May 31, 2025 •

edited

Loading