Skip to content

[FEATURE] Startup Module Validate and Debug Functionality #210

@GregorSchroeder

Description

@GregorSchroeder

Is your feature request related to a problem? Please describe

The current Startup Module currently executes just executes a single SQL statement that both gets data and inserts it into the production database. See sql/insert_mgra.sql. This does not do any validation of negative or NULL values, nor does it validate that the correct number of MGRAs are inserted. It also does not do anything if run in debug mode. Every other module, except for Staging, does some form of validation and also writes csv files locally in debug mode.

Describe the solution you'd like

Perform data validation and write csv files locally if in debug mode.

Describe alternatives you've considered

The [shape] field adds an interesting wrinkle as Python does not easily read in geometry attributes from SQL using Pandas.

Additional context

This issue has been brought to the forefront as @GregorSchroeder encountered a silent error developing #209 wherein 0 records were inserted into the [inputs].[mgra] table but the module executed successfully and the process failed on a subsequent module.

Metadata

Metadata

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions