-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Is your feature request related to a problem? Please describe
The current Startup Module currently executes just executes a single SQL statement that both gets data and inserts it into the production database. See sql/insert_mgra.sql. This does not do any validation of negative or NULL values, nor does it validate that the correct number of MGRAs are inserted. It also does not do anything if run in debug mode. Every other module, except for Staging, does some form of validation and also writes csv files locally in debug mode.
Describe the solution you'd like
Perform data validation and write csv files locally if in debug mode.
Describe alternatives you've considered
The [shape] field adds an interesting wrinkle as Python does not easily read in geometry attributes from SQL using Pandas.
Additional context
This issue has been brought to the forefront as @GregorSchroeder encountered a silent error developing #209 wherein 0 records were inserted into the [inputs].[mgra] table but the module executed successfully and the process failed on a subsequent module.