Skip to content

hdfeos5_or_csv_2json_mbtiles.py --num-workers 6 overwrites chunk.json files #9

Open
Open
hdfeos5_or_csv_2json_mbtiles.py --num-workers 6 overwrites chunk.json files#9
@falkamelung

Description

@falkamelung
falkamelung
opened on Mar 24, 2026

Hi @epehlivanli , hdfeos5_or_csv_2json_mbtiles.py does not properly use the specified --num-workers . ingest_insarmaps.bash uses a default of 6. It seems to overwrite and creating only 3 chunk.json files:

ingest_insarmaps.bash TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.csv
sourcing /Users/famelung/code/minsar/minsar/bin/../lib/minsarApp_specifics.sh ...
sourcing /Users/famelung/code/minsar/minsar/bin/../lib/utils.sh ...
sourcing minsar/bin/../lib/utils.sh ...
####################################
20260324:18-51 * ingest_insarmaps.bash TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.csv
####################################
Processing: TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.csv
####################################
Running.... hdfeos5_or_csv_2json_mbtiles.py "TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.csv" "./JSON" --num-workers 6
20260324:18-51 * hdfeos5_or_csv_2json_mbtiles.py "TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.csv" "./JSON" --num-workers 6
[INFO] Detected point_ID column: point_id (non-null=335044)
Using columns: lat = Y, lon = X
[INFO] Set data_type: LOS Displacement
[INFO] slcStack.h5 not found at: /Users/famelung/scratch/inputs/slcStack.h5
data_footprint:  POLYGON((-80.1147308349609 25.7637500762939,-80.1593627929688 25.7637500762939,-80.1593627929688 25.9763336181641,-80.1147308349609 25.9763336181641,-80.1147308349609 25.7637500762939))
 Check: read_from_csv_file output
 - timeseries_datasets keys: ['20170923', '20171004', '20171015'] ...
 - sample slice shape: (578, 580)
 - lat_grid shape: (578, 580)
 - lon_grid shape: (578, 580)
 - Number of dates: 200
 - Attributes: {'PROJECT_NAME': 'CSV_IMPORT', 'WIDTH': '580', 'LENGTH': '578', 'processing_type': 'LOS_TIMESERIES', 'look_direction': 'R', 'collection': 'sarvey', 'data_type': 'LOS Displacement', 'PLATFORM': 'S1', 'MISSION': 'S1', 'mission': 'Tsx', 'relative_orbit': 36, 'beam_mode': 'SM', 'platform': 'S1', 'REF_LAT': 25.849037629616618, 'REF_LON': -80.13136049066759, 'first_date': '20170923', 'last_date': '20251008', 'history': '2026-03-24', 'data_footprint': 'POLYGON((-80.1147308349609 25.7637500762939,-80.1593627929688 25.7637500762939,-80.1593627929688 25.9763336181641,-80.1147308349609 25.9763336181641,-80.1147308349609 25.7637500762939))', 'scene_footprint': 'POLYGON((-80.1147308349609 25.7637500762939,-80.1593627929688 25.7637500762939,-80.1593627929688 25.9763336181641,-80.1147308349609 25.9763336181641,-80.1147308349609 25.7637500762939))', 'atmos_correct_method': None, 'beam_swath': 1, 'post_processing_method': 'MintPy', 'prf': 1717.128973878037, 'processing_software': 'isce', 'wavelength': 0.05546576, 'first_frame': 556, 'last_frame': 557}
columns: 580
rows: 578
converted chunk 0
converted chunk 0
converted chunk 0
converted chunk 0
converted chunk 0
converted chunk 0
converted chunk 1
converted chunk 1
converted chunk 1
converted chunk 1
converted chunk 1
converted chunk 1
converted chunk 2
converted chunk 2
converted chunk 2
converted chunk 2
converted chunk 2
WIDTH                       580
LENGTH                      578
processing_type             LOS_TIMESERIES
look_direction              R
mission                     Tsx
relative_orbit              36
beam_mode                   SM
REF_LAT                     25.849037629616618
REF_LON                     -80.13136049066759
first_date                  20170923
last_date                   20251008
history                     2026-03-24
data_footprint              POLYGON((-80.1147308349609 25.7637500762939,-80.1593627929688 25.7637500762939,-80.1593627929688 25.9763336181641,-80.1147308349609 25.9763336181641,-80.1147308349609 25.7637500762939))
scene_footprint             POLYGON((-80.1147308349609 25.7637500762939,-80.1593627929688 25.7637500762939,-80.1593627929688 25.9763336181641,-80.1147308349609 25.9763336181641,-80.1147308349609 25.7637500762939))
atmos_correct_method        None
beam_swath                  1
post_processing_method      MintPy
prf                         1717.128973878037
processing_software         isce
wavelength                  0.05546576
first_frame                 556
last_frame                  557
Now running tippecanoe with command tippecanoe *.json -P -l chunk_1 -x d -pf -pk -o TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.mbtiles 2> tippecanoe_stderr.log
time elapsed: 63.928025957997306
####################################
Running json_mbtiles2insarmaps.py...
Running.... json_mbtiles2insarmaps.py --num-workers 6 -u "insaradmin" -p "insaradmin" --host "149.165.153.50" -P "insarmaps" -U "insarmaps@insarmaps.com" --json_folder "./JSON" --mbtiles_file "./JSON/TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.mbtiles"
20260324:18-52 * json_mbtiles2insarmaps.py --num-workers 6 -u "insaradmin" -p "insaradmin" --host "149.165.153.50" -P "insarmaps" -U "insarmaps@insarmaps.com" --json_folder "./JSON" --mbtiles_file "./JSON/TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.mbtiles"
Uploading json chunks...
Clearing old dataset, if it is there
Creating index on TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011
Inserted chunk_2.json to db
Inserted chunk_0.json to db
Inserted chunk_1.json to db

in contrast, for --num-workers 1 it works fine:

ingest_insarmaps.bash TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.csv --num-workers 1
sourcing /Users/famelung/code/minsar/minsar/bin/../lib/minsarApp_specifics.sh ...
sourcing /Users/famelung/code/minsar/minsar/bin/../lib/utils.sh ...
sourcing minsar/bin/../lib/utils.sh ...
####################################
20260324:18-55 * ingest_insarmaps.bash TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.csv --num-workers 1
####################################
Processing: TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.csv
####################################
Running.... hdfeos5_or_csv_2json_mbtiles.py "TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.csv" "./JSON" --num-workers 1
20260324:18-55 * hdfeos5_or_csv_2json_mbtiles.py "TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.csv" "./JSON" --num-workers 1
[INFO] Detected point_ID column: point_id (non-null=335044)
Using columns: lat = Y, lon = X
[INFO] Set data_type: LOS Displacement
[INFO] slcStack.h5 not found at: /Users/famelung/scratch/inputs/slcStack.h5
data_footprint:  POLYGON((-80.1147308349609 25.7637500762939,-80.1593627929688 25.7637500762939,-80.1593627929688 25.9763336181641,-80.1147308349609 25.9763336181641,-80.1147308349609 25.7637500762939))
 Check: read_from_csv_file output
 - timeseries_datasets keys: ['20170923', '20171004', '20171015'] ...
 - sample slice shape: (578, 580)
 - lat_grid shape: (578, 580)
 - lon_grid shape: (578, 580)
 - Number of dates: 200
 - Attributes: {'PROJECT_NAME': 'CSV_IMPORT', 'WIDTH': '580', 'LENGTH': '578', 'processing_type': 'LOS_TIMESERIES', 'look_direction': 'R', 'collection': 'sarvey', 'data_type': 'LOS Displacement', 'PLATFORM': 'S1', 'MISSION': 'S1', 'mission': 'Tsx', 'relative_orbit': 36, 'beam_mode': 'SM', 'platform': 'S1', 'REF_LAT': 25.849037629616618, 'REF_LON': -80.13136049066759, 'first_date': '20170923', 'last_date': '20251008', 'history': '2026-03-24', 'data_footprint': 'POLYGON((-80.1147308349609 25.7637500762939,-80.1593627929688 25.7637500762939,-80.1593627929688 25.9763336181641,-80.1147308349609 25.9763336181641,-80.1147308349609 25.7637500762939))', 'scene_footprint': 'POLYGON((-80.1147308349609 25.7637500762939,-80.1593627929688 25.7637500762939,-80.1593627929688 25.9763336181641,-80.1147308349609 25.9763336181641,-80.1147308349609 25.7637500762939))', 'atmos_correct_method': None, 'beam_swath': 1, 'post_processing_method': 'MintPy', 'prf': 1717.128973878037, 'processing_software': 'isce', 'wavelength': 0.05546576, 'first_frame': 556, 'last_frame': 557}
columns: 580
rows: 578
converted chunk 0
converted chunk 1
converted chunk 2
converted chunk 3
converted chunk 4
converted chunk 5
converted chunk 6
converted chunk 7
converted chunk 8
converted chunk 9
converted chunk 10
converted chunk 11
converted chunk 12
converted chunk 13
converted chunk 14
converted chunk 15
converted chunk 16
WIDTH                       580
LENGTH                      578
processing_type             LOS_TIMESERIES
look_direction              R
mission                     Tsx
relative_orbit              36
beam_mode                   SM
REF_LAT                     25.849037629616618
REF_LON                     -80.13136049066759
first_date                  20170923
last_date                   20251008
history                     2026-03-24
data_footprint              POLYGON((-80.1147308349609 25.7637500762939,-80.1593627929688 25.7637500762939,-80.1593627929688 25.9763336181641,-80.1147308349609 25.9763336181641,-80.1147308349609 25.7637500762939))
scene_footprint             POLYGON((-80.1147308349609 25.7637500762939,-80.1593627929688 25.7637500762939,-80.1593627929688 25.9763336181641,-80.1147308349609 25.9763336181641,-80.1147308349609 25.7637500762939))
atmos_correct_method        None
beam_swath                  1
post_processing_method      MintPy
prf                         1717.128973878037
processing_software         isce
wavelength                  0.05546576
first_frame                 556
last_frame                  557
Now running tippecanoe with command tippecanoe *.json -P -l chunk_1 -x d -pf -pk -o TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.mbtiles 2> tippecanoe_stderr.log
time elapsed: 90.06259633299851
####################################
Running json_mbtiles2insarmaps.py...
Running.... json_mbtiles2insarmaps.py --num-workers 6 -u "insaradmin" -p "insaradmin" --host "149.165.153.50" -P "insarmaps" -U "insarmaps@insarmaps.com" --json_folder "./JSON" --mbtiles_file "./JSON/TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.mbtiles"
20260324:18-56 * json_mbtiles2insarmaps.py --num-workers 6 -u "insaradmin" -p "insaradmin" --host "149.165.153.50" -P "insarmaps" -U "insarmaps@insarmaps.com" --json_folder "./JSON" --mbtiles_file "./JSON/TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011.mbtiles"
Uploading json chunks...
Clearing old dataset, if it is there
Creating index on TSX_036_20170923_20251008_N2598W08016_N2576W08016_N2576W08011_N2598W08011
Inserted chunk_8.json to db
Inserted chunk_14.json to db
Inserted chunk_4.json to db
Inserted chunk_5.json to db
Inserted chunk_9.json to db
Inserted chunk_15.json to db
Inserted chunk_2.json to db
Inserted chunk_12.json to db
Inserted chunk_13.json to db
Inserted chunk_3.json to db
Inserted chunk_0.json to db
Inserted chunk_10.json to db
Inserted chunk_11.json to db
Inserted chunk_1.json to db
Inserted chunk_16.json to db
Inserted chunk_6.json to db
Inserted chunk_7.json to db
Indexing table

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions