- when saying
******************** step - generate_ifgram ******************** etc, give the number of the step
- This logging :
cat qunittestGalapagosSenDT128/minopy/log
20220111:175132 * minopyApp.py /work2/05861/tg851601/stampede2/insarlab/infiles/famelung/TEMPLATES/qunittestGalapagosSenDT128.template --dir minopy
should be in qunittestGalapagosSenDT128/log
- When running without --jobfiles (on idev) do we need
run_files directory? Could we write the commands that are written into e.g. run_files/run_files/run_02_minopy_phase_inversion into minopy/log?
- Consider to clearly say when mintpy is used. E.g.
*********** step 7 - ifgram_correction using MintPy ************
- rename
phase_series.h5 to phase_timeseries.h5
- in
minopyApp.cfg use minopy.network.type and title it "select the network to unwrap" (maybe the run_file run_04_minopy_generate_ifgram_network)
- make the numbers is
minopyApp.cfg consistent with the step numbers. E.g. don't use number for minopy.textCmd and
minopy.multiprocessing.numProcessor (see MintPy) but give a number for concatenate_patch (just explaining, no options)
- in the steps description , explain which files are created in each step (for the pure minopy steps, of course), also why an empty
minopy/inverted/interferograms_sequential is created as well as what weighting s used in the network_inversion step.
- introduce
minopy.inversion.startDate and minopy.inversion.endDate (to be overwritten by mintpy.network.startDate if given (there should be a mintpy.load.startDate but unfortunately it does not exist).
- somewhere document properly which mintpy option are used if given (will
mintpy.subset be used if given)
- how about
minopy.plot.coherenceMatrix option. You can give lat,lon or num. For num=3 it will divide equally space the subset to generate a 3x3 grid of points (side note: it would be could to have the ability to give multiple points (minopy.plot.coherenceMatrix=lat1,lon1;lat2,lon2 -- does MintPy has a convention for this? ';' is not great)
- weird unwrap/snaphu behavior. When running out of memory it starts again.
top - 22:50:50 up 315 days, 14:07, 1 user, load average: 5.50, 6.04, 6.00
Tasks: 1101 total, 1 running, 1100 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
GiB Mem : 187.3 total, 173.9 free, 3.5 used, 9.9 buff/cache
GiB Swap: 0.0 total, 0.0 free, 0.0 used. 179.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27574 tg851601 20 0 152932 3512 1676 R 0.7 0.0 0:02.59 top
21830 tg851601 20 0 177064 4364 1056 S 0.0 0.0 0:00.07 sshd
21831 tg851601 20 0 115836 2668 1796 S 0.0 0.0 0:00.05 bash
29020 tg851601 20 0 150760 5404 4208 S 0.0 0.0 0:00.04 ssh
29023 tg851601 20 0 177064 4356 1052 S 0.0 0.0 0:00.00 sshd
29024 tg851601 20 0 113312 1828 1464 S 0.0 0.0 0:00.01 bash
top - 23:01:47 up 315 days, 14:18, 1 user, load average: 9.12, 6.57, 6.16
Tasks: 1118 total, 17 running, 1101 sleeping, 0 stopped, 0 zombie
%Cpu(s): 15.4 us, 1.0 sy, 0.0 ni, 82.7 id, 0.9 wa, 0.0 hi, 0.0 si, 0.0 st
GiB Mem : 187.3 total, 0.8 free, 167.5 used, 19.0 buff/cache
GiB Swap: 0.0 total, 0.0 free, 0.0 used. 15.5 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32387 tg851601 20 0 18.9g 18.9g 628 R 100.0 10.1 1:15.89 snaphu
32388 tg851601 20 0 18.9g 18.9g 628 R 100.0 10.1 1:15.93 snaphu
32392 tg851601 20 0 18.9g 18.9g 628 R 100.0 10.1 1:15.62 snaphu
32461 tg851601 20 0 2041844 1.9g 608 R 100.0 1.0 0:23.74 snaphu
32385 tg851601 20 0 18.9g 18.9g 628 R 99.7 10.1 1:19.60 snaphu
32390 tg851601 20 0 18.9g 18.9g 628 R 99.7 10.1 1:16.33 snaphu
32394 tg851601 20 0 18.9g 18.9g 628 R 99.7 10.1 1:13.46 snaphu
32396 tg851601 20 0 18.9g 18.9g 628 R 99.7 10.1 1:12.70 snaphu
32397 tg851601 20 0 18.9g 18.9g 628 R
top - 23:02:32 up 315 days, 14:19, 1 user, load average: 5.00, 5.89, 5.96
Tasks: 1102 total, 1 running, 1101 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
GiB Mem : 187.3 total, 173.9 free, 3.5 used, 9.9 buff/cache
GiB Swap: 0.0 total, 0.0 free, 0.0 used. 179.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27574 tg851601 20 0 152932 3552 1676 R 0.7 0.0 0:07.10 top
21830 tg851601 20 0 177064 4364 1056 S 0.0 0.0 0:00.12 sshd
21831 tg851601 20 0 115836 2668 1796 S 0.0 0.0 0:00.05 bash
32492 tg851601 20 0 150760 5404 4208 S 0.0 0.0 0:00.04 ssh
32495 tg851601 20 0 177064 4356 1044 S 0.0 0.0 0:00.00 sshd
32496 tg851601 20 0 113312 1828 1464 S 0.0 0.0 0:00.01 bash
32573 tg851601 20 0 113184 1540 1268 S 0.0 0.0 0:00.00 init_launcher
32575 tg851601 20 0 113320 1580 1292 S 0.0 0.0 0:00.00 launcher
32576 tg851601 20 0 113320 1580 1292 S
Check memory requirement running only one unwrap.py job
top - 23:13:37 up 8 days, 9:49, 1 user, load average: 1.99, 1.09, 0.70
Tasks: 1090 total, 3 running, 1087 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.1 us, 0.0 sy, 0.0 ni, 97.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
GiB Mem : 187.3 total, 142.4 free, 39.7 used, 5.3 buff/cache
GiB Swap: 0.0 total, 0.0 free, 0.0 used. 144.5 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
380871 tg851601 20 0 18.2g 18.2g 636 R 99.7 9.7 4:00.52 snaphu
380915 tg851601 20 0 18.2g 18.2g 636 R 99.7 9.7 3:09.16 snaphu
380911 tg851601 20 0 152840 3464 1656 R 0.7 0.0 0:01.16 top
380570 tg851601 20 0 113300 1620 1316 S 0.0 0.0 0:00.00 slurm_script
380577 tg851601 20 0 107960 364 284 S 0.0 0.0 0:00.00 sleep
To discuss:
- How to deal with cases where we are interested in SHP over a long period but deformation only for a subset of this period?
interferograms_sequential-3for 3 connections if it makes it easier to use different networks.- what is
remove filter after unwrap ?
minopy.unwrap.method = auto (snaphu for auto)?- (For the logging, can we have after giving the step the command? Not sure...
******************** step - phase_inversion ********************
Generate /scratch/05861/tg851601/qunittestGalapagosSenDT128/minopy/run_files/run_02_minopy_phase_inversion
Total number of PATCHES: 30
Number of tasks for step phase inversion: 8
20220111:175541 * phase_inversion.py --work_dir /scratch/05861/tg851601/qunittestGalapagosSenDT128/minopy --range_window 19 --azimuth_window 9 --patch_size 200 --method sequential_EMI --test ks --num_worker 4 --mini_stack_size 10 --time_lag 10 --ps_num_shp 10 --index 0 --slc_stack /scratch/05861/tg851601/qunittestGalapagosSenDT128/minopy/inputs/slcStack.h5
)
******************** step - generate_ifgram ********************etc, give the number of the stepshould be in
qunittestGalapagosSenDT128/logrun_filesdirectory? Could we write the commands that are written into e.g.run_files/run_files/run_02_minopy_phase_inversionintominopy/log?*********** step 7 - ifgram_correction using MintPy ************phase_series.h5tophase_timeseries.h5minopyApp.cfguseminopy.network.typeand title it "select the network to unwrap" (maybe the run_filerun_04_minopy_generate_ifgram_network)minopyApp.cfgconsistent with the step numbers. E.g. don't use number forminopy.textCmdandminopy.multiprocessing.numProcessor(see MintPy) but give a number forconcatenate_patch(just explaining, no options)minopy/inverted/interferograms_sequentialis created as well as what weighting s used in thenetwork_inversionstep.minopy.inversion.startDateandminopy.inversion.endDate(to be overwritten bymintpy.network.startDateif given (there should be amintpy.load.startDatebut unfortunately it does not exist).mintpy.subsetbe used if given)minopy.plot.coherenceMatrixoption. You can give lat,lon or num. For num=3 it will divide equally space the subset to generate a 3x3 grid of points (side note: it would be could to have the ability to give multiple points (minopy.plot.coherenceMatrix=lat1,lon1;lat2,lon2 -- does MintPy has a convention for this? ';' is not great)Check memory requirement running only one unwrap.py job
To discuss:
interferograms_sequential-3for 3 connections if it makes it easier to use different networks.remove filter after unwrap?minopy.unwrap.method = auto (snaphu for auto)?)