coadd preprocessing on TACC

[changhoonhahn]

issue to track progress on deploying the DESI coadd preprocessing on TACC.

• implement SLURM script
• deploy for test sample of 1,000
• deploy for all 110,000 candidates

[yuni1218]

coadd-4-80609-20201217-39627853670650224.fits

It has only BR_* extensions:
BR_WAVELENGTH, BR_FLUX, BR_IVAR, BR_MASK, BR_RESOLUTION
It is missing all BRZ_*:
BRZ_WAVELENGTH, BRZ_FLUX, BRZ_IVAR, BRZ_MASK

It is also missing separate camera sets:
B_, R_, Z_* (WAVELENGTH/FLUX/IVAR/MASK)

[yuni1218]

I'm assuming that there is no Z flux in this file, which is not a problem so I'm gonna change my script to skip those files but also we can skip coadding files if there is missing flux.

This is HDU header of a valid file:

[3] B_WAVELENGTH
EXTNAME: B_WAVELENGTH
NAXIS: 1
NAXIS1: 2751
BUNIT: Angstrom

[4] B_FLUX
EXTNAME: B_FLUX
NAXIS: 2
NAXIS1: 2751
NAXIS2: 1
BUNIT: 10**-17 erg/(s cm2 Angstrom)

[5] B_IVAR
EXTNAME: B_IVAR
NAXIS: 2
NAXIS1: 2751
NAXIS2: 1
BUNIT: 10**+34 (s2 cm4 Angstrom2) / erg2

[6] B_MASK
EXTNAME: B_MASK
NAXIS: 2
NAXIS1: 2751
NAXIS2: 1

[7] B_RESOLUTION
EXTNAME: B_RESOLUTION
NAXIS: 3
NAXIS1: 2751
NAXIS2: 11

[8] R_WAVELENGTH
EXTNAME: R_WAVELENGTH
NAXIS: 1
NAXIS1: 2326
BUNIT: Angstrom

[9] R_FLUX
EXTNAME: R_FLUX
NAXIS: 2
NAXIS1: 2326
NAXIS2: 1
BUNIT: 10**-17 erg/(s cm2 Angstrom)

[10] R_IVAR
EXTNAME: R_IVAR
NAXIS: 2
NAXIS1: 2326
NAXIS2: 1
BUNIT: 10**+34 (s2 cm4 Angstrom2) / erg2

[11] R_MASK
EXTNAME: R_MASK
NAXIS: 2
NAXIS1: 2326
NAXIS2: 1

[12] R_RESOLUTION
EXTNAME: R_RESOLUTION
NAXIS: 3
NAXIS1: 2326
NAXIS2: 11

[13] Z_WAVELENGTH
EXTNAME: Z_WAVELENGTH
NAXIS: 1
NAXIS1: 2881
BUNIT: Angstrom

[14] Z_FLUX
EXTNAME: Z_FLUX
NAXIS: 2
NAXIS1: 2881
NAXIS2: 1
BUNIT: 10**-17 erg/(s cm2 Angstrom)

and of the problematic file:

[1] FIBERMAP
EXTNAME: FIBERMAP
NAXIS: 2
NAXIS1: 341
NAXIS2: 1
TTYPE1: TARGETID
TFIELDS: 70

[2] EXP_FIBERMAP
EXTNAME: EXP_FIBERMAP
NAXIS: 2
NAXIS1: 142
NAXIS2: 2
TTYPE1: TARGETID
TFIELDS: 22

[3] BR_WAVELENGTH
EXTNAME: BR_WAVELENGTH
NAXIS: 1
NAXIS1: 5026
BUNIT: Angstrom

[4] BR_FLUX
EXTNAME: BR_FLUX
NAXIS: 2
NAXIS1: 5026
NAXIS2: 1
BUNIT: 10**-17 erg/(s cm2 Angstrom)

[5] BR_IVAR
EXTNAME: BR_IVAR
NAXIS: 2
NAXIS1: 5026
NAXIS2: 1
BUNIT: 10**+34 (s2 cm4 Angstrom2) / erg2

[6] BR_MASK
EXTNAME: BR_MASK
NAXIS: 2
NAXIS1: 5026
NAXIS2: 1

[7] BR_RESOLUTION
EXTNAME: BR_RESOLUTION
NAXIS: 3
NAXIS1: 5026
NAXIS2: 11

[8] SCORES
EXTNAME: SCORES
NAXIS: 2
NAXIS1: 8
NAXIS2: 1
TTYPE1: TARGETID
TFIELDS: 1

[changhoonhahn]

I vaguely recall there were some spectra without one of the spectrograph outputs.

Can you post what survey this coadd was observed in? Was it main, sv1, sv3? If it's sv1 or sv3 it wouldn't be a big surprise, as SV stands for survey validation.

[yuni1218]

It was sv3
Target ID 39627859723032583
Tile ID 6
Night 20210417

[yuni1218]

login1.ls6(999)$ cd /work/10579/prisha/ls6/desi_project/output_coadds/
login1.ls6(1000)$ find . -maxdepth 1 -mindepth 1 -type d | wc -l
19313

There are only ~20,000 targets

[yuni1218]

check how IVAR is generated