Time labels of avhrr SST 2020 data are inconsistent with 2019

[kdraeder]

2019-12-31-00000 failed at the start
because the SST file does not contain 2020-01-01 (0Z).
I made a 2020 file for Benjamin through 2020-05
(see /glade/scratch/raeder/SST/avhrr_v2.1),
which I've concatenated with the 2011-2019 file.
The relevant files are in
/glade/work/raeder/Models/CAM_init/SST:
avhrr-only-v2.2011-2019_0Z_filled_c200219.nc
avhrr_v2.1-only-v2.20200101-20200607_filled_c200622.nc
avhrr-only-v2.2011-202005_0Z_filled_c200807.nc
and the same name roots with .time replacing .nc.

A (slight) problem has re-emerged; the AVHRR data times for 2020
are now (correctly) set to be the middle of the day,
so the concatenated file has times (days since 1978-01-01 0Z)
... 15337, 15338, 15339, 15340.5, 15341.5, ...
I'm guessing that the data were all created in the same way,
but there could be a slight inconsistency in the hindcasts
during the last day of 2019 and the first half day of 2020
as the SST interpolation is spread over 36 hours, instead of 24.

So it would be great if someone(s) else could check these files
(and the avhrr procedure?) today while I'm out to see whether
the new SST file is safe to use.

Thanks,
Kevin

[timhoar]

So here is my dilemma ... as Kevin states, the processing of the AVHRR SSTs did not change, just the metadata. So, to be consistent with the existing reanalysis SST data avhrr-only-v2.2011-2019_0Z_filled_c200219.nc, I think we should just append a single day with a time that is consistent with the existing SST data.

The question I now have is ... is the last day of the reprocessed 2019 the same as the last day that we have. If that is the case, then we can be sure the first day of the reprocessed 2020 (1 Jan 12Z) is the data we should be calling 2020, 1, 1, 0Z.

In /glade/scratch/raeder/SST/avhrr_v2.1, I do not see the data needed to determine if the end of the reprocessed 2019 is the same as what we have. Even if people are comfortable with the more accurate but discontinuous time, the only way I know to check the data is to compare the last day of the new datastream to our existing datastream.

[timhoar]

There does seem to be a concerning blip just off the west coast of South America.
I put avhrr-only-v2.2011-202005_0Z_filled_c200807.nc in a loop in ncview and got to the transition point ... seems like a kinda large change, and then there's the blip ...

[kdraeder]

That's ugly. And it lasts for several days.

In addition, I used ncview to see the new ds277.7 SST for the last day of 2019
and our 2011-2019 (old ds277.?) file for the same time.
That point (aka [1077,354]; a cold spot around the Galapagos)
has values 19.01 in the ds277.7 and 15.21 in ours. I can imagine the opposite happening if ours
was smoothed so that cold spot was less cold than originally, but not 3.8 C colder.
Additional areas are visibly different (I displayed them on the same scale), while others look identical.
I've not done an ncdiff on the 2 files because they have different variable names and metadata,
and so far it seems unnecessary because they are clearly different.

It looks like our file has a moving average and used something cold (0 C?) as the value
for missing 2020 data when we generated the 2019 file.
The 277.7 files use -9.99 as the special value for land areas, so that would fit,
but that would need to be spread to the nearby ocean points.
And when I generated the 2020 file for Benjamin, there would have been no 2019
data in its moving average, so the first few days would be bad, but it would recover.
It may be related the the "fill" process that was done to our file there because of the Galapagos.

I'll try regenerating the SSTs for late 2019 continuously through early 2020
to see if that looks healthy. If so, we can return to deciding what to do about the 0Z vs 12Z time stamp.

Then I'll rerun the last few days of 2019, not just the last day.

[kdraeder]

The first response to Tim's comments I posted had some misunderstandings in it, so please be sure that you read the comment as posted in github, which I edited, not what appears in the emails. Thanks, Kevin

…

On Fri, Aug 7, 2020 at 10:28 AM Tim Hoar ***@***.***> wrote: There does seem to be a concerning blip just off the west coast of South America. I put avhrr-only-v2.2011-202005_0Z_filled_c200807.nc in a loop in ncview and got to the transition point ... seems like a kinda large change, and then there's the blip ... [image: sst_check_timeseries] <https://user-images.githubusercontent.com/15146042/89666871-5a4f5a80-d898-11ea-9090-dea7d997ae64.png> — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#84 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADFFGEBVN2QGPZMK3I74ZMTR7QTRXANCNFSM4PXUALKA> .

[kdraeder]

I've rerun the SST processing programs again, and they appear to reproduce
the data that's in the ds277.7 files, with the continents filled in.
In particular, the point near the Galapagos now has an SST of 19.01
in our file, just like the ds277.7. That multi-day dip in SST seems to
be either a real thing, or at least what's in the ds277.7 analysis.
I haven't generated a difference picture. If anyone wants to,
the (new) ds277.7 file is
/gpfs/fs1/collections/rda/data/ds277.7/avhrr_v2.1/2019/oisst-avhrr-v02r01.20191231.nc
and the new one I've been comparing against is in
/glade/scratch/raeder/SST/avhrr_v2.1/avhrr_v2.1-only-v2.201912_filled_c200808.nc
(be sure to use the last time slot).

I also have not found the ds277.? file that had times labeled 00Z.
It seems that they replaced them with 12Z files.
Do we need to ask them for the old ones to see if the SSTs are different,
along with the time assigned to them? It appears to me that they are different
based on what's in our 2011-2019 SST file.

[kdraeder]

Ji Zaihua, who maintains the ds277.x data sets wrote:

the data files in ds277.7 are updated to version 2.1 now.
The middle time of daily mean is 12Z.
For the older version, the time set to 0Z by mistake,
The middle daily mean should be 12Z.

We do have backup copies of the oder version 2.0 and saved by years under
/gpfs/fs1/collections/rda/decsdata/ds277.7/V/avhrr

I've confirmed that the data in those files are what we used in our 2011-2019 file.
The data for 2020-01-01 00Z are available, so we can tack that on to the 2011-2019 file
(after the usual conversion of meta data and filling the land areas).
Woot!

[timhoar]

Excellent ... sounds like a good solution. Tim Hoar Data Assimilation Research Section National Center for Atmospheric Research thoar@ucar.edu 303.497.1708 *As of 3/17/20, NCAR is on mandatory telework.* *Please reach me via email or Google Chat/Meet instead of my office phone.*

…

On Mon, Aug 10, 2020 at 11:24 AM kdraeder ***@***.***> wrote: Ji Zaihua, who maintains the ds277.x data sets wrote: the data files in ds277.7 are updated to version 2.1 now. The middle time of daily mean is 12Z. For the older version, the time set to 0Z by mistake, The middle daily mean should be 12Z. We do have backup copies of the oder version 2.0 and saved by years under /gpfs/fs1/collections/rda/decsdata/ds277.7/V/avhrr I've confirmed that the data in those files are what we used in our 2011-2019 file. The data for 2020-01-01 00Z are available, so we can tack that on to the 2011-2019 file (after the usual conversion of meta data and filling the land areas). Woot! — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#84 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTRYOVSZJRQK4GL4TB4Y7LSAAUNPANCNFSM4PXUALKA> .

[kdraeder]

The reason that the times jumped from 15399 to 15400.5
is that the new avhrr files have time:units = days from 1978-01-01 12Z.
When I concatenated the files with ncrcat, it took that extra 12 hours
into account and added a time for the new data that is relative to the old start time;
1978-01-01 0Z.

[kdraeder]

Even if the new data at 15400.5 is the same as the old data at 15400,
that would still lead to an interpolation over 36 hours, instead of 24.
It appears that the new data are NOT just a shift of the old data by 12 hours.
This issue is being investigated with the original data set creators at NOAA.