Change "time" value for averaged quantities to midpoint of averaging period

[strandwg]

A longstanding quirk of CAM is that fields saved as averages have the "time" coordinate value of the end of the averaging period. A January monthly average has a timestamp of 1 February, for example. Changing the "time" coordinate value to represent the middle of the averaging period assists our users in understanding CAM output and doing proper analysis.

Fields saved as instantaneous values don't need to be changed at all.

Having one time axis for instantaneous fields and one for averaged fields would allow both kinds of output to reside in the same file, but netCDF-3 doesn't allow for more than one UNLIMITED dimension in a single file. netCDF-4 allows multiple UNLIMITED dimensions, however, pnetcdf is built on netCDF-3, so multiple time axes aren't possible.

[gold2718]

I have several questions about this issue that can help us process it.

• Why is a center time considered more correct? Does this apply to MIN and MAX fields as well, even if that is not when the minimum or maximum value was captured?
• If a center time is desired, the average of the time_bnds variable for that time frame is the correct value so the data is already on the file. Why is this not a solution?
• Why is another dimension needed?
  • If we want a variable that describes the middle of a time-averaged field, why can't we just define a new variable, center_time (or similar name)? Both time and center_time will be defined using the time dimension, as does time_bnds now.

[brian-eaton]

I take offense (not really:) to the suggestion that CAM's time coordinate is quirky. It is completely in conformance with the CF Metadata conventions!

If the timestamp of a time interval is desired to be at the interval midpoint, then the average of the interval endpoints is a value, say '15.5 days since 1850-01-01 00:00:00', which may then be converted to a calendar representation, or not, if say the time coordinate is used for plotting relative values along a time axis. Typically the thing that causes the biggest headache has been converting to a calendar representation. That's because standard tools for doing this kind of conversion don't recognize a 365-day year. I haven't kept up with the postprocessing toolchains, but surely by now this is no longer a problem? Bottom line: I'm agreeing with Steve's comment: Why is this not a solution?

I also agree with Steve's comment on the unlimited dimension. There only needs to be one unlimited dimension and it should be used for any alternate time axis, e.g., center_time. The only issue here is that center_time(time) is not a coordinate variable (in the NetCDF conventions sense) and so connecting it to the time_bnds from which it is derived would in fact be "quirky", that is there's no convention for doing that. So it would be a CAM specific thingy that our postprocessing tools could be hardcoded to recognize, but nothing else would recognise it. I don't recomment this solution.

[strandwg]

1. CMIP3/CMIP5/CMIP6 have all required that the "time" value for averaged quantities be the midpoint of the period, with the "time_bnds" values representing the endpoints of the averaging period. Many of our users are familiar with MIPn data, and expect that CAM output is the same. I've gotten many emails over the years asking about this exact issue.

Fields that are MIN and MAX should have their time value represent the period over which the extrema has been found, analogous to an average. I don't think CAM has ever kept track of which timestep an extreme was found.

2. Averaging "time_bnds" is what's done for the translation from CESM output to MIP-compliant format, and I know at least a few users who've written their own software to do that average and realign their data to the new time axis. CAM can do that natively and thus prevent the issue.

3. If averaged and instantaneous fields were forced to be in different output streams, then an additional time dimension wouldn't be needed, but CAM allows outputting both instantaneous and averaged fields in the same stream. The "time" dimension is overloaded in a sense by using the same single axis to attempt to serve both kinds of fields.

[brian-eaton]

There is a lot of postprocessing that needs to be done to produce files that are in compliance with CMIP. And there is no issue in that case about putting both instantaneous and time averaged fields into a single file (as far as I know).

I think having CAM produce CMIP compliant output directly would be a substantial task which is far better left to postprocessing. If CAM isn't going to produce CMIP compliant output is there really a significant gain in going to the partway solution of splitting CAM's instantaneous and time interval based output into separate streams? What we're doing now is simpler.

I agree with your point about MIN and MAX fields. CAM has never kept track of the timestep where the extreme value occurs and the mid-point time is no more correct than the endpoint in thas case.

If the argument is made that the mid-point time coordinate is scientifically more correct than the endpoint then that could be a valid reason for adding the complication of separate output streams for instantaneous and time interval based output. I think that's really the discussion that the scientists need to have. From an SE perspective there is no real difficulty in doing this. Just a bit of work to separate out the currently mixed fields to create separate streams, and adding some checking code in CAM to prevent mixed output in the future.

[strandwg]

Basically, CAM is overloading "time" by using one axis to represent both averaged and instantaneous values. Since CESM uses pnetcdf, and pnetcdf is built on netCDF-3, CESM is allowed only one unlimited dimension in a file. If CESM could use netCDF-4, then additional unlimited time axes could be defined - one for instantaneous values, one for averaged values, in the same file.

I'm asking that CAM (and CESM as a whole) write MIP compliant output directly, since CESM now dumps out so much data that keeping copies (CESM output and CMIP compliant format) of the same data is no longer practical and makes managing CESM data much more difficult.

[brian-eaton]

Let's not mix up this issue with direct output in MIP format. That's a much larger task than dealing with the time coordinate. I think you should open another issue if you want to keep that on the radar. My initial thought is that the ongoing exercise by CAM to upgrade its postprocessing toolchain should address this issue. If our tools dealt with the MIP format then there would be no reason to save the original history files once the conversion was done. Obviously though not all CAM history output would need to be converted to MIP format and ideally the new tools would deal with either format.

[strandwg]

Yes, MIP format is another issue. Sorry to inject it here.

[gold2718]

Fields that are MIN and MAX should have their time value represent the period over which the extrema has been found, analogous to an average.

I'm not even sure how that would work given that the minimum or maximum value is per grid box which means that the capture could represent hundreds of different times in a one-month sample (up to (number of grid cells * number of levels) for longer intervals) . What am I missing?

[brian-eaton]

Good point. Clearly the time coordinate being at the end of the interval is as good a choice as any in this case. The midpoint would also be as good a choice as any if we end up resolving this issue by making that change for the sake of the time averaged variables.

[strandwg]

Fields that are MIN and MAX should have their time value represent the period over which the extrema has been found, analogous to an average.

I'm not even sure how that would work given that the minimum or maximum value is per grid box which means that the capture could represent hundreds of different times in a one-month sample (up to (number of grid cells * number of levels) for longer intervals) . What am I missing?

Nothing - you're correct. A min or max over a month is over many timesteps - thousands in a month.

[strandwg]

I've been thinking and I believe using a meaningful value for "time" for a temporal average is better than using the instantaneous value for "time" - even for instantaneous output.

I've not often seen instantaneous output in the monthly-mean stream, but if there is, obviously all that's wanted is just a value, and it doesn't matter when it happened.

Actually, the same thing applies to daily output, 6-hourly output, 3-hourly output, etc. What's really wanted is the preservation of variability, which is important for fields like precipitation and a few others. The exact value for "time" doesn't matter, and any value is as good as any other, and just so there are 365 values per year, one day apart, for daily data, 365*4, 6 hours apart, for 6-hourly data, etc. "time_bnds" has no meaning.

For averages, however, the value for "time" does matter. By convention (and by de facto standard in MIPs), the value for "time" of an average is the midpoint of the times, with "time_bnds" representing those endpoints. Makes perfect sense given the limitations of netCDF's representation of spans of time.

By using the instantaneous value for "time" for average data (especially monthly-mean output), we just confuse users and lose meaning to our data.

Unless I've overlooked something, and I'm entirely wrong.

[gold2718]

@strandwg,
Thanks for the discussion, we will run this by the AMP scientists to see if keeping mixed fields (instantaneous fields in an averaged file) is really important to anyone or if we should try and phase them out.

[strandwg]

@gold2718, Thanks.

[swrneale]

I've always felt that variables were described correctly in non monthly mean files., i.e. 'time' describes the instantaneous time step for an instantaneous field. For a time averaged variable, the meta data of 'time_averaged' attached to the variable tells you the averaging period is in time_bnds.

Monthly averages are messy because the file naming is more informative for the time averaging obviously. I have just always resisted the center time thinking for the time stamp. 31-day months give you 12Z on the 16th day, which just feels clunky to me.

[brian-eaton]

I've been thinking and I believe using a meaningful value for "time" for a temporal average is better than using the instantaneous value for "time" - even for instantaneous output.

I've not often seen instantaneous output in the monthly-mean stream, but if there is, obviously all that's wanted is just a value, and it doesn't matter when it happened.

Actually, the same thing applies to daily output, 6-hourly output, 3-hourly output, etc. What's really wanted is the preservation of variability, which is important for fields like precipitation and a few others. The exact value for "time" doesn't matter, and any value is as good as any other, and just so there are 365 values per year, one day apart, for daily data, 365*4, 6 hours apart, for 6-hourly data, etc. "time_bnds" has no meaning.

For averages, however, the value for "time" does matter. By convention (and by de facto standard in MIPs), the value for "time" of an average is the midpoint of the times, with "time_bnds" representing those endpoints. Makes perfect sense given the limitations of netCDF's representation of spans of time.

By using the instantaneous value for "time" for average data (especially monthly-mean output), we just confuse users and lose meaning to our data.

Unless I've overlooked something, and I'm entirely wrong.

I don't agree.

Consider an example of yearly averaged TS (surface temperature) data. Let's say time_bnds=[0,365] (days since 1850-01-01) and time=182.5. The annual average field doesn't look like any actual state of TS, so why is using a time coordinate of July 1 more correct than Dec 31 (MIP convention aside)? In fact the only relevent time information in this example comes from the bounds which tells you it's an annual average for 1850.

Now consider instantaneous once yearly output of TS. Again let time_bnds=[0,365] and time=182.5. You claim it doesn't matter what the instantaneous time value is, but now we have a field which is clearly NH winter labeled with the time July 1.

I'm afraid we can't solve the problem of confused users, but the time coordinate convention used by CAM does not sacrifice any meaning of the data.

My conclusion is that the only reasonable way to move the time coordinate value to the interval midpoint is to not allow instantaneous fields in files that contain interval based fields.