Input Data StreamsOverview
An input data stream is a time-series of input data files
where all the fields in the stream are located in the same data file
and all share the same spatial and temporal coordinates
(ie. are all on the same grid and share the same time axis).
Normally a time axis has a uniform dt, but this is not a requirement.
The data models can have multiple input streams.
The data for one stream may be all in one file or may be spread over several files.
For example, 50 years of monthly average data might be contained all in one data file
or it might be spread over 50 files, each containing one year of data.
The data models can loop over stream data --
repeatedly cycle over some subset of an input stream's time axis.
When looping, the models can only loop over whole years.
For example, an input stream might have SST data for years 1950 through 2000,
but a model could loop over the data for years 1960 through 1980.
A model cannot loop over partial years, for example,
from 1950-Feb-10 through 1980-Mar-15.
The input data must be in a netcdf file and the time axis in that file
must be CF-1.0 compliant.
There are two main categories of information that the data models need
to know about a stream:
data that describes what a user wants --
what streams to use and how to use them -- things that can be changed by a user. data that describes the stream data -- meta-data about the
inherent properties of the data itself -- things that cannot be
changed by a user.
Generally, information about what streams a user wants to use and how
to use them is input via the strdata ("stream data") Fortran namelist,
while meta-data that describes the stream data itself is found in an
xml-like text file called a "stream description file."
Stream Data
The strdata (short for "stream data") input is
set via a fortran namelist called shr_strdata_nml. That namelist, the
strdata datatype, and the methods are contained in the share source
code file, models/csm_share/shr/shr_strdata_mod.F90. In general,
strdata input defines an array of input streams and operations to
perform on those streams. Therefore, many namelist inputs are arrays
of character strings. Different variable of the same index are
associated. For instance, mapalgo(1) spatial interpolation will be
performed between streams(1) and the target domain.
The following namelist are available with the strdata namelist.
dataMode - component specific modedomainFile- final domainstreams - input filesvectors - paired vector field namesfillalgo - fill algorithmfillmask - fill maskfillread - fill mapping file to readfillwrite - fill mapping file to writemapalgo - spatial interpolation algorithmmapmask - spatial interpolation maskmapread - spatial interpolation mapping file to readmapwrite - spatial interpolation mapping file to write tintalgo - time interpolation algorithm taxMode - time interpolation modedtlimit - delta time axis limit
The set of shr_strdata_nml namelist keywords are the same for all data
models. As a result, any of the data model namelist documentation can
be used to view a full description. For example, see stream specific namelist
settings .
Specifying What Streams to Use
The data models have a namelist variable that specifies which input
streams to use and, for each input stream, the name of the
corresponding stream description file, what years of data to use, and
how to align the input stream time axis with the model run time axis.
This input is set in the strdata namelist input.
General format:
&shr_strdata_nml
streams = 'stream1.txt year_align year_first year_last ',
'stream2.txt year_align year_first year_last ',
...
'streamN.txt year_align year_first year_last '
/
where:
the stream description file, a plain text file containing details about the input stream (see below)
the first year of data that will be used
the last year of data that will be used
a model year that will be aligned with data for year_first
The stream text files for a given data model mode are automatically
generated by the corresponding data model
build-namelist with present names. As an example
we refer to the following datm_atm_in example
file (that would appear in both
$CASEROOT/CaseDocs and
$RUNDIR):
datamode = 'CLMNCEP'
domainfile = '/glade/proj3/cseg/inputdata/share/domains/domain.lnd.fv1.9x2.5_gx1v6.090206.nc'
dtlimit = 1.5,1.5,1.5,1.5
fillalgo = 'nn','nn','nn','nn'
fillmask = 'nomask','nomask','nomask','nomask'
mapalgo = 'bilinear','bilinear','bilinear','bilinear'
mapmask = 'nomask','nomask','nomask','nomask'
streams = "datm.streams.txt.CLM_QIAN.Solar 1895 1948 1972 ",
"datm.streams.txt.CLM_QIAN.Precip 1895 1948 1972 ",
"datm.streams.txt.CLM_QIAN.TPQW 1895 1948 1972 ",
"datm.streams.txt.presaero.trans_1850-2000 1849 1849 2006"
taxmode = 'cycle','cycle','cycle','cycle'
tintalgo = 'coszen','nearest','linear','linear'
vectors = 'null'
As is discussed in the CESM1.1 User's Guide, to change the contents of
datm_atm_in, you can edit
$CASEROOT/user_nl_datm to change any of the above
settings EXCEPT FOR THE NAMES
datm.streams.txt.CLM_QIAN.Solar,
datm.streams.txt.CLM_QIAN.Precip,
datm.streams.txt.CLM_QIAN.TPQW and
datm.streams.txt.presaero.trans_1850-2000. Note
that any namelist variable from shr_strdata_nml and datm_nml can be
modified by adding the appropriate keyword/value pairs to
user_nl_datm. As an example, the following could
be the contents of $CASEROOT/user_nl_datm:
!------------------------------------------------------------------------
! Users should ONLY USE user_nl_datm to change namelists variables
! Users should add all user specific namelist changes below in the form of
! namelist_var = new_namelist_value
! Note that any namelist variable from shr_strdata_nml and datm_nml can
! be modified below using the above syntax
! User preview_namelists to view (not modify) the output namelist in the
! directory $CASEROOT/CaseDocs
! To modify the contents of a stream txt file, first use preview_namelists
! to obtain the contents of the stream txt files in CaseDocs, and then
! place a copy of the modified stream txt file in $CASEROOT with the string
! user_ prepended.
!------------------------------------------------------------------------
streams = "datm.streams.txt.CLM_QIAN.Solar 1895 1948 1900 ",
"datm.streams.txt.CLM_QIAN.Precip 1895 1948 1900 ",
"datm.streams.txt.CLM_QIAN.TPQW 1895 1948 1900 ",
"datm.streams.txt.presaero.trans_1850-2000 1849 1849 2006"
and the contents of shr_strdata_nml (in both $CASEROOT/CaseDocs and $RUNDIR)
would be
datamode = 'CLMNCEP'
domainfile = '/glade/proj3/cseg/inputdata/share/domains/domain.lnd.fv1.9x2.5_gx1v6.090206.nc'
dtlimit = 1.5,1.5,1.5,1.5
fillalgo = 'nn','nn','nn','nn'
fillmask = 'nomask','nomask','nomask','nomask'
mapalgo = 'bilinear','bilinear','bilinear','bilinear'
mapmask = 'nomask','nomask','nomask','nomask'
streams = "datm.streams.txt.CLM_QIAN.Solar 1895 1948 1900 ",
"datm.streams.txt.CLM_QIAN.Precip 1895 1948 1900 ",
"datm.streams.txt.CLM_QIAN.TPQW 1895 1948 1900 ",
"datm.streams.txt.presaero.trans_1850-2000 1849 1849 2006"
taxmode = 'cycle','cycle','cycle','cycle'
tintalgo = 'coszen','nearest','linear','linear'
vectors = 'null'
As is discussed in the User's Guide, you should use
preview_namelists to view (not modify) the output
namelist in CaseDocs.
Stream Description File
The stream description file is not Fortran
namelist, but a locally built xml-like parsing implementation.
Sometimes it is called a "stream dot-text file" because it has a
".txt." in the filename. Stream description files contain data that
specifies the names of the fields in the stream, the names of the
input data files, and the file system directory where the data files
are located. In addition, a few other options are available such as
the time axis offset parameter.
In CESM1.1, each data model's build-namelist
utility (e.g. models/atm/datm/bld/build-namelist) automatically
generates these stream description files. The directory contents of
each data model will look like the following (using DATM as an
example)
models/atm/datm/bld/build-namelist
models/atm/datm/bld/namelist_files/namelist_definition_datm.xml
models/atm/datm/bld/namelist_files/namelist_defaults_datm.xml
The namelist_definition_datm.xml file defines all
the namelist variables and associated groups. The
namelist_defaults_datm.xml provides the out of
the box settings for the target data model and target stream.
build-namelist utilizes these two files to
construct the stream files for the given compset settings. You can
modify the generated stream files for your particular needs by doing
the following:
Call setup OR
preview_namelists. Copy the relevant description file (from
$CASEROOT/CaseDocs to
$CASEROOT and pre-pend a "user_nl_" string to the
filename. Change the permission of the file to write. For example,
assuming you are in $CASEROOT
cp CaseDocs/datm.streams.txt.CLM_QIAN.Solar user_nl_datm.streams.txt.CLM_QIAN.Solar
chmod u+w user_nl_datm.streams.txt.CLM_QIAN.Solar
Edit
user_nl_datm.streams.txt.CLM_QIAN.Solar with your
desired changes. Be sure not to put any tab characters in
the file: use spaces instead Also, in contrast to other
user_nl_xxx files, be sure to set all relevant data model settings in
the xml files, before editing
user_nl_datm.streams.txt.CLM_QIAN.Solar.
Once you have created a user_nl_* streams file, further
modifications to the relevant data model settings in the xml files
will be ignored. If you later realize that you need to
change some settings in an xml file, you should remove the user_nl_*
stream file, make the modifications in the xml file, rerun
preview_namelists, then reintroduce your
modifications into a new user_nl_* stream file. Call preview_namelists Verify that your changes do indeed appear in the
resultant stream description file appear in
CaseDocs/datm.streams.txt.CLM_QIAN.Solar. These
changes will also appear in
$RUNDIR/datm.streams.txt.CLM_QIAN.Solar.
The data elements found in the stream description file are:
A comment about the source of the data -- always set to GENERIC in
CESM1.1 and not used by the model. This is there only for backwards
compatibility.
Information about the field data for this stream...
A list of the field variable names. This is a paired list
with the name of the variable in the netCDF file on the left and the
name of the corresponding model variable on the right.
This is the list of fields to read in from the data file, there may be
other fields in the file which are not read in (ie. they won't be used).
The file system directory where the data files are located.
The list of data files to use. If there is more than one file,
the files must be in chronological order, that is, the dates in time axis
of the first file are before the dates in the time axis of the second file.
The option is obsolete and no longer performs a function. Control
of the time interpolation algorithm is in the strdata namelists,
tinterp_algo and taxMode
.
This offset allows a user to shift the time axis of a data stream
by a fixed and constant number of seconds. For instance, if a data
set contains daily average data with timestamps for the data at the
end of the day, it might be appropriate to shift the time axis by
12 hours so the data is taken to be at the middle of the day
instead of the end of the day. This feature supports only simple
shifts in seconds as a way of correcting input data time axes
without having to modify the input data time axis manually. This
feature does not support more complex shifts such as end of month
to mid-month. But in conjunction with the time interpolation
methods in the strdata input, hopefully most user needs can be
accommodated with the two settings. Note that a positive offset
advances the input data time axis forward by that number of
seconds.
Information about the domain data for this stream...
A list of the domain variable names. This is a paired list
with the name of the variable in the netCDF file on the left and the
name of the corresponding model variable on the right.
This data models require five variables in this list.
The names of model's variables (names on the right) must be:
"time," "lon," "lat," "area," and "mask."
The file system directory where the domain data file is located.
The name of the domain data file.
Often the domain data is located in the same file as the field data (above),
in which case the name of the domain file could simply be the name of the
first field data file. Sometimes the field data files don't contain the
domain data required by the data models, in this case, one new file can
be created that contains the required data.
Actual example:
<stream>
<dataSource>
GENERIC
</dataSource>
<domainInfo>
<variableNames>
time time
lon lon
lat lat
area area
mask mask
</variableNames>
<filePath>
/glade/proj3/cseg/inputdata/atm/datm7/NYF
</filePath>
<fileNames>
nyf.ncep.T62.050923.nc
</fileNames>
</domainInfo>
<fieldInfo>
<variableNames>
dn10 dens
slp_ pslv
q10 shnum
t_10 tbot
u_10 u
v_10 v
</variableNames>
<filePath>
/glade/proj3/cseg/inputdata/atm/datm7/NYF
</filePath>
<offset>
0
</offset>
<fileNames>
nyf.ncep.T62.050923.nc
</fileNames>
</fieldInfo>
</stream>