The following describes many of the files and directories in the $CASEROOT directory. Buildconf/ is the directory where the buildnml and buildexe component scripts reside and where the input_data_list files are generated by the buildnml scripts. CaseDocs/ is the directory where copies of the latest namelist/text input files from invoking preview_namelists are placed. These files should not be edited and exist only to help document the case setup and run. SourceMods/ contains directories for each component where case specific source code modifications can be included. The source files in these directories will always be used in preference to the source code in $CCSMROOT. This feature allows users to modify CESM source code on a case by case basis if that preferable over making modifications in the $CCSMROOT sandbox. LockedFiles/ is the directory that holds copies of the locked files. MachinesHist/ is a directory where previous case configurations are stored. In other words, when cesm_setup -clean is run, the current batch run script and env_mach_pes.xml file are copied into this directory, so when &cesm_setup; is subsequently run, there is an opportunity to review and compare previous setups. Macros is the Makefile Macros file for the current configuration. The Makefile is located in the Tools directory and is identical on all machines. The Macros file is a machine and compiler dependent file. This file is locked during the build step. README.case provides a summary of the commands used to generate this case. $CASE.build is the script that is run interactively to build the CESM model. $CASE.clean_build is the script that cleans the CESM build. $CASE.l_archive is the script that is submitted to the batch queue to archive CESM data to the long-term archive disk, like an hpss or mass storage system. $CASE.run is the script that is submitted to the batch queue to run a CESM job. This script could also be run interactively if resources allow. check_input_data is a tool that checks for missing input datasets and provides a capability for exporting them to local disk. cesm_setup is the script that is run to generate the $CASE.run script for the target &env_mach_pes.xml; file and if they have not already been created, the user_nl_xxx files for the target components. create_production_test is a tool that generates an exact restart test in a separate directory based on the current case. env_*.xml files contain variables used to set up, build, and run CESM. logs/ is the directory that contains a copy of the component log files from successful case runs. timing/ is the directory that contains timing output from each successful case run. xmlchange is a utility that supports changing xml variables in the $CASEROOT xml files. $CASEROOT/Tools/ a directory containing many scripts that are used to setup the CESM model as well as run it. Some of particular note are Makefile is the Makefile that will be used for the build. cesm_buildexe.csh is invoked by $CASEROOT/$CASE.build to generate the model executable. This script calls the component buildexe scripts in Buildconf. cesm_buildnml.csh is invoked by $CASEROOT/$CASE.build to generate the component namelists in $RUNROOT. This script calls the component buildnml scripts in Buildconf. ccsm_check_lockedfiles checks that any files in the $CASEROOT/LockedFiles/ directory match those in the $CASEROOT directory. This helps protect users from overwriting variables that should not be changed. ccsm_getenv converts the xml variables in $CASEROOT to csh environmental variables. getTiming.csh generates the timing information. getTiming.pl generates timing information and is used by getTiming.csh. mkDepends generates Makefile dependencies in a form suitable for inclusion into a Makefile. perf_summary.pl generates timing information. st_archive.sh is the short-term archive script. It moves model output out of run directory to the short-term archive directory. Associated with DOUT_S and DOUT_S_ROOT env variables in env_run.xml. taskmaker.pl derives pe counts and task and thread geometry info based on env var values set in the &env_mach_pes.xml; file. xml2env converts env_*xml files to shell environment variable files that are then sourced for inclusion in the model environment. Used by the ccsm_getenv script. Like in CESM1.0, CESM1.1 cases are customized, built and run largely through setting what CESM1 calls "environment variables". These actually appear to the user as variables defined in xml files. Those files appear in the case directory once a case is created and are named something like env_*.xml. They are converted to actual environment variables via a csh script called ccsm_getenv. That script calls a perl script called xml2env that converts the xml files to shell files that are then sourced and removed. The ccsm_getenv and xml2env exist in the $CASEROOT/Tools directory. The environment variables are specified in xml files to support extra automated error checking and automatic generation of env variable documentation. If you want to have the ccsm environment variables in your local shell environment, do the following > cd $CASEROOT > source ./Tools/ccsm_getenv You must run the ccsm_getenv from the CASEROOT directory exactly as shown above. There are multiple env_*.xml files including &env_case.xml;, &env_mach_pes.xml;, &env_build.xml;, and &env_run.xml;. To a large degree, the different env files exist so variables can be locked in different phases of the case setup, build, and run process. For more info on locking files, see . The important point is that env_case.xml variables cannot be changed after create_newcase is invoked. &env_mach_pes.xml; cannot be changed after &cesm_setup; is called unless you plan to invoke the commands cesm_setup -clean, &cesm_setup;. &env_build.xml; variables cannot be changed after the model is built unless you plan to clean and rebuild. env_run.xml variables can be changed anytime. The CESM1 scripting software checks that xml files are not changed when they shouldn't be. CESM recommends using the xmlchange tool to modify env variables. This will decrease the chance that typographical errors will creep into the xml files. Conversion of the xml files to environment variables can fail silently with certain xml format errors. To use xmlchange, do, for instance, > cd $CASEROOT > ./xmlchange -id STOP_OPTION -val nmonths > ./xmlchange -id STOP_N -val 6 which will change the variables STOP_OPTION and STOP_N in the file env_run.xml to the specified values. The xml files can be edited manually, but users should take care not to introduce any formatting errors that could lead to incomplete env settings. If there appear to be problems with the env variables (i.e. if the model doesn't seem to have consistent values compared to what's set in the xml files), then confirm that the env variables are being set properly. There are a couple of ways to do that. First, run the ccsm_getenv script as indicated above and review the output generated by the command "env". The env variables should match the xml settings. Another option is to edit the $CASEROOT/Tools/ccsm_getenv script and comment out the line "rm $i:r". That should leave the shell env files around, and they can then be reviewed. The latter approach should be undone as soon as possible to avoid problems running ccsm_getenv later. CESM recommends using the xmlchange tool to modify env variables. xmlchange supports error checking as part of the implementation. Also, using xmlchange will decrease the chance that typographical errors will creep into the xml files. Conversion of the xml files to environment variables can fail silently with certain xml format errors. To use xmlchange, do, for instance, > cd $CASEROOT > ./xmlchange -id STOP_OPTION -val nmonths > ./xmlchange -id STOP_N -val 6 which will change the variables STOP_OPTION and STOP_N in the file env_run.xml to the specified values. The xml files can be edited manually, but users should take care not to introduce any formatting errors that could lead to incomplete env settings. See also . It's possible that a formatting error has been introduced in the env xml files. This would lead to problems in setting the env variables. If there appear to be problems with the env variables (i.e. if the model doesn't seem to have consistent values compared to what's set in the xml files), then confirm that the env variables are being set properly. There are a couple of ways to do that. First, run the ccsm_getenv script via > cd $CASEROOT > source ./Tools/ccsm_getenv > env and review the output generated by the command "env". The env variables should match the xml settings. Another option is to edit the $CASEROOT/Tools/ccsm_getenv script and comment out the line "rm $i:r". That should leave the shell env files around, and they can then be reviewed. The latter approach should be undone as soon as possible to avoid problems running ccsm_getenv later. In CESM, there are several different $&CASEROOT; xml files. These include env_case.xml, env_mach_pes.xml, env_build.xml, and env_run.xml. These files are organized so variables can be locked during the case setup, build, and run. Locking variables is a feature of CESM that prevents users from changing variables after they have been used in other parts of the scripts system. The variables in env_case.xml are locked when create_newcase is called. The env_mach_pes.xml variables are locked when cesm_setup is called. The env_build.xml variables are locked after the model is built, and the env_run.xml variables are never locked and can be changed anytime. In addition, the Macros file is locked as part of the model build. The $CASEROOT/LockedFiles directory saves copies of the xml files to facilitate the locking feature. In summary: &env_case.xml; variables are locked upon invoking &create_newcase; and cannot be unlocked. To change settings in env_case.xml, a new case has to be generated with create_newcase. &env_mach_pes.xml; variables are locked after running cesm_setup. After changing variable values in this file, you need to invoked cesm_setup -clean and then cesm_setup. Macros and &env_build.xml; are locked upon the successful completion of $CASE.build. Both Macros and &env_build.xml; can be unlocked by invoking $CASE.cleanbuild and then the model should be rebuilt. See or the use case . The parallel IO (PIO) library is included with CESM1 and is automatically built as part of the CESM build. Several CESM1 components use the PIO library to read and/or write data. The PIO library is a set of interfaces that support serial netcdf, parallel netcdf, or binary IO transparently. The implementation allows users to easily modify the pio setup on the fly to change the method (serial netcdf, parallel netcdf, or binary data) as well as various parameters associated with PIO to optimize IO performance. CESM1 prefers that data be written in CF compliant netcdf format to a single file that is independent of all parallel decomposition information. Historically, data was written by gathering global arrays on a root processor and then writing the data from the root processor to an external file using serial netcdf. The reverse process (read and scatter) was done for reading data. This method is relatively robust but is not memory scalable, performance scalable, or performance flexible. PIO works as follows. The PIO library is initialized and information is provided about the method (serial netcdf, parallel netcdf, or binary data), and the number of desired IO processors and their layout. The IO parameters define the set of processors that are involved in the IO. This can be as few as one and as many as all processors. The data, data name and data decomposition are also provided to PIO. Data is written through the PIO interface in the model specific decomposition. Inside PIO, the data is rearranged into a "stride 1" decomposition on the IO processors and the data is then written serially using netcdf or in parallel using pnetcdf. There are several benefits associated with using PIO. First, even with serial netcdf, the memory use can be significantly decreased because the global arrays are decomposed across the IO processors and written in chunks serially. This is critical as CESM1 runs at higher resolutions where global arrays need to be minimized due to memory availability. Second, pnetcdf can be turned on transparently potentially improving the IO performance. Third, PIO parameters such as the number of IO tasks and their layout can be tuned to reduce memory and optimize performance on a machine by machine basis. Fourth, the standard global gather and write or read and global scatter can be recovered by setting the number of io tasks to 1 and using serial netcdf. CESM1 uses the serial netcdf implementation of PIO and pnetcdf is turned off in PIO by default. Several components provide namelist inputs that allow use of pnetcdf in PIO. To use pnetcdf, a pnetcdf library (like netcdf) must be available on the local machine and PIO pnetcdf support must be turned on when PIO is built. This is done as follows Locate the local copy of pnetcdf. It must be version 1.1.1 or library Set LIB_PNETCDF in the Macros file to the directory of the pnetcdf library (ie. /contrib/pnetcdf1.1.1/lib). Add PNETCDF_PIO to the pio CONFIG_ARGS variable in the Macros file, and set it to the directory of the top level of a standard pnetcdf installation (ie /contrib/pnetcdf1.1.1). Run the clean_build script if the model has already been built. Run the build script to rebuilt pio and the full CESM1 system. Change component IO namelist settings to pnetcdf and set appropriate IO tasks and layout. The PNETCDF_PIO variable tells pio to build with pnetcdf support turned on. The LIB_PNETCDF variable tells the CESM Makefile to link in the pnetcdf library at the link step of the CESM1 build. There is an ongoing effort between CESM, pio developers, pnetcdf developers and hardware vendors to understand and improve the IO performance in the various library layers. To learn more about pio, see http://code.google.com/p/parallelio. See Additional implementation details are provided in the the CESM coupler user guide about sequencing, parallel IO, performance, grids, threading, budgets, and other items. Several compsets are hardwired in the CESM1 release. "create_newcase -l" provides a current listing of supported "out-of-the-box" compsets. To create a customized compset, > cd $CCSMROOT/scripts Now copy sample_compset_file.xml to another file, e.g. my_compset.xml. > cp sample_compset_file.xml my_compset.xml Edit the file, my_compset.xml, to create your own compset configuration. In particular, the NAME, SHORTNAME, DESC, and COMP_ variables should be specified. The STATUS and CCSM_CCOST variables can be ignored. Note: Other CESM env variables can also be added here. See scripts/ccsm_utils/Case.template/config_compsets.xml for other variables that might be related to compset configuration. Next run create_newcase with the optional -compset_file argument. > create_newcase -case mycase -res f19_g16 -compset MYCS -mach mymach -compset_file my_compset.xml The case mycase should have been generated and the configuration should be consistent with settings from the my_compset.xml file. In general, the only supported calendar in the CESM time manager is the 365 day or no-leap calendar. This calendar has the standard 12 months, but it has 365 days every year and 28 days in every February. Monthly averages in CESM are truly computed over varying number of days depending on the month of the year. A gregorian calendar is available if the ESMF time manager is used. To use the ESMF time manager, the ESMF library has to be available locally and the ESMF library has to be turned on in CESM. See using ESMF in CESM for more information about setting up CESM with the ESMF library. Support for specific components (ie. cam, pop, cice, clm, datm, etc) is hardwired into the scripts. To add a new component model, specifically a new atmosphere, land, ocean, or sea ice component, several things need to be done. This section only summarizes the tasks to complete without providing details. This is an advanced exercise, and CESM may be able to provide addition assistance if resources are available. In the directions below, the model "cxyz" is a new land model that is going to be added. There are two major parts. First, the component needs to be supported in the CESM scripts. Second, the component needs to be able to run under the CESM driver. Add the new model under the appropriate models/"component" directory. For instance add the cxyz model under models/lnd/cxyz. Add scripts of the form buildnml.csh, buildexe.csh and add a template file to the new model under the bld directory. For instance, models/lnd/cxyz/bld/cxyz.buildnml.csh, models/lnd/cxyz/bld/cxyz.buildexe.csh and models/lnd/cxyz/bld/cxyz.cpl7.template. Use another models' files as a starting point. Edit scripts/create_newcase. Add support for the new model under the definition of "my @comps". Add the new model as a valid option in the scripts/ccsm_utils/Case.template/config_definition.xml file for the specific component. For instance, for COMP_LND, add cxyz as a valid option. Add any new env variables that are needed specifically for that component and create a new group for them called, for instance, "conf_cxyz". Add the new model to the list of components in scripts/ccsm_utils/Case.template/ConfigCase.pm if any component specific env variables are added to the config_definition.xml file. Add a new compset that supports the new component to scripts/ccsm_utils/Case.template/config_compsets.xml. For instance, add a new compset called IZ that is based on the I_2000 compset but has cxyz instead of clm as the land component. Review the generated Macros files in $CASEROOT to provide any compiler modifications specifically for the new component. This might be a step that is done after further testing. Add support for output files for the new component in the scripts/ccsm_utils/Tools/st_archive.sh script. This might be done after the new component is in "production". For the new component to run under the CESM driver, a new top level *_comp_mct.F90 file needs to be added to the component. For instance, a lnd_comp_mct.F90 file should exist in the cxyz model source code. This file will provide the init, run, and final interfaces that the CESM driver require. To generate one of these for the new component, copy an existing one from another component and modify it to run with the new component. There are several inherent requirements that make this work such as the driver provides the mpi communicator for the component at initialization. The component must save and use this mpi communicator internally. The top level "program" file for the new component should be disabled. the component must set various parameters at initialization such as the present and prognostic flags, and the nx and ny size values. the component must pass the grid and decomposition to the driver at initialization in the particular format required. the component must unpack and pack data from the coupling datatype at initialization and during runtime. the fields used must be set in the seq_flds_mod.F90 module in the driver. the component must stay synchronized in time with the provided driver time and should abort if time coordination is incorrect. the component must advance the correct amount of time whenever the run method is called. the time is provided to the component by the driver. The pop and cice models both have similar decompositions and strategies for specifying the decomposition. Both models support decomposition of the horizontal grid into two-dimensional blocks, and these blocks are then allocated to individual processors inside each component. The decomposition must be specified when the models are built. There are four environment variables in env_build.xml for each model that specify the decomposition used. These variables are POP or CICE followed by _BLCKX, _BLCKY, _MXBLCKS, and _DECOMP. BLCKX and BLCKY specify the size of the local block in grid cells in the "x" and "y" direction. MXBLCKS specifies the maximum number of blocks that might be on any given processor, and DECOMP specifies the strategy for laying out the blocks on processors. The values for these environment variables are set automatically by the scripts in the $CASEROOT/Buildconf directory whenever the model is built or run is run. The scripts that generate the decompositions are CASEROOT/Buildconf/generate_pop_decomp.pl and $CASEROOT/Buildconf/generate_cice_decomp.pl. Those tools leverage decompositions stored in xml files, $CCSMROOT/models/ocn/pop2/bld/pop_decomp.xml and $CCSMROOT/models/ice/cice/bld/cice_decomp.xml, respectively. These utilities set the decomposition for a given resolution and total processor count. The decomposition used can have a significant effect on the model performance, and the decompositions specified by the tools above generally provide optimum or near optimum values for the given resolution and processor count. More information about cice and pop decompositions can be found in each of those user guides. The decompositions can be specified manually by setting the environment variable POP_AUTO_DECOMP or CICE_AUTO_DECOMP to false in env_build.xml (which turns off use of the scripts above) and then setting the four BLCKX, BLCKY, MXBLCKS, and DECOMP environment variables in env_build.xml. In general, relatively square and evenly divided Cartesian decompositions work well for pop at low to moderate resolution. Cice performs best with "tall and narrow" blocks because of the load imbalance for most global grids between the low and high latitudes. At high resolutions, more than one block per processor can result in land block elimination and non-Cartesian decompositions sometimes perform better. Testing of several decompositions is always recommended for performance and validation before a long run is started. If you want to change the frequency of output for CAM or CLM (i.e. generate output every 6 model hours instead of once a model day) in the middle of a run, or if you want to change the fields that are output, in the middle of a run, you need to stop the run, rebuild and rerun it with the same casename and branch from the same casename. See the steps below for doing a branch run while retaining the casename. Rebuilding the case and restarting it where you left off, are necessary because CAM and CLM only read namelist variables once, at the beginning of a run. This is not the case for POP and CICE, they read the namelist input on every restart, and therefore for POP and CICE, you can change output fields and frequency by modifying the appropriate namelist variables and then doing a restart. The following example shows case B40.20th.1deg which runs from 1850 to 2005, and will generate high frequency output for years 1950 through 2005. CAM will output data every six hours instead of once a day. Also starting at year 1950 additional fields will be output by the model. The first step is to create case b40.20th.1deg and run the case for years 1850 through 1949 with your initial settings for output. Next move your entire case directory, $CASEDIR, somewhere else, because you need to rebuild and rerun the case using the same name. > cd $CASEDIR > mv b40.20th.1deg b40.20th.1deg.1850-1949 Now move your run directory, $RUNDIR, somewhere else as well. > cd $RUNDIR > mv b40.20th.1deg b40.20th.1deg.1850-1949 Next create a new case in your case directory with the same name, b40.20th.1deg. > cd $CASEDIR/scripts > create_newcase -mach bluefire -compset B_1850-2000_CN -res f09_g16 -case b40.20th.1deg cd $RUNDIR Next invoke the following commands > cd $CASEROOT > xmlchange -id RUN_TYPE -val 'branch' > xmlchange -id RUN_REFCASE -val 'b40.20th.1deg' > xmlchange -id RUN_REFDATE -val '1948-01-01' > xmlchange -id CAM_NML_USE_CASE -val '1850-2005_cam4' > xmlchange -id BRNCH_RETAIN_CASENAME -val 'TRUE' > xmlchange -id GET_REFCASE -val 'FALSE' Next set up the case and edit the coupler and CAM namelists. Set up the case. > ./cesm_setup -case Edit user_nl_cpl. Add the following to the end of the file. brnch_retain_casename = .true. Edit user_nl_cam. Check that bndtvghg = '$DIN_LOC_ROOT' and add the following to the end of the file doisccp = .true. isccpdata = '/fis/cgd/cseg/csm/inputdata/atm/cam/rad/isccp.tautab_invtau.nc' mfilt = 1,365,30,120,240 nhtfrq = 0,-24,-24,-6,-3 fincl2 = 'TREFHTMN','TREFHTMX','TREFHT','PRECC','PRECL','PSL' fincl3 = 'CLDICE','CLDLIQ','CLDTOT','CLOUD','CMFMC','CMFMCDZM','FISCCP1', 'FLDS','FLDSC','FLNS','FLUT','FLUTC','FSDS','FSDSC','FSNS', 'FSNSC','FSNTOA','FSNTOAC','LHFLX','OMEGA','OMEGA500', 'PRECSC','PRECSL','PS','Q','QREFHT','RELHUM','RHREFHT','SHFLX', 'SOLIN','T','TGCLDIWP','TGCLDLWP','U','V','Z3' fincl4 = 'PS:I','PSL:I','Q:I','T:I','U:I','V:I','Z3:I' fincl5 = 'CLDTOT','FLDS','FLDSC','FLNS','FLNSC','FSDS','FSDSC','FSNS', 'LHFLX','PRECC','PRECL','PRECSC','PRECSL','SHFLX', 'PS:I','QREFHT:I','TREFHT:I','TS:I' / Now build and run the case. > b40.20th.1deg.build > bsub < b40.20th.1deg.run TODO: Fill this in TODO: Fill this in