The CESM1 driver manages the main clock in the system. That clock advances at the shortest coupling period and uses alarms to trigger component coupling and other events. In addition, the driver maintains a clock that is associated with each component. The driver's component clocks have a timestep associated with the coupling period of that component. The main driver clock and the component clocks in the driver advance in a coordinated manor and are always synchronized. The advancement of time is managed as follows in the main run loop. First, the main driver clock advances one timestep and the component clocks are advanced in a synchronous fashion. The clock time represents the time at the end of the next model timestep. Alarms may be triggered at that timestep to call the the atmosphere, land, sea ice, land ice, or ocean run methods. If a component run alarm is triggered, the run method is called and the driver passes that component's clock to that component. The component clock contains information about the length of the next component integration and the expected time of the component at the end of the integration period. Generally, the component models have indepedent time management software. When a component run method is called, the component must advance the proper period and also check that their internal clock is consistent with the coupling clock before returning to the driver. The clock passed to the component by the driver contains this information. Component models are also responsible for making sure the coupling period is consistent with their internal timestep. History files are managed independently by each component, but restart files are coordinated by the driver (see ). The clocks in CESM1 are based on the ESMF clock datatype are are supported in software by either an official ESMF library or by software included in CESM called esmf_wrf_timemgr. The esmf_wrf_timemgr software is a much simplified Fortran implementation of a subset of the ESMF time manager interfaces. The driver time loop is hardwired to sequence the component models in a specific way to meet scientific requirements and to otherwise provide the maximum amount of potential concurrency of work. The results of the model integration are not dependent on the processor layout of the components. In addition, the driver is currently configured to couple the atmosphere, land, and sea ice models using the same coupling frequency while the ocean model can be coupled at the same or at a lower frequency. To support this feature, the driver does temporal averaging of coupling inputs to the ocean and the driver also computes the surface ocean albedo at the higher coupling frequency. There is no averaging of coupling fields for other component coupling interactions and the land and sea ice models' surface albedos are computed inside those components. Averaging functionality could be added to the driver to support alternative relative coupling schemes in the future if desired with the additional caveat that the interaction between the surface albedo computation in each component and the atmospheric radiation calculation have to be carefully considered. In addition, some other features may need to be extended to support other coupling schemes and still allow model concurrency. The coupler processors (pes) handle the interaction of data between components, so there are separate tasks associated with deriving fields on the coupler pes, transfering data to and from the coupler pes and other components, and then running the component models on their processors. The driver time loop is sequenced as follows, The driver clock is advanced first. Initial data is computed on the coupler pes. Ocean data is rearranged from the coupler to the ocean pes. Land data is rearranged from the coupler to the land pes. Ice data is rearranged from the coupler to the ice pes. The ocean model is run. The ice model is run. The land model is run. The ocean inputs are accumulated, and the atmosphere/ocean fluxes are computed on the coupler pes based on the results from the previous coupled timestep. Land data is rearranged from the land pes to the coupler pes. Ice data is rearranged from the ice pes to the coupler pes. Atmospheric forcing data is computed on the coupler pes. Atmospheric data is rearranged from the coupler pes to the atmosphere pes. The atmosphere model is run. Atmospheric data is rearranged from the atmosphere pes to the coupler pes. Ocean data is rearranged from the ocean pes to the coupler pes. The loop returns Within this loop, as much as possible, coupler work associated with mapping data, merging fields, diagnosing, applying area corrections, and computing fluxes is overlapped with component work. The land ice model interaction has not been discussed in the above context because the coupling sequencing of that component is still being developed, but the land ice model is likely to be coupled at much longer periods that the current components. The driver sequencing in CESM1 has been developed over nearly two decades, and it plays a critical role in conserving mass and heat, minimizing lags, and providing stability in the system. The above description is consistent with the concurrency limitations described here. Just to reiterate, the land and sea ice models will always run before the atmospheric model, and the coupler and ocean models are able to run concurrently with all other components. The coupling between the atmosphere, land, sea ice, and atmosphere/ocean flux computation incurs no lags but the coupling to the ocean state is lagged by one ocean coupling period in the system. Mass and heat are conserved in the system with more description here. It is possible to reduce the ocean lag in the system. There is a namelist variable, ocean_tight_coupling, that moves the step where ocean data is rearranged from the ocean pes to the coupler pes from the end of the loop to before the atmosphere/ocean flux computation. If ocean_tight_coupling is set to true, then the ocean lag is reduced by one atmosphere coupling period, but the ability of the ocean model to run concurrently with the atmosphere model is also reduced or eliminated. This flag is most useful when the ocean coupling frequency matches the other components. In the current implementation, the coupling period must be identical for the atmosphere, sea ice, and land components. The ocean coupling period can be the same or greater. All coupling periods must be multiple integers of the smallest coupling period and should evenly divide a day of time. The coupling periods are set in the driver namelist for each component via variables called something like atm_cpl_dt and atm_cpl_offset. The units of these inputs are seconds. The coupler template file derives these values from CESM1 script variable names like ATM_NCPL which is the coupling frequency per day. The *_cpl_dt input specifies the coupling period in seconds and the *_cpl_offset input specifies the temporal offset of the coupling time relative to initial time. An example of an offset might be a component that couples every six hours. That would normally be on the 6th, 12th, 18th, and 24th hour of every day. An offset of 3600 seconds would change the coupling to the 1st, 7th, 13th, and 19th hour of every day. The offsets cannot be larger than the coupling period and the sign of the offsets is such that a positive offset shifts the alarm time forward by that number of seconds. The offsets are of limited use right now because of the limitations of the relative coupling frequencies. Offsets play an important role in supporting concurrency. There is an offset of the smallest coupling period automatically introduced in every coupling run alarm for each component clock. This is only mentioned because it is an important but subtle point of the implementation and changing the coupling offset could have an impact on concurrency performance. Without this explicit automatic offset, the component run alarms would trigger at the end of the coupling period. This is fine for components that are running at the shortest coupling period, but will limit the ability of models to run concurrently for models that couple at longer periods. What is really required for concurrency is that the run alarm be triggered as early as possible and that the data not be copied from that component to the coupler pes until the coupling period has ended. The detailed implementation of this feature is documented in the seq_timemgr_mod.F90 file and the impact of it for the ocean coupling is implemented in the ccsm_driver.F90 code via use of the ocnrun_alarm and ocnnext_alarm variables. The standard implementation for grids in CESM has been that the atmosphere and land models are run on identical grids and the ocean and sea ice model are run on identical grids. The ocean model mask is used to derive a complementary mask for the land grid such that for any given combination of atmosphere/land and ocean/ice grids, there is a unique land mask. This approach for dealing with grids is still used a majority of the time in CESM1. But there is a new capability, called trigrid, that allows the atmosphere and land grids to be unique. A typical grid in CESM1 is named something like 1.9x2.5_gx1v6 which is the finite volume "2 degree" atmosphere/land grid matched with the gx1v6 "1 degree" ocean/ice grid. This also has a shortname of f19_g16. The "out-of-the-box" supported grids, compsets, and machines in CESM1 are generated automatically by running create_newcase -list from the scripts directory. Historically, the ocean grid has been the higher resolution grid in CESM. While that is no longer always the case, the current implementation largely reflects that presumption. The atmosphere/ocean fluxes in the coupler are computed on the ocean grid. A new namelist input which is not yet validated called aoflux_grid will allow the user to specify the atmosphere/ocean flux computation grid in the coupler in the future. In addition, the mapping approach used in CESM1 also reflects the presumption that the ocean is generally higher resolution. Fluxes are always mapped using a locally conservative area average methods to preserve conservation. However, states are mapped using bilinear interpolation from the atmosphere grid to the ocean grid to better preserve gradients, while they are mapped using a locally conservative area average approach from the ocean grid to the atmosphere grid. These choices are based on the presumption that the ocean grid is higher resolution. There has always been an option that all grids (atmosphere, land, ocean, and ice) could be identical, and this is still supported. There are a couple of namelist variables, samegrid_ao, samegrid_al, and samegrid_ro that tell the coupler whether to expect that the following grids; atmosphere/ocean, atmosphere/land, and runoff/ocean respectively are identical. These are set automaticaly in the driver namelist depending on the grid chosen and impact mapping as well as domain checking. A new feature in CESM1 allows the atmosphere and land grids to be unique. One example in the CESM1 scripts is the ne30np4_1.9x2.5_gx1v6 (ne30_f19_g16) grid. This grid configuration consists of the ne30np4 homme grid for the atmosphere, the "2 degree" grid for the land, and the "1 degree" grid for the ocean/ice. Note that the trigrid implementation has undergone a reasonable amount of testing, but as of April, 2010, it is still considered a relatively new feature and continues to undergo careful scrutiny. The trigrid implementation introduces an ambiguity in the definition of the mask. This ambiguity is associated with an inability to define an absolutely consistent ocean/land mask across all grids in the system. A decision was made in CESM1 to support the trigrid as follows. The land mask is defined on the atmosphere grid as the complement of the ocean mask mapped conservatively to the atmosphere grid. Then the land and ocean masks are exactly complementary on the atmosphere grid where conservative merging are critical. No precise land fraction needs to be defined in the land grid. The only requirement is that the land model compute data on a masked grid such that when mapped to the atmosphere grid, all atmosphere grid points that contain some fraction of land have valid values computed in the land model. There are an infinite number of land fraction masks that can accomplish this including a fraction field that is exactly one at every grid cell. In the land model, all land fraction masks produce internally conservative results. So there is no problem with conservation in the system with this approach with one caveat. The runoff model in the clm active land model is run on a unique half degree grid. In order to interpolate the land forcing data for the runoff grid, a land fraction mask that is consistent with the ocean mask in the system is required. To support the land to runoff mapping in clm, the land fraction on the atmosphere grid is conservatively mapped to the land grid in the coupler and sent to the land model for use. This field can then be used with the land fraction field from within clm to generated properly scaled runoff forcing which is also conservative. The component grid fractions in the coupler are defined and computed in CESM1 in models/drv/driver/seq_frac_mct.F90. A slightly modified version of the notes from this file is pasted below. Just to clarify some of the terms. fractions_a, fractions_l, fractions_i, and fractions_o are the fractions on the atmosphere, land, ice, and ocean grids. afrac, lfrac, ifrac, and ofrac are the atmosphere, land, ice, and ocean fractions on those grids. so fractions_a(lfrac) is the land fraction on the atmosphere grid. lfrin in the land fraction defined in the land model. This can be different from lfrac because of the trigrid implementation. lfrac is the land fraction consistent with the ocean mask and lfrin is the land fraction in the land model. ifrad and ofrad are fractions at the last radiation timestep. These fractions preserve conservation of heat in the net shortwave calculation because the net shortwave calculation is one timestep behind the ice fraction evolution in the system. When the variable "dom" is mentioned below, that refers to a field sent from a component at initialization. ! the fractions fields are now afrac, ifrac, ofrac, lfrac, and lfrin. ! afrac = fraction of atm on a grid ! lfrac = fraction of lnd on a grid ! ifrac = fraction of ice on a grid ! ofrac = fraction of ocn on a grid ! lfrin = land fraction defined by the land model ! ifrad = fraction of ocn on a grid at last radiation time ! ofrad = fraction of ice on a grid at last radiation time ! afrac, lfrac, ifrac, and ofrac are the self-consistent values in the ! system. lfrin is the fraction on the land grid and is allowed to ! vary from the self-consistent value as descibed below. ifrad ! and ofrad are needed for the swnet calculation. ! the fractions fields are defined for each grid in the fraction bundles as ! needed as follows. ! character(*),parameter :: fraclist_a = 'afrac:ifrac:ofrac:lfrac:lfrin' ! character(*),parameter :: fraclist_o = 'afrac:ifrac:ofrac:ifrad:ofrad' ! character(*),parameter :: fraclist_i = 'afrac:ifrac:ofrac' ! character(*),parameter :: fraclist_l = 'afrac:lfrac:lfrin' ! character(*),parameter :: fraclist_g = 'gfrac' ! ! we assume ocean and ice are on the same grids, same masks ! we assume ocn2atm and ice2atm are masked maps ! we assume lnd2atm is a global map ! we assume that the ice fraction evolves in time but that ! the land model fraction does not. the ocean fraction then ! is just the complement of the ice fraction over the region ! of the ocean/ice mask. ! we assume that component domains are filled with the total ! potential mask/fraction on that grid, but that the fractions ! sent at run time are always the relative fraction covered. ! for example, if an atm cell can be up to 50% covered in ! ice and 50% land, then the ice domain should have a fraction ! value of 0.5 at that grid cell. at run time though, the ice ! fraction will be between 0.0 and 1.0 meaning that grid cells ! is covered with between 0.0 and 0.5 by ice. the "relative" fractions ! sent at run-time are corrected by the model to be total fractions ! such that ! in general, on every grid, ! fractions_*(afrac) = 1.0 ! fractions_*(ifrac) + fractions_*(ofrac) + fractions_*(lfrac) = 1.0 ! where fractions_* are a bundle of fractions on a particular grid and ! *frac (ie afrac) is the fraction of a particular component in the bundle. ! ! the fractions are computed fundamentally as follows (although the ! detailed implementation might be slightly different) ! initialization (frac_init): ! afrac is set on all grids ! fractions_a(afrac) = 1.0 ! fractions_o(afrac) = mapa2o(fractions_a(afrac)) ! fractions_i(afrac) = mapa2i(fractions_a(afrac)) ! fractions_l(afrac) = mapa2l(fractions_a(afrac)) ! initially assume ifrac on all grids is zero ! fractions_*(ifrac) = 0.0 ! fractions/masks provided by surface components ! fractions_o(ofrac) = dom_o(frac) ! ocean "mask" ! fractions_l(lfrin) = dom_l(frac) ! land model fraction ! then mapped to the atm model ! fractions_a(ofrac) = mapo2a(fractions_o(ofrac)) ! fractions_a(lfrin) = mapl2a(fractions_l(lfrin)) ! and a few things are then derived ! fractions_a(lfrac) = 1.0 - fractions_a(ofrac) ! this is truncated to zero for very small values (< 0.001) ! to attempt to preserve non-land gridcells. ! fractions_l(lfrac) = mapa2l(fractions_a(lfrac)) ! one final term is computed ! dom_a(ascale) = fractions_a(lfrac)/fractions_a(lfrin) ! dom_l(ascale) = mapa2l(dom_a(ascale)) ! these are used to correct land fluxes in budgets and lnd2rtm coupling ! and are particularly important when the land model is running on ! a different grid than the atm model. in the old system, this term ! was treated as effectively 1.0 since there was always a check that ! fractions_a(lfrac) ~ fractions_a(lfrin), namely that the land model ! provided a land frac that complemented the ocean grid. this is ! no longer a requirement in this new system and as a result, the ! ascale term can be thought of as a rescaling of the land fractions ! in the land model to be exactly complementary to the ocean model ! on whatever grid it may be running. ! run-time (frac_set): ! update fractions on ice grid ! fractions_i(ifrac) = i2x_i(Si_ifrac) ! ice frac from ice model ! fractions_i(ofrac) = 1.0 - fractions_i(ifrac) ! note: the relative fractions are corrected to total fractions ! fractions_o(ifrac) = mapi2o(fractions_i(ifrac)) ! fractions_o(ofrac) = mapi2o(fractions_i(ofrac)) ! fractions_a(ifrac) = mapi2a(fractions_i(ifrac)) ! fractions_a(ofrac) = mapi2a(fractions_i(ofrac)) ! ! fractions used in merging are as follows ! mrg_x2a uses fractions_a(lfrac,ofrac,ifrac) ! mrg_x2o needs to use fractions_o(ofrac,ifrac) normalized to one ! normalization happens in mrg routine ! ! fraction corrections in mapping are as follows ! mapo2a uses *fractions_o(ofrac) and /fractions_a(ofrac) ! mapi2a uses *fractions_i(ifrac) and /fractions_a(ifrac) ! mapl2a uses *fractions_l(lfrin) and /fractions_a(lfrin) ! mapa2* should use *fractions_a(afrac) and /fractions_*(afrac) but this ! has been defered since the ratio always close to 1.0 ! ! budgets use the standard afrac, ofrac, ifrac, and lfrac to compute ! quantities except in the land budget which uses lfrin multiplied ! by the scale factor, dom_l(ascale) to compute budgets. ! ! fraction and domain checks ! initialization: ! dom_i = mapo2i(dom_o) ! lat, lon, mask, area ! where fractions_a(lfrac) > 0.0, fractions_a(lfrin) is also > 0.0 ! this ensures the land will provide data everywhere the atm needs it ! and allows the land frac to be subtlely different from the ! land fraction specified in the atm. ! dom_a = mapl2a(dom_l) ! if atm/lnd same grids ! dom_a = mapo2a(dom_o) ! if atm/ocn same grids ! dom_a = mapi2a(dom_i) ! if atm/ocn same grids ! 0.0-eps < fractions_*(*) < 1.0+eps ! fractions_l(lfrin) = fractions_l(lfrac) ! only if atm/lnd same grids (but this is not formally required) ! this is needed until dom_l(ascale) is sent to the land model ! as an additional field for use in l2r mapping. ! run time: ! fractions_a(lfrac) + fractions_a(ofrac) + fractions_a(ifrac) ~ 1.0 ! 0.0-eps < fractions_*(*) < 1.0+eps Domain checking is a very important initialization step in the system. The domain checking verifies that the longitudes, latitudes, areas, masks, and fractions of different grids are consistent with each other in a way that is required by the CESM1 implementation. The subroutine that carries out domain checking is in models/drv/driver/seq_domain_mct.F90 and is called seq_domain_check_mct. Tolerances for checking the domains can be set in the drv_in driver namelist via the namelist variables, eps_frac, eps_amask, eps_agrid, eps_aarea, eps_omask, eps_ogrid, and eps_oarea. These values are derived in the coupler namelist from the script env variables, EPS_FRAC, EPS_AMASK, EPS_AGRID, EPS_AAREA, EPS_OMASK, EPS_OGRID, and EPS_OAREA in the env_run.xml file. If an error is detected in the domain checking, the model will write an error message and abort. The domain checking is dependent on the grids and in particular, the samegrid input namelist settings. But it basically does the following, ocean/ice grid comparison: - verifies the grids are the same size - verifies the difference in longitudes and latitudes is less than eps_ogrid. - verifies the difference in masks is less than eps_omask - verifies the difference in areas is less than eps_oarea atmosphere/land grid comparison (if samegrid_al): - verifies the grids are the same size - verifies the difference in longitudes and latitudes is less than eps_agrid. - verifies the difference in masks is less than eps_amask - verifies the difference in areas is less than eps_aarea atmosphere/ocean grid comparison (if samegrid_ao): - verifies the grids are the same size - verifies the difference in longitudes and latitudes is less than eps_agrid. - verifies the difference in masks is less than eps_amask - verifies the difference in areas is less than eps_aarea fractions - verifies that the land fraction on the atmosphere grid and the ocean fraction on the atmosphere grid add to one within a tolerance of eps_frac. There are a number of subtle aspects in the domain checking like whether to check over masked grid cells, but these issues are less important than recognizing that errors in the domain checking should be treated seriously. It is easy to make the errors go away by changing the tolerances, but it is also easy to overlook a critical grid error that can impact conservation and consistency in a simulation. Mapping files to support interpolation of fields between grids are computed offline. General, this is done using the SCRIP package, but any package that generates a mapping file of valid format can be used in CESM. Several different mapping approaches are used in CESM1. First, note that historically, the ocean grid has been the higher resolution grid in CESM. While that is no longer always the case, the current implementation largely reflects that presumption. In general, mapping of fluxes is done using a locally conservative area average approach to preserve conservation. State fields are generally mapped using bilinear interpolation from the atmosphere grid to the ocean grid to better preserve gradients, but state fields are generally mapped using the conservative area average approach from the ocean grid to the atmosphere grid. But this is not a requirement of the system. The individual state and flux mapping files are specified at runtime using the seq_maps.rc input file, and any valid mapping file using any mapping approach can be specified in that input file. The seq_maps.rc file contains information about the mapping files as well as the mapping type. There are currently two types of mapping implementations, "X" and "Y". The "X" mapping rearranges the source data to the destination grid decomposition and then a local mapping is done from the source to the destination grid on the destination decomposition. The "Y" mapping does a local mapping from the source grid to the destination grid on the source grid decomposition. That generates a partial sum of the destination values which are then rearranged to the destination decomposition and summed. Both options produce reasonable results, although they may differ in value by "roundoff" due to differences in order or operations. The type chosen impacts performance. In both implementations, the number of flops is basically identical. The difference is the communication. In the "X" type, the source grid is rearranged. In the "Y" type, the destination grid is rearranged. Since historically, the ocean grid is higher resolution than the atmosphere grid, "X" mapping is used for atmosphere to ocean/ice mapping and "Y" mapping is used from ocean/ice to atmosphere mapping to maximize mapping performance. Mapping corrections are made in some cases in the polar region. In particular, the current bilinear and area conservative mapping approaches introduce relatively large errors in mapping vector fields around the pole. The current coupler can correct the interpolated surface wind velocity near the pole when mapping from the atmosphere to the ocean and ice grids. There are several options that correct the vector mapping and these are set in the env variable VECT_MAP. The npfix option only affects ocean and ice grid cells that are northward of the last latitude line of the atmospheric grid. The algorithm is contained in the file models/drv/driver/map_atmocn_mct.F90 and is only valid when the atmosphere grid is a longitude/latitude grid. This feature is generally on by default. The other alternative is the cart3d which converts the surface u and v velocity to 3d x,y,z vectors then maps those three vectors before coverting back to u and v east and north directions on the surface. Both vector mapping methods introduce errors of different degrees but are generally much better than just mapping vector fields as if they were individual scalars. The vector mapping namelist input is set in the drv_in file. The input mapping files are assumed to be valid for grids with masks of value zero or one where grids points with a mask of zero are never considered in the mapping. Well defined, locally conservative area mapping files as well as bilinear mapping files can be generated using this masked approach. However, there is another issue which is that a grid fraction in an active cell might actually change over time. This is not the case for land fraction in CESM, but it is the case for relative ice and ocean fractions in CESM. The ice fraction is constantly evolving in the system in general. To improve the accuracy of the ice and ocean mapping, the ocean/ice fields are scaled by the local fraction before mapping and unscaled by the mapped fraction after mapping. The easiest way to demonstate this is via an example. Consider a case where two ice cells of equal area underlie a single atmosphere cell completely. The mapping weight of each ice cell generated offline would be 0.5 in this case and if ice temperatures of -1.0 and -2.0 in the two cells respectively were mapped to the atmosphere grid, a resulting ice temperature on the atmosphere grid of -1.5 would result. Consider the case where one cell has an ice fraction of 0.3 and the other has a fraction of 0.5. Mapping the ice fraction to the atmospheric cell results in a value of 0.4. If the same temperatures are mapped in the same way, a temperature of -1.5 results which is reasonable, but not entirely accurate. Because of the relative ice fractions, the weight of the second cell should be greater than the weight of the first cell. Taking this into account properly results in a fraction weighted ice temperature of -1.625 in this example. This is the fraction correction that is carried out whenever ocean and ice fields are mapped to the atmosphere grid. Time varying fraction corrections are not required in other mappings to improve accuracy because their relative fractions remain static. To improve conservation in the system, all fluxes sent to and received from components are corrected for the area differences between the components. There are many reasonable ways to compute an area of a grid cell, but they are not generally consistent. One assumption with respect to conservation of fluxes is that the area acting upon the flux is well defined. Differences in area calculations can result in differences of areas up to a few percent and if these are not corrected, will impact overall mass and heat conservation. In CESM1, areas are extracted for each grid from the mapping files. In this implementation, it is assumed that the areas in all mapping files are computed reasonably and consistently for each grid and on different grids. Those mapping areas are used to correct the fluxes for each component by scaling the fluxes sent to and received by the component by the ratio of the mapping area and the component area. The areas from the components are provided to the coupler by the component at initialization. The minimum and maximum value of each area corrections is written to the coupler log file at initialization. One critical point is that if mapping files are generated by different tools offline and used in CESM, an error could be introduced that is related to inconsistent areas provided by different mapping files. The CESM initialization has been developed over the last two decades to meet the scientific goals, minimize the communication required, and ensure a consistent and well defined climate system. The order of operations is critical. The CESM1 initialization is basically as follows, The ccsm_pes namelist is read and mpi communicators are initialized. The seq_infodata namelist is read and configuration settings are established. The prof_inparm namelist is read and the timing tool is initialized. The pio_inparm namelist is read and the driver IO is initialized. The seq_timemgr namelist is read and the driver time manager and clocks are initialized. The atmosphere init routine is called, the mpi communicator and clock are sent, and the atmosphere grid is returned. The land init routine is called, the mpi communicator and clock are sent, and the land grid is returned. The ocean init routine is called, the mpi communicator and clock are sent, and the ocean grid is returned. The ice init routine is called, the mpi communicator and clock are sent, and the ice grid is returned. The land ice init routine is called, the mpi communicator and clock are sent, and the land ice grid is returned. The infodata buffer is synchronized across all processors. This buffer contains many model configuration settings set by the driver but also sent from the components. The mapping areas are initialized to the component areas on the component pes. The atmosphere, land, runoff, ice, land ice, and ocean rearrangers are initialized. These rearrangers move component data between the component pes and the coupler pes. Initialize the coupler to component fields and zero them out. The ocean accumulator is initialized and zeroed out. This is used to accumulate and average the ocean input. Datatypes associated with merging are allocated. The atm2ocn, ocn2atm, ocn2ice, ice2ocn, ice2atm, rof2ocn, atm2lnd, and lnd2atm mapping weights and areas are read. Component grids are checked using the domain checking method. The flux area corrections are initialized on the component pes and applied to the initial fields sent by each component on the component pes. Those initial fields are then rearranged to the coupler pes. The fractions are initialized on the coupler pes. The atmosphere/ocean flux computation is initialized and initial ocean albedos are computed on the coupler pes. The land, ocean, and ice initial data is mapped to the atmosphere grid and merged to generate initial atmosphere forcing data. This is really only needed for the albedo fields. The initial atmosphere forcing data (albedos) is rearranged from the coupler pes to the atmosphere pes, and the area corrections are applied. The atmosphere init method is called to run the second phase which initializes the atmosphere radiation based on the surface albedos sent. The new atmosphere initial data is area corrected and rearranged to the coupler pes. The budget diagnostics are zeroed out. Initialization is complete. CESM1 supports both CESM designed coupling interfaces based on MCT datatypes and ESMF coupling interfaces based on the ESMF design. In both cases, a top level driver calls init, run, and finalize methods for each gridded component. The primary and default method for running CESM1 is with the MCT based interfaces and an ESMF library is not required in that case. ESMF interfaces are supported via translation methods that are instantiated in the models/drv/shr_esmf directory. These methods translate between the datatypes in the MCT interfaces and the datatypes in the ESMF interfaces. In the current CESM1 ESMF interface implementation, the top level driver still calls the init, run, and finalize component methods using the MCT interfaces. That interface does not match the ESMF interface provided by the gridded components. To facilitate translation, each component provides an additional layer where the MCT interfaces are translated to the ESMF interfaces and data is copied between datatypes on entry and exit of each method. The translation of MCT to ESMF datatypes and vice versa is supported via the shared source code in the models/drv/shr_esmf directory. In the future, the CESM1 driver could be modified and the ESMF interfaces called directly thereby eliminating the extra translation layer. All CESM1 components support both MCT interfaces and ESMF interfaces at the top level independently. This is specifically implemented using two distinct directories of source code to separate the interfaces, so one or the other can be compiled with the component model. These directories exist in all components and are generally called cpl_mct and cpl_esmf. The cpl_mct directory contains the MCT coupling interface. The cpl_esmf directory contains the ESMF coupling interfaces as well as the additional translation layer. At the present time, these interfaces are maintained independently and modifications within any of the interface methods will likely require modifications to the alternative interface for consistency. To use the ESMF interfaces, the ESMF version 4 release library must be installed locally and the USE_ESMF_LIB environment variable must be set to TRUE in the case env_build.xml file. In addition, the COMP_INTERFACE value in env_build.xml must be set to ESMF. When those variables are set, components compile the source code in cpl_esmf instead of cpl_mct and the ESMF library is linked to the CESM executable. Support of the ESMF interfaces is a relatively new feature in CESM1. Results using either interface are bit-for-bit identical for several configurations tested, and validation of the implementation is ongoing. OpenMP thread counts are controlled at three levels in CESM1. The coarsest level is prior to launching the model in the CESM run script. The environment variable OMP_NUM_THREADS is usually set to the largest value any mpi task will use in CESM1. At a minimum, this will ensure threading is turned on to the maximum desired value in the run. The next level is during initialization in CESM1. When the mpi communicators are initialized, the maximum number of threads per mpi task can be computed based on the ccsm_pes namelist input. At that point, there is an initial fortran call to the intrinsic, omp_set_num_threads. When that happens and if that call is successful, the number of threads will be set to the maximum needed in the system on an mpi task by task basis. Finally, there is the ability of CESM to change the thread count per task as each component is individually called and as the model integrates through the driver run loop. In other words, for components that share the same hardware processor but have different threads per task, this feature allows those components to run with the exact value set by the user in the ccsm_pes namelist. This final level of thread control is turned off by default, but it can be turned on using the drv_threading namelist input. This fine control of threading is likely of limited use at this point given the current CESM1 driver implementation. There is a bit-for-bit flag in the CESM drv_in namelist called bfbflag. This flag modifies some coupler computations to preserve bit-for-bit results on different coupler processor counts. This flag has no impact on other components and their ability to generate bit-for-bit results on different pe counts. When this flag is set, all mappings become "X" types where the source data is rearranged to the destination processor and then local mapping is carried out. The order of operations of this mapping is independent of the pe count or decomposition of the grids. The other feature that is changed by the bfbflag is the global sum diagnostics. When the bfbflag is set to false, a partial sum is done on each processors and those partial sums are added together to form a global sum. This is generally not order of operations independent for different pe counts or decompositions. When the bfbflag is set, the global sums are computed by gathering the global field on the root processor and doing an ordered sum there. In addition to log files, component models also produce history and restart files. History files are generally netcdf format and contain fields associated with the state of the model. History files are implemented and controlled independently in the component models, although support for monthly average history files is a standard output of most CESM production runs. CESM has a file naming standard for history files which includes the case names, component name, and model date. All component models in CESM must be able to stop in the middle of a run and then subsequently restart in a bit-for-bit fashion. For most models, this requires the writing of a restart file. The restart file can be any format, although netcdf has become relatively standard, and it should contain any scalars, fields, or information that is required to restart the component model in exactly the same state as when the restart was written and the model was stopped. The expectation in CESM is that a restart of a model run will be bit-for-bit identical and this is regularly tested as part of CESM development by running the model 10 days, writing a restart at the end of 5 days, and then restarting at day 5 and comparing the result with the 10 day run. Unlike history files, restart files must be coordinated across different components. The restart frequency is set in the driver time manager namelist and the driver triggers a restart alarm in clocks when a coordinated restart is requested. The components are required to check this alarm whenever they are called and to write a restart file at the end of the current coupling period. This method ensures all components are writing restart files at a consistent timestamp. The restart filenames are normally set in a generic rpointer file. The rpointer file evolves over the integration and keeps track of the current restart filenames. When a model is restarted, both the rpointer file and the actual restart file are generally required. Many models are also able to restart accumulating history exactly files in the middle of an accumulation period, but this is not a current requirement in CESM1. In production, the model is usually started and stopped on monthly boundaries so monthly average history files are produced cleanly. The run length of a CESM1 production run is usually specified using the nmonths or nyears option and restart files are normally written only at the end of the run. Mass and heat are conserved in the coupler to several digits over centuries. Several steps have been taken to ensure this level of conservation, and these are described in other sections of the document. In addition, efforts have been made to make sure each component is internally conservative with respect to mass and heat. The budgets can be turned on and off using the namelist variable do_budgets. The value of that namelist is set by the env variable, BUDGETS in env_run.xml. By default, the do_budgets flag is false and budgets are not generated. The CESM1 coupler can diagnose the CESM1 budget at several levels and over different periods. The periods are instantenous, daily average, monthly average, annual average, or since the start of the run. The budget output for each of these periods is controlled by the namelist input budget_inst, budget_daily, budget_month, budget_ann, budget_ltann, and budget_ltend. budget_ltann and budget_ltend are used to write the long term budget at either the end of every year or the end of every run. Other budgets are written at their period interval. The namelist input is an integer specifying what to write. The budget flags are controlled by env variables in env_run.xml named BUDGET_INST, BUDGET_DAILY, BUDGET_MONTHLY, BUDGET_ANNUAL, BUDGET_LONGTER_EOY, and BUDGET_LONGTERM_STOP respectively. Valid values are 0, 1, 2, or 3. If 0 is set, no budget data is written. The value 1 generates a net heat and water budget for each component, 2 adds a detailed heat and water budget for each component, and 3 adds a detailed heat and water budget of the different conmponents on the atmosphere grid. Normally values of 0 or 1 are specified. Values of 2 or 3 are generally used only when debugging problems involving conservation.