CESM1 is accompanied by utilities that support automated testing of the model. In general, these should be used only after the model has been ported to the target machine (see ). The tools are create_production_test, create_test, and create_test_suite. create_production_test tool is executed from $&CASEROOT;, and tests exact restartability of the case setup in a separate directory. create_test is executed from &CCSMROOT/scripts and allows you to quickly set up and run one of several supported tests. Finally, create_test_suite is primarily for CSEG developers and is there to easily set up and run a list of supported tests. Each of these tools will be described below. In general, after configuring and testing a case and before starting a long production job based on that case, it's important to verify that the model restarts exactly. This is a standard requirement of the system and will help demonstrate stability of the configuration technically. The tool create_production_test is located in the case directory, and it sets up an ERU two month exact restart test in a separate directory based on the current case. To use it, do the following > cd $CASEROOT > ./create_production_test > cd ../$CASE_ERT > $CASE_ERT.test_build submit $CASE_ERT.test Check your test results. A successful test produces "PASS" as the first word in the file, $CASE_ERT/TestStatus If the test fails, see for test debugging guidance. The create_test tool is located in the scripts directory and can be used to set up a standalone test case. The test cases are fixed and defined within the CCSM scripts. To see the list of test cases or for additional help, type "create_test -help" from the scripts directory. To use create_test, do something like > cd $CCSMROOT/scripts > ./create_test -testname ERS.f19_g16.X.bluefire_ibm -testid t01 > cd ERS.f19_g16.X.bluefire_ibm.t01 > ERS.f19_g16.X.bluefire_ibm.t01.test_build submit ERS.f19_g16.X.bluefire.t01.test Check your test results. A successful test produces "PASS" as the first word in the file TestStatus The above sets up an exact restart test (ERS) at the 1.9x2.5_gx1v6 resolution using a dead model compset (X) for the machine bluefire. The testid provides a unique tag for the test in case it needs to be rerun (i.e. using -testid t02). Some things to note about CCSM tests For more information about the create_test tool, run "create_test -help". Test results are set in the TestStatus file. The TestStatus.out file provides additional details. Tests are not always easily re-runable from an existing test directory. Rather than rerun a previous test case, it's best to set up a clean test case (i.e. with a new testid). Tests are built using the .test_build script. This is different from cases which are built using the .build script. Some tests require more than one executable, the .test_build script builds all required executables upfront interactively. The costs of tests vary widely. Some are short and some are long. If a test fails, see . There are -compare and -generate options for the create_test tool that support regression testing. There are extra test options that can be added to the test such as _D, _E, or _P*. These are described in more detail in the create_test -help output. The test status results have the following meaning Test ResultDescription BFAIL compare test couldn't find base result BUILD build succeeded, test not submitted CFAIL env variable or build error CHECK manual review of data is required ERROR test checker failed, test may or may not have passed FAIL test failed GEN test has been generated PASS test passed PEND test has been submitted RUN test is currently running OR it hung, timed out, or exitied ungracefully SFAIL generation of test failed in scripts TFAIL test setup error UNDEF undefined result The following tests are available at the time of writing TestDescription SMS smoke test ERS exact restart from startup, default 6 days + 5 days ERB branch/exact restart test ERH hybrid/exact restart test ERI hybrid/branch/exact restart test ERT exact restart from startup, default 2 months + 1 month SEQ sequencing bit-for-bit test PEA single processor testing with mpi and mpi-serial PEM pe counts mpi bit-for-bit test PET pe counts mpi/openmp bit-for-bit test CME compare mct and esmf interfaces test NCK single vs multi instance validation test The create_test_suite tool is located in the scripts directory and can be used to set up a suite of standalone test cases automatically. To use this tool, a list of tests needs to exist in a file. Some examples can be found in the directory scripts/ccsm_utils/Testlists. create_test_suite in invoked on a list of tests and then the full list of tests is generated. In addition, an automated submission script and reporting script are created. The cs.submit script reduces the time to submit multiple test cases significantly. To use this tool, do something like the following create a list of desired tests in some filename, i.e. my_lists > create_test_suite -input_list my_lists -testid suite01 > ./cs.status.suite01 > ./cs.submit.suite01.$MACH > ./cs.status.suite01 The cs.status script is generated by create_test_suite, and it reports the status of all the tests in the suite. The cs.submit script builds and submits all the tests sequentially. The cs.submit script should only be executed once to build and submit all the tests. The cs.status script can be executed as often as needed to check the status of the tests. When all the tests have completed running, then the results are static and complete. To help debug failed tests, see .. This section describes what steps can be taken to try to identify why a test failed. The primary information associated with reviewing and debugging a run can be found in . First, verify that a test case is no longer in the batch queue. If that's the case, then review the possible test results and compare that to the result in the TestStatus file. Next, review the TestStatus.out file to see if there is any additional information about what the test did. Finally, go to the troubleshooting section and work through the various log files. Finally, there are a couple other things to mention. If the TestStatus file contains "RUN" but the job is no longer in the queue, it's possible that the job either timed out because the wall clock on the batch submission was too short, or the job hung due to some run-time error. Check the batch log files to see if the job was killed due to a time limit, and if it was increase the time limit and either resubmit the job or generate a new test case and update the time limit before submitting it. Also, a test case can fail because either the job didn't run properly or because the test conditions (i.e. exact restart) weren't met. Try to determine whether the test failed because the run failed or because the test did not meet the test conditions. If a test is failing early in a run, it's usually best to set up a standalone case with the same configuration in order to debug problems. If the test is running fine, but the test conditions are not being met (i.e. exact restart), then that requires debugging of the model in the context of the test conditions. Not all tests will pass for all model configurations. Some of the issues we are aware of are All models are bit-for-bit reproducible with different processor counts EXCEPT pop. The BFBFLAG must be set to TRUE in the &env_run.xml; file if the coupler is to meet this condition. There will be a performance penalty when this flag is set. Some of the active components cannot run with the mpi serial library. This library takes the place of mpi calls when the model is running on one processors and MPI is not available or not desirable. The mpi serial library is invoked by setting the xml variable MPILIBS to mpiserial in &env_build.xml;. An effort is underway to extend the mpi serial library to support all components' usage of the mpi library with this standalone implementation. Also NOT all machines/platforms are set up to enable setting MPILIBS to mpiserial. See the summary of supported machines for details. To use the mpi serial feature on your machine, you also need to make changes in the Macros and env_machopts files for that machine. The best way to do this is to use a machine where MPILIBS can be set to mpiserial and look at the type of changes needed to make it work. Those same changes will need to be introduced for your machine. For the Macros file this includes the name of the compiler, possibly options to the compiler, and the settings of the MPI library and include path. For the env_machopts file you may want/need to modify the setting of MPICH_PATH. There also maybe many settings of MPI specific environment variables that don't matter when the mpiserial setting is used.