Computing on catfishComputing on catfish"The catfish is a plenty good enough fish for anyone" - Mark Twain
Jump to: general info | compilers | libraries | software | NCAR-CCSM | miscellaneous
Click here for Basic UNIX help.Catfish is a dual CPU AMD Opteron system, running Redhat linux, and the master node of a cluster. There are 8 compute nodes which can accessed only from catfish (cf01-cf08), and are almost identical hardware to catfish. This page represents a starting point for setting up account environments, compiling and running programs.
I recommend you download a recommended additions to you .cshrc file here. Paste this text somewhere after the default environment settings. Alternatively, you can save this in your home directory (as .cshrc_catfish) then add source .cshrc_catfish to your .cshrc
Each user has a home directory which is backed up every night. Also, a scratch (/scratch) disk can be used to store temporary data, and is a convenient place to have models dump output. A large RAID disk, broken into two partitions (/data1 and /data2) are used for longer term storage (such as good model runs), and the place for ongoing work. Although not backed up, the RAID is more reliable hardware than the single scratch disk. /data1 contains the NCEP (/data1/NCEP) and ECMWF (/data1/ERA40) reanalysis data. /data2 is contains the NCAR model source code and data (/data2/csm).
See below for details on recommended additions to your default .cshrc and .login files.
All tasks to be run on the cluster nodes must be submitted via the PBS queue system. For this to run ensure /usr/bin/pbs is in your path. To use the batch queue, make sure you have ssh set up (use ssh-keygen -t rsa, then copy id_pub.rsa to authorized_keys2 in you ~/.ssh directory). Also, you should place a .rhosts file in your home directory that contains the names of all nodes (copy this from David's home directory). The present queue configuration is:
| queue name | time limit | cpu limit | priority | comments |
| short | 24 hours | none | high | |
| normal | 4 days | 8 | medium | |
| long | none | 8 | low | |
| testing | 10 minutes | (8 virtual) | medium | Runs on master to debug scripts only |
| special | none | none | very high | For special projects |
In general jobs should be run as short, normal or long. Example job scripts can be generated with the commands pbsserial or pbsspmd, the latter being for jobs that require MPI. You can also see examples: myruns.job and myrunmpi.job .
The PBS commands qstat shows the status (try -a and -q options), and qsub submits jobs, and qdel removes jobs from the queue.
When running interactive jobs on catfish, please be mindful of other. In particular, hen jobs will take more than a few hours run, submit these to the queue. Otherwise, for jobs that will take more than a few minutes, remember to "nice" them using the renice command. This includes long numerically intensive IDL sessions.
There are a number of Fortran and C compilers.
GNU compilers (catfish and nodes): gcc, g++ and g77 are in thire default locations. Be aware that both 32bit and 64 bit C libraries exist in /usr/lib, and /usr/lib64.
Portland Group compilers (catfish only): pgcc, pgf77, pgf90 and pghpf, as well the the pgdbg debugger (versions 5.2) are all available in both 32 but and 64 bit versions. I recommend 32 bit, which is set as the default for various libraries listed below (although 64 bit version of the libraries also exist). To enable this compiler, you must add the following to you .cshrc file
setenv PGI /usr/local/pgi
setenv LM_LICENSE_FILE $PGI/license.dat
setenv PATH $PGI/linux86/5.2/bin:${PATH}
setenv MANPATH $PGI/linux86/5.2}/man:$MANPATH
setenv LD_LIBRARY_PATH $PGI/linux86/5.2/lib:$LD_LIBRARY_PATH
setenv PGI_JAVA /usr/local/java/jre/bin/java
Intel compilers(catfish and nodes): ifort, icc and idb are available (version 8.1).
setenv INTEL_LICENSE_FILE /usr/local/intel/licenses #setenv PATH /usr/local/intel/bin:$PATH setenv MANPATH /usr/local/intel/man:$MANPATH setenv LD_LIBRARY_PATH /usr/local/intel/lib:$LD_LIBRARY_PATH
A Java compilers (catfish and nodes): installed in /usr/bin.
NCAR Graphics: Version 4.2 installed, in /usr/local/ncarg. To enable use of NCARG compiler front ends (ncargf77, etc), and use of binaries (ctrans, etc) the NCARG_ROOT environment variable needs to be set, and PATH updated.
Add the following to .cshrc
setenv PATH $/usr/local/ncarg/bin:${PATH}
setenv NCARG_ROOT /usr/local/ncarg
Note that NCL is installed as a separate software package in /usr/local/ncl. To use NCL, you will net to set NCARG_ROOT as: setenv NCARG_ROOT /usr/local/ncl. You will also need to add /usr/local/ncl/bin in your path, which has the ncl binary, as well as copies of standard NCARG utilities (ctrans, etc).
NETCDF (version 3.5) is installed as three different compilation options, GNU, Intel, Portland Group 32 bit and Portland Group 64 bit. The GNU versions are in /usr/local/netcdf-3.5.1, and needed for linking with programs compiled with, for instance, gcc. For locally compiled program (especially Fortran), use the 32 bit Portland Group version stored in netcdf-3.5.0_pgi32. The netcdf.mod file, needed for fortran 90 compilations is stored in the "include" subdirectory. It is convenient to define the following in you .cshrc file:
set NETCDF_ROOT = '/usr/local/netcdf-3.5.0_pgi32'
setenv PATH ${NETCDF_ROOT}/bin:${PATH}
setenv NETCDF_LIB $NETCDF_ROOT/lib
setenv NETCDF_INC $NETCDF_ROOT/include
setenv NETCDF_MOD $NETCDF_ROOT/include
MPI (mpich-1.2.6) is installed, like NETCDF, in three flavors for compatibility.
set MPI_ROOT = '/usr/local/mpich-1.2.6_pgi32'
setenv PATH ${MPI_ROOT}/bin:${PATH}
setenv MPI_LIB ${MPI_ROOT}/lib
setenv MPI_INC ${MPI_ROOT}/include
HDF libraries and GRIBEX libraries are also installed, and can be found in /usr/local. UDUNITS is installed in /usr/local/udunits.
Catfish has the usual assortment of linux software (netpbm, perl, python, ImageMagick OpenOffice, LaTeX, tcl/tk, etc). Also, The NETCDF operators and ncview are installed. Add the following to your .cshrc:
setenv PATH /usr/local/nco/bin:/usr/local/ncview/bin:${PATH}
Conmap: "Contour Mapper" has grown into an extensive suite of analysis programs and a comprehensive 2d plotting utility for latitude longitude data. Other than the data structure this package has little resemblance to the original inspirational "conmap.f" program from Melbourne University. The data format itself is a simple unformatted 2d data structure, which can be layered in an multi-record file:
integer nlon, nlat real lons(nlon),lats(nlat),field(nlon,nlat) character(len=80) header open(1,file=myfile,form='unformatted') read(1) nlat read(1) lats(:) read(1) nlon read(1) lons(:) read(1) header read(1) field(:,:)
Some common data processing routines include, cmpls, cmpdiff, cmpmean, cmpslice, cmpstrip, cmptrend, cmppca. There about 150 of them. The plotting program, cmpplt, is a fortran front-end to the NCAR graphics library. It can handle contour plots (lines of color filled), pixel fills, shading (stippling or hatching), vector arrows, wind barbs, streamlines, with annotation (markers, words, lines), with varying colors line widths, etc. Also all geophysical projections supported by NCAR Graphic can be used. cmpplt can layer multiple input fields (such as contours layered on top of a contour fill, with wind vectors on top, and stippling on top again), and multi-record input files can be used to generate a sequence of frames to build up animations. There is also a 3d rendering version for surfaces and isosurfaces and trajectories cmpsurf3d - it even does sterographic plots. Output is written to a file "gmeta" which is an NCAR graphic meta file, which can be viewed (or converted to postscript) with ctrans.
setenv PATH /usr/local/conmap/bin:{PATH}
The NCAR Community Climate System Model source code, input data, diagnostic processor and miscellaneous analysis tools and scripts are stored centrally in /data2/csm. All boundary data and initial conditions are stored in /data/csm/inputdata. Source codes for each of the models (CCM3, CAM2, CAM3, CCSM2, CCSM3) are in separate subdirectories. Example run scripts have been placed in each each.
Example: Run CAM3: Make a temporary directory somewhere (say, on /scratch), copy the example run script (run-cam3_catfish.csh) to this directory. Edit the run script as needed, paying special attention to variable "case", which determines the name of the directory where the model will run (the default is /scratch/YOUR_USER_NAME/camrun). Execute the script. You can check that it is running with top. For more details see the README file and/or check the documentation on the NCAR CAM website..
A copy of the AWG diagnostic package is also included in the csm directory, and has been modified for catfish. Some modification is still required for specific model runs (such as naming your model case and run duration, see documentation). Also, in /data2/csm/utils you will find programs written to perform various pre- and post-processing tasks on CCSM (CAM) history files. Further details on local processing and CAM on catfish can be found here.
Printing will default to the Stadium laserjet
For additional information email David