Introduction
Logging On
Programming Environment
Compilers & Deduggers
Queue Commands & PBS

Introduction

The Cray XT3 will be named Marron, after a Western Australian crayfish. It is 164 Node Cray XT3 with 2.6 GHz, single core AMD Opertons and 330GB memory and a .85 TFlop capability; and 32TB of RAID data storage based on the Lustre file system. The Marron is configured in a torus with a topology of 6 x 4 x 8. Provision for two (2) login PE’s is incorporated in this design, each enabling eight (8) simultaneous users to access the system, depending upon the characteristics of the computational requirements of those users.

When you log onto Marron please review the Message of The Day (MOTD) for the latest information. Make sure that you have have copies of your important files on your local system. If you have problems using Marron please submit them to: support@wasp.uwa.edu.au

Logging On

If you are on the WASP UWA network you can logon to marron using marron.wasp.uwa.edu.au. For other network, please logon to marron via the IVEC network marron.ivec.org.

The IP for logon via the WASP network is 130.95.112.34. Logon via the IVEC network is 192.54.130.65

Your login directory on marron is /home/*/$USERID

Your WASP home directory "/home/*/$USERID" is mounted on all login nodes. The total size of /home is 40GB. Your quota on /home is set to 2GB.

Programming Environment

Cray XT3 Programming Environment includes:

• PGI Compilers for C, C++ and Fortran
• GNU GCC compilers for C, C++ and FORTRAN 77
• Parallel Programming models:
• 1. Cray MPICH2, the Message-Passing Interface 2 (MPI-2)
  2. Cray SHMEM logically shared, distributed memory access
• AMD Core Math Library (ACML)
  1. Level 1, 2, and 3 Basic Linear Algebra
  2. Linear Algebra (LAPACK)
  3. Fast Fourier Transform (FFT)
  4. Math transcendental library
  5. Random number generators
  6. GNU Fortran libraries
• Cray XT3 LibSci scientific library
  1. ScaLAPACK, a set of LAPACK routines redesigned for use in MPI applications
  2. BLACS, a set of communication routines used by ScaLAPACK and the user to set up a problem and handle the communications
  3. SuperLU, a set of routines that solve large, sparse, nonsymmetric systems of linear equations
• A special port of the glibc GNU C Library routines for compute node
  applications
• The Performance API (PAPI) for measuring the efficiency of an application's use of processor functions
• The yod command for launching applications
• Lustre parallel and UFS-like file systems
• The xtshowmesh utility for determining the availability of batch and interactive compute nodes
• The xtshowcabs command, which shows the current allocation and status of the system's nodes and gives information about each job that is running
• Single-system view (SSV) commands (such as xtps and xtkill) for
  managing multinode processes
• Portals, the low-level message-passing interface

Compilers

The default compiler environment is PGI. Use the module command to load additional require module. "man module" has more info

To list all avialable modules

$ module avial

To list modules that have been loaded. Some modules are already loaded by default. Please list module before adding modules that are already loaded.

$ module list

PGI Compiler Commands

Compiler	Command	Source File
C compiler	cc	filename.c
C++ compiler	CC	filename.c
Fortran compiler for Fortran 90 & Fortran 95	ftn	filename.f (fixed source) filename.f90, filename.f95, filename.F95 (free source)
Fortran 77	f77	filename.f77

To use the GCC compilers, load the PrgEnv-gnu module:

$ module load PrgEnv-gnu

To switch from PGI Programming Environment to the GNU Programming Environment, enter:

$ module swap PrgEnv-pgi PrgEnv-gnu

This loads the product modules that defines the system paths and environment variables needed to use a GNU environment.

GCC Compiler Commands

Compiler	Command	Source File
C compiler	cc	filename.c
C++ compiler	CC	filename.cc filename.c++ filename.c
Fortran 77	f77	filename.f

Example: PGI - C Make file

all: XXX
INCLUDE=
LFLAGS= -lm -lgmalloc -Mlarge_arrays
CC = cc
CDEBUGFLAGS = -O
CFLAGS = $(INCLUDE)
XXX: XXX.c XXX.o
cc -o XXX $(CFLAGS) XXX.o $(LFLAGS)

Example: PGI - Fortran77 Make file

all: XXX
INCLUDE=
LFLAGS= -lm -lgmalloc -Mlarge_arrays
FFLAGS = $(INCLUDE)
XXX: XXX.f XXX.o
f77 -o XXX $(FFLAGS) XXX.o $(LFLAGS)

Queue Commands & PBS

The batch scheduler on marron is PBS(Portable Batch Scheduler). Man pages for PBS are avialable on marron's login modes. The 3 PBS commands that you will find most useful are:

qsub for submitting a batch job
qstat for getting status on a batch job
qdel for deleting a batch job

Example: PBS Job Shell Script

# Lustre dir is /lus/nid00036/<user>
#
# Ensure this is a C-shell script
#!/bin/csh
#
# Set number of nodes
#PBS -l size=4
#PBS -N XXX

#launch script with working directory
echo $PBS_O_WORKDIR
cd $PBS_O_WORKDIR
setenv YOD_STRACE 1
yod -sz 4 -strace ./XXX