Main Page

Carpet Developer Workshop Announcement

Dear Colleagues,

We would like to invite you to attend a Carpet developer workshop to be hosted by the Center for Computational Relativity and Gravitation at the Rochester Institute of Technology on August 23th, 24th, 25th and 26th (2010). This workshop is aimed at experienced Carpet users that want to gain further expertise on the internal workings of the Carpet driver, and, thereby, to be able to contribute to the infrastructure development. This workshop will be led by Erik Schnetter, the main architect for the Carpet AMR infrastructure, and will count with the presence of an HPC specialist. One important goal of this workshop will be to discuss and implement ways to improve the performance for Cactus-based applications for current and future high-end petascale architectures. Due to space limitations, we encourage you to contact the organizers as soon as possible to register your presence.

Sincerely, the organizers:

• Manuela Campanelli (manuela@astro.rit.edu)
• Bruno C. Mundim (bcmsma@astro.rit.edu)
• Erik Schnetter (schnetter@cct.lsu.edu)

Program

The first two days of the workshop will be dedicated to the theory of adaptive mesh refinement (AMR) and related algorithms. The last two days is reserved for hands-on work on the development of the Carpet infrastructure. Details of the program are still to be announced. The program will be agreed upon during the first day of the workshop.

We will break for lunch sometime around 12PM EDT.

Cookies and coffee will always be served at 9:30AM, and---most likely---the workshop will start at around 10AM.

First Day:
Workshop commences at 10AM.
Talk: Erik Schnetter "Introduction to Carpet/AMR" (1:30pm)
Talk: Justin Luitjens "Introduction to Uintah/AMR" (2:30pm) Uintah_Framework.pdf
Talk: Erik Schnetter "Carpet Details" (3:30pm)

Second Day:

Third Day:
Hands-on Carpet work:

• 10am: Round table where each participant reported what they have been working on so far.

Fourth Day:
Hands-on Carpet work.

Discussions

Please vote on which of these you want and will spend time helping to implement by adding your name after an item.

Wish list:

• Carpet Mercurial
  • Buffer zone mask (ICH, CDO, FL, BCM).
  • Shadow hierarchy.
  • Support for high number of refinement levels. [FL]
  • Simple AMR scheme that uses CarpetRegrid2 and just enlarges/shrinks refined regions based on some criterion.
  • Cell centred algorithm. (SCN)
    • Refluxing. (SCN)
  • Better diagnostics:
    • System and library memory use: Ian has a freely available thorn SystemStatistics in AEIThorns for monitoring memory usage (ICH).
    • More user friendly error messages (RH).
    • Mask for interpolation errors.
    • Mask telling which the finest level would have a particular coordinate (there is a new function in Mercurial version: locate_position).
    • Better error reporting: Output stderr from all processors to the same file, prefixed with the MPI rank of the process. Make sure there is no stderr unless there is a really serious problem that needs to be addressed.
    • Output a backtrace on crash
• Scalability
  • More efficient regridding (recomposing?)(BCM).
  • Load balance (ICH, PD, FL).
  • Performance model for communication in Carpet (ICH, PD).
• Efficient I/O
  • Write an API for F5 file format (ES, CR).
  • Faster recovery; Processor decomposition on recovery (ICH).
  • Use a single file for all variables.
  • Add an option to keep the hdf5 open longer (ICH).
  • Omit buffer zones when outputting (optionally) (ICH, RH).
  • Better documentation - both for users and for developers of Carpet (ICH)

Carpet Overview

MONDAY

Ian has played a bit with generating code documentation for Carpet (Mercurial version) using Doxygen. Initial attempt is at

 http://www.aei.mpg.de/~ianhin/carpet-doc/html/index.html

• Update: Have added graphical class hierarchy
• Update: Have added search box

Ideas:

• Replace malloc/new to poison memory
• See if there is a compiler option to poison the stack

TUESDAY

• tt is a "time hierarchy" which keeps track of which level has which time
• Ordering of scheduled functions is respected on a given level, but because of the ordering of levels in Berger-Oliger, not between levels
• Carpet_Regrid(Maps) is the function that you provide to determine the new grid structure. It is called automatically by Carpet in the CCTK_REGRID bin. The existing grid structure is passed in and you should modify them to change the grid structure. In this function you need to do the processor decomposition as well.
• Would be nice to have a function which took in a bboxset on each processor and returned the bboxsets on all the processors (normalised) - though this would have scaling problems
• In order to determine mesh refinement boxes locally, you currently need something like the above. What we would like is to only have each process know about its own local bboxes. Then you could locally construct the refined regions that you need and give them to Carpet, and Carpet would do whatever is necessary.
• Modes in Carpet:
  • Modes are a hack introduced because Cactus scheduler doesn't know about mesh refinement
  • They are a way of telling Carpet how you want a routine to be called (on all levels? just once? etc)
  • Meta mode: Called once per process; this is not used currently so is equivalent to Global mode for our purposes.
  • Global mode: Called once per "resolution". The "time" is the time of the finest existing grid.
  • Level mode: Called once per level
  • Singlemap mode: Called once per patch
  • Local mode: Called once per component. Only here do you have access to the grid function data.
  • In any mode, you can switch mode, both up and down. There are three sets of macros for modes: ENTER_X_MODE, BEGIN_Y_LOOP, and a variant which allows you to select just a single Z (component, level etc) You also need to call the looping macros. Moving up needs thought, but is perfectly fine and can be useful for some things.
• You can tell Carpet the minimum number of points per process; empty components will be created
• Recompose.cc does the load balancing - there is a function called cost which computes the cost of a region with a particular shape

• Carpet perform 3 general operations:
  • Initialization
  • Evolve
  • Shutdown

• For now assume "mg" = 1 (number of multigrid levels)

• Evolve opeation:
  • 1) user routine that defines the regridding algorithm; (Carpet_Regrid Carpet_RegridMaps)
  • 2) Regrid (in Carpet) (handles interface between that user routine and Carpet)
  • 3) Recompose (handles grid function initialization on grid structure and load balancing, Recompose.cc)

• Definitions:
  • component = 3D volume of points that live on a processor
  • patch/map = collection of components that follow "arbitrary" (e.g. cubed sphere) coordinate systems
  • level = set of all components/patches at a given spatial discretization
  • simulation/grid-hierarchy (global view) = set of levels

• Vertex-centered vs. Cell-centered AMR:
  • There are routines for restricting and prologing face-centered, vertex-cented and cell-centered functions, but no gereral interface
  • It is up to the user (right now) to interpret where in the cell the grid function is located, so the user must choose how each function is restricted/prolonged and how the evolution scheme interprets a particular element's meaning

• MASK:
  • In order to create a mask that flags those cells, which need to be "fixed" from a failed update, we need to locate the coarse points that lie sufficiently within refined regions and so lie outside the sphere of influence of the rest of the coarse domain; these coarse points are part of the "shrunk" domain and can be safely ignored when an error occurs in it is updated.
  • The final mask will include all the problematic points minus those "causally disconnected" points.

WEDNESDAY

• Workshoppers logged their work flow in the below section called "Work". We are individually working on our own tasks today.

• Erik provided a brief description of how to use Mercurial (hg). Please see Mercurial for notes on "hg".

Work

Please add a paragraph here to describe what you have been working on and what you plan to work on in the coming days.

Bruno C. Mundim (BCM)

• I am looking at the data structure for the class bboxset. At the moment it is just a list of bounding boxes distributed over all processors. This data structure may be responsible for the regridding bottleneck identified on the strong scaling. Maybe the best approach would change it to a binary tree data structure.

Also it would be interesting that each of the processes would receive (or generate) a local tree only, saving then the overhead of sending this data structure over thousands of processes.

Ian Hinder (ICH)

• Have generated doxygen documentation for Carpet. This describes all the Carpet files, functions and classes and their relationships based on the C++ source. It uses comments in a particular format in the code. There is a search box and a tree view for the classes, as well as links to HTML versions of the source files. See current status at http://www.aei.mpg.de/~ianhin/carpet-doc/html/index.html. To-do: Decide if this should go on the Carpet web page and be automatically generated each night.

• Designed a mechanism for arranging the timer information in a tree. Currently the timer information is stored in a flat list of timers so it is very difficult to see the hierarchical relationship between the timers. The tree would be written out in XML format so that existing tools can present it in tree form easily. To-do: Write the code.

• Have checked out latest Carpet and built it. Tried to run BBH test-case but it is throwing a std::out_of_range exception. Core dump does not have a valid address. This used to work in March so something must have changed somewhere.

• To-do: Make it easy to look at actual load imbalance in RHS functions

• To-do: Try to solve the "processor decomposition changing on recovery" bug.

SCN

• Contributing to notes

• Will help test cell-centered AMR and refluxing

FL

• Work on 'unused mask'. Work in CarpetLib done (mask setup as ibset), Left: Setup GF for Mask
• Added some checks (e.g. malloc) here and there, some documentation here and there
• Looked into higher numbers of refinement levels: seems the code does use 'int' for the size which is 32 bit, even on 64 bit machines

PD

• Trying to understand the load balancing/processor decomposition routine in Carpet. Interested in helping with improving this.
• Interested in developing a performance model of the Carpet communication.

Roland Haas (RH)

• optionally omit buffer and ghost zones from 3d output (ie. honour out3d_ghosts)
• more user friendly error messages

Christian Reisswig (CR)

• Implement new high-level library for reading/writing simulation data to HDF5 files, very similar to Werner Benger's F5 format. The reason for reimplementing this is that Werner's light++ license is somewhat ill-posed for our purposes and restricts the application of F5 such that we would be unable to use it for e.g. VisIt.
• First, I will design an API for dumping and reading datasets (including coordinates and other metadata).
• Second, this API can be directly used within Carpet and VisIt to write or read data in a very simple manner.

Extended Program

After the carpet workshop, on Friday 27th, CCRG is hosting two talks for a general audience:

• Black Hole Formation in Massive Star Collapse by Christian Ott
• The Einstein Toolkit by Erik Schnetter

References

We list below the main references on adaptive/guided/fixed mesh refinement:

• Adaptive Mesh Refinement for Hyperbolic Partial Differential Equations - by Marsha J Berger, Joseph Oliger Journal of Computational Physics 53, March 1984, pp 484-512 (SPIRES) For your convenience, the current citations for this pioneer paper are grouped here

• Evolutions in 3D numerical relativity using fixed mesh refinement - by Erik Schnetter, Scott H. Hawley and Ian Hawke - Class.Quant.Grav.21:1465-1488,2004. e-Print: gr-qc/0310042 (SPIRES) (arXiv).

• Adaptive mesh refinement for characteristic codes - by Frans Pretorius, Luis Lehner J.Comput.Phys.198:10-34,2004. - gr-qc/0302003 (SPIRES) (arXiv)

• Adaptive mesh refinement for coupled elliptic-hyperbolic systems - by Frans Pretorius, Matthew W. Choptuik J.Comput.Phys.218:246-274,2006 - gr-qc/0508110 (SPIRES) (arXiv)

• AMR, stability and higher accuracy - by Luis Lehner, Steven L. Liebling, Oscar Reula - Class.Quant.Grav.23:S421-S446,2006. - gr-qc/0510111 (SPIRES) (arXiv)

Internal Links

Some useful information to help you to navigate around RIT/Rochester:

• Workshop Location
• Lodging
• Restaurants
• RIT annotated campus map
• RIT google map
• CCRG links

External Links

Some useful links about AMR codes and their applications:

• Carpet — AMR for the Cactus Framework
• Cactus code
• Einstein toolkit
• Einstein toolkit wiki
• Enzo — AMR Cosmological Simulation Code
• Uintah — AMR software components and libraries
• Chombo — Infrastructure for Adaptive Mesh Refinement

Some development/debugging/profiling tools and libraries of possible interest:

• HDF5 Group
• MPICH2 : High-performance and Widely Portable MPI
• Charm++ library
• PAPI — Performance Application Programming Interface
• mpiP: Lightweight, Scalable MPI Profiling
• Valgrind — an instrumentation framework

Participants

RIT

Visitors

Via EVO teleconference

Group photo -- working...

From right to left: Marcelo Ponce (RIT), Christian Reisswig (Caltech), Chritian Ott (Caltech), Justin Luitjens (Utah), Ian Hinder (Max-Planck-Institute), Joshua Faber (RIT), Yosef Zlochower (RIT), Hiroyuki Nakano (RIT), Prabath Pereis (RIT), Frank Loeffer (LSU), Erick Schnetter (LSU), Peter Diener (LSU), Bruno Mundim (RIT), Scott Noble (RIT), Manuela Campanelli's laptop (RIT).

Getting started

• Consult the User's Guide for information on using the wiki software.
• Configuration settings list
• MediaWiki FAQ
• MediaWiki release mailing list

Sponsors

The workshop is co-sponsored by:

• The RIT's Center for Computational Relativity and Gravitation ([1])
• NSF PRAC award OCI-0941653 [2]
• NSF PRAC award OCI-0832606 [3].