Main Page

Carpet Developer Workshop Announcement

Dear Colleagues,

We would like to invite you to attend a Carpet developer workshop to be hosted by the Center for Computational Relativity and Gravitation at the Rochester Institute of Technology on August 23th, 24th, 25th and 26th (2010). This workshop is aimed at experienced Carpet users that want to gain further expertise on the internal workings of the Carpet driver, and, thereby, to be able to contribute to the infrastructure development. This workshop will be led by Erik Schnetter, the main architect for the Carpet AMR infrastructure, and will count with the presence of an HPC specialist. One important goal of this workshop will be to discuss and implement ways to improve the performance for Cactus-based applications for current and future high-end petascale architectures. Due to space limitations, we encourage you to contact the organizers as soon as possible to register your presence.

Sincerely, the organizers:

• Manuela Campanelli (manuela@astro.rit.edu)
• Bruno C. Mundim (bcmsma@astro.rit.edu)
• Erik Schnetter (schnetter@cct.lsu.edu)

Program

The first two days of the workshop will be dedicated to the theory of adaptive mesh refinement (AMR) and related algorithms. The last two days is reserved for hands-on work on the development of the Carpet infrastructure. Details of the program are still to be announced. The program will be agreed upon during the first day of the workshop.

We will break for lunch sometime around 12PM EDT.

Cookies and coffee will always be served at 9:30AM, and---most likely---the workshop will start at around 10AM.

First Day:
Workshop commences at 10AM.
Talk: Erik Schnetter "Introduction to Carpet/AMR" (1:30pm)
Talk: Justin Luitjens "Introduction to Uintah/AMR" (2:30pm) Uintah_Framework.pdf
Talk: Erik Schnetter "Carpet Details" (3:30pm)

Second Day:

Third Day:
Hands-on Carpet work.

Fourth Day:
Hands-on Carpet work.

Discussions

Please vote on which of these you want and will spend time helping to implement by adding your name after an item.

Wish list:

• Carpet Mercurial
  • Buffer zone mask (ICH, CDO, FL).
  • Shadow hierarchy.
  • Support for high number of refinement levels.
  • Simple AMR scheme that uses CarpetRegrid2 and just enlarges/shrinks refined regions based on some criterion.
  • Cell centred algorithm.
    • Refluxing.
  • Better diagnostics:
    • System and library memory use: Ian has a freely available thorn SystemStatistics in AEIThorns for monitoring memory usage (ICH).
    • More user friendly error messages.
    • Mask for interpolation errors.
    • Mask telling which the finest level would have a particular coordinate (there is a new function in Mercurial version: locate_position).
• Scalability
  • More efficient regridding (recomposing?).
  • Load balance (ICH, PD, FL).
  • Performance model for communication in Carpet (ICH, PD).
• Efficient I/O
  • Write an API for F5 file format (ES, CR).
  • Faster recovery; Processor decomposition on recovery (ICH).
  • Use a single file for all variables.
  • Add an option to keep the hdf5 open longer (ICH).
  • Omit buffer zones when outputting (optionally) (ICH).
  • Better documentation - both for users and for developers of Carpet (ICH)

Carpet Overview

Ian has played a bit with generating code documentation for Carpet (Mercurial version) using Doxygen. Initial attempt is at

 http://www.aei.mpg.de/~ianhin/carpet-doc/html/index.html

• Update: Have added graphical class hierarchy
• Update: Have added search box

Ideas:

• Replace malloc/new to poison memory
• See if there is a compiler option to poison the stack

Notes from Tuesday:

• tt is a "time hierarchy" which keeps track of which level has which time
• Ordering of scheduled functions is respected on a given level, but because of the ordering of levels in Berger-Oliger, not between levels
• Carpet_Regrid(Maps) is the function that you provide to determine the new grid structure. It is called automatically by Carpet in the CCTK_REGRID bin. The existing grid structure is passed in and you should modify them to change the grid structure. In this function you need to do the processor decomposition as well.
• Would be nice to have a function which took in a bboxset on each processor and returned the bboxsets on all the processors (normalised) - though this would have scaling problems
• In order to determine mesh refinement boxes locally, you currently need something like the above. What we would like is to only have each process know about its own local bboxes. Then you could locally construct the refined regions that you need and give them to Carpet, and Carpet would do whatever is necessary.
• Modes in Carpet:
  • Modes are a hack introduced because Cactus scheduler doesn't know about mesh refinement
  • They are a way of telling Carpet how you want a routine to be called (on all levels? just once? etc)
  • Meta mode: Called once per process; this is not used currently so is equivalent to Global mode for our purposes.
  • Global mode: Called once per "resolution". The "time" is the time of the finest existing grid.
  • Level mode: Called once per level
  • Singlemap mode: Called once per patch
  • Local mode: Called once per component. Only here do you have access to the grid function data.
  • In any mode, you can switch mode, both up and down. There are three sets of macros for modes: ENTER_X_MODE, BEGIN_Y_LOOP, and a variant which allows you to select just a single Z (component, level etc) You also need to call the looping macros. Moving up needs thought, but is perfectly fine and can be useful for some things.
• You can tell Carpet the minimum number of points per process; empty components will be created
• Recompose.cc does the load balancing - there is a function called cost which computes the cost of a region with a particular shape

Extended Program

After the carpet workshop, on Friday 27th, CCRG is hosting two talks for a general audience:

• Black Hole Formation in Massive Star Collapse by Christian Ott
• The Einstein Toolkit by Erik Schnetter

References

We list below the main references on adaptive/guided/fixed mesh refinement:

• Adaptive Mesh Refinement for Hyperbolic Partial Differential Equations - by Marsha J Berger, Joseph Oliger Journal of Computational Physics 53, March 1984, pp 484-512 (SPIRES) For your convenience, the current citations for this pioneer paper are grouped here

• Evolutions in 3D numerical relativity using fixed mesh refinement - by Erik Schnetter, Scott H. Hawley and Ian Hawke - Class.Quant.Grav.21:1465-1488,2004. e-Print: gr-qc/0310042 (SPIRES) (arXiv).

• Adaptive mesh refinement for characteristic codes - by Frans Pretorius, Luis Lehner J.Comput.Phys.198:10-34,2004. - gr-qc/0302003 (SPIRES) (arXiv)

• Adaptive mesh refinement for coupled elliptic-hyperbolic systems - by Frans Pretorius, Matthew W. Choptuik J.Comput.Phys.218:246-274,2006 - gr-qc/0508110 (SPIRES) (arXiv)

• AMR, stability and higher accuracy - by Luis Lehner, Steven L. Liebling, Oscar Reula - Class.Quant.Grav.23:S421-S446,2006. - gr-qc/0510111 (SPIRES) (arXiv)

Internal Links

Some useful information to help you to navigate around RIT/Rochester:

• Workshop Location
• Lodging
• Restaurants
• RIT annotated campus map
• RIT google map
• CCRG links

External Links

Some useful links about AMR codes and their applications:

• Carpet — AMR for the Cactus Framework
• Cactus code
• Einstein toolkit
• Einstein toolkit wiki
• Enzo — AMR Cosmological Simulation Code
• Uintah — AMR software components and libraries
• Chombo — Infrastructure for Adaptive Mesh Refinement

Some development/debugging/profiling tools and libraries of possible interest:

• HDF5 Group
• MPICH2 : High-performance and Widely Portable MPI
• Charm++ library
• PAPI — Performance Application Programming Interface
• mpiP: Lightweight, Scalable MPI Profiling
• Valgrind — an instrumentation framework

Participants

RIT

Visitors

Via EVO teleconference

Getting started

• Consult the User's Guide for information on using the wiki software.
• Configuration settings list
• MediaWiki FAQ
• MediaWiki release mailing list

Sponsors

The workshop is co-sponsored by:

• The RIT's Center for Computational Relativity and Gravitation ([1])
• NSF PRAC award OCI-0941653 [2]
• NSF PRAC award OCI-0832606 [3].