Tag Archives: carpet

The Eighteenth Release of the Einstein Toolkit

Posted on by

The Eighteenth Release of the Einstein Toolkit

We are pleased to announce the eighteenth release (code name “Proca”) of the Einstein Toolkit, an open, community developed software infrastructure for relativistic astrophysics. The highlights of this release are:

New arrangements and thorns have been added:

* Proca

– NPScalars_Proca
– ProcaBase
– ProcaEvolve
– Proca_simpleID
– TwoPunctures_KerrProca

* lean_public

– LeanBSSNMoL
– NPScalars

* wvuthorns_diagnostics

– particle_tracerET
– Seed_Magnetic_Fields_BNS
– smallbPoynET
– VolumeIntegrals_GRMHD
– VolumeIntegrals_vacuum

In addition, bug fixes accumulated since the previous release in September 2018 have been included.

The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems that builds on numerous software efforts in the numerical relativity community including CactusEinstein, the Carpet AMR infrastructure and the relativistic magneto-hydrodynamics codes GRHydro and IllinoisGRMHD. For parts of the toolkit, the Cactus Framework is used as the underlying computational infrastructure providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The toolkit includes modules to build complete codes for simulating black hole spacetimes as well as systems governed by relativistic magneto-hydrodynamics.

The Einstein Toolkit uses a distributed software model and its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers, or by individual groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve postdocs and faculty from six different institutions, and host weekly meetings that are open for anyone to join in.

Guiding principles for the design and implementation of the toolkit include: open, community-driven software development; well thought out and stable interfaces; separation of physics software from computational science infrastructure; provision of complete working production code; training and education for a new generation of researchers.

For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages at http://einsteintoolkit.org.

The Einstein Toolkit is primarily supported by NSF 1550551/1550461/1550436/1550514 (Einstein Toolkit Community Integration and Data Exploration). The Einstein Toolkit contains about 400 regression test cases. On a large portion of the tested machines, almost all of these tests pass, using both MPI and OpenMP parallelization.

The changes between this and the previous release include:

Larger changes since last release

* The Proca arrangement has been added: This repository provides the tools to evolve the Einstein-Proca system as first described in https://arxiv.org/abs/1505.00797.

– NPScalars_Proca: Implementation of the spin-1 (electromagnetic) and spin-2 (gravitational) Newman-Penrose scalars
– Proca_simpleID: Create analytic initial data for a non-rotating black hole surrounded by a Proca field with mass mu.
– TwoPunctures_KerrProca: A modified TwoPunctures thorn to construct initial data for a single rotating black hole coupled to a massive vector field.

* The Lean arrangement has been added:

– LeanBSSNMoL: Implementation to evolve Einstein’s Equations using the W-version of the BSSN formulation together with the puncture gauge. Also available, in the “new_gauge” branch, is a modified “Gamma- driver” that stabilizes highly rotating black hole spacetimes (adapted from Figueras et al; see: https://arxiv.org/abs/1512.04532).
– NPScalars: Implementation of the spin-2 Newman-Penrose scalars

* The WVU Diagnostics arrangement has been added: These thorns are designed primarily to add useful diagnostics for binary neutron star simulations performed with IllinoisGRMHD.

– NSNS_parameter_files Contains parameter files for magnetized and unmagnetized BNS evolutions.
– Seed_Magnetic_Fields_BNS Extended Seed_Magnetic_Fields thorn for binary neutron stars.
– VolumeIntegrals_GRMHD: GRMHD volume integration thorn, currently depends on IllinoisGRMHD and Carpet. Performs volume integrals on arbitrary “Swiss-cheese”-like topologies, and even interoperates with Carpet to track NS centers of mass.
– VolumeIntegrals_vacuum: Same functionality as VolumeIntegrals_GRMHD, but designed for integration of spacetime quantities. Depends on ML_BSSN and ADMBase for integrands.
– particle_tracerET Solves the ODE \partial_t x^i=v^i for typically thousands of tracer particles, using an RK4 integration atop the current time stepping.
– smallbPoynET Computes b^i, b^2, and three spatial components of Poynting flux. It also computes (-1-u_0), which is useful for tracking unbound matter.

* Ticket tracking system moved to bitbucket: https://bitbucket.org/einsteintoolkit/tickets/
* Subversion infrastructure for thorns is no longer maintained at LSU. Instead, the svn checkout mechanism supported by github.com is used.
* Llama supports tensorweights other than 1.0 or 0.0
* Added EinsteinAnalysis/Hydro_Analysis/Hydro_Analysis_Masses.F90 in order to compute the total baryonic mass and baryonic mass within user defined radii.
* A summary of changes Carpet:

– add support for very large grids where 64bit integer are needed for grid indices and sizes of transfer buffers
– fix how physical_time_per_hour is computed
– add functionality to align interior of grid functions to cache boundaries. This requires changes t. Cactus and PUGH as well.
– add a parameter “granularity” to make sure the interior of components is a multiple of N points in each direction

* The version of MPI bundled with the ET is now OpenMPI 1.10.7
* The SystemTopology thorn now supports hwloc 2.0

How to upgrade from Chien-Shiung Wu (ET_2018_09)

To upgrade from the previous release, use GetComponents with the new thornlist to check out the new version.

See the Download page (http://einsteintoolkit.org/download/) on the Einstein Toolkit website for download instructions.

Machine notes

Supported (tested) machines include:

* Default Debian, Ubuntu, Fedora, CentOS, Mint, OpenSUSE and MacOS (MacPorts) installations
* Bluewaters
* Comet
* Golub
* Stampede 2
* Shelob
* Wheeler

* TACC machines: defs.local.ini needs to have sourcebasedir=$WORK and basedir=$SCRATCH/simulations configured for this machine. You need to determine $WORK and $SCRATCH by logging in to the machine.

All repositories participating in this release carry a branch ET_2019_03 marking this release. These release branches will be updated if severe errors are found.

The “Proca” Release Team on behalf of the Einstein Toolkit Consortium (2019-03-29)

* Steven R. Brandt
* Samuel D. Cupp
* Peter Diener
* Zachariah Etienne
* Roland Haas
* Helvi Witek

Mar, 2019

Einstein Toolkit Release

Posted on by

We are pleased to announce the fifth release (code name Lovelace (http://en.wikipedia.org/wiki/Ada_Lovelace) of the Einstein Toolkit, an open, community developed software infrastructure for relativistic astrophysics. This release includes beginning support for OpenCL (disabled by default). In addition, bug fixes accumulated since the previous release in October 2011 have been included.

The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems that builds on numerous software efforts in the numerical relativity community including CactusEinstein, the Carpet AMR infrastructure and the relativistic hydrodynamics code GRHydro (an updated and extended version of the public release of the Whisky code). The Cactus Framework is used as the underlying computational infrastructure providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The toolkit includes modules to build complete codes for simulating black hole spacetimes as well as systems governed by relativistic hydrodynamics.

The Einstein Toolkit uses a distributed software model and its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers, or by individual groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve postdocs and faculty from five different institutions, and host weekly meetings that are open for anyone to join in.

Guiding principles for the design and implementation of the toolkit include: open, community-driven software development; well thought out and stable interfaces; separation of physics software from computational science infrastructure; provision of complete working production code; training and education for a new generation of researchers.

For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages at http://einsteintoolkit.org.

The Einstein Toolkit is primarily supported by NSF 0903973/0903782/0904015 (CIGR), and also by NSF 0701566/0855892 (XiRel), 0721915 (Alpaca), 0905046/0941653 (PetaCactus), and 0710874 (LONI Grid).

The Einstein Toolkit contain over 170 regression test cases. On a large portion of the tested machines, all of these testsuites pass, using both MPI and OpenMP parallelization.

The changes between this and the previous release include:

– Accelerator Support

This release of the Einstein Toolkit adds support for GPUs and other accelerators. This support comprises three levels of abstraction, ranging from merely building and running both CUDA and OpenCL code, to automated code generation targeting GPUs instead of CPUs. As with any other programming paradigm (such as MPI or OpenMP), the performance benefits depend on the particular algorithms used and optimizations that are applied. In addition, the Simulation Factory greatly aids portability to a wide range of computing systems.

At the lowest level, Cactus now supports compiling, building, and running with either CUDA or OpenCL. CUDA is supported as new language in addition to C, C++, and Fortran; OpenCL is supported as an external library, and builds and executes compute kernels via run-time calls. Details are described in the user’s guide (for CUDA) and in thorn ExternalLibraries/OpenCL (for OpenCL).

Many accelerator platforms today separate between host memory and device memory, and require explicit copy or map operations to transfer data. An intermediate level of abstraction aids transferring grid variables between host and device, using schedule declarations to keep track of which data are needed where, and minimizing expensive data transfers. For OpenCL, there is a compact API to build and execute compute kernels at run time. Details are described in thorns CactusUtils/Accelerator and CactusUtils/OpenCLRunTime (with example parameter file).

Finally, the code generation system Kranc has been extended to be able to produce either C++ or OpenCL code, based on the infrastructure described above. This allows writing GPU code in a very high-level manner. However, it needs to be stated that the efficiency of the generated code depends on many variables, including e.g. the finite differencing stencil radius and the number of operations in the generated compute kernels. Non-trivial kernels typically require system-dependent tuning to execute efficiently, as GPUs and other accelerators generally show a rather unforgiving performance behavior. The thorns McLachlan/ML_WaveToy and McLachlan/ML_WaveToy_CL are examples, generated from the same Kranc script, showing the generated C++ and OpenCL code.

– SimFactory
– Machine database and optionlists updated due to system changes on HPC resources
– Simfactory’s capability of running the testsuites is properly tested on a lot of systems.
– IOUtil: checkpoint_dir is now steerable
– SphericalSurface: added functionality to name spherical surfaces
– Formaline: Support a “local repository” that collects all machine-local repositories
– TimerReport: Allow different timers on different processes
– WeylScal4: Enable use of LoopControl, and hence OpenMP
– EOS_Omni: use C interface for HDF5 to avoid needing Fortran HDF5 bindings
– EOSG_*: Support for the so-called ‘general EOS interface’ has been dropped from the Einstein Toolkit
– A new arrangement EinsteinExact has been added to the toolkit, providing a wide range of exact initial data, which will eventually replace the ‘Exact’ thorn.
– The *_O2 versions of McLachlan have been removed from the toolkit. This functionality is already provided by the regular McLachlan thorns now.
– A new thorn ADMMass has been added to the Einstein Toolkit, which can calculate approximations of the ADM mass using a finite surface or volume integral.
– The old library mechanism in Cactus (e.g. HDF5=yes) is now deprecated. Expect it to be removed in one of the next releases.
– The thorns ADM and LegoExcision are deprecated and will be removed in one of the next releases.
– GRHydro:
– use atmosphere integer mask instead of bitmask
– remove (now) unused old Tmunu interface
– Implemented enhanced PPM scheme by Colella & Sekora 2008, McCorquodale & Colella 2011. Can be activated by setting
use_enhanced_ppm = yes
– External Libraries: several updates and configuration improvements
– Cactus
– implement per-variable tolerances for Cactus testsuites, for long discussion, see ET ticket #114
– Allow arithmetic expression in ParameterSet: parameter files can now contain a limited set of expressions
– Handles requirements recursively
– A lot of smaller bug fixes
– McLachlan: Implement CCZ4 formulation
– CarpetMask: Keep track of the volume that is masked out
– CarpetLib: Define MPI reduction operators for complex numbers
– CarpetIOASCII: Add new “compact” output format
– Csrpet: Support accelerator data transfer
– CarpetRegrid2: Add periodic boundary conditions
– Simfactory
– Use OpenMP by default
– Make running testsuites using Simfactory possible
– Updated a lot of configurations

All repositories participating in this release carry a branch ET_2012_05 marking this release. These release branches will be updated if severe errors are found.

For more detailed information about the “Lovelace” release please read the long release announcement on the Einstein Toolkit web pages: http://einsteintoolkit.org/about/releases/ET_2012_05_announcement.php.

On behalf of the Einstein Toolkit Consortium: the “Lovelace” Release Team

Eloisa Bentivegna
Tanja Bode
Peter Diener
Roland Haas
Ian Hinder
Frank Löffler
Bruno Mundim
Christian D. Ott
Erik Schnetter

May 28, 2012

Einstein Toolkit Release

Posted on by

We are pleased to announce the second release (code name “Chandrasekhar”) of the Einstein Toolkit, an open, community developed software infrastructure for relativistic astrophysics. This release is mainly a maintenance release incorporating fixes accumulated since the previous release in June 2010, as well as additional test suites.

The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems that builds on numerous software efforts in the numerical relativity community including CactusEinstein, the Carpet AMR infrastructure and on the public version of the Whisky hydrodynamics code (now modified and called GRHydro). The Cactus Framework is used as the underlying computational infrastructure providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The toolkit includes modules to build complete codes for simulating black hole spacetimes as well as systems governed by relativistic hydrodynamics. Current development in the consortium is targeted at providing additional infrastructure for general relativistic magnetohydrodynamics.

The Einstein Toolkit uses a distributed software model and its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers, or by individual groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve postdocs and faculty from five different institutions, and host weekly meetings that are open for anyone to join in.

Guiding principles for the design and implementation of the toolkit include: open, community-driven software development; well thought out and stable interfaces; separation of physics software from computational science infrastructure; provision of complete working production code; training and education for a new generation of researchers.

For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages http://einsteintoolkit.org.

The Einstein Toolkit is primarily supported by NSF 0903973/0903782/0904015 (CIGR), and also by NSF 0701566/0855892 (XiRel), 0721915 (Alpaca), 0905046/0941653 (PetaCactus) and 0710874 (LONI Grid).

The “Chandrasekhar” Release Team on behalf of the Einstein Toolkit Consortium (2010-11-23)

Einstein Toolkit

Posted on by

The Einstein Toolkit is a collection of software components and tools for simulating and analyzing general relativistic astrophysical systems. Such systems include gravitational wave space-times, collisions of compact objects such as black holes or neutron stars, accretion onto compact objects, core collapse supernovae and Gamma-Ray Bursts.

The Einstein Toolkit builds on numerous software efforts in the numerical relativity community including CactusEinstein, Whisky, and Carpet. The Einstein Toolkit currently uses the Cactus Framework as the underlying computational infrastructure that provides large-scale parallelization, general computational components, and a model for collaborative, portable code development.

The Toolkit includes a set of mix-and-match components (or Thorns) that support the development of codes for relativistic astrophysics. A number of full examples provide prototype, production examples of complete astrophysical codes including black hole spacetimes and relativistic hydrodynamical spacetimes. There is also a step-by-step tutorial for new users.

Einstein Toolkit Release

Posted on by

We are pleased to announce the first release (code name “Bohr”) of the Einstein Toolkit, an open, community developed software infrastructure for relativistic astrophysics. The Einstein Toolkit is a collection of over 130 software components and tools for simulating and analyzing general relativistic astrophysical systems that builds on numerous software efforts in the numerical relativity community including CactusEinstein, the Whisky hydrodynamics code, and the Carpet AMR infrastructure. The Cactus Framework is used as the underlying computational infrastructure providing large-scale parallelization, general computational components, and a model for collaborative, portable code development. The toolkit includes modules to build complete codes for simulating black hole spacetimes as well as systems governed by relativistic hydrodynamics. Current development in the consortium is targeted at providing additional infrastructure for general relativistic magnetohydrodynamics.

The Einstein Toolkit uses a distributed software model and its different modules are developed, distributed, and supported either by the core team of Einstein Toolkit Maintainers, or by individual groups. Where modules are provided by external groups, the Einstein Toolkit Maintainers provide quality control for modules for inclusion in the toolkit and help coordinate support. The Einstein Toolkit Maintainers currently involve postdocs and faculty from five different institutions, and hold weekly meetings that are open for anyone to join in.

Guiding principles for the design and implementation of the toolkit include:

1: Open, community-driven software development that encourages the sharing of code across the community, prevents code duplication, and leads to sustainable support and development of essential code.

2: Well thought out and stable interfaces between components that enable multiple implementations of physics capabilities, and allow groups or individuals to concentrate on their areas of interest.

3: Separation of physics software from computational science infrastructure so that new technologies for large scale computing, processor accelerators, or parallel I/O can be easily integrated with science codes.

4: The provision of complete working production codes to provide: prototypes, standard benchmarks, and testcases; codes that are available for and usable by the general astrophysics community; tools for new researchers and groups to enter this field; training and education for a new generation of researchers.

For more information about using or contributing to the Einstein Toolkit, or to join the Einstein Toolkit Consortium, please visit our web pages at <http://einsteintoolkit.org>.

We thank the numerous people who contributed to this software over the past many years; there are too many to be listed here. We also gratefully acknowledge those who helped in the past months to make this release happen. The Einstein Toolkit is primarily supported by NSF 0903973/0903782/0904015 (CIGR), and also by NSF 0701566/0855892 (XiRel), 0721915 (Alpaca), 0725070 (Blue Waters), and 0905046/0941653 (PetaCactus).

The “Bohr” Release Team on behalf of the Einstein Toolkit Consortium
(2010-06-17)

Whisky

Posted on by

Whisky is a code to evolve the equations of general relativistic hydrodynamics (GRHD) and magnetohydrodynamics (GRMHD) in 3D Cartesian coordinates on a curved dynamical background. It was originally developed by and for members of the EU Network on Sources of Gravitational Radiation and is based on the Cactus Computational Toolkit. Whisky can also implement adaptive mesh refinement (AMR) if compiled together with Carpet.

Carpet

Posted on by

Carpet is an adaptive mesh refinement and multi-patch driver for the Cactus Framework. Cactus is a software framework for solving time-dependent partial differential equations on block-structured grids, and Carpet acts as driver layer providing adaptive mesh refinement, multi-patch capability, as well as parallelisation and efficient I/O.

Cactus

Posted on by

Cactus is an open source problem solving environment designed for scientists and engineers. Its modular structure easily enables parallel computation across different architectures and collaborative code development between different groups. Cactus originated in the academic research community, where it was developed and used over many years by a large international collaboration of physicists and computational scientists.

The name Cactus comes from the design of a central core (or “flesh”) which connects to application modules (or “thorns”) through an extensible interface. Thorns can implement custom developed scientific or engineering applications, such as computational fluid dynamics. Other thorns from a standard computational toolkit provide a range of computational capabilities, such as parallel I/O, data distribution, or checkpointing.

Cactus runs on many architectures. Applications, developed on standard workstations or laptops, can be seamlessly run on clusters or supercomputers. Cactus provides easy access to many cutting edge software technologies being developed in the academic research community, including the Globus Metacomputing Toolkit, HDF5 parallel file I/O, the PETSc scientific library, adaptive mesh refinement, web interfaces, and advanced visualization tools.