AMR14

Matt’s notes on (HPC)^3 2014

2014-11-09

Overview

• Tweaked PyBoxLib’s setup.py to support the develop command through setuptools.
• Created a pyboxlib HashStack. Works on the Mac machine (see below).
• Created a PR for HashStack to get this up.

PyBoxLib HashStack

extends:
- name: hashstack
  urls: ['https://github.com/hashdist/hashstack.git']
  key: 'git:c5e2c7a8cad8751fe5a9ab130f267652bf2e09e0'
  file: osx.yaml

parameters:
  fortran: true
  PATH: /Users/mwemmett/opt/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin

packages:
  mpi:
    use: host-mpi
  blas:
    use: host-osx-framework-accelerate
  pyboxlib:

package_dirs:
- pkgs
- base

Note that I set the PATH so that HashDist picked up my local installation of MPICH. The pyboxlib.yaml file currently contains:

extends: [setuptools_package]

dependencies:
  build: [mpi, numpy]
  run: [mpi, numpy]

build_stages:
- name: enter_pyboxlib_dir
  after: prologue
  before: install
  handler: bash
  bash: |
    cd Src/Python

sources:
- url: https://ccse.lbl.gov/pub/Downloads/BoxLib.git
  key: git:9abcfa56867347305f3a0ad9674fb5ae1ec98ef6

2014-11-10

Overview

• Reverted PyBoxLib’s setup.py file to it’s distutils state instead of setuptools.
• Talked with Kyle about domain/patch/grid design in PyClaw and how we will approach regridding.

AMR

• Kyle will create/import an AMR Solver in PyClaw.
• Regridding will be driven by a user settable routine that, given a state, returns a char tagging array. This will be passed down to BoxLib which will take care of regridding.
• Started working on mock Amr and AmrLevel classes to bridge between PyClaw and BoxLib…

2014-11-11

Overview

• Worked with Aron to get the PyBoxLib HashStack up to date.
• Merged Aron’s patch to PyBoxLib to improve the detection of libraries for gfortran.
• Made a mirror of BoxLib on github.com.

AMR

• Will continue to work on regridding…
• Slowly realising that the C++ version of BoxLib is fairly awkward to use as a library, whereas the Fortran version is much more lightweight. As such, I am going to see if I can swap out the backend of BoxClaw to use the Fortran version of BoxLib instead of the C++ version.

2014-11-12

Overview

• Participated in morning discussions regarding the future and sustainability of Clawpack.
• Continued work on BoxClaw and AMR in the afternoon.

BoxClaw

• Spent most of the day getting the Fortran version of PyBoxLib back in working order, and working on a Mac.
• Seems to be working… needs some tidying.

The current FPyBoxLib install creates two dynamic libraries. One contains the Fortran BoxLib, and the other contains a small shim to expose a NumPy view of a FAB. Ideally I could include the shim and BoxLib into one library, but when I try to do that I get segfaults when creating NumPy views. Apparently this is due to complications around import_array(), but I can’t figure this out.

2014-11-13

Overview

• Continued working on regridding. Seems to be working!

FPyBoxLib

FPyBoxLib is very fragile, and some aspects of it are not very Pythonic. In particular

• the __getattr__ in base.py is very fragile;
• the two stage constructors are silly;
• initialization (fboxlib.open()) is fragile.

These need to be addressed.

I think there is also something bad happening with respect to the NumPy views outliving the underlying FAB arrays. Is there a better API for getting and releasing these views?

Chatted with Aron about this and he suggested using Cython to do the wrapping instead of ctypes. Will try this out…

I have started creating a more robust wrapper and will be contained in one Python extension module.

Tagging

Currently BoxLib assumes that a function called tag_boxes is defined to determine which cells on a given level should be refined. This model doesn’t work well when trying to use BoxLib as a library. We’ll have to write a tag_boxes.f90 specific to FPyBoxLib and allow this to call back into Python.

I have created local regrid, make_new_grids and tag_boxes routines. These now pass C function points (c_funptr) down and eventually call back to do the actual tagging.

Now to try attaching a Python routine!

MAN ALIVE! IT’S WORKING! THERE’S MEN ALIVE IN THERE!

Here is a sample script

import fboxlib
fboxlib.open()

ba = fboxlib.boxarray(boxes=[[ (1,1),(100,100) ]])
la = fboxlib.layout(ba)
mf = fboxlib.multifab(la)

fab = mf.fab(1)
print fab.array.shape
fab.array[...] = 1.0
fab.array[10:20,10:20] = 1.02
fab.array[50:60,50:60] = 1.02

def tag_boxes(mf, tb, dx, lev):
    print "TAGGING FROM PYTHON", dx, lev
    # ...

mfs = fboxlib.regrid([la], [mf], [0.5], tag_boxes)

2014-11-14

Overview

• Reworked and tidied up the PyBoxLib wrapper quite a bit.
• Tidied up the pyboxlib HashStack and created a PR against Aron’s ahmadia/hpc3_2014_stack branch.

Victory

After a final push, witness the magic…

• Create a HashDist profile and bring in the pyboxlib package.
• Run the following script (see BoxLib/Src/Python/F90/tests/regrid.py)…

We first create a 2d multifab from (0,0) to (100,100) and fill it with 1.0, except in two blocks from (11,11) to (21,21) and (51,51) to (61,61) where we fill it with 2.0.

We then calls BoxLib’s regrid routine and pass it the tag_boxes routine. This tag_boxes routine gets called by BoxLib to mark cells for refinement. Currently we refine on the first level if the first component of the original multifab is > 1.0 (which corresponds to the two blocks that we set to 2.0).

The underlying layout are printed… note that after regridding there are two high-resolution boxes around the two blocks that we tagged!

import fboxlib
fboxlib.open()

ba = fboxlib.boxarray([[(0,0), (100,100)]])
la = fboxlib.layout(ba)
mf = fboxlib.multifab(la, nc=3, ng=0)

print "#"*80
print "# before regridding"
la.echo()

fab = mf.fab(1)
fab.array[...] = 1.0
fab.array[10:20,10:20] = 2.0
fab.array[50:60,50:60] = 2.0

def tag_boxes(mf, tb, dx, lev):
    if lev > 1:
        return
    mf = fboxlib.multifab(cptr=mf)
    tb = fboxlib.lmultifab(cptr=tb)
    mfab = mf.fab(1)
    tfab = tb.fab(1)
    tfab.array[mfab.array[:,:,0] > 1.0] = 1

mfs = fboxlib.regrid([la], [mf], [0.5], tag_boxes)

print "#"*80
print "# after regridding"
for mf in mfs:
    mf.layout.echo()

Here are the results…

################################################################################
# before regridding
LAYOUT[(*
 ID = 1 DIM     =  2
 NBOXES  =   1
 PD      = BOX[{0, 0}, {100, 100}]
 1: BOX[{0, 0}, {100, 100}] 0
 *)]
################################################################################
# after regridding
LAYOUT[(*
 ID = 1 DIM     =  2
 NBOXES  =   1
 PD      = BOX[{0, 0}, {100, 100}]
 1: BOX[{0, 0}, {100, 100}] 0
 *)]
LAYOUT[(*
 ID = 7 DIM     =  2
 NBOXES  =   2
 PD      = BOX[{0, 0}, {201, 201}]
 1: BOX[{16, 16}, {47, 47}] 0
 2: BOX[{96, 96}, {127, 127}] 0
 *)]