Subject: [Fenics-ufc] ufc-many-core blueprint

From: Florian Rathgeber <[email-masked]>
Date: Wed, 04 Aug 2010 12:25:53 +0200

Hi!

I was wondering who is currently working on extending UFC for many-core
architectures. I know that Garth and Graham Markall (Imperial) had some
thoughts about it, which resulted in a blueprint, see [1]

Is anyone else currently investigating this issue? And are there any new
findings, ideas or plans that have maybe not yet found their way into
this blueprint?

In the meantime I added some more thoughts I came up with during my
recent work (and will probably add more to come). Please feel free to
comment and add more!

In short, I'm doing assembly and solve for FEniCS on the GPU. For that I
have a CUDA kernel for element tensor tabulation for all cells of the
mesh at once (auto-generated from the form by a hacked FFC, see [2]). I
currently programmed around UFC for that since I did not want to tackle
the (maybe significant) changes to DOLFIN resulting from a change to the
UFC interface.

That works pretty well so far with the restriction that I am currently
limited to P1 scalar elements, since I use plain dolfin::Mesh::cells()
as the dofmap. I would want to add an auto-generated GPU kernel for dof
tabulation for all cells of the mesh at once (similar to the way I did
for tensor tabulation) to support vector elements of arbitrary order.
Again I would circumvent UFC for the moment, but that is more involved
since the dofmap tabulation touches FunctionSpace and DofMap.

I'm not quite sure where is the right place to
1) hold the pointer to the dofmap data on the GPU
2) call the kernel to do the actual tabulation
For tensor tabulation I did both from a wrapper around dolfin::Form
(neglecting subdomains and assuming there is only 1 cell integral per
form), but dof tabulation is per function space. If I wrapped
dolfin::FunctionSpace, I still cannot use DofMap without significant
changes (that I don't want to do right now) since it uses STL data
structures in CPU memory and relies on dof tabulation cell-by-cell.

For the moment I thought about deriving from FunctionSpace and just
adding the GPU data pointer as member and a member function calling the
GPU kernel for dofmap tabulation. Comments or suggestions on the issue
are very welcome!

Best regards,
Florian

[1] https://blueprints.launchpad.net/ufc/+spec/ufc-many-core
[2] https://code.launchpad.net/~florian-rathgeber/ffc/gpu-wrappers