Subject: [FFC-dev] UFL and new release

From: Anders Logg <[email-masked]>
Date: Thu, 26 Mar 2009 21:44:50 +0100

FFC is now fully functional w.r.t. compilation of UFL forms, at least
judging by the demos in FFC which now all produce correct results.

The Poisson demo in DOLFIN has also been moved to UFL.

I therefore suggest we make new releases of FFC, UFL and DOLFIN:

  DOLFIN 0.9.2
  FFC 0.6.2
  UFL 0.1.0 ?
  SyFi/SFC?

This batch of releases will support both the old .form format and the
new .ufl format. After the release, we can get started on cleaning up
and removing the .form format.

In it's simplest form, the UFL form language looks very much like the
FFC form language. Here's a comparison for the Poisson demo:

FFC
---

  element = FiniteElement("Lagrange", "triangle", 1)

  v = TestFunction(element)
  u = TrialFunction(element)
  f = Function(element)

  a = dot(grad(v), grad(u))*dx
  L = v*f*dx

UFL
---

  element = FiniteElement("Lagrange", triangle, 1)

  v = TestFunction(element)
  u = TrialFunction(element)
  f = Function(element)

  a = inner(grad(v), grad(u))*dx # dot also works
  L = v*f*dx

However, UFL does much more, like treatment of tensor expressions,
nonlinear operators like sin, cos, exp, sqrt, differentiation of forms
w.r.t. variables etc. For more information, look at Martin's summary:

  http://www.fenics.org/wiki/UFL

UFL includes a script form2ufl that can be used to convert from the
old FFC form language to the new UFL form language. The script is not
foolproof but it may help in converting to the new format.

So, let's make a new release now (after getting the buildbot green
again for DOLFIN) and then make the full transition after the release.

It's also time to promote UFL from a development project (/dev) to a
proper project (/hg) since now FFC, SFC and DOLFIN will depend on it.
Any objections?

-- 
Anders

---

From: "Garth N. Wells" <[email-masked]>
Date: Fri, 27 Mar 2009 13:04:01 +0900

> proper project (/hg) since now FFC, SFC and DOLFIN will depend on it.

It would be worth checking with Kristian how far away the quadrature 
optimisations are for FFC + UFL. If it's only a few days, it would be 
good to wait on the FFC release since the new quadrature code is quite a 
bit slower than the old code (around a factor of 5 in some of my tests).

Garth

> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> UFL-dev mailing list
> [email-masked]
> http://fenics.org/mailman/listinfo/ufl-dev

---

From: Kristian Oelgaard <[email-masked]>
Date: Fri, 27 Mar 2009 09:49:47 +0100

> bit slower than the old code (around a factor of 5 in some of my tests).

It's already there, just uncomment the optimise_options lines at the top of the
uflquadraturegenerator.py file.
However, there might still be bugs in the generated code and it compiles slower
(I've seen a factor of around 6-7 in some cases)

Some benchmark tests:

PressureEquation       FFC comp.    num. ops  assemble 2.5e6 times
Old FFC (*.form),+opt: 45.0s         26679          28.2s
New FFC (*.ufl), -opt:  0.9s        166086         110.0s
New FFC (*.ufl), +opt:  3.4s         15771          21.8s

DGElastoDynamics  FFC comp. | ops. a, dx | ops. a, dS | ops. L, dx | ops. L, dS
(*.form), +opt:     16m20s        791        2098         1571         4305
(*.ufl),  -opt:         7s      25776       93600        10008        23280
(*.ufl),  -opt:        45s       1217        2529         1375         2923

For the elasto-dynmics case, running 10 time steps on a 9000 element mesh takes
roughly 14.5s for both of the two optimised versions. The new FFC without
optimisation runs 1 second slower (15.6s).

Kristian

---

From: Anders Logg <[email-masked]>
Date: Fri, 27 Mar 2009 12:11:39 +0100

> optimisation runs 1 second slower (15.6s).

I don't think we need to wait for further testing / optimization. The
old .form format is still functioning and it's the default choice.

If we make a release, it will be easier to add the optimizations since
we can remove all the old stuff and reset the references for the
regression tests to the new format (simplifying and speeding up
testing).

-- 
Anders

---

From: "Garth N. Wells" <[email-masked]>
Date: Fri, 27 Mar 2009 20:16:52 +0900

> old .form format is still functioning and it's the default choice.

If the optimisations are implemented with UFL, it's easier to test the 
new UFL-based code for correctness. It seems that the optimisations are 
largely in place anyway.

Garth

> DOLFIN-dev mailing list
> [email-masked]
> http://www.fenics.org/mailman/listinfo/dolfin-dev

---

From: Johan Hake <[email-masked]>
Date: Fri, 27 Mar 2009 16:33:16 +0100

> The Poisson demo in DOLFIN has also been moved to UFL.

Should we also include the ufl version of the Poission python demo too?

This could be done by putting:

from dolfin import *
try:
    from dolfin.ufl import *
except:
    pass
    
at the top of the poission demo. 

If we want that we should probably release viper too, as a recent patch made 
it possible to plot ufl based PyDOLFIN Functions.

> I therefore suggest we make new releases of FFC, UFL and DOLFIN:

The added ufc.build_ufc_module is needed for the PyDOLFIN/ffc.jit so we will 
also need a UFC 1.1.2 release.

Johan

---

From: "Garth N. Wells" <[email-masked]>
Date: Sat, 28 Mar 2009 10:30:12 +0900

> uflquadraturegenerator.py file.

How about turning this on when the FFC optimisation flag is used (-O, 
--optimize)?

Garth

---

From: Kristian Oelgaard <[email-masked]>
Date: Sat, 28 Mar 2009 11:26:43 +0100

> How about turning this on when the FFC optimisation flag is used (-O,

This was on my TODO list and now it has been done.

Kristian

---

From: Anders Logg <[email-masked]>
Date: Mon, 30 Mar 2009 20:06:04 +0200

> at the top of the poission demo. 

I don't think it's necessary. It would generally be good if we could
avoid try/except in demos as it obfuscates the code.

> it possible to plot ufl based PyDOLFIN Functions.

It's also needed for C++ plotting which has changed and now relies
directly on Viper.

For those who haven't noticed, Viper can now plot all kinds of data
(Function(PlotData), Mesh, MeshFunction) directly from the command-line:

  viper mesh.xml

> also need a UFC 1.1.2 release.

Yes.

So the list is

  DOLFIN 0.9.2
  FFC 0.6.2
  Viper 0.4.2 ?
  UFL 0.1.0 ?
  SyFi/SFC ?

The buildbot is not responding at the moment so I can't check the
status.

-- 
Anders

---

From: Johan Hake <[email-masked]>
Date: Mon, 30 Mar 2009 20:19:49 +0200

On Monday 30 March 2009 20:06:04 Anders Logg wrote:
> > If we want that we should probably release viper too, as a recent patch
> > made it possible to plot ufl based PyDOLFIN Functions.
>
> > The added ufc.build_ufc_module is needed for the PyDOLFIN/ffc.jit so we
> > will also need a UFC 1.1.2 release.
>
> Yes.
>
> So the list is
>
>   DOLFIN 0.9.2
>   FFC 0.6.2
>   Viper 0.4.2 ?
>   UFL 0.1.0 ?
>   SyFi/SFC ?

    UFC 1.1.2! ;)

> The buildbot is not responding at the moment so I can't check the

All is green except:

The DOLFIN macbot how is complaining about the trilinos demo and the DOLFIN 
linux64-exp which complains about:

Traceback (most recent call last):
  File "./demo.py", line 8, in <module>
    from dolfin import *
  
File "/work/jhbuildbot/fenics/lib/python2.5/site-packages/dolfin/__init__.py", 
line 16, in <module>
    from assemble import *
  
File "/work/jhbuildbot/fenics/lib/python2.5/site-packages/dolfin/assemble.py", 
line 25, in <module>
    import cpp
  File "/work/jhbuildbot/fenics/lib/python2.5/site-packages/dolfin/cpp.py", 
line 25, in <module>
    import _cpp
ImportError: /work/jhbuildbot/local/lib/openmpi/mca_paffinity_linux.so: 
undefined symbol: mca_base_param_reg_int

I know Johannes has tried to get into this but as far as I know with no 
success.

Johan

---

From: [email-masked]
Date: Mon, 30 Mar 2009 21:23:49 +0200 (CEST)

> avoid try/except in demos as it obfuscates the code.

In the long run I guess
  from dolfin import *
does the same as
  from dolfin.ufl import *
does now.

SFC also works with UFL now and it is sometimes a nasty mix of
FFC and SFC. However, Poisson, advection-diffusion, Elasticity, cbc-flow
all work now.
I have not tested the others.

Kent

>> made
> FFC-dev mailing list
> [email-masked]
> http://www.fenics.org/mailman/listinfo/ffc-dev
>

---

From: Johan Hake <[email-masked]>
Date: Mon, 30 Mar 2009 21:28:00 +0200

> > avoid try/except in demos as it obfuscates the code.

Of course!

I will just move all the files from from dolfin.ufl to dolfin and its done. 
Then we will also remove from 

  from ffc import *

that now polute the namespace when also doing an 

  from dolfin.ufl import *

Johan

---

From: Anders Logg <[email-masked]>
Date: Mon, 30 Mar 2009 23:40:53 +0200

> that now polute the namespace when also doing an 

Well, that's the main point. To make a release now which supports both
FFC (.form) and UFL (.ufl). Then we can freely remove all the FFC
specific stuff.

-- 
Anders

---

From: "Johannes Ring" <[email-masked]>
Date: Tue, 31 Mar 2009 09:45:33 +0200 (CEST)

On Mon, March 30, 2009 20:19, Johan Hake wrote:
> File
> "/work/jhbuildbot/fenics/lib/python2.5/site-packages/dolfin/__init__.py",
> line 16, in <module>
>     from assemble import *
>
> File
> "/work/jhbuildbot/fenics/lib/python2.5/site-packages/dolfin/assemble.py",
> line 25, in <module>
>     import cpp
>   File
> "/work/jhbuildbot/fenics/lib/python2.5/site-packages/dolfin/cpp.py",
> line 25, in <module>
>     import _cpp

This is a problem with PyDOLFIN and Open MPI 1.3. We (me and Johan) had
success with a small hack this morning. By adding

  import ctypes
  ctypes.CDLL('libmpi.so', ctypes.RTLD_GLOBAL)

before loading the cpp module in site-packages/dolfin/__init__.py the
Python poisson demo ran just fine. Alternatively one should be able to use
this instead:

  import dl
  import sys
  flags = sys.getdlopenflags()
  sys.setdlopenflags(flags | dl.RTLD_GLOBAL)

but because of a bug in the python2.5 package in Hardy (missing dl module)
we couldn't test this.

It might also be possible to do this in C++ directly. Any suggestion on
the best way to fix this issue? See also:

http://fenics.org/pipermail/deb-dev/2009-March/000210.html
http://www.open-mpi.org/faq/?category=running#loading-libmpi-dynamically

Open MPI 1.3 is in Debian unstable so it would be great if we could fix
this before the release.

Johannes

---

From: Johan Hake <[email-masked]>
Date: Tue, 31 Mar 2009 10:17:45 +0200

> Open MPI 1.3 is in Debian unstable so it would be great if we could fix

The point is that the shared mpi library needs to be loaded using the flag 
RTLD_GLOBAL. Otherwise the loaded mpi libraries will be loaded into private 
namespaces causing the undefined symbol link error.

I do not understand why this is not a problem when only using DOLFIN and not 
PyDOLFIN. Probably the mpi library is only loaded one time to the right 
namespace? Well that's how deep my understanding is of this :P

The hack consist of: 
Before we load the compiled c++ module, i.e., the _cpp.do, we load the shared 
mpi library using the above mentioned calls. The library stays in memory and 
is not loaded a second time when the _cpp.so module is loaded. 

This is my naive view of the problem and the why the hack works. It could 
probably be solved in a more gracious manner, e.g. before PetscInitialize is 
called in DOLFIN or maybe even better on the PETSc side?

Cheers!

Johan

---

From: "Garth N. Wells" <[email-masked]>
Date: Tue, 31 Mar 2009 16:23:51 +0800

> namespace? Well that's how deep my understanding is of this :P

Things went sour when DOLFIN was compiled with Trilinos and Trilinos had 
MPI enabled. Could this be related?

Garth

---

From: "Johannes Ring" <[email-masked]>
Date: Tue, 31 Mar 2009 10:27:02 +0200 (CEST)

On Tue, March 31, 2009 10:23, Garth N. Wells wrote:
>>>> ImportError:
>>>> /work/jhbuildbot/local/lib/openmpi/mca_paffinity_linux.so:

Probably not. Trilinos is built with --disable-mpi on all the buildslaves.

Johannes

---

From: "Garth N. Wells" <[email-masked]>
Date: Tue, 31 Mar 2009 16:35:42 +0800

> Probably not. Trilinos is built with --disable-mpi on all the buildslaves.

I mean that if this issue is resolved, using --enable-mpi for Trilinos 
will work.

Garth

---

From: Johan Hake <[email-masked]>
Date: Tue, 31 Mar 2009 10:53:03 +0200

On Tuesday 31 March 2009 10:23:51 Garth N. Wells wrote:
> >>> "/work/jhbuildbot/fenics/lib/python2.5/site-packages/dolfin/__init__.py
> >>>", line 16, in <module>
> >>>     from assemble import *
> >>>
> >>> File
> >>> "/work/jhbuildbot/fenics/lib/python2.5/site-packages/dolfin/assemble.py
> >>>", line 25, in <module>
> >>>     import cpp
> >>>   File
> >> Python poisson demo ran just fine. Alternatively one should be able to
> >> use this instead:
> >>
> >>   import dl
> >>   import sys
> >> but because of a bug in the python2.5 package in Hardy (missing dl
> >> module) we couldn't test this.
> >>
> > The point is that the shared mpi library needs to be loaded using the
> > flag RTLD_GLOBAL. Otherwise the loaded mpi libraries will be loaded into
> > private namespaces causing the undefined symbol link error.
> >
> > I do not understand why this is not a problem when only using DOLFIN and
> > not PyDOLFIN. Probably the mpi library is only loaded one time to the
> > right namespace? Well that's how deep my understanding is of this :P
>

I have think this is a known issue introduced by OpenMPI for libraries that 
dynamically load mpi, which we do when the _cpp.so module is loaded in 
python.

  <http://www.open-mpi.org/faq/?category=running#loading-libmpi-dynamically>

Johan

> Garth
>
> > The hack consist of:
> > Before we load the compiled c++ module, i.e., the _cpp.do, we load the
> > shared mpi library using the above mentioned calls. The library stays in
> > memory and is not loaded a second time when the _cpp.so module is loaded.
> >
> > probably be solved in a more gracious manner, e.g. before PetscInitialize
> > is called in DOLFIN or maybe even better on the PETSc side?
> >
> > Cheers!
> >
> > Johan

---

From: "Johannes Ring" <[email-masked]>
Date: Tue, 31 Mar 2009 11:25:31 +0200 (CEST)

On Tue, March 31, 2009 10:35, Garth N. Wells wrote:
>>>>> use
>>>> flag
>>>> not

OK, I see. Yes, that might be true.

Johannes

>>>> shared
>>>> mpi library using the above mentioned calls. The library stays in
>>>> memory
>>>> and