Subject: [fenics-dev] Application domains

From: Johan Jansson <[email-masked]>
Date: Tue, 27 Sep 2005 11:03:45 +0200

Hi!

I think it's a very nice discussion going on right now, many good
points. I just want to make an addition with a slightly different
perspective.

The application domain we seem to be discussing is scientific
computing on supercomuters, i.e. the target "customer" for Fenics is a
scientist at some lab with a supercomputer.

I think this view is far too limited, but I guess is a symptom of the
particular academic environment most of us live in. PDEs and the need
for discretization appears in computer games, image manipulation,
video processing, computer animation, medical training simulators,
flight simulators, etc. etc. Some of these involve supercomputers, but
some do not. The vision algorithm for a robot probably runs on an
embedded system, but likely involves discretizing a PDE (or at least
could be modeled as one). There's no reason Fenics should not be used
to generate the code for that algorithm.

What is common for all these application domains however is the need
for efficiency and the need for conceptual abstraction (I don't think
a developer of a filter for the Gimp or Photoshop wants to worry about
the details of a finite element solver). I believe Fenics solves both
of these in a very nice way. The input is the form and the element(s),
I can't think of a better abstraction representation, and we can
generate an optimal solver from that completely automatically.

Of course supercomputers are important, and of course parallelism is
important, the embedded system in the robot could probably be parallel
as well. What I'm saying is, let's not tailor Fenics only for
scientists with supercomputers, and in doing so, sabotage Fenics for
other application domains. I believe historically this has been true,
FEM "codes" have been specifically tailored for supercomputers, and
this has limited the spread of the FEM into other domains. If you ask
a typical engineer what the FEM is, he will say that it's a method for
dividing a domain into triangles. As far as I know, the mechanical
engineers at Chalmers don't come into contact with the FEM unless they
specifically choose that track.

We have the opportunity to create a FEM system which can generate
optimal solvers for essentially _all_ domains. We can solve the
accessibility problem of the FEM, which is the complexity explosion of
coefficients and indices once and for all and actually make it
available to normal people (engineers at least), without having to
sacrifice _any_ performance. Or perhaps we should say that "we have
already solved" this problem (with regards to assembly at least), now
it's essentially a question of packaging and further developing it.

  Johan

---

From: Matthew Knepley <[email-masked]>
Date: Tue, 27 Sep 2005 08:41:26 -0500

Johan Jansson <[email-masked]> writes:

      More than 50% of Petsc downloads (we estimate) are for the
Windows version, which we also surmise is not a supercomputer. There
is no difference in our application code that runs on a supercomputer
from what I run on my laptop. In fact, I do all development on the
laptop, and only move to the supercomputer for VERY large runs (the
overwhelming amount of supercomputer usage is robbery of the taxpayer).

      The above paragraph is meant to support the thesis that there
need not be any difference between development for a laptop and a
supercomputer. In my view, parallelism is recognition of structure
inherent in the problem. Witness the difference between "parallelism"
in densse and sparse linear algebra. They bear almost no relation to
each other. In fact, the parallelism in sparse linear algebra has the
same root as parallelism in FEM. This point is made forcefully in the
Sieve paper.

      Thus, I would be interested in any aspect of FENICS software that:

  1) Makes it hard for a laptop user to code/run

  2) Prevents it from running on a supercomputer

I do not see antagonism between these goals.

  Thanks,

        Matt

> _______________________________________________
> fenics-dev mailing list
> [email-masked]
> http://www.fenics.org/cgi-bin/mailman/listinfo/fenics-dev
>
>
>

-- 
"Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness

---

From: Anders Logg <[email-masked]>
Date: Tue, 27 Sep 2005 08:45:17 -0500

Let me also point out that Johan has done a great job with the FEniCS
component Ko, which sits on top of DOLFIN/FFC and provides a
domain-specific interface and utilities for simulating mechanical
systems that could be used in animation, gaming etc.

We could have more domain-specific interfaces that target specific
application areas.

/Anders

-- 
Anders Logg
Research Assistant Professor
Toyota Technological Institute at Chicago
http://www.tti-c.org/logg/

---

From: Anders Logg <[email-masked]>
Date: Tue, 27 Sep 2005 08:50:23 -0500

On Tue, Sep 27, 2005 at 08:41:26AM -0500, Matthew Knepley wrote:

> [...]

So these are your two goals? Making FEniCS hard to run on laptops and
preventing it from running on supercomputers... ;-)

/Anders

---

From: "tomtzigt" <[email-masked]>
Date: Tue, 27 Sep 2005 12:28:05 -0400

Johan:

  I love your concept of 'hiding' the complexity of the FEM. In this
context, what in your mind would be required to hook Fenics up in an
environment like FEMLAB where you would be able to write the equations,
handle the boundary conditions, and automate design studies? In my taste,
there are a little bit too many clicks in FEMLAB to set parameters of the
equations but it does bundle enough components to create a usable FEM
environment for normal people.

Theo

---

From: Matthew Knepley <[email-masked]>
Date: Tue, 27 Sep 2005 11:33:49 -0500

> preventing it from running on supercomputers... ;-)

  Helps keep my workload down.

        Matt

---

From: Johan Jansson <[email-masked]>
Date: Wed, 28 Sep 2005 16:41:23 +0200

> environment for normal people.

We already have what FEMLAB has sans the GUI in DOLFIN (this also
applies to Sundance). I haven't used FEMLAB much, but from what I've
tested and seen, you select a pre-defined equation from a library and
then specify coefficients. DOLFIN has modules which correspond to an
equation, and where you can set coefficients. What is missing is some
polish (GUI, more coefficients, more equations).

Fenics can go much further though. Instead of selecting an equation,
you can simply write an arbitrary equation yourself, with any
coefficients you like. Fenics then generates the assembly routine for
you. This already exists in Fenics, in the FIAT/FFC/DOLFIN
chain. Sundance also does this, but generates the assembly routine at
run-time, where FFC generates code which needs to be compiled (and
will likely always give higher performance).

Again, to challenge FEMLAB, some polish is needed (GUI, some system
for JIT, i.e. the assembly code needs to be compiled and loaded at
run-time).

What would be needed is also a way to generate a solver. If an
equation is pre-defined, then the solver can also be pre-defined to go
with that equation. But if an arbitrary equation is input, then the
solver will look different depending on if the problem is
time-dependent, non-linear etc. I made a suggestion how to handle that
in an automatic way in a previous post, I don't think it should be too
hard to make it work.

  Johan

---

From: Johan Jansson <[email-masked]>
Date: Wed, 28 Sep 2005 17:02:23 +0200

> overwhelming amount of supercomputer usage is robbery of the taxpayer).

That sounds great, and is probably contributing to keeping PETSc
healthy.

> I do not see antagonism between these goals.

The main point I was attacking with my post was the suggestion that we
should sacrifice performance by not pre-compiling code, since the
Linux version Blue Gene runs (and perhaps other supercomputer
operating systems) apparently can't do dynamic loading. This would
impact your laptop user, since his code would run slower. A better
solution as far as I can see would be to fix Linux on Blue Gene so
that it can do dynamic loading, or alternatively avoid dynamic loading
by recompiling and reuploading the code to the nodes between runs.

As soon as DOLFIN can do parallel assembly and mesh manipulation with
Sieve, all Fenics components should run equally well on laptops and
supercomputers.

  Johan

---

From: "Robert C. Kirby" <[email-masked]>
Date: Wed, 28 Sep 2005 10:26:55 -0500

> should sacrifice performance by not pre-compiling code, since the
1.) We don't know that not-precompiling code sacrifices significant  
performance.  This is not a settled question.  It could be that the  
optimal run-time code is only epsilon more expensive than the optimal  
compiled code.  It largely depends on how the systems are  
engineered.  An interesting comparison (don't want to ruffle any  
feathers though) would be to make a run of solving some problem in  
DOLFIN and in Sundance and compare the time to assemble the matrix.

> operating systems) apparently can't do dynamic loading. This would
Your comment about "only running on supercomputers" is well-taken but  
slightly overstated -- there are lots of places besides the laptop/ 
desktop/cluster/supercomputer spectrum where we might be interested  
in running.  Kevin's design decision were to make something portable  
across the spectrum of machines that he runs on -- it can run on a  
linux box/cluster, but he doesn't have to do too much to do his  
production runs.

> by recompiling and reuploading the code to the nodes between runs.

The first is neither possible nor practical.
1.) Blue Gene runs a very stripped-down kernel (parts of it are  
highly guarded as they contain proprietary IBM secrets, I believe) to  
allow maximum memory usage per node.  The claim is that when you  
start adding stuff to the kernel (like dynamic loading) you will lose  
available memory for the highest-end runs.  Hence, even if you wanted  
to spend all your time rewriting the BlueGene kernel, they wouldn't  
let you.

2.) cross-compiling on the other hand, including generating code for  
forms instead of having code to interpret them, could be an eventual  
possibility.

Rob Kirby

"Mathematical software should be mathematical."

---

From: Anders Logg <[email-masked]>
Date: Wed, 28 Sep 2005 11:04:13 -0500

> DOLFIN and in Sundance and compare the time to assemble the matrix.

That would be great. If would think DOLFIN/FFC does the form
evaluation faster (computing the element stiffness matrix) but
Sundance might very well beat us when you count in insterting into the
global matrix (don't know how PETSc compares to Trilinos here) and
interacting with the mesh (our mesh is probably a little slow).

I think a benchmark would be constructive since it would help both
codes find any bottlenecks we're not aware of.

/Anders

---

From: Anders Logg <[email-masked]>
Date: Wed, 28 Sep 2005 11:30:59 -0500

On Wed, Sep 28, 2005 at 11:24:46AM -0500, Robert C. Kirby wrote:
> We don't know that it does.  It has more symbolics, but it uses level  
> 3 BLAS also.  Probably depends heavily on the operator.

Yes, we don't know. That's why we should try.

> I don't know how efficient Kevin's mesh communication is.  I know he  
> does "SumIntoRows" which is just Trilinos' Matt, Set Values!

I like that. Maybe you could update PETSc to support things like

    Matt, SetValues!
    Matt, Assemble!

You can overload ',' in C++ so it should be possible, I don't know
about '!' though...

> If we could really straighten out who's spending time where and get  
> an accurate comparison of time to build the matrix, it might also  
> tell us how much is lost (if anything) by run-time stuff.  Or, it  
> could very well tell us that even though DOLFIN is compiled, it's  
> embarassingly slow  :)

Yes, that's the point.

/Anders

---

From: Matthew Knepley <[email-masked]>
Date: Wed, 28 Sep 2005 12:13:10 -0500

> by recompiling and reuploading the code to the nodes between runs.

  Yes, those are really build issues that can be separated from the code
generation issues I think. And, yes, we need better systems support on BG/L.

> Sieve, all Fenics components should run equally well on laptops and

  Excellent.

        Matt

---

From: Matthew Knepley <[email-masked]>
Date: Wed, 28 Sep 2005 12:16:43 -0500

"Robert C. Kirby" <[email-masked]> writes:

> 1.) We don't know that not-precompiling code sacrifices significant  performance.  This is not a settled question.  It
> could be that the  optimal run-time code is only epsilon more expensive than the optimal  compiled code.  It largely
> depends on how the systems are  engineered.  An interesting comparison (don't want to ruffle any  feathers though) would
> be to make a run of solving some problem in  DOLFIN and in Sundance and compare the time to assemble the matrix.

  I have talked to Kevin about this many times. The problems he solves are EXPLICITLY solve dominated. This is by no
means true for all problems. Many, many problems are dominated by assembly time, particularly for complex, nonlinear
physics since you are constantly computing the residual. Here, precompilation clearly wins (I think Kevin agrees).

> 1.) Blue Gene runs a very stripped-down kernel (parts of it are  highly guarded as they contain proprietary IBM secrets,
> I believe) to  allow maximum memory usage per node.  The claim is that when you  start adding stuff to the kernel (like
> dynamic loading) you will lose  available memory for the highest-end runs.  Hence, even if you wanted  to spend all your
> time rewriting the BlueGene kernel, they wouldn't  let you.

  This is complete crap I think. IBM is lazy and unmotivated here. If you can load a program, you can load my damn
dynamic library with the exact same system call (patching is a litle different, but easy). Memory is also a red herring
since this code will be loaded by the static executable anyway. These guys are blowing smoke.

      Matt

---

From: "tomtzigt" <[email-masked]>
Date: Wed, 28 Sep 2005 13:48:54 -0400

Would there be interest in a Java-based GUI that tries to do this for
Fenics. If there is interest, I can give this a try.

Theo

-----Original Message-----
From: Johan Jansson [mailto:[email-masked]] 
Sent: Wednesday, September 28, 2005 10:41 AM
To: tomtzigt
Cc: 'Johan Jansson'; [email-masked]
Subject: Re: [fenics-dev] Application domains

  Johan

---

From: Johan Jansson <[email-masked]>
Date: Wed, 28 Sep 2005 21:03:20 +0200

> Fenics. If there is interest, I can give this a try.

Yes, definitely!

I have started (this monday) to experiment with SWIG to try to get a
Python interface to Ko (a multibody solid mechanics component of
Fenics I'm developing), and it would have to make interfaces to DOLFIN
as well (meshes, coefficient functions, etc.). If you wanted to work
on a GUI it would be a great addition, and both of us would have to
solve some common problems. I think you also would require a
SWIG-generated interface to similar parts of DOLFIN/FFC.

This is the first time I'm using SWIG, and I've only been coding
Python for a few weeks, so it will take me some time before I get up
to speed. Have you used SWIG before? It seems to be the way to go for
this type of thing.

Most of Fenics tends to be written in C++ or Python, but I don't see a
problem using Java for the GUI. It should be straightforward to
interface to the Python components, and something like SWIG can
generate interfaces for Java as well as Python.

  Johan

---

From: Anders Logg <[email-masked]>
Date: Wed, 28 Sep 2005 14:10:17 -0500

Any contributions are welcome. Applications that sit on top of the
current libraries are particularly useful, since they will force us to
specify proper interfaces.

My only worry about Java is that it should work with gcj/kaffe and GNU
classpath (so one doesn't have to download the java runtime from Sun
which is not available on most Linux distros).

/Anders

---

From: "tomtzigt" <[email-masked]>
Date: Wed, 28 Sep 2005 15:11:55 -0400

Cool.

 I haven't used SWIG, so I'll take a look at it andtry to catch up with you.

 Another thought would be to leverage the Eclipse eco-system with its well
thought out plug-in architecture and solutions to editor-model and context
sensitive help collaborations.

  We probably want to li-fo the basic components in the user interface and
generate consensus.

Theo

-----Original Message-----
From: Johan Jansson [mailto:[email-masked]] 
Sent: Wednesday, September 28, 2005 3:03 PM
To: tomtzigt
Cc: 'Johan Jansson'; [email-masked]
Subject: Re: [fenics-dev] Application domains

Yes, definitely!

  Johan

---

From: "tomtzigt" <[email-masked]>
Date: Wed, 28 Sep 2005 15:19:22 -0400

Kaffe: is this a real requirement? Our development systems (red-hat, suse,
windows) are all SUN run-times and are dedicated to other projects that rely
on the regular SUN distributions so if Kaffe is a requirement I am not the
guy that can deliver.

Theo

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of Anders Logg
Sent: Wednesday, September 28, 2005 3:10 PM
To: Discussion of FEniCS development
Subject: Re: [fenics-dev] Application domains

/Anders

---

From: Anders Logg <[email-masked]>
Date: Wed, 28 Sep 2005 14:59:34 -0500

I'm not in a position to specify requirements for anyone, but it would
be easier for me (as a Debian user) if it could work out of the box on
Debian without needing to download the Java runtime from Sun.

There was a long discussion recently about the use of Java in
OpenOffice. The conclusion seemed to be that developers would make
sure that OpenOffice is compatible with free implementations of
Java.

It's been a while since I did any Java (and all work I did was based
on Sun's Java...) but I imagine that if it works for OpenOffice, then
it should work for FEniCS?

/Anders