Subject: [FFC-dev] Re: [DOLFIN-dev] BLAS mode verified

From: "Garth N. Wells" <[email-masked]>
Date: Wed, 05 Oct 2005 15:41:33 +0200

For higher-order (p > 1) complicated forms, I'm seeing big improvements in
compile time when using BLAS mode. Run-time performance appears very similar. I
can't test big problems because BLAS mode has *huge* memory requirements
when using higher order elements. For Poisson equation with p = 10, I can't use
any more than 35x35 elements without having a PETSc memory error.

Separate issue - with more complicated forms, I have trouble with anything
around p > 2 due to the memory requirements of ffc. Are there any simple
solutions to this? 

Garth

Quoting Anders Logg <[email-masked]>:

> I have verified that the new FFC BLAS mode works for the stabilized
> convection-diffusion module in DOLFIN (which is sufficiently complex
> to catch any obvious bugs).
> 
> I still need to run some more extensive benchmarks, but the first
> indications seem to be that the new BLAS mode is actually slower and
> the generated code actually takes longer to compile. Possible
> explanations:
> 
> 1. For the convection-diffusion form, most of the work is computing
> the geometry tensor and that is performed in the same way in both
> cases. Maybe we could generate some loop-based code for computing the
> geometry tensor also.
> 
> 2. Assigning to array entries (G[i] = ... in BLAS mode) seems to take
> longer to compile than assigning to variables (double Gi = ... in
> default mode).
> 
> 3. It actually matters that the default mode of FFC removes any
> products with zeros in the tensor product. BLAS does not know about
> zeros.
> 
> However, for high-order forms and possibly Navier-Stokes, the BLAS
> mode could be faster.
> 
> To compile a form for use with BLAS mode, just add the flag -f blas
> when compiling the form:
> 
>     ffc -f blas Poisson.form
> 
> Note that any generated XML data files (containing the values of the
> reference tensor) must be present in the directory where you run the
> executable.
> 
> Please try out the new BLAS mode and report any errors/performance
> issues.
> 
> /Anders
> 
> _______________________________________________
> DOLFIN-dev mailing list
> [email-masked]
> http://www.fenics.org/cgi-bin/mailman/listinfo/dolfin-dev
>

---

From: Anders Logg <[email-masked]>
Date: Wed, 5 Oct 2005 10:32:23 -0500

On Wed, Oct 05, 2005 at 03:41:33PM +0200, Garth N. Wells wrote:

> compile time when using BLAS mode.

Great!

> Run-time performance appears very similar.

ok. I'm about to start with some more extensive benchmarking. I'll
report the results later.

> I can't test big problems because BLAS mode has *huge* memory
> requirements when using higher order elements. For Poisson equation
> with p = 10, I can't use any more than 35x35 elements without having
> a PETSc memory error.

p = 10 is pretty high... you can add a line

   blas.disp();

somewhere in your eval() function in the form generated by FFC and see
how many entries you have in the matrix that gets shipped off to BLAS.

It sounds strange that you get a PETSc memory error. Degree 10 on
tetrahedra has 286 degrees of freedom, so rougly 286*35*35 = 350350
degrees of freedom total. That should be no problem for PETSc (if you
have sufficient memory).

> around p > 2 due to the memory requirements of ffc. Are there any simple

I wasn't aware of this. With degree 10 Poisson, memory usage stays
around 30MB. Can you give me an example (as simple as possible) that
requires a lot of memory?

/Anders

---

From: "Garth N. Wells" <[email-masked]>
Date: Wed, 05 Oct 2005 17:43:14 +0200

> p = 10 is pretty high... you can add a line

If I use p =3 on triangles, I can't use more than 120k elements - it requires
more than 1GB of memory. Without BLAS, assembly is no problem. Also, with the
p=10 example I was referring to triangles.

> > around p > 2 due to the memory requirements of ffc. Are there any simple

With Poisson, I can go really high. I'll send the Cahn-Hilliard form shortly to
try out. With Cahn-Hilliard I'm quite limited.

---

From: Anders Logg <[email-masked]>
Date: Wed, 5 Oct 2005 11:00:59 -0500

> p=10 example I was referring to triangles.

1GB with p = 3 on triangles for Poisson, and only in BLAS mode?

That sounds like a memory leak. BLAS mode should only be a little
overhead in memory (if any). The only difference is that data is
stored in arrays rather than in the code itself which should be
about the same total memory usage. 

> try out. With Cahn-Hilliard I'm quite limited.

ok.

/Anders