Subject: [Fenics] Documentation effort

From: Anders Logg <[email-masked]>
Date: Tue, 16 Mar 2010 20:41:12 +0100

Dear all,

As you are all aware, FEniCS is lacking good documentation. The
situation will improve when the FEniCS book comes out, but it will
not replace a comprehensive user manual.

A very good solution to this problem has just presented itself. I
have some grant money that could go towards creating good
documentation and I have also found an ideal candidate in Kristian
Oelgaard. He should really be finishing up his thesis but has kindly
accepted to work part-time on the manual. :-)

So, let's hear some opinions on what kind of documentation users need.

Kristian, Garth, Hans Petter and I have had some initial discussions
and here are some thoughts so far. Let's get the discussion going.

1. The new documentation will consist of two parts, one called
"FEniCS User Manual" and one called "FEniCS Tutorial". The first one
will be a comprehensive manual and the second will be a tutorial based
on Hans Petter's tutorial chapter for the FEniCS book.

2. Both the user manual and tutorial will come in two different
flavors, one C++ and Python. With some clever use of LaTex \input, it
should be possible to handle with not too much extra work.

3. The user manual will replace the current user manuals for DOLFIN,
FFC, UFL and UFC. What we have now in those manuals can be used as
input.

4. The manual will focus on the user experience and therefore
concentrate on the DOLFIN interface. Specific details for FFC, UFL and
UFC will be made appendices in the new manual.

5. The manual will be available both as a PDF file and online HTML.
Hans Petter has pointed out a new tool called doconce that might be
useful for generating PDF, HTML and docstrings from a common source.
This is worth investigating as docstrings are also important and it's
extra work to maintain both docstrings and manual.

6. We need well-documented demos. It's currently unclear what the
relation is between the manual, documented demos, current demos in
DOLFIN and the examples in Hans Petter's tutorial so we need to work
out a model for this.

--
Anders

PS: The project page is here, so make sure to subscribe if you want to
follow the development of the documentation:

  https://launchpad.net/fenics-doc

There is (yet) no special mailing list for the documentation but
perhaps it will be enough to use this list.

---

From: Marie Rognes <[email-masked]>
Date: Tue, 16 Mar 2010 21:35:31 +0100

I think this sounds awesome!

Some comments:

(a) A programmer's reference is also needed.

(b) About demos: It would be good to separate demos from tests.
That is, it is good to have the demos as a subset of the tests. But
I imagine that it would be easier to have well-documented demos
if there are less such. For instance, having the Mass, Poisson,
ReactionDiffusion and NeumannProblem demos in FFC adds clutter
rather than clarification (imo).

(c) "There is often a FEniCS way of doing it."
Some operations can often be surprisingly easily and compactly expressed.
(For instance, the current adaptive-poisson-demo could benefit from using
a form to assemble the error indicators instead of the current approach.)
I think we should illustrate this by maybe comparing a naive-way, better-way
and the ultimate-fenics-way for some nifty examples.
(I have not read Hans Petter's tutorial carefully, so if this is
already included, please disregard.)

--
Marie

Anders Logg wrote:
> ------------------------------------------------------------------------
>
> _______________________________________________
> Mailing list: https://launchpad.net/~fenics
> Post to     : [email-masked]
> Unsubscribe : https://launchpad.net/~fenics
> More help   : https://help.launchpad.net/ListHelp
>

---

From: Johan Hake <[email-masked]>
Date: Wed, 17 Mar 2010 15:04:11 -0700

Nice effort!

I like the choice of bundle the documentation into one FEniCS documentation, 
and letting the existing documentation of the sub-projects be Appendices, or 
just incorporated into the new one.

I agree with Marie that we also need a programmers reference, which is more or 
less the Doxygen generated code we have today (where is this online btw?). As 
such it is nice but the minimalistic approach to the C++ comments for many 
member functions makes the documentation a bit sparse. André has taken a more 
verbose approach when he documented the methods in IntersectionOperator. The 
code looks a bit cluttered (using /* */ instead of // on each line would 
probably improve this), but it generates more verbose documentation. 

If we decides to be more verbose, which I think we should, we need to define a 
common way to do this. I kind of like how PETSc does it. 

We have also discussed using a standard format for the Python Docstrings. So 
nice programmers references can be generated from these. We haven't decided 
which one we are going to use (epydoc or others). I couldn't find doconce (too 
close to many other meanings of doc once :P) at the net so I do not know what 
this software does or how well adapted it is.

Can someone(tm) explain what "well documented demos" means? Should the 
equation the demo solve be explained more? Should an underlying structure a 
demo show be explained in more detail (for example: one demo explaining the 
mixed method, and an other one explaining mesh refining, or the use of JIT 
compiled expressions in Python?)

Johan

---

From: "Garth N. Wells" <[email-masked]>
Date: Thu, 18 Mar 2010 05:17:35 +0000

> common way to do this. I kind of like how PETSc does it.

Personally, I dislike the (overly) verbose approach in the code since I 
often can't find the actual code amongst all the comments. I think that 
we do a pretty good job in DOLFIN on the C++ side of having most 
functions perform one simple task, which makes the function name + 
argument list + comment enough to describe what's going. Python involves 
more magic and there is no type information, so more elaborate doc 
strings are helpful.

> equation the demo solve be explained more?

Firstly, to document better the equation and the solution method. Then, 
to make clear how each 'step' in the solution process can be implemented.

> Should an underlying structure a

Yes, I think that if there is a gap in the sense that we don't have a 
demo to illustrate the use of a particular mainstream feature, then we 
should formulate a demo, preferably something interesting, to 
demonstrate the feature. For more complicated features, we might want 
the demo to focus strongly on a particular feature.

Garth

---

From: [email-masked]
Date: Thu, 18 Mar 2010 08:20:17 +0100

> the demo to focus strongly on a particular feature.

It could be an idea to order the demos like they've done in deal.II. Then
the second
demo does only need to document how it is different from the first. Of
course, the
name of the demos are more descriptive so they should not be taken away,
but there
could be a suggested path through the demos. The path could be then one in
the tutorial of HPL or some other document.

And I think it is important that we get an online reference, automatically
generated for
both C++ and Python for both stable and developement versions.

Kent

---

From: Chris Richardson <[email-masked]>
Date: Thu, 18 Mar 2010 09:40:00 +0000

As a user who has come to FEniCS in the last month or so, I have some 
direct experience of
the existing documentation. I just wanted to make a few points, which do 
represent a personal point of view, but are probably also representative 
of a certain type of user.

Paper-style documentation in pdf or book form is not very useful. Whilst 
I would read something like that if studying maths or physics, HTML is 
much more useful for a computing language. Examples can be downloaded, 
and it is much more easily searchable.

The demos are great, and the best thing you've got at the moment. I find 
myself using "grep" on them to find the specific feature which I am 
interested in. I agree that they shouldn't be too heavily commented.

The doxygen generated documentation is fine, but it is a bit 
overwhelming, and it has no examples in it.

My preferred documentation model would be something like 
perldoc.perl.org or php.net which have
well-written pages on each function etc. with sections on parameters, 
return values, links to related functions, and examples. If you have 
something like this, plus the demos, that's all you need.

Another model would be to use something like "dokuwiki" which would 
allow collaboration on generating  searchable, crosslinked documentation 
with embedded examples.

Chris Richardson

P.S. I'm happy to help with the documentation effort in any way, e.g. 
proof-reading, road-testing, etc. once it
is decided.

-- 
Chris Richardson
BP Institute, Madingley Rd,
Cambridge CB3 0EZ, UK.
Tel: 01223 765704
Fax: 01223 765701
Web: www.bpi.cam.ac.uk <http://www.bpi.cam.ac.uk/users/chris>

---

From: "Anna Blechingberg" <[email-masked]>
Date: Thu, 18 Mar 2010 10:40:00 +0100

>>> member functions makes the documentation a bit sparse. André has taken
>>> a
>>> more
>>> verbose approach when he documented the methods in
>>> IntersectionOperator.
>>> The
>>> close to many other meanings of doc once :P) at the net so I do not
>>> know
>>> what

I agree with the previous commentaries. One idea could also be to have
references in the demos to the tutorial or the documentation. For example
"see page 40 in the Tutorial for an explanation and other variants of
coding this." Or "compare with the ?? demo that solves this problem with
the ?? method" or "this code is explaned in the ?? demo".

Anna

---

From: Andre Massing <[email-masked]>
Date: Thu, 18 Mar 2010 14:47:48 +0100

On Thursday 18. March 2010 06.17.35 Garth N. Wells wrote:
> > I like the choice of bundle the documentation into one FEniCS
> > documentation, and letting the existing documentation of the
> > sub-projects be Appendices, or just incorporated into the new one.
> > 
> > I agree with Marie that we also need a programmers reference, which is
> > more or less the Doxygen generated code we have today (where is this
> > online btw?). As such it is nice but the minimalistic approach to the
> > C++ comments for many member functions makes the documentation a bit
> > sparse. André has taken a more verbose approach when he documented the
> > methods in IntersectionOperator. The code looks a bit cluttered (using
> > /* */ instead of // on each line would probably improve this), but it
> > generates more verbose documentation.
> > 
> > If we decides to be more verbose, which I think we should, we need to
> > define a common way to do this. I kind of like how PETSc does it.
> 

I must admit that I strongly prefer more detailed comments over too little 
comments, mainly with the user in mind.  Firstly for the user it is easier to 
jump over a comment in documentation when he/she understands what happens then 
to think about and find out what a certain function might do if there is too 
little or no description. Secondly as a developer we often tend to be too much 
into the topic. We think that meaningful names of functions and arguments 
should be almost self-explanatory, but for the average user it is often not 
the case.  He/she does not want to study source code but only want to *use* 
the lib and interface, so I think few additional lines can clarify a lot. For 
example, QT's success is amongst other things also largely based on its well 
documented library. Almost everybody appreciates the detailed documentation.
If comments hide the source code I  would rather think about another way of 
adding descriptions then just reducing it. And at least for me with a good 
editor with syntax highlighting and maybe text folding and a ctags-plugin it 
is no problem to immediately find the line of code I am looking for (yepp it is 
gvim ;-P )

> 
> > We have also discussed using a standard format for the Python Docstrings.
> > So nice programmers references can be generated from these. We haven't
> > decided which one we are going to use (epydoc or others). I couldn't
> > find doconce (too close to many other meanings of doc once :P) at the
> > net so I do not know what this software does or how well adapted it is.
> > 
> > demo show be explained in more detail (for example: one demo explaining
> > the mixed method, and an other one explaining mesh refining, or the use
> > of JIT compiled expressions in Python?)
> 

I think that sounds pretty good to have in a sense gradually varying demos and 
it seems to me that DOLFIN has already a pretty cool demo base for accomplish 
this. So what about adding some kind of "big picture" demo, solving some 
rather sophisticated problems, e.g fluid-structure-interaction, which involve 
quite a few features of DOLFIN? The documentation of these ones could point to 
corresponding specialized demos dedicated for single feature classes. Such 
kind of examples might also added to the tutorial as some good "Now we put 
everything together example" similiar as Hans-Petter Langtangen already do  
(e.g. " A general d-Dimensional Multi-Material Test Problem", section 7). I 
really like that style of gradually increasing complexity of Hans Petters 
tutorial and I aggree with Anne that it would be a good idea to map tutorial 
and demos closely. 

Andre

---

From: Andy Ray Terrel <[email-masked]>
Date: Thu, 18 Mar 2010 09:22:24 -0500

I've never heard of doconce.  Is there a link somewhere?  Just to
throw this out there you might check out the way Numpy and Scipy does
theirs (I believe with sphinx [0] which has that nice feature of
publishing to pdf or html).  If there was some style guide set up and
the key areas pointed out, it might be good to have a documentation
marathon for FEniCS'10.

[0] http://sphinx.pocoo.org/

---

From: Johan Hake <[email-masked]>
Date: Thu, 18 Mar 2010 08:21:17 -0700

On Wednesday 17 March 2010 22:17:35 Garth N. Wells wrote:
> > documentation, and letting the existing documentation of the sub-projects
> > be Appendices, or just incorporated into the new one.
> >
> > online btw?). As such it is nice but the minimalistic approach to the C++
> > comments for many member functions makes the documentation a bit sparse.
> > André has taken a more verbose approach when he documented the methods in
> > IntersectionOperator. The code looks a bit cluttered (using /* */ instead
> > of // on each line would probably improve this), but it generates more
> > verbose documentation.
> >

I see your point and I tend to agree with you. However most people (is this 
true?) look at the doxygen generated documentation and here a bit more 
verbosity would not hurt. We could agree on the syntax and the level of 
verbosity which keep the code readable. I also think text editors come with 
comment foldings so you wont be exposed to the comments when you actually do 
code.

The Python documentation also lack in one large respect and that is for 
overloaded C++ methods. They are all lumped into one in Python. The auto-
generated docstrings for the single Python method is then only based on the 
method declared last in the .h file. It would be nice to come around this 
limitation.

> > decided which one we are going to use (epydoc or others). I couldn't find
> > doconce (too close to many other meanings of doc once :P) at the net so I
> > do not know what this software does or how well adapted it is.
> >

Sounds good!

> the demo to focus strongly on a particular feature.

Also sounds good!

Johan

---

From: Johan Hake <[email-masked]>
Date: Thu, 18 Mar 2010 09:34:02 -0700

On Thursday 18 March 2010 07:22:24 Andy Ray Terrel wrote:
> I've never heard of doconce.  Is there a link somewhere?  

Couldn't find it either. We probably need a well adapted documentation tool. 

> Just to
> publishing to pdf or html).  

Looks nice, but maybe a bit Python centric? I see that there are efforts [1] 
for using it to parse Doxygen generated xml, but then what's the point of not 
using Doxygen? Is the generated documentation nicer using Sphinx? They 
certainly look nice!

Johan

[1] http://github.com/michaeljones/breathe/tree/master

> If there was some style guide set up and

---

From: Johannes Ring <[email-masked]>
Date: Thu, 18 Mar 2010 18:29:52 +0100

> Couldn't find it either. We probably need a well adapted documentation tool.

doconce is written by Hans Petter Langtangen. It is currently only
available for Simula internals (in the scriptingsuite subversion
repository). We have used it for the documentation of SciTools
(scitools.googlecode.com).

Johannes

---

From: Anders Logg <[email-masked]>
Date: Sat, 20 Mar 2010 10:00:28 +0100

On Thu, Mar 18, 2010 at 05:17:35AM +0000, Garth N. Wells wrote:
> Personally, I dislike the (overly) verbose approach in the code
> since I often can't find the actual code amongst all the comments. I
> think that we do a pretty good job in DOLFIN on the C++ side of
> having most functions perform one simple task, which makes the
> function name + argument list + comment enough to describe what's
> going. Python involves more magic and there is no type information,
> so more elaborate doc strings are helpful.

I tend to agree here. I like having just a single line of comment in
the code to not clutter the code, especially if we start to use
special formatting symbols for input / output arguments.

Another reason for avoiding very verbose comments in the code is that
if we should base the documentation on automatic extraction of
comments from the code, then we would need *much more* verbosity, even
more than what Andre has put in the mesh intersection classes. Then we
end up with header files that are 90% comments and 10% code.

Then it seems more efficient to document each function twice: first
very compact as part of the code (as we do now) and then very verbose
in a written manual.

One could also consider some middle ground where we put a semicompact
documentation as part of the code, then somehow automatically append
additional information as part of the generation of the documentation,
but that seems complicated.

> >this software does or how well adapted it is.

Perhaps Hans Petter could say something about doconce?

It might have some good features we can use. But one issue would be
the status of doconce. To be useful, it needs to be actively
maintained and widely available (as in having a Debian package).

> Firstly, to document better the equation and the solution method.
> Then, to make clear how each 'step' in the solution process can be
> implemented.

deal.II has well documented examples as part of their tutorial that
could serve as a model:

  http://www.dealii.org/6.2.1/doxygen/tutorial/index.html

I think we need to separate demos from tutorial/example programs. The
demos should be as they are now: short, simple and self-explaining.

Then we can have a different set of programs which are explained very
carefully. It would be natural for these to be part of the tutorial
(based on the existing tutorial written by Hans Petter).

--
Anders

> Yes, I think that if there is a gap in the sense that we don't have
> a demo to illustrate the use of a particular mainstream feature,
> then we should formulate a demo, preferably something interesting,
> to demonstrate the feature. For more complicated features, we might
> want the demo to focus strongly on a particular feature.
>
> Garth
>
>
> >Johan
> >

---

From: Anders Logg <[email-masked]>
Date: Sat, 20 Mar 2010 10:04:50 +0100

> [0] http://sphinx.pocoo.org/

sphinx looks nice, but can it do C++?

--
Anders

---

From: Anders Logg <[email-masked]>
Date: Sat, 20 Mar 2010 10:15:52 +0100

On Thu, Mar 18, 2010 at 09:40:00AM +0000, Chris Richardson wrote:
>
> As a user who has come to FEniCS in the last month or so, I have
> some direct experience of
> the existing documentation. I just wanted to make a few points,
> which do represent a personal point of view, but are probably also
> representative of a certain type of user.
>
> Paper-style documentation in pdf or book form is not very useful.
> Whilst I would read something like that if studying maths or
> physics, HTML is much more useful for a computing language. Examples
> can be downloaded, and it is much more easily searchable.
>
> The demos are great, and the best thing you've got at the moment. I
> find myself using "grep" on them to find the specific feature which
> I am interested in. I agree that they shouldn't be too heavily
> commented.
>
> well-written pages on each function etc. with sections on
> parameters, return values, links to related functions, and examples.
> If you have something like this, plus the demos, that's all you
> need.

The perldoc looks nice, but it illustrates the problem of
autogenerating documentation from the code. For example, look at the
documentation for caller:

  http://perldoc.perl.org/functions/caller.html

This is the level of verbosity we need and it would clutter up the
header files. Here's an example from IntersectionOperator.h that I
personally think looks ugly (sorry Andre...):

    ///Compute all id of all cells which are intersects by a \em entity.
    ///\param[out] ids_result The ids of the intersected entities are
    saved in a vector.
    ///This allows is more efficent than using a set and allows a map
    between
    //the (external) cell and the intersected cell of the mesh. If you
    //are only interested in intersection with a list of cells without
    caring about which
    //cell what intersected by which one, use
    // void IntersectionOperator::all_intersected_entities(const
    std::vector<Cell> &, uint_set &) const;
    /////@internal
    ///@todo This function has to improved: 1) it requires the object
    the
    //mesh is to be cut with to be another mesh entitiy instead of
    being just a
    //kind of geometric object. 2) Requires a runtime switch 3) would
    require a
    //implementation for each geometric  primitive if they have no
    common base
    //class.
    void all_intersected_entities(const MeshEntity & entity, std::vector<uint> & ids_result) const;

Perhaps it is just a question of formatting and it could be made to
look better.

> allow collaboration on generating  searchable, crosslinked
> documentation with embedded examples.
>
> Chris Richardson
>
> P.S. I'm happy to help with the documentation effort in any way,
> e.g. proof-reading, road-testing, etc. once it
> is decided.

Great.

--
Anders

---

From: Anders Logg <[email-masked]>
Date: Sat, 20 Mar 2010 10:20:21 +0100

> is no problem to immediately find the line of code I am looking for (yepp it is

There's no question we need much more detailed documentation, the
question is just where to put it. Either as part of the code or
separately.

Perhaps it is just a matter of figuring out how to write the verbose
comments in a way that looks good.

--
Anders

---

From: Andy Ray Terrel <[email-masked]>
Date: Sat, 20 Mar 2010 08:56:07 -0500

> sphinx looks nice, but can it do C++?

I don't think it can do things automatically like it can with python,
but you can add the calls you want to document (for example the NumPy
array c-api [0]).

[0] http://docs.scipy.org/doc/numpy/reference/c-api.array.html

---

From: Hans Petter Langtangen <[email-masked]>
Date: Sat, 20 Mar 2010 16:10:53 +0100

> Perhaps Hans Petter could say something about doconce?

I wrote doconce during the FEniCS'06 (?) meeting in Delft :-) as a
solution to the problem that I wanted to document software in one
place only, and still be able to put that single piece of text into
pure source code, latex manuals, html manuals, other formats like
docbook, etc. There is and was a relevant tool, docutils and the reST
(restructured text) markup, for doing this. The problem with docutils
was that I think reST has too much markup compared to the plain text
you want to see in the source code and with pydoc in a
terminal. Writing a new backend in docutils that filters reST to plain
(almost) untagged text is the obvious solution, but that seemed
very complicated. I also wanted better support for source code and
mathematics in documents. The attached tutorial explains what doconce is.

> maintained and widely available (as in having a Debian package).

Right now it's living in Simula's svn repository only. It has been
frequently used by myself and others (Johannes, Ilmar). We have thought
of putting it out on googlecode, but that requires maintenance...

For example, my latest book has half a chapter where the source is in
doconce.  That source is automatically copied into a doc string in the
Python code, so that we have the book material also as documentation
of a Python module (via pydoc), the source is filtered to Epytext and
included in the Python module to produce an Epydoc manual of the
module, the source is also filtered to HTML for an online tutorial,
and to latex for fancy typesetting in the book (and in fact also
Johannes Ring's master thesis). The idea is to document once and only
once: if the tutorial is updated, it's the doconce source, and I have
automatic update of the book, the Python source code, and other
manuals :-)

Personally, I use doconce for almost everything I write (except slides
and complete books) since that enables me to postpone the choice of
format.

Johannes (who have made a wiki filter to doconce and done many other
adjustments) and I could maintain a more official version and maybe
try to get it into Debian. However, we should carefully consider the
work associated with a more official version. Doconce is very limited
(no footnotes, no citations and bibliography, for instance) and as
soon as it gets users, they will demand extensions.  The point was to
make it minimal. Debugging errors can also be somewhat frustrating
and challenging.

FEniCS should definitely adopt the doconce *strategy*, i.e., write a
piece of text only once, but maybe distutils and reST is a better
technical choice?  The downside is no support for math, you cannot
copy code directly from a source file and into a manual, and the reST
tagged text (sometimes a bit ugly) appears in Doxygen text or Python
doc strings. The upside is strong support.

> (based on the existing tutorial written by Hans Petter).

I share this view. You copy demos and modify to solve a new problem.
Tutorial examples are made for teaching ideas, syntax, etc. The code
therefore often becomes quite different from a self-explaining demo.

> > >>should be possible to handle with not too much extra work.

I use a program preprocess (in Python :-) to allow for #ifdef and other
C preprocessor type of commands in latex, shell, doconce, and almost
whatever. Then it's easy to compile a tutorial in one mode. (At the
University of Oslo this is used for compiling course manuals with
Python or Matlab.)

But I doubt that #ifdef or clever \input is the way to go with Python
and C++ in a tutorial. The text will naturally be quite different for
the Python and C++ versions. I think it's better to explain most of
the things in Python first and then provide add-on material for
C++. This can be made quite short with a focus on syntax, and reference
back to where the example is introduced in the Python part. However,
if you have a substantial amount of users who don't want to use or
read about the Python interface, you need a stand-alone C++
tutorial. In my view, FEniCS users should normally use Python for
rapid development, but turn to the C++ interface for speed in a more
comprehensive code. That is, the C++ FEniCS programmers are quite
advanced users already and you can take advantage of that when writing
tutorial material. (For the Python tutorial, I forsee very novice
users who don't know finite elements and Python, but they have a PDE
to solve...it seems that FEniCS can, maybe uexpectedly, work well
for this group of users.)

> > >>DOLFIN and the examples in Hans Petter's tutorial so we need to work

I suggest to view the manual, tutorial and demos as three different
types of documentation. The tutorial has its example programs, so may
the manual (but I guess there will be more of short code snippets in a
manual), and the demo selection is a third source of information. To
me it seems difficult to maintain these three types of documentation
if they are closely related to each other.

Hans Petter

---

From: Anders Logg <[email-masked]>
Date: Sun, 21 Mar 2010 00:15:07 +0100

> mathematics in documents. The attached tutorial explains what doconce is.

It looks pretty cool and I very much like the idea of a simple
text-based format that looks natural (very similar to Publish).

But how would it be used to document code? Is the doconce input
written as part of the code, or is the code generated as the result of
some preprocessing stage?

> for this group of users.)

I think we have some C++ only users that don't care much for the
Python interface and some Python users that absolutely don't want to
see the C++ interface, so I think it would be good to split the two
manuals. But you are probably right that the two texts will be
different. Instead of one document with #ifdefs, it should perhaps be
two different documents that share some common chapters that are
language independent.

> if they are closely related to each other.

Agree. And the "well documented examples" discussed above should be
part of the tutorial. I think we are looking at the following pieces:

  1.a FEniCS Tutorial (C++)
  1.b FEniCS Tutorial (Python)
  2.a FEniCS User Manual (C++)
  2.b FEniCS User Manual (Python)
  3.a Demos (C++)
  3.b Demos (Python)

1 and 2 should be available in both HTML and PDF.

1.a and 1.b can be created based on the existing C++/Python tutorial.

3.a and 3.b are in pretty good shape already and can remain where they
are (as part of the DOLFIN source tree), but some demos are missing in
C++ and others in Python. We might also consider making the demos more
easily accessible via the web page. They may be difficult to find,
especially for someone using the Debian/Ubuntu packages (where they
are located somewhere under /usr/share/doc).

The question is how to design 2, the user manual, whether it should be
generated from the code, or if it should be written manually.

A related question is if we need a FEniCS Programmer's Reference and
how that differs from the User Manual, or if the User Manual should
contain everything and make a Programmer's Reference unnecessary.

--
Anders

---

From: Anders Logg <[email-masked]>
Date: Mon, 22 Mar 2010 11:40:11 +0100

Chris Richardson pointed me to the PHP documentation which looks
pretty good:

  http://www.php.net/manual/en/function.explode.php

Note the sections ChangeLog, Examples and Notes. Being able to add
notes seems very useful.

It is generated using DocBook:

  http://doc.php.net/php/dochowto/docbook-whatis.php

The problem with it seems to be it's a nightmare to write the input:

  http://doc.php.net/php/dochowto/chapter-skeletons.php

--
Anders

---

From: Andre Massing <[email-masked]>
Date: Mon, 22 Mar 2010 11:47:45 +0100

> personally think looks ugly (sorry Andre...):

No way to talk it away, it looks ugly, no question.

> Perhaps it is just a question of formatting and it could be made to

Even better formatting would probably not prevent from having a (too) high 
comment to code line ratio :(. I agree that the code looks overly cluttered, I 
would also be happy to have a solution at hand which allows both for verbose 
documentation (and maybe even internal notes, which the the comment example 
above contains) and keeping the code clean. On the other hand the main 
advantage of source embedded documentation is that you can document the code 
while you are writing it, meaning it is easily embedded into the 
implementation workflow. It would be nice to have a kind out of source 
documentation tool, which does not add too much extra work to write 
documentation.

---

From: Chris Richardson <[email-masked]>
Date: Mon, 22 Mar 2010 10:58:28 +0000

>   http://doc.php.net/php/dochowto/chapter-skeletons.php
Docbook does have the advantage of being mature... if you look at an 
actual example, e.g.

http://svn.php.net/repository/phpdoc/en/trunk/language/types/array.xml

it does not seem too bad, just XML, (OK - you might think this is 
terrible)... and it would allow a certain amount
of auto-generation (e.g. function prototypes) from the code.

Chris

---

From: Hans Petter Langtangen <[email-masked]>
Date: Mon, 22 Mar 2010 12:19:06 +0100

> text-based format that looks natural (very similar to Publish).

It's very similar to Publish (simple format, some options not found
elsewhere), and as with Publish, you can at any time convert to a
standard format such as BibTeX and go with that for the future.
Doconce can at any time be transformed to reST, which is the standard
markup language for sphinx and Python documentation in general.
It should be straightforward to have a script that automates the transition
from doconce to reST in the documentation.
 
> some preprocessing stage?

I usually put the doconce documentation in separate files and
preprocess the source code. The text can be put in the source instead,
but then you need a little script to extract the text such that you
can generate LaTeX and HTML manuals, etc. 

Personally, I like plain text with minimal tagging in the source code,
which means that I filter doconce to plain text before the source code
files are preprocessed (and sometimes I filter to Epytext and insert
in doc strings to make Epydoc manuals, or to reST for sphinx manuals).

> contain everything and make a Programmer's Reference unnecessary.

I think this will be the same. Anyway, just start writing the manual and see
what need it covers.

Hans Petter

---

From: Hans Petter Langtangen <[email-masked]>
Date: Mon, 22 Mar 2010 12:38:54 +0100

>   http://doc.php.net/php/dochowto/chapter-skeletons.php

There is a tool rst2docbook.py, as part of experimental docutils, for
converting reST to docbook. (This also means that you can go from
doconce to reST to docbook.)

Hans Petter

---

From: Anders Logg <[email-masked]>
Date: Mon, 22 Mar 2010 13:46:16 +0100

> > text-based format that looks natural (very similar to Publish).

Footnote: http://bitbucket.org/logg/publish/

> from doconce to reST in the documentation.

I haven't seen reST before. I took a closer look now and it looks
pretty nice. Was the motivation for doconce to make it look even
nicer? reST already looks pretty nice to me.

> in doc strings to make Epydoc manuals, or to reST for sphinx

I still don't get it. So you have foo.do with the documentation, but
where is the code? Is it in foo.h.pre and then you generate foo.h from
foo.do and foo.h.pre?

  foo.do + foo.h.pre --> foo.h
  foo.do --> foo.html + foo.pdf

--
Anders

---

From: Anders Logg <[email-masked]>
Date: Mon, 22 Mar 2010 14:50:58 +0100

> [0] http://sphinx.pocoo.org/

I looked a bit more on Sphinx, and it looks really good.

The documentation for Sphinx states the following:

  The focus is on hand-written documentation, rather than
  auto-generated API docs.

This seems to fit very well with our discussion regarding whether or
not to put the documentation as part of the code.

So Sphinx would be a tool we could use to produce good looking,
cross-referenced, indexed documentation from a set of simple input
files (in reST or doconce format), but we would not extract anything
from the code.

With this approach, I wonder where the limitations are for documenting
the C++ interface. Are there any? If we don't use Doxygen, we would
just write the documentation in the same way as for the Python
interface.

And I just thought of another reason for splitting the documentation
from the code which is that it makes it possible to separate write
access for the code and the documentation.

--
Anders

---

From: Kristian Oelgaard <[email-masked]>
Date: Mon, 22 Mar 2010 14:59:16 +0100 (CET)

>  foo.do --> foo.html + foo.pdf

That's how I understood it. I guess that way we can keep foo.h.pre pretty clean such that developers will not see all the documentation, but on the downside the foo.h has to be generated everytime something has changes in the implementation.
Is the other option to write documentation completely independent of the source code in doconce (or docbook, the output of which I think looks very nice). If we can write it in doconce and translate it to prettymuch any format we want later that is ok for me too.

Kristian

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
>
> iEYEARECAAYFAkunZpgACgkQTuwUCDsYZdFqJwCeKeD1GrZFc0XNVyl3+TNGnyxg
> kUEAoIhNXiLhgVn6WeinWNxEjo9DUSuP
> =a+I6
> -----END PGP SIGNATURE-----
>

---

From: Kristian Oelgaard <[email-masked]>
Date: Mon, 22 Mar 2010 15:08:32 +0100 (CET)

>  The focus is on hand-written documentation, rather than

So we will abandon Doxygen and similar tools? Sounds good to me, but I guess it will make it slightly more difficult to ensure that all functions are documented and not forgotten. But maybe there are tools for this?

> not to put the documentation as part of the code.

So we only put short comments in the code like we do now, and then put documentation in sphinx source files.

> access for the code and the documentation.

Good point.

Kristian

> iEYEARECAAYFAkundcIACgkQTuwUCDsYZdGbegCgkvje5skw8pkEHWWKRgW4CrFh
> zjMAn15eImocRibmFiLQExZkClQ8oTdg
> =p16A

---

From: Anders Logg <[email-masked]>
Date: Mon, 22 Mar 2010 15:16:15 +0100

On Mon, Mar 22, 2010 at 03:08:32PM +0100, Kristian Oelgaard wrote:
> So we will abandon Doxygen and similar tools? Sounds good to me, but
> I guess it will make it slightly more difficult to ensure that all
> functions are documented and not forgotten. But maybe there are
> tools for this?

I imagine we can write a script that checks for functions that need to
be documented. We could also have a script that checks that the
one-line compact comment in the code is the same as the summary/first
sentence in the hand-written documentation.

> So we only put short comments in the code like we do now, and then
> put documentation in sphinx source files.

That would be my suggestion. Let's hear what people think.

--
Anders

---

From: Kristian Oelgaard <[email-masked]>
Date: Mon, 22 Mar 2010 15:22:24 +0100 (CET)

> sentence in the hand-written documentation.

Sure, I was just wondering if someone out there on the big Internet had already done this job for us.

> That would be my suggestion. Let's hear what people think.

OK.

Kristian

> iEYEARECAAYFAkune68ACgkQTuwUCDsYZdG2AACgioXxnxrq1eKeU6pPWOin8Yg6
> /yIAniukS2aro5GY1WxufNObgjC2zlBK
> =nHn8

---

From: Hans Petter Langtangen <[email-masked]>
Date: Mon, 22 Mar 2010 20:04:49 +0100

> nicer? reST already looks pretty nice to me.

The reason is about the same as why develop Publish when BibTeX exists
;-) I wanted to have plain untagged ASCII as a format and found it too
demanding to write a new backend for docutils. I also wanted to have
support for math in the text (there is upcoming suppert for LaTeX in
reST, but not in any official version), plus support for computer code
copied directly from source files and typeset in a variety of formats.
The LaTeX and HTML source files generated from reST are quite ugly
too. Finally, it's simpler to add certain features to a simple markup
language than to reST.

>   foo.do --> foo.html + foo.pdf

Yes! 

Hans Petter

---

From: Hans Petter Langtangen <[email-masked]>
Date: Mon, 22 Mar 2010 20:17:51 +0100

> That would be my suggestion. Let's hear what people think.

Sphinx is a great tool, standardized and well supported, so if it can
be used for the C++ code, it's probably a very good choice for
FEniCS. The documentation and the source code are then separated, but
the documentation does not go into doc strings (which is the purpose
of preprocessing source code files).

Hans Petter

---

From: Johan Hake <[email-masked]>
Date: Mon, 22 Mar 2010 13:04:49 -0700

> of preprocessing source code files).

Shouldn't it be possible to add doc strings to a library based on some 
external reference documentation? Something like:

for names in dolfin.__dict__:
   dolfin.__dict__[name].__doc__ = documentation[name]

BTW: I have to admit that I am not too keen on having the documentation and 
source code on two different places, yet... 

Johan

---

From: Anders Logg <[email-masked]>
Date: Mon, 22 Mar 2010 21:15:31 +0100

>    dolfin.__dict__[name].__doc__ = documentation[name]

Everything should be possible with Python. :-) I like that idea rather
than needing to run a preprocessor.

> source code on two different places, yet...

I think it makes a lot of sense, especially for Python where the
declaration and implementation is mixed up in the same file. Consider
for example the important documentation of the Expression class. It is
currently buried deep in site-packages/dolfin/function/expression.py
on lines 332-470, just below class ExpressionMetaClass and def
expression__call__(self, *args, **kwargs) btw... ;-)

Having the documentation spread out and buried deep across multiple
files makes it difficult to edit. For example, it would be difficult
to ask someone to go through the documentation and check
grammar/spelling.

--
Anders

---

From: Johan Hake <[email-masked]>
Date: Mon, 22 Mar 2010 13:28:59 -0700

On Monday 22 March 2010 13:15:31 Anders Logg wrote:
> > BTW: I have to admit that I am not too keen on having the documentation
> > and source code on two different places, yet...
> 

Ok, slowly seeing your point :)

> to ask someone to go through the documentation and check

Getting there!

Johan

> > > > > >files (in reST or doconce format), but we would not extract
> > > > > > anything from the code.
> > > > > >
> > > > > >With this approach, I wonder where the limitations are for
> > > > > > documenting the C++ interface. Are there any? If we don't use
> > > > > > Doxygen, we would just write the documentation in the same way as
> > > > > > for the Python interface.
> > > > > >
> > > > > >And I just thought of another reason for splitting the
> > > > > > documentation from the code which is that it makes it possible to
> > > > > > separate write access for the code and the documentation.
> > > > >
> > > > > Good point.
> > > > >
> > > > > Kristian
> > > >

---

From: Anders Logg <[email-masked]>
Date: Mon, 22 Mar 2010 21:56:04 +0100

> too. Finally, it's simpler to add certain features to a simple markup

I'm probably missing something here, but I don't see much difference
between the reST and doconce input. Take for example the following page:

  http://sphinx.pocoo.org/rest.html

The reST input can be viewed by clicking "Show Source" in the right
column. What would the corresponding input look like for doconce?

You don't need to retype the entire page... :-) but maybe some interesting
paragraph that would look different in doconce.

> >   foo.do --> foo.html + foo.pdf

I'm not too keen on not editing the source files directly, but from
what I understand, you do this to make things work with docstrings in
Python? Then this can be avoided by Johan Hake's trick (assigning some
appropriate output from doconce to __doc__).

--
Anders

---

From: Hans Petter Langtangen <[email-masked]>
Date: Tue, 23 Mar 2010 08:28:52 +0100

> column. What would the corresponding input look like for doconce?

Very, very similar! There are only some small differences for making
it easier to parse the text with regular expressions (which I use for
simplicity, but that's sometimes a real hazzle). 

There are a couple of features of doconce that I wanted to have: 
1) copying of source code from files, with specification of a certain
style of typesetting the code (in LaTeX), and 2) better support for
equations and inline math so that the math comes nicely out in LaTeX.
Doconce has special paragraphs for code and math, so this is the
biggest (and almost only) syntax difference from reST.  These
constructions are nice things if you work on some document where the
final format is unclear, but LaTeX (=book/paper) is very likely. Over
the years I've written a lot of educational material that have ended
up in LaTeX books, but started it's life as doconce text and lived for
a while in HTML, some parts also as Epytext and plain text for Python
(module/package) doc strings. The ugly LaTeX from reST was not
suitable for (my) books ;-)

For FEniCS documentation you don't need that flexibility, and you have
more emphasis on HTML output than LaTeX or other formats. Then reST
is definitely the answer! This is my clear conclusion now after having
thought about it for a few days :-) Sphinx + reST! (If it's okay for C++ too)

> appropriate output from doconce to __doc__).

Sure, Python can preprocess itself - I do that a lot. One example is
to list all options of various kinds in static class variables, use
these lists (or dicts) for checking input in the code and provide nice
error messages, while these lists can also be formatted nicely and
inserted automatically in doc strings so that you always have an
updated list of what's possible in a class or function when you
process the doc string (pydoc, help in the interpreter, Epydoc, I
guess it works with sphinx too). Just the import statement does all
the processing. Large sections of text that goes into various
documents, can either be preprocessed (#include) or loaded into doc
strings at import time (the latter strategy takes more time and is not
my personal favorite, but that's just a matter of taste - I'm very
much a preprocessing guy).

Hans Petter

---

From: Anders Logg <[email-masked]>
Date: Tue, 23 Mar 2010 20:54:37 +0100

> suitable for (my) books ;-)

ok! I haven't seen the LaTeX output from Sphinx.

> thought about it for a few days :-) Sphinx + reST! (If it's okay for C++ too)

ok, let's start with reST! Perhaps we will realize the benefits of
doconce somewhere along the road.

> much a preprocessing guy).

I personally prefer import. Hopefully the overhead is not noticeable.

--
Anders