Subject: [Fenics] Docstrings etc

From: Anders Logg <[email-masked]>
Date: Thu, 26 Aug 2010 18:22:30 +0200

I've thought some more on how to organize/synchronize the FEniCS
documentation (in fenics-doc) with the documentation we have in the
code.

I think it is important that

(1) the strings we have in the code are the same as those that appear
on in the HTML documentation (which we write in Sphinx).

(2) the strings we have in the code are short (so they don't clutter
up the code)

If we look at these two, it seems that (1) implies that we should
write the documentation as part of the code and then extract it using
some tool.

But (2) prevents that since we don't want to constrain the
documentation for all functions to be very short.

How about the following solution.

* Write short docstrings in the code

* Auto-generate all the .rst input files for the Programmer's
  Reference using a simple Python script that looks for '///'

* The script looks at the code to generate the signature of the
  function and the text that comes immediately after.

  But it also looks in a hand-written .rst file that contains any
  additional stuff we want to put below.

So for the code example in the style manual, the things that get
picked up from the code are

  // Return the cell which is closest to the given point
  uint closest_cell(const Point & point) const

which gets converted to

.. cpp:function:: uint closest_cell(const Point & point) const

    Return the cell which is closest to the given point

The script also looks in a file for "closest_cell" below which we have
written all the *Arguments* stuff that will be thrown in below.

Will that work?

Another solution would be to just write everything as part of the
code, and just add some settings to our editors that will fold the
extra stuff away so we don't need to see it. Maybe that is the most
robust solution?

--
Anders

---

From: Kristian Ølgaard <[email-masked]>
Date: Thu, 26 Aug 2010 19:42:35 +0200

> (2) the strings we have in the code are short (so they don't clutter

I disagree. The whole idea of the documentation effort was to document
in one place
(using carefully handwritten and elaborate explanations including
examples and links to demos etc.) and code in another.
The comments in the code should be very short and precise such that
together with the class/function definition and type info the
developer can complete the task without looking elsewhere. These kind
of comments, I expect, will look weird when put next to an elaborate
explanation on how the class/function works including all the bells
and whistles.

>  function and the text that comes immediately after.

This might be possible for a simple
'change-order-of-comment-and-function' script where you manipulate the
output manually afterwards, but if you want to run this more than once
you will have to pick up nested class/struct definitions templates and
all kinds of crap.
I tried to write a parser like this to check if all classes and
functions were documented, but gave up and let Doxygen do the dirty
work. (But do we want to do this just to generate 20 characters of
docstring automatically?)

> written all the *Arguments* stuff that will be thrown in below.

Yes, but the work flow is getting complex, and you'll need to know
what you get from the source code so you don't repeat yourself.
It is much easier to have the documentation in one place.

> extra stuff away so we don't need to see it. Maybe that is the most

The general consensus the last time this issue came up was not to
clutter the code with documentation markup.

Kristian

> --
> Anders
>
> _______________________________________________
> Mailing list: https://launchpad.net/~fenics
> Post to     : [email-masked]
> Unsubscribe : https://launchpad.net/~fenics
> More help   : https://help.launchpad.net/ListHelp
>

---

From: Anders Logg <[email-masked]>
Date: Thu, 26 Aug 2010 19:51:06 +0200

> clutter the code with documentation markup.

I agree it's good to have the documentation in one place, but it would
be good if we found a way to keep it in sync. Helper scripts can do
some of that work, but we probably won't be able to pick up things
like having

  "Compute the number of neighbors"

in one place and

  "Return the number of neighbors"

in other places. Things like this will creep in over time. It might
not be a big issue but I find it a bit annoying.

Won't we always have the documentation string (in .rst) start with a
simple thing like this? "Return this or that", or "Compute foo based
on...". Then a more elaborate explanation follows.

--
Anders

---

From: Kristian Ølgaard <[email-masked]>
Date: Thu, 26 Aug 2010 20:09:56 +0200

> not be a big issue but I find it a bit annoying.

I see. A simpler approach, rather than generating docstrings would be
to have a script that
simply looks for '///' comments in dolfin/mesh/Mesh.h and check if the
EXACT same strings are present in
programmers-reference/cpp/mesh/Mesh.rst, if not crash test and let
user figure out manually why it failed and which comment/docstring
should be changed.
This won't be completely bulletproof, but much much simpler than
parsing a C++ library.
I currently check if the docstrings of the documentation for the
Python interface is equal to the docstrings of the DOLFIN module after
import so that sort of works in the same way, only in this case I know
that the docstring I check belongs to function 'bar' of class 'foo'.

Then we use the stub-generator that you have know to give us the first
set of *.rst files and then add the '///' comments check to the
verify_cpp_documentation.py script.

Kristian

> on...". Then a more elaborate explanation follows.

Probably, I just copied the docstrings which were already there.

Kristian

---

From: Anders Logg <[email-masked]>
Date: Thu, 26 Aug 2010 20:16:41 +0200

> This won't be completely bulletproof, but much much simpler than

Yes, that might be a good solution.

> verify_cpp_documentation.py script.

It's almost there now, I just need to do some polishing.

Sphinx is currently crashing when it generates the documentation from
the .rst files I generate.

Exception occurred:
  File "/usr/lib/pymodules/python2.6/docutils/nodes.py", line 1898, in
  dupname
    node['names'].remove(name)
ValueError: list.remove(x): x not in list

Any ideas what this might be?

--
Anders

---

From: Anders Logg <[email-masked]>
Date: Thu, 26 Aug 2010 20:35:17 +0200

> Any ideas what this might be?

Looks like this happens when there are multiple functions with the
same signature.

--
Anders

---

From: Kristian Ølgaard <[email-masked]>
Date: Thu, 26 Aug 2010 21:34:01 +0200

> Looks like this happens when there are multiple functions with the

Very likely,  and that's probably because you need to extract 'const'
information too, and that's just the tip of the iceberg if we proceed
down this road....

Kristian

---

From: Anders Logg <[email-masked]>
Date: Thu, 26 Aug 2010 21:38:44 +0200

> information too, and that's just the tip of the iceberg if we proceed

It's working alright now. It's looking pretty good. I'll be pushing
soon so you can check.

--
Anders

---

From: Anders Logg <[email-masked]>
Date: Thu, 26 Aug 2010 22:04:49 +0200

> information too, and that's just the tip of the iceberg if we proceed

Try now.

You need to set DOLFIN_DIR to the DOLFIN source tree.

Then run

  python utils/generate_cpp_doc.py
  make html

The generated stuff is in {source/build}/programmers-reference/test/cpp

I'll be moving it to {source/build}/programmers-reference/cpp and make
sure not to overwrite the Mesh and Point class documentation that you
have written.

--
Anders

---

From: Kristian Ølgaard <[email-masked]>
Date: Thu, 26 Aug 2010 22:09:16 +0200

> The generated stuff is in {source/build}/programmers-reference/test/cpp

OK, I'm just finishing a DOLFIN build to test the docstrings in the
Python interface. Will test soon.

> sure not to overwrite the Mesh and Point class documentation that you

There is no C++ documentation for Point, only for the Python interface
and that was just to see how some of the autodoc functions worked.
Anyway, we can always dig it up by reverting the repo to hack away.

Kristian

---

From: Anders Logg <[email-masked]>
Date: Thu, 26 Aug 2010 22:13:21 +0200

> Anyway, we can always dig it up by reverting the repo to hack away.

I noticed that. I just remember seeing something about the Point
class.

Anyway, it seems to work now. What is missing is to generate the
index.rst files for each module.

--
Anders

---

From: Kristian Ølgaard <[email-masked]>
Date: Thu, 26 Aug 2010 22:34:02 +0200

> index.rst files for each module.

Looks pretty good to me. Do you need to generate the index.rst files?
Can't you just add the output from 'ls *.h' in the modules to the
index.rst files?
Once we're finished editing the *.rst files you have generated we
should be able to run the script verify_cpp_documentation.py which
should tell us if we missed any.
BTW, I'm done for today.

Kristian

---

From: Anders Logg <[email-masked]>
Date: Fri, 27 Aug 2010 08:43:01 +0200

> should tell us if we missed any.

I think we should think hard on this one more time. Is it really that
bad do write the documentation as part of the code?

The stuff that you have written for the Mesh class could easily go in
to Mesh.h without causing too much clutter (reST looks nice), and I
imagine it would be easy to add a folding mode to Emacs and other
editors that will hide all lines starting with /// except for the
first line.

The simple script I wrote seems to work pretty well to extract the
documentation. If it breaks somewhere, we could either improve the
script or learn to write the code so the script does not break.

The point here is that now the generated .rst files are in sync with
the code, but in a day or two someone will edit one of the .h files in
DOLFIN and the documentation and code will start to diverge.

--
Anders

---

From: "Garth N. Wells" <[email-masked]>
Date: Fri, 27 Aug 2010 07:54:45 +0100

On 27/08/10 07:43, Anders Logg wrote:
>> On 26 August 2010 22:13, Anders Logg<[email-masked]>  wrote:
>>>> On 26 August 2010 22:04, Anders Logg<[email-masked]>  wrote:
>>>>>> On 26 August 2010 20:35, Anders Logg<[email-masked]>  wrote:
>>>>>>>>> On 26 August 2010 19:51, Anders Logg<[email-masked]>  wrote:
>>>>>>>>>>> On 26 August 2010 18:22, Anders Logg<[email-masked]>  wrote:
>>>>>>>>>>>>   uint closest_cell(const Point&  point) const
>>>>>>>>>>>>
>>>>>>>>>>>> which gets converted to
>>>>>>>>>>>>
>>>>>>>>>>>> .. cpp:function:: uint closest_cell(const Point&  point) const
>>>>>>>>>>>>

It's good to have it in the code as long as its not too long and not 
full of mark up. The thing I look for most are function declarations, so 
I find it annoying when I can't find a declaration for all the markup. 
It's also hard to get an overview of a class when only a few 
declarations fit on the screen amongst the markup with funny symbols.

I do like long docstrings in Python. Because the argument list is not 
statically typed and there's more magic in Python, a good docstring is 
essential.

Garth

---

From: Kristian Ølgaard <[email-masked]>
Date: Fri, 27 Aug 2010 09:06:49 +0200

On 27 August 2010 08:54, Garth N. Wells <[email-masked]> wrote:
>>>>>> The generated stuff is in
>>>>>> {source/build}/programmers-reference/test/cpp
>>>>>
> It's good to have it in the code as long as its not too long and not full of
> mark up. The thing I look for most are function declarations, so I find it
> annoying when I can't find a declaration for all the markup. It's also hard
> to get an overview of a class when only a few declarations fit on the screen
> amongst the markup with funny symbols.

The markup that we plan to use will be pretty simple (see the source
for the C++ Mesh.rst), but it will add a lot of extra lines to the
source code.
I too would find this annoying.

>> DOLFIN and the documentation and code will start to diverge.

Yes, but this problem is already there for the Python interface and it
won't go away.
I guess the key thing to this is that a new feature or a change in
DOLFIN source code is not complete until the documentation has been
updated.

Kristian

---

From: "Garth N. Wells" <[email-masked]>
Date: Fri, 27 Aug 2010 08:39:21 +0100

On 27/08/10 08:06, Kristian Ølgaard wrote:
> On 27 August 2010 08:54, Garth N. Wells<[email-masked]>  wrote:
>>
>>

To save ourselves work for now, we could just let doxygen create the C++ 
programmers reference and provide a link to it. It doesn't seem very 
sensible that we write our own parser to document the C++ code. With 
doxygen, we also get class diagrams. We can then scan the doxygen 
documentation for each class and improve it iteratively.

Garth

---

From: Kristian Ølgaard <[email-masked]>
Date: Fri, 27 Aug 2010 11:24:41 +0200

On 27 August 2010 09:39, Garth N. Wells <[email-masked]> wrote:
>>>>>>>>>>>>>>> documentation (in fenics-doc) with the documentation we have
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> code.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (1) the strings we have in the code are the same as those
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> appear
>>>>>>>>>>>>>> developer can complete the task without looking elsewhere.
>>>>>>>>>>>>>> These
>>>>>>>>>>>>>> kind
>>>>>>>>>>>>>>> write the documentation as part of the code and then extract
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>> some tool.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> work. (But do we want to do this just to generate 20
>>>>>>>>>>>>>> characters
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> The script also looks in a file for "closest_cell" below
>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>> we have
>>>>>>>>>>>>>>> code, and just add some settings to our editors that will
>>>>>>>>>>>>>>> fold
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> extra stuff away so we don't need to see it. Maybe that is
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>> robust solution?
>>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree it's good to have the documentation in one place, but
>>>>>>>>>>>>> it
>>>>>>>>>>>>> would
>>>>>>>>>>>>> be good if we found a way to keep it in sync. Helper scripts
>>>>>>>>>>>>> can
>>>>>>>>>>>>> do
>>>>>>>>>>>> I see. A simpler approach, rather than generating docstrings
>>>>>>>>>>>> would
>>>>>>>>>>>> be
>>>>>>>>>>>> to have a script that
>>>>>>>>>>>> simply looks for '///' comments in dolfin/mesh/Mesh.h and check
>>>>>>>>>>>> if
>>>>>>>>>>>> the
>>>>>>>>>>>> import so that sort of works in the same way, only in this case
>>>>>>>>>>>> I
>>>>>>>>>>>> know
>>>>>>>>>>>   File "/usr/lib/pymodules/python2.6/docutils/nodes.py", line
>>>>>>>>>>> 1898,
>>>>>>>>>>> in
>>>>>>>>>>>   dupname

On second thought, what do you mean by diverge?
I have test scripts in place the checks if a function in *.h is
documented in *.rst, and if a function in *.rst is still present in
*.h.

If you mean the docstrings might change, we can perform the additional
check where we test if the one liner docstring in *.h is present in
the documentation in *.rst, then there can be no divergence and we can
have short comments in the DOLFIN source code.

> documentation for each class and improve it iteratively.

Do you mean improve the Doxygen output, or the source  code (*.h
files)? If we improve the output we can get diverging docs and code.

I had a look at the Breathe generated docs from Doxygen, it doesn't
look that great and we won't have all the links from the index page to
the classes.

Kristian

---

From: "Garth N. Wells" <[email-masked]>
Date: Fri, 27 Aug 2010 10:31:20 +0100

> files)? If we improve the output we can get diverging docs and code.

I mean improve the strings following '///' in the .h files. In quite
some cases, just a few extra words would make a big difference.

I'm coming around to putting all programming reference doc in the code.
I don't like lots of markup, but I don't see any other robust and easily
maintainable solution.

Perhaps we should settle on well crafted, handwritten docs for the demos
(which is the most important), and all programming reference docs coming
automatically from the source code. When scanning auto generated docs,
it easy to pick up quickly where more detail needs to be added, and we
can open open up the source and add it.

Garth

---

From: Kristian Ølgaard <[email-masked]>
Date: Fri, 27 Aug 2010 11:51:21 +0200

> I don't like lots of markup, but I don't see any other robust and easily

As I wrote above, a test script is in place to pick up
missing/obsolete docs, very little extra work is needed to also test
if the short docstring in the source  code is correct. Then we run the
tests as part of building the documentation.

I admit that the Doxygen output is much more detailed and the type
information/links in argument lists is better compared to what is in
Sphinx now, but that might change in the future (in Sphinx). On the
downside, I personally find the Doxygen documentation overwhelming and
I never use it for just that reason.

> can open open up the source and add it.

A benefit of having the manually written programmer's reference is
that it is easy to link to demos for additional information (and the
other way around).

If we use automatically generated docs, we can only link to the
programmer's reference when documenting demos (it won't be well
integrated in Sphinx though and it might break more easily).

I agree that well documented demos is most important since this is
what users will be copying anyway.

Kristian

---

From: "Garth N. Wells" <[email-masked]>
Date: Fri, 27 Aug 2010 11:00:19 +0100

> tests as part of building the documentation.

I just can't see myself hopping back and forth between the code and the
documentation when implementing and testing something new.

> I never use it for just that reason.

Doxygen is an (imperfect) ready made solution for the programmers
reference - so one of my points is that we can forget about the C++
programmers reference for now and get on with the more importance task
of documenting demos. We can return to the C++ programmers reference
later (which, as you say, may improve in Sphinx in the future).

We can some some very limited work to improve the doxygen output which
will make it easier to navigate.

Garth

---

From: Kristian Ølgaard <[email-masked]>
Date: Fri, 27 Aug 2010 12:12:59 +0200

> documentation when implementing and testing something new.

I don't see why that would be necessary, the documentation can be
updated and built later once the feature is in place and tested.
But the feature can't be 'official' until it has been documented, it
will require more self-discipline from the developers, which I don't
think is necessarily a bad idea.

> later (which, as you say, may improve in Sphinx in the future).

I'm fine with using Doxygen and simply put a link to the index page, I
just think it is worthwhile to carefully discuss the pros and cons.

The Python interface still has to be documented manually since there
is no way to extract docstrings from the source code since the
intention is to add docstrings to the module.

> will make it easier to navigate.

I just don't see how this can be integrated easily with the output from Doxygen.
We don't want to manipulate the output files since they will be
re-generated whenever we build the docs. It is possible though to link
to the html pages of classes/functions, but it won't be naturally
supported like it would be if everything is in Sphinx.

Kristian

---

From: Kristian Ølgaard <[email-masked]>
Date: Fri, 27 Aug 2010 13:09:06 +0200

> just think it is worthwhile to carefully discuss the pros and cons.

Let's wait for someone else (Anders?) to comment on this before making
a decision.
On a related note, it is interesting that only the three of us appears
to have an opinion about the look and feel of the documentation of the
programmer's reference. Are we perhaps making a big deal out of
something which nobody will use anyway?

> intention is to add docstrings to the module.

Since we do need this, I'll start documenting the Python interface
starting with the functions and classes in the cpp.py module.

Kristian

---

From: Anders Logg <[email-masked]>
Date: Fri, 27 Aug 2010 13:09:31 +0200

> supported like it would be if everything is in Sphinx.

I think that the simple script we have now does a fairly good job at
extracting the documentation. I like having it as part of the
reST-based documentation (so it looks like it's part of the
documentation), rather than as a separate set of pages generated by
Doxygen. But I wouldn't mind having Doxygen-generated pages in
addition.

I agree with Garth that it will be difficult to jump between two
different repositories to make updates every time we change the name
of an argument to a function, or add a new function.

Maybe we could try to put the documentation Kristian has written in
reST for the Mesh class into Mesh.h and see how that works out.

I don't think it will be much of a problem to have it there,
especially if we employ some folding mode in our editors.

--
Anders

---

From: "Garth N. Wells" <[email-masked]>
Date: Fri, 27 Aug 2010 12:11:10 +0100

> Doxygen. But I wouldn't mind having Doxygen-generated pages in

The doc generated by the script lacks links, which is a big drawback.

Garth

---

From: Anders Logg <[email-masked]>
Date: Fri, 27 Aug 2010 13:14:35 +0200

> Let's wait for someone else (Anders?) to comment on this before making

I think we can have both side by side (since both are working), but I
prefer having the reST comments as part of the code and using our own
script to extract it (or in the future built-in support in Sphinx if
that should come). If we use Doxygen, we will need to use some special
Doxygen-markup.

> something which nobody will use anyway?

I would be good to get some more comments from others.

> starting with the functions and classes in the cpp.py module.

I still haven't understood completely how the Python documentation is
organized, but won't we be documenting Python code (the stuff we write
in site-packages/dolfin as part of the code?

--
Anders

--
Anders

---

From: Anders Logg <[email-masked]>
Date: Fri, 27 Aug 2010 13:17:30 +0200

> The doc generated by the script lacks links, which is a big drawback.

Links to other classes you mean?

That would be easy to add to the script. We could just insert labels
and references into the generated code.

Perhaps we'll find that we should use a fancy tool like Doxygen, but
it seems it's not that complicated and it took a very short time to
write our own script.

--
Anders

---

From: Kristian Ølgaard <[email-masked]>
Date: Fri, 27 Aug 2010 13:30:56 +0200

>> > Doxygen. But I wouldn't mind having Doxygen-generated pages in

I agree that having it in reST makes it look more like it is a part of
the documentation.

> and references into the generated code.

What I have done with the links in the Mesh.rst so far is to put it in
the *Arguments* section like:
:cpp:class:`Point`, we can have still have this as part of the source
code in the *.h files.

Kristian

---

From: Kristian Ølgaard <[email-masked]>
Date: Fri, 27 Aug 2010 13:43:17 +0200

> in site-packages/dolfin as part of the code?

The idea was to simply use the docstrings from the pseudo docstrings
module since we need those anyway for everything which is in cpp.py.
At least that's what we agreed upon a long time ago.

In short we document everything in the fenics-doc/docstrings/dolfin
module which has the exact same layout as the real dolfin module, but
it only contains docstrings (no code).

We use the docstrings module to create the docstrings.i file such that
we have the correct docstrings for the stuff in the dolfin/cpp.py
module because we can't assign to docstrings on import for certain
classes/functions.

We also copy the docstrings module to site-packages/dolfin such that
we in e.g., dolfin/fem/form.py can do:

import dolfin.docstrings

class Form(cpp.Form):
   __doc__ = dolfin.docstrings.fem.Form.__doc__

it might seem silly, but the big advantage is that we have all docs in
one place which was one of the arguments for splitting the
documentation into a different repository in the first place.

Kristian

---

From: Anders Logg <[email-masked]>
Date: Fri, 27 Aug 2010 18:53:15 +0200

> At least that's what we agreed upon a long time ago.

That might still be the best conclusion. The reason this is coming up
again (although we discussed it at length before) is I understand more
after having actually worked with the documentation over the last
couple of days.

> it only contains docstrings (no code).

Can we make this part of DOLFIN instead of copying it from the
documentation repo? That would make it easier to keep in sync. So all
documentation that goes into the programmer's reference (as part of
the documentation) is in some way copied from DOLFIN to the
documentation (not the opposite).

> module because we can't assign to docstrings on import for certain

And this replaces the current system where the stuff in docstrings.i
is extracted from C++? So all Python functions will have their own
documentation and that is always unrelated to the C++ documentation?

Wouldn't it be good to have the C++ extracted stuff and then overwrite
it when necessary, to avoid duplication?

> documentation into a different repository in the first place.

ok.

--
Anders

---

From: Anders Logg <[email-masked]>
Date: Fri, 27 Aug 2010 18:55:41 +0200

> :cpp:class:`Point`, we can have still have this as part of the source

ok. Does everyone agree that as a first iteration, it would be good to
at least try to:

1. Move over the stuff from Mesh.rst (in the doc repo) to Mesh.h (in
DOLFIN)

2. Generate all .rst files in the doc repo from the C++ code

3. Evaluate if we can live with the more heavy commenting in the .h
files (or find a folding mode)

4. Evaluate if the simple extract script does its job well enough

?

--
Anders

---

From: "Garth N. Wells" <[email-masked]>
Date: Fri, 27 Aug 2010 19:28:22 +0100

On 27/08/10 17:55, Anders Logg wrote:
>> On 27 August 2010 13:17, Anders Logg<[email-masked]>  wrote:
>>>>>> On 27 August 2010 12:00, Garth N. Wells<[email-masked]>  wrote:
>>>>>>>
>>>>>>>
>>>>>>>> On 27 August 2010 11:31, Garth N. Wells<[email-masked]>  wrote:
>>>>>>>>>

OK with me.

Garth

---

From: Marie Rognes <[email-masked]>
Date: Fri, 27 Aug 2010 22:17:08 +0200

> I would be good to get some more comments from others.

Frankly, I don't really give a darn.

I think we should have a programmer's reference. I will use it
frequently. I will use it mainly to see what kind of functions that are
available from a class. I think it is important that it is up-to-date.

Aside from that, I trust you to devise a clever setup.

--
Marie

---

From: "Garth N. Wells" <[email-masked]>
Date: Fri, 27 Aug 2010 21:29:52 +0100

> available from a class. I think it is important that it is up-to-date.

Sounds like a vote for doxygen ;).

Garth

---

From: Johan Hake <[email-masked]>
Date: Fri, 27 Aug 2010 14:10:24 -0700

On Friday August 27 2010 09:55:41 Anders Logg wrote:
> > >> >>>>>>>>>> The stuff that you have written for the Mesh class could
> > >> >>>>>>>>>> easily go in to Mesh.h without causing too much clutter
> > >> >>>>>>>>>> (reST looks nice), and I imagine it would be easy to add a
> > >> >>>>>>>>>> folding mode to Emacs and other editors that will hide all
> > >> >>>>>>>>>> lines starting with /// except for the first line.
> > >> >>>>>>>>>> 
> > >> >>>>>>>>>> The simple script I wrote seems to work pretty well to
> > >> >>>>>>>>>> extract the documentation. If it breaks somewhere, we
> > >> >>>>>>>>>> could either improve the script or learn to write the code
> > >> >>>>>>>>>> so the script does not break.
> > >> >>>>>>>>>> 
> > >> >>>>>>>>>> The point here is that now the generated .rst files are in
> > >> >>>>>>>>>> sync with the code, but in a day or two someone will edit
> > >> >>>>>>>>>> one of the .h files in DOLFIN and the documentation and
> > >> >>>>>>>>>> code will start to diverge.
> > >> >>>>>> 
> > >> >>>>>> documented in *.rst, and if a function in *.rst is still
> > >> >>>>>> present in *.h.
> > >> >>>>>> 
> > >> >>>>>> If you mean the docstrings might change, we can perform the
> > >> >>>>>> additional check where we test if the one liner docstring in
> > >> >>>>>> *.h is present in the documentation in *.rst, then there can
> > >> >>>>>> be no divergence and we can have short comments in the DOLFIN
> > >> >>>>>> source code.
> > >> >>>>>> 
> > >> >>>>>>>> Yes, but this problem is already there for the Python
> > >> >>>>>>>> interface and it won't go away.
> > >> >>>>>>>> I guess the key thing to this is that a new feature or a
> > >> >>>>>>>> change in DOLFIN source code is not complete until the
> > >> >>>>>>>> documentation has been updated.
> > >> >>>>>>> 
> > >> >>>>>>> To save ourselves work for now, we could just let doxygen
> > >> >>>>>>> create the C++ programmers reference and provide a link to
> > >> >>>>>>> it. It doesn't seem very sensible that we write our own
> > >> >>>>>>> parser to document the C++ code. With doxygen, we also get
> > >> >>>>>>> class diagrams. We can then scan the doxygen documentation
> > >> >>>>>>> for each class and improve it iteratively.
> > >> >>>>>> 
> > >> >>>>>> Do you mean improve the Doxygen output, or the source  code
> > >> >>>>>> (*.h files)? If we improve the output we can get diverging
> > >> >>>>>> docs and code.
> > >> >>>>> 
> > >> >>>>> I mean improve the strings following '///' in the .h files. In
> > >> >>>>> quite some cases, just a few extra words would make a big
> > >> >>>>> difference.
> > >> >>>>> 
> > >> >>>>> I'm coming around to putting all programming reference doc in
> > >> >>>>> the code. I don't like lots of markup, but I don't see any
> > >> >>>>> other robust and easily maintainable solution.
> > >> >>>> 
> > >> >>>> missing/obsolete docs, very little extra work is needed to also
> > >> >>>> test if the short docstring in the source  code is correct. Then
> > >> >>>> we run the tests as part of building the documentation.
> > >> >>> 
> > >> >>> I just can't see myself hopping back and forth between the code
> > >> >>> and the documentation when implementing and testing something
> > >> >>> new.
> > >> >> 
> > >> >> But the feature can't be 'official' until it has been documented,
> > >> >> it will require more self-discipline from the developers, which I
> > >> >> don't think is necessarily a bad idea.
> > >> >> 
> > >> >>>> I admit that the Doxygen output is much more detailed and the
> > >> >>>> type information/links in argument lists is better compared to
> > >> >>>> what is in Sphinx now, but that might change in the future (in
> > >> >>>> Sphinx). On the downside, I personally find the Doxygen
> > >> >>>> documentation overwhelming and I never use it for just that
> > >> >>>> reason.
> > >> >>> 
> > >> >>> reference - so one of my points is that we can forget about the
> > >> >>> C++ programmers reference for now and get on with the more
> > >> >>> importance task of documenting demos. We can return to the C++
> > >> >>> programmers reference later (which, as you say, may improve in
> > >> >>> Sphinx in the future).
> > >> >> 
> > >> >> I'm fine with using Doxygen and simply put a link to the index
> > >> >> page, I just think it is worthwhile to carefully discuss the pros
> > >> >> and cons.
> > >> >> 
> > >> >> The Python interface still has to be documented manually since
> > >> >> there is no way to extract docstrings from the source code since
> > >> >> the intention is to add docstrings to the module.
> > >> >> 
> > >> >>> We can some some very limited work to improve the doxygen output
> > >> >>> which will make it easier to navigate.
> > >> >> 
> > >> >> I just don't see how this can be integrated easily with the output
> > >> >> from Doxygen. We don't want to manipulate the output files since
> > >> >> they will be re-generated whenever we build the docs. It is
> > >> >> possible though to link to the html pages of classes/functions,
> > >> >> but it won't be naturally supported like it would be if everything
> > >> >> is in Sphinx.
> > >> >> 
> > >> >> Kristian
> > >> > 

Sounds good. I was in favour of a more heavily commented .h files in the first 
place. Then using Doxygen. But I agree that Doxygen isn't the nices markup 
language, and using just one is probably a strength. I also agree with 
Kristian that Doxygen generated references can be too overwhelming...

If we can just deal with the extraction of the comments, which I figure Anders 
script allready does(?), we should be able to handle this in a nice way. 
Slightly over crowded .h files has never been an issue for me. You have the 
progammers reference for looking into class definitions and such ;)

It also leaves the possibilities that Sphinx get better in parsing C++. I see 
that there where some GSOC project description for this year adressing this 
issue, but not sure if it got accepted or how it went.

What does not the script of Anders extract? I think you have mentioned it in 
the other thread but I lost sight...

Johan

---

From: Kristian Ølgaard <[email-masked]>
Date: Fri, 27 Aug 2010 23:43:44 +0200

> documentation (not the opposite).

We can easily make the docstrings module part of DOLFIN, we just add
it to site-packages/dolfin/docstrings and write all documentation for
the Python interface in this module. This approach even makes more
sense compared to what we have now.
The only reason I put the docstrings module in fenics-doc is that we
initially wanted to separate the two completely, but I guess we have
found out it is not possible to do in a sensible way.

> documentation and that is always unrelated to the C++ documentation?

Yes and yes. We will still need to generate the docstrings.i file
because some of the Swig output in cpp.py makes it impossible to
assign to certain docstrings dynamically (when importing dolfin). If
we have the docstrings module locally in DOLFIN we no longer have to
worry about where to find it when running the script
dolfin/swig/generate.py which is good.

> it when necessary, to avoid duplication?

Maybe, but it is not straightforward because:
* overloaded functions in Python have to be dealt with in a special way.
* all Sphinx links to classes like :cpp:class`Foo` has to change to
:py:class:`Foo` (which is the simplest problem)
* the example code needs to be different.
* some other stuff that I didn't think of

Kristian

---

From: Kristian Ølgaard <[email-masked]>
Date: Fri, 27 Aug 2010 23:53:36 +0200

> 4. Evaluate if the simple extract script does its job well enough

OK, let's give it a go.

But in case it fails (I do hope it doesn't) I vote for writing *.rst
files manually and test if the documentation is missing/obsolete (this
is already in place) and testing if the one-line docstrings in *.h
files is present in the *.rst files (could be ready in a day).

The last resort, in my book, would be a link to Doxygen pages.

Kristian

---

From: "Garth N. Wells" <[email-masked]>
Date: Fri, 27 Aug 2010 22:54:14 +0100

On 27/08/10 22:43, Kristian Ølgaard wrote:
> On 27 August 2010 18:53, Anders Logg<[email-masked]>  wrote:
>>> On 27 August 2010 13:14, Anders Logg<[email-masked]>  wrote:
>>>>> On 27 August 2010 12:12, Kristian Ølgaard<[email-masked]>  wrote:

I've lost track of this thread. Either I'm not paying enough attention 
or the approach to documentation is getting too complicated.

Garth

---

From: Andy Ray Terrel <[email-masked]>
Date: Fri, 27 Aug 2010 20:09:00 -0500

A better tool for editting your rst:

http://blog.enthought.com/enthought-tool-suite/a-renewed-restructured-text-editor/

---

From: Anders Logg <[email-masked]>
Date: Sun, 29 Aug 2010 21:23:39 +0200

> the other thread but I lost sight...

The script reads all .h files and finds:

  /// Comments

followed by either a class declaration or a function declaration. Then
just generates .rst for that.

--
Anders

---

From: Anders Logg <[email-masked]>
Date: Sun, 29 Aug 2010 21:26:56 +0200

> http://blog.enthought.com/enthought-tool-suite/a-renewed-restructured-text-editor/

I don't see how this is better than Emacs with rst-mode. :-)

--
Anders

---

From: Kristian Ølgaard <[email-masked]>
Date: Sun, 29 Aug 2010 21:49:25 +0200

> I don't see how this is better than Emacs with rst-mode. :-)

Which in turn doesn't compare to Vi(m) of course. :)

Kristian

---

From: Anders Logg <[email-masked]>
Date: Sun, 29 Aug 2010 22:07:43 +0200

> Which in turn doesn't compare to Vi(m) of course. :)

Yes, this is where we turn this 40+ thread into an Emacs/Vim
discussion!

--
Anders