Subject: [Fenics] Cleanup of repositories

From: Anders Logg <[email-masked]>
Date: Thu, 21 Mar 2013 15:07:08 +0100

I have done some work on cleaning up the DOLFIN repository.

- All data files have been removed from the lp:dolfin bzr repository.

- All generated code has been removed from the lp:dolfin bzr repository.

- With the help of Jed Brown's git-fat, I have generated a list of
  files that I suggest we strip from the history (once the new
  conversion has taken place).

Please look at the files to be removed here (files_to_strip.txt):

  https://gist.github.com/alogg/5213171

Btw, is there some equivalent of the Gist thing for Bitbucket? I found
it very convenient for sharing the above data. Actually, I couldn't
figure out any other convenient way to do it.

Garth is on his way to Australia, Florian needs some time to think
about what this means for conversion of branches (like his FFC
branch), Johannes is about to make a new release and I need feedback
on the file removal. So the new message is to keep on working on the
Launchpad repositories (with focus on cleanup and bug fixes) until
this has settled, and I'll run the final script for conversion of the
repositories again when we have worked out all the issues.

--
Anders

---

From: Johan Hake <[email-masked]>
Date: Thu, 21 Mar 2013 18:52:58 +0100

Nice work!

So this is how git get so fast, we first remove all the large files ;)

>   conversion has taken place).

How will this affect the possibility to compile legacy versions of
dolfin to test for example speed ups? Are these done with cached tarballs?

> repositories again when we have worked out all the issues.

Sounds good.

Johan

> --
> Anders
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~fenics
> Post to     : [email-masked]
> Unsubscribe : https://launchpad.net/~fenics
> More help   : https://help.launchpad.net/ListHelp
>

---

From: Andy Ray Terrel <[email-masked]>
Date: Thu, 21 Mar 2013 14:54:14 -0500

>> figure out any other convenient way to do it.

I don't think there is one run by bitbucket.  But before gist there
was dpaste.org and pastebin.org.  The feature has been requested
https://bitbucket.org/site/master/issue/516/mercurial-backed-pastebin-bb-672

Gists are really nice since they stay around and are good for sharing
small bits of code and instructions.

I've also gotten little traction with my questions about pull requests
between different forks:
https://answers.atlassian.com/questions/149653/pull-request-between-two-different-repositories

-- Andy

---

From: Anders Logg <[email-masked]>
Date: Thu, 21 Mar 2013 21:58:21 +0100

> So this is how git get so fast, we first remove all the large files ;)

Yes, exactly.

I didn't think it would be such a hassle. I was imagining a simple
conversion (which took just a few minutes to figure out), but then
suddenly we are in the git camp and need to rewrite our history to
present a beatiful repository to the world. :-)

> dolfin to test for example speed ups? Are these done with cached tarballs?

Those can be built (with a lot of effort) using the tarballs, and the
converted unchanged repository as well as the present bzr repository
will be stored for eternity on the web server.

What needs to happen now before we can make the transition is:

1. Get the buildbot green again. It is broken as a result of some
renaming and removing of files that the buildbot needs to generate.

2. Release 1.2.

3. Get comments from Florian or anyone else interested in merging in
converted repos on the Git side.

4. Get comments from developers on the list of proposed files to be
deleted from the history.

--
Anders

---

From: Martin Sandve Alnæs <[email-masked]>
Date: Thu, 21 Mar 2013 22:14:11 +0100

Comments on removal of files:

a) Reference data for regression tests can't be removed from the
repository. With a significantly larger unit test suite the number of
regression tests could safely be reduced, but that is a lot of work. We
could strip older versions of reference files and re-add fresh ones.

b) The list contains files like Changelog, CMakeLists.txt from the build
system which obviously need to be retained, as well as meshes used by demos.

c) In any event we should keep checked in the few generated files that are
part of the dolfin source, so _building_ dolfin does not have dependencies
to the entire fenics toolchain.

d) How does the build system know when to/which files to regenerate? I
guess calling ffc as part of the building of C++ demos is no worse than the
jitting in Python demos. This will add quite a bit to the demo/test build
time though.

Martin

---

From: Florian Rathgeber <[email-masked]>
Date: Thu, 21 Mar 2013 21:15:24 +0000

On 21/03/13 20:58, Anders Logg wrote:
>> Nice work! So this is how git get so fast, we first remove all
>> the large files ;)
> 
> Yes, exactly.
> 

See it the other way: we now have the *opportunity* to undo some
mistakes/sins from the past.

>>> - With the help of Jed Brown's git-fat, I have generated a list
>>> of files that I suggest we strip from the history (once the
>>> new conversion has taken place).
>> How will this affect the possibility to compile legacy versions
>> of dolfin to test for example speed ups? Are these done with
>> cached tarballs?
> 
> Those can be built (with a lot of effort) using the tarballs, and
> the converted unchanged repository as well as the present bzr
> repository will be stored for eternity on the web server.

I'm not sure I understand what your plan for the clean-up is: I was
assuming you want to take the opportunity to purge accidentally
committed files, large binary files or generated files that should
have never been committed in the first place etc.

However all of this should not affect the ability to build old
versions of FEniCS tools: I would advise against cleaning out old
files which were legitimately committed at the time and simply became
obsolete and were removed again later. Git is really good at
compressing text files.

> 3. Get comments from Florian or anyone else interested in merging
> in converted repos on the Git side.

I'm hoping git-fast-filter [1] will do the trick. If not it'll require
a bit of hackery. I'm not very worried about our FFC branch (I'm
comfortable rebasing it on whatever final repository), but ideally I'd
like to not slam the door in the faces of all those having feature
branches lying around (Andre?).

> 4. Get comments from developers on the list of proposed files to
> be deleted from the history.

See my comments above: don't be too radical, only get rid of things
that shouldn't have been there in the first place.

[1]:
https://github.com/mojavelinux/jbosstools-gitmigration/tree/master/git_fast_filter

Florian

> -- Anders
>>> Btw, is there some equivalent of the Gist thing for Bitbucket?
>>> I found it very convenient for sharing the above data.
>>> Actually, I couldn't figure out any other convenient way to do
>>> it.
>>> 
>>> Garth is on his way to Australia, Florian needs some time to
>>> think about what this means for conversion of branches (like
>>> his FFC branch), Johannes is about to make a new release and I
>>> need feedback on the file removal. So the new message is to
>>> keep on working on the Launchpad repositories (with focus on
>>> cleanup and bug fixes) until this has settled, and I'll run the
>>> final script for conversion of the repositories again when we
>>> have worked out all the issues.
>> Sounds good. Johan

---

From: Anders Logg <[email-masked]>
Date: Thu, 21 Mar 2013 22:31:31 +0100

>   could strip older versions of reference files and re-add fresh

I agree with this.

>   b) The list contains files like Changelog, CMakeLists.txt from the
>   build system which obviously need to be retained, as well as meshes
>   used by demos.

Did you look at the wrong file? The files suggested for removal are in
the file 'files_to_strip.txt'. The others are used to generated
candidates for that list and I have manually gone through the list to
make the selection. There will likely be some mistakes in that file,
so I would appreciate more eyes looking at it.

And I don't see why we need to keep the meshes. They will be
downloaded and bundled as part of tarballs and distributions. And can
be easily downloaded by users who clone the repository via a simple
script.

>   c) In any event we should keep checked in the few generated files that
>   are part of the dolfin source, so _building_ dolfin does not have
>   dependencies to the entire fenics toolchain.

I see the point but have heard different opinions on this issue. More
opinions? We have a few files currently in dolfin/ale and many more
generated files in the past being part of the library.

>   guess calling ffc as part of the building of C++ demos is no worse than
>   the jitting in Python demos. This will add quite a bit to the demo/test
>   build time though.
>   Martin

My suggestion to Johannes was to make it part of calling cmake. Then
it gets built only once, and developers will know where to look for a
script to rebuild them when necessary (inside cmake/scripts).

--
Anders

>   repository.
>   repository.

>     the
>   found
>   the

>   Mailing list: [3]https://launchpad.net/~fenics
>   Post to     : [4][email-masked]
>   Unsubscribe : [5]https://launchpad.net/~fenics
>   More help   : [6]https://help.launchpad.net/ListHelp
> Referenser
>   1. mailto:[email-masked]
>   2. https://gist.github.com/alogg/5213171
>   3. https://launchpad.net/~fenics
>   4. mailto:[email-masked]
>   5. https://launchpad.net/~fenics
>   6. https://help.launchpad.net/ListHelp

---

From: Anders Logg <[email-masked]>
Date: Thu, 21 Mar 2013 22:37:05 +0100

> obsolete and were removed again later. Git is really good at

The files I've put up for removal are meshes and generated code, some
of it quite large.

> branches lying around (Andre?).

Can anyone with a feature branch comment on this issue? Will you want
to convert it to Git, then merge. Or are you happy to clone a fresh
branch on the Git side, then patch your changes manually?

> that shouldn't have been there in the first place.

Here's the list again for reference:

https://gist.github.com/alogg/5213171#file-files_to_strip-txt

The simple version is that the list contains the following:

- .h if generated
- .xml.gz
- .inp
- .stl
- .data files
- the old manual with figures, tex, pdf
- some big autotools files

--
Anders

---

From: Martin Sandve Alnæs <[email-masked]>
Date: Thu, 21 Mar 2013 22:52:54 +0100

> Did you look at the wrong file?

Yes...

> be easily downloaded by users who clone the repository via a simple

The consequence is that when downloading an earlier dolfin version the
meshes may not be available or may have changed, unless we version
the downloadable meshes as well.

> generated files in the past being part of the library.

Not only does it make dolfin depend on the entire fenics toolchain being
installed, but that compatible versions of ufl/ffc are installed.
This makes it harder to check out and build different versions of dolfin
in a bughunt where reverting ufl/ffc would not be necessary.

> script to rebuild them when necessary (inside cmake/scripts).

Ok, I actually wondered once why that wasn't the case to begin with :)
It can't easily know when e.g. ffc has changed though, so keeping the
forced rebuild script will be useful.

Martin

---

From: Anders Logg <[email-masked]>
Date: Thu, 21 Mar 2013 22:58:25 +0100

>   the downloadable meshes as well.

If you download a tarball for a release, then you will get everything
bundled. It is only if you do 'git clone' and then revert to an
earlier version that you run into trouble.

I'm not planning to unpack all the old tarballs and remove
stuff from them... :-)

>   in a bughunt where reverting ufl/ffc would not be necessary.

Hmm... Need to think about that.

>   forced rebuild script will be useful.

Yes.

--
Anders

---

From: Martin Sandve Alnæs <[email-masked]>
Date: Thu, 21 Mar 2013 23:06:14 +0100

> I'm not planning to unpack all the old tarballs and remove

Make that "when reverting to an earlier dolfin version" then, same issue
as the checked in code when debugging and looking for the version that
fails to run a demo.

Martin

---

From: Anders Logg <[email-masked]>
Date: Thu, 21 Mar 2013 23:08:00 +0100

>   as the checked in code when debugging and looking for the version that

By this argument, there isn't much we can remove.

--
Anders

---

From: Andre Massing <[email-masked]>
Date: Fri, 22 Mar 2013 00:34:12 +0100

On 03/21/2013 10:37 PM, Anders Logg wrote:
>>> Those can be built (with a lot of effort) using the tarballs,
>>> and the converted unchanged repository as well as the present
>>> bzr repository will be stored for eternity on the web server.
>> I'm not sure I understand what your plan for the clean-up is: I
>> was assuming you want to take the opportunity to purge
>> accidentally committed files, large binary files or generated
>> files that should have never been committed in the first place
>> etc. However all of this should not affect the ability to build
>> old versions of FEniCS tools: I would advise against cleaning out
>> old files which were legitimately committed at the time and
>> simply became obsolete and were removed again later. Git is
>> really good at compressing text files.
> 
> The files I've put up for removal are meshes and generated code,
> some of it quite large.
> 
>>> 3. Get comments from Florian or anyone else interested in
>>> merging in converted repos on the Git side.
>> I'm hoping git-fast-filter [1] will do the trick. If not it'll
>> require a bit of hackery. I'm not very worried about our FFC
>> branch (I'm comfortable rebasing it on whatever final
>> repository), but ideally I'd like to not slam the door in the
>> faces of all those having feature branches lying around
>> (Andre?).

Personally it would be fine with me, but one should keep in mind that
there are 78 branches of DOLFIN at the moment, 31 of which have
updated in 2013. As Martin pointed out, one could not expected every
single branch owner to follow the rapid pace of changes right now :)
so it might be advantageous to have a "post-bzr" merge possibility if
possible and if this does not cause too much of a hazzle for those who
are doing the dirty work right now.

Cheers,
- --
Andre

> 
> Can anyone with a feature branch comment on this issue? Will you
> want to convert it to Git, then merge. Or are you happy to clone a
> fresh branch on the Git side, then patch your changes manually?

> 
>>> 4. Get comments from developers on the list of proposed files
>>> to be deleted from the history.
>> See my comments above: don't be too radical, only get rid of
>> things that shouldn't have been there in the first place.
> 
> - .h if generated - .xml.gz - .inp - .stl - .data files - the old
> manual with figures, tex, pdf - some big autotools files
> 
> -- Anders
> 
> _______________________________________________ Mailing list:
> https://launchpad.net/~fenics Post to     :
> [email-masked] Unsubscribe :
> https://launchpad.net/~fenics More help   :
> https://help.launchpad.net/ListHelp
>

---

From: Florian Rathgeber <[email-masked]>
Date: Fri, 22 Mar 2013 00:58:43 +0000

On 21/03/13 22:08, Anders Logg wrote:
>> If you download a tarball for a release, then you will get 
>> everything bundled. It is only if you do 'git clone' and then
>> revert to an earlier version that you run into trouble. I'm not
>> planning to unpack all the old tarballs and remove stuff from
>> them... :-) Make that "when reverting to an earlier dolfin
>> version" then, same issue as the checked in code when debugging
>> and looking for the version that fails to run a demo.
> 

How about using git-fat to handle large files that we don't want in
the repository, but still be able to pull in on demand? Jed has a
recipe for filtering an existing repository and taking a list of files
to be handled by git-fat:
https://github.com/jedbrown/git-fat/blob/master/test-retroactive.sh

Florian

---

From: Anders Logg <[email-masked]>
Date: Fri, 22 Mar 2013 09:06:10 +0100

On Fri, Mar 22, 2013 at 12:58:43AM +0000, Florian Rathgeber wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
>> On Thu, Mar 21, 2013 at 11:06:14PM +0100, Martin Sandve Alnæs
>>
>>

For me as a newcomer to git, it seems like an extra complication, and
the problem is not that we have a bunch of files which are very
large. It's more that we have very many files of moderate size that we
generate and these change frequently. Furthermore, we can only a apply
a filter to a few of these files so we would need to list them
manually.

It seems there a differing opinions on a few matters:

- Should we care about conversion of branches?

  My opinion is yes. There are quite a few active branches on
  Launchpad, more than I had thought.

- Should we strip files from the history? Or just leave it as it is?

  My opinion is yes. We will archive the full repository on the web
  page and possibly some archive site so that historians can go back
  and excavate the history in full detail.

  It needs to be discussed exactly which files to remove. See earlier
  post for a suggestion.

- Should we remove meshes from the repository?

  My opinion is yes. The meshes have already been removed and
  a working system for easily downloading the meshes is in place.
  This system will encourage the use of more interesting meshes (since
  they no longer need to be very small) and will encourage
  contribution of meshes to the gallery on the web page.

- Should we remove generated code from the repository?

  Martin has some good arguments in favor of keeping the generated
  code, but I'm not fully convinced. A compromise would be to include
  all files needed to build the library itself, which essentially
  means keeping the generated code in dolfin/ale.

  But does that mean we should also keep all the old generated code
  that was part of the library at some point?

--
Anders

---

From: Kristian Ølgaard <[email-masked]>
Date: Fri, 22 Mar 2013 09:55:02 +0100

> - some big autotools files

I have no problems with these files being removed, for the reference
files in FFC I think Martin's solution sounds good.

Kristian

---

From: Martin Sandve Alnæs <[email-masked]>
Date: Fri, 22 Mar 2013 10:03:47 +0100

>   Launchpad, more than I had thought.

Agree.

>   It needs to be discussed exactly which files to remove. See earlier

Ok by me in principle, with some caveats.

>   contribution of meshes to the gallery on the web page.

We currently have 3 MB of .xml.gz files eacho in data/ and demo/, and 48 KB
in test/. Small(!) meshes used by the unit tests should definitely be kept,
for test coverage of e.g. mesh reading. Data not used by demos should
definitely be moved. With meshes used by demos we run into versioning
problems if they are removed.

>   means keeping the generated code in dolfin/ale.

I think the compromise is good and important, since it essentially means
keeping the dolfin library build ffc-independent. Having ffc generating
fresh code for C++ demos regularly has advantages as well.

>   that was part of the library at some point?

I personally don't care a lot about code archeology, as long as we have
fairly recent history available for debugging. Is it feasible to pick a
date and strip before that? E.g. stripping generated code from before the
1.0 release?

Martin

---

From: Johan Hake <[email-masked]>
Date: Fri, 22 Mar 2013 10:07:22 +0100

Agree.

>   contribution of meshes to the gallery on the web page.

Just to clarify. The meshes you are talking about are meshes not used in
demos, right? If so, I agree.

>   means keeping the generated code in dolfin/ale.

I think we should keep these in the repo.

>   that was part of the library at some point?

No strong opinion.

All the swig code can be regenerated by dolfin provided tools so that is
OK to remove. However the docstring generation code is really slow, and
is pretty annoying to generate. But as others have mentioned. This could
be a cmake-cached process, which is only done once, unless the
CMakeCache is removed or one trigger the generation script under
cmake/scripts.

Johan

---

From: Martin Sandve Alnæs <[email-masked]>
Date: Fri, 22 Mar 2013 10:10:38 +0100

Having outdated docstrings if you forget to run a magic script in some
folder is not very convenient. (And why is it so slow?)

Martin

---

From: Martin Sandve Alnæs <[email-masked]>
Date: Fri, 22 Mar 2013 10:21:10 +0100

Which solution is that? :)

I just checked the lp:ffc repo, we have 58 MB in ffc/test/, 46 of which is
.h files, 2.6 MB .json files, 1.8 MB .out files.

But the .bzr directory is only 44 MB, so I guess the compression ratio is
pretty good for the generated code (most of it doesn't change much).

If we archived the current ffc repository for archeologists, and stripped
old generated code references but kept adding new references, there would
be little to gain.

There are some MB to gain by reducing the size of the P5tet element code,
if someone feels up for the task :)

The biggest gain would come from making a more careful selection of
references, as there's a lot of duplication here (and still far from
perfect coverage). In particular, generating element code shareable between
forms like I did with sfc would give a massive reduction in total .h size
here.

Martin

---

From: Anders Logg <[email-masked]>
Date: Fri, 22 Mar 2013 10:27:10 +0100

My suggestion would be to only strip files from DOLFIN and keep FFC as is.

--
Anders

On Fri, Mar 22, 2013 at 10:21:10AM +0100, Martin Sandve Alnæs wrote:
>   I just checked the lp:ffc repo, we have 58 MB in ffc/test/, 46 of which
>   is .h files, 2.6 MB .json files, 1.8 MB .out files.
>   But the .bzr directory is only 44 MB, so I guess the compression ratio
>   is pretty good for the generated code (most of it doesn't change much).
>   If we archived the current ffc repository for archeologists, and
>   stripped old generated code references but kept adding new references,
>   there would be little to gain.
>   There are some MB to gain by reducing the size of the P5tet element
>   code, if someone feels up for the task :)
>   perfect coverage). In particular, generating element code shareable
>   between forms like I did with sfc would give a massive reduction in
>   total .h size here.
>   Martin
>   On 22 March 2013 09:55, Kristian Ølgaard <[1][email-masked]>

>   became

>   require
>   I'd

>   > [3]https://gist.github.com/alogg/5213171#file-files_to_strip-txt
>   >

>   > Mailing list: [4]https://launchpad.net/~fenics
>   > Post to     : [5][email-masked]
>   > Unsubscribe : [6]https://launchpad.net/~fenics
>   > More help   : [7]https://help.launchpad.net/ListHelp
>   Mailing list: [8]https://launchpad.net/~fenics
>   Post to     : [9][email-masked]
>   Unsubscribe : [10]https://launchpad.net/~fenics
>   More help   : [11]https://help.launchpad.net/ListHelp
> Referenser
>   1. mailto:[email-masked]
>   2. mailto:[email-masked]
>   3. https://gist.github.com/alogg/5213171#file-files_to_strip-txt
>   4. https://launchpad.net/~fenics
>   5. mailto:[email-masked]
>   6. https://launchpad.net/~fenics
>   7. https://help.launchpad.net/ListHelp
>   8. https://launchpad.net/~fenics
>   9. mailto:[email-masked]
>  10. https://launchpad.net/~fenics
>  11. https://help.launchpad.net/ListHelp

---

From: Anders Logg <[email-masked]>
Date: Fri, 22 Mar 2013 10:29:46 +0100

> demos, right? If so, I agree.

Not exactly:

- Meshes in demos --> remove (already done)
- Meshes in data/meshes --> remove (already done)
- Meshes in unit tests --> keep

> I think we should keep these in the repo.

ok.

> CMakeCache is removed or one trigger the generation script under

I think it would be good to always (when first running CMake) generate
the docstrings since now they only generated when someone randomly
remembers to run that script.

--
Anders

---

From: Johan Hake <[email-masked]>
Date: Fri, 22 Mar 2013 10:30:00 +0100

On 03/22/2013 10:10 AM, Martin Sandve Alnæs wrote:
> folder is not very convenient. 

That has been the behavior up to now anyhow. I agree it is suboptimal.

> (And why is it so slow?)

I did some profiling a while ago bu I do not remember. Kristian wrote
the algorithm, but it should be pretty straightforward to optimize. It
is just not on the top of my list...

Johan

> Martin
> 
> 
> On 22 March 2013 10:07, Johan Hake <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     <mailto:[email-masked]>
>     <mailto:[email-masked]>

---

From: Johan Hake <[email-masked]>
Date: Fri, 22 Mar 2013 10:32:50 +0100

> - Meshes in demos --> remove (already done)

I suggest we keep these. There aren't any big files anyhow, are there?

> remembers to run that script.

Ok.

Johan

---

From: Anders Logg <[email-masked]>
Date: Fri, 22 Mar 2013 10:36:44 +0100

> I suggest we keep these. There aren't any big files anyhow, are there?

They have already been removed and there's a good system in place for
handling them. Keeping the meshes elsewhere will encourage use of the
mesh gallery and keeping better track of which meshes to use. There
were lots of meshes named 'mesh.xml' or 'mesh2d.xml' which were really
copies of other meshes used in other demos, some of them were gzipped,
some not etc. That's all very clean now. Take a look at how it's done
in trunk. I think it looks quite nice.

Most of the example meshes are not that big, but multiply that by 30
and then some when meshes are moved around or renamed.

--
Anders

---

From: Anders Logg <[email-masked]>
Date: Fri, 22 Mar 2013 10:40:28 +0100

PS: Martin, can you turn off HTML? I can't really read your emails.

--
Anders

On Fri, Mar 22, 2013 at 10:03:47AM +0100, Martin Sandve Alnæs wrote:
>   >
>   files
>   [2]https://github.com/jedbrown/git-fat/blob/master/test-retroactive.sh

>   We currently have 3 MB of .xml.gz files eacho in data/ and demo/, and
>   48 KB in test/. Small(!) meshes used by the unit tests should
>   definitely be kept, for test coverage of e.g. mesh reading. Data not
>   used by demos should definitely be moved. With meshes used by demos we
>   run into versioning problems if they are removed.

>   I think the compromise is good and important, since it essentially
>   means keeping the dolfin library build ffc-independent. Having ffc
>   generating fresh code for C++ demos regularly has advantages as well.

>   date and strip before that? E.g. stripping generated code from before
>   the 1.0 release?
>   Martin
> Referenser
>   1. mailto:[email-masked]
>   2. https://github.com/jedbrown/git-fat/blob/master/test-retroactive.sh

---

From: "Garth N. Wells" <[email-masked]>
Date: Fri, 22 Mar 2013 09:46:37 +0000

On Friday, 22 March 2013, Martin Sandve Alnæs wrote:

> On 22 March 2013 09:06, Anders Logg <[email-masked] <javascript:_e({},
> 'cvml', [email-masked]');>> wrote:
> >
> We currently have 3 MB of .xml.gz files eacho in data/ and demo/, and 48
> KB in test/. Small(!) meshes used by the unit tests should definitely be
> kept, for test coverage of e.g. mesh reading. Data not used by demos should

This is my take too.  Release tarballs provide snapshots of the distant
past.

We do need a strategy for generated code because keeping it all puts us on
an unsustainable path re repository size. The DOLFIN git repo was of the
order of 350MB (as reported by Bitbucket). For reference, PETSc is around
90MB.

Garth

-- 
Garth N. Wells
Department of Engineering, University of Cambridge
http://www.eng.cam.ac.uk/~gnw20

---

From: Kristian Ølgaard <[email-masked]>
Date: Fri, 22 Mar 2013 10:48:32 +0100

> Which solution is that? :)

The one of stripping older versions and adding new ones.
I thought the gain would be bigger than what you report below, so my
inclination is to keep FFC as it is, as Anders suggest below.

Kristian

> old generated code references but kept adding new references, there would be
> little to gain.
>
> There are some MB to gain by reducing the size of the P5tet element code, if
> someone feels up for the task :)
>
> references, as there's a lot of duplication here (and still far from perfect
> coverage). In particular, generating element code shareable between forms
> like I did with sfc would give a massive reduction in total .h size here.
>
> Martin
>
>

---

From: Johan Hake <[email-masked]>
Date: Fri, 22 Mar 2013 10:52:25 +0100

> in trunk. I think it looks quite nice.

Nice and clean, but it really is just 30 meshes. Duplications are mostly
related to dolfin_fine.xml.gz, which there are 7 copies of, and that
file is 86K.

> and then some when meshes are moved around or renamed.

I just question if it is worth it. Seems convenient to just have the
meshes there.

If we keep them out of the repo I think we should include some automagic
downloading when building the demos.

Also should we rename the script to download-demo-meshes, or something
more descriptive, as this is what that script now basically does?

Johan

---

From: Anders Logg <[email-masked]>
Date: Fri, 22 Mar 2013 10:53:27 +0100

Could we agree on the following?

- Keep FFC repository as is for now

  (nothing to do)

- Remove meshes from DOLFIN except those in the test directory

  (nothing to do, already done)

- Remove generated code from DOLFIN except when part of the library

  (add back the files in dolfin/ale after conversion)

- Strip history of big files

  (please look at suggested list and edit)

- Convert to git in a way that allows for conversion of branches

  (need help from Florian with this)

Did I miss something? I can't read Martin's comments except for when
he's top-posting. ;-)

-- 
Anders

---

From: Anders Logg <[email-masked]>
Date: Fri, 22 Mar 2013 10:57:13 +0100

> I just question if it is worth it. Seems convenient to just have the

Keeping the meshes there will put a limit on which demos we can add. I
think it would be good to allow for more complex demos requiring
bigger meshes (not necessarily run on the buildbot every day).

> downloading when building the demos.

Yes, or at least a message stating: "You have not downloaded demo
data. Please run the script foo."

> more descriptive, as this is what that script now basically does?

It is not only meshes, but also markers and velocity fields. Perhaps
it can be renamed download-demo-data?

-- 
Anders

---

From: Johan Hake <[email-masked]>
Date: Fri, 22 Mar 2013 10:59:56 +0100

Ok.

Sounds good.

Johan

---

From: "Marie E. Rognes" <[email-masked]>
Date: Fri, 22 Mar 2013 11:55:45 +0100

> Did I miss something? I can't read Martin's comments except for when

Sounds good to me.

Independently of this, I fully agree with Martin in that there is 
separate value in revising the FFC tests.

--
Marie

---

From: Anders Logg <[email-masked]>
Date: Fri, 22 Mar 2013 13:02:02 +0100

>>  (need help from Florian with this)

More thoughts on this? It would be good to get this in place over the
weekend if possible. I'll be in a meeting in the beginning of next
week and can't really do much then.

It looks like we mostly agree so I just need to get comments on this list:

Then I hope I can get help from Florian with how to do the conversion
in combination with file stripping.

--
Anders

---

From: Johan Hake <[email-masked]>
Date: Fri, 22 Mar 2013 13:17:55 +0100

>   https://gist.github.com/alogg/5213171#file-files_to_strip-txt

Maybe keep:

src/pydolfin/dolfin_docstrings.i
src/pydolfin/PoissonTest.h

These are all files from the ancient SWIG interface and it might be that
these are needed to compile that version?

Johan

---

From: Anders Logg <[email-masked]>
Date: Fri, 22 Mar 2013 13:21:30 +0100

> these are needed to compile that version?

ok. I'll keep those.

But note that we won't be able to compile old versions since we remove
generated code which was then part of the library.

--
Anders

---

From: Florian Rathgeber <[email-masked]>
Date: Mon, 25 Mar 2013 00:31:04 +0000

On 22/03/13 09:59, Johan Hake wrote:
>>>> They have already been removed and there's a good system in 
>>>> place for handling them. Keeping the meshes elsewhere will 
>>>> encourage use of the mesh gallery and keeping better track
>>>> of which meshes to use. There were lots of meshes named 
>>>> 'mesh.xml' or 'mesh2d.xml' which were really copies of other 
>>>> meshes used in other demos, some of them were gzipped, some 
>>>> not etc. That's all very clean now. Take a look at how it's 
>>>> done in trunk. I think it looks quite nice.
>>> 
>>> Nice and clean, but it really is just 30 meshes. Duplications 
>>> are mostly related to dolfin_fine.xml.gz, which there are 7 
>>> copies of, and that file is 86K.

If they're bit-by-bit identical git will only store a single copy in
the repository anyway, regardless of how many copies you happen to
have in the working tree.

On the note of storing gzipped meshes: Do they change frequently? Why
are they stored gzipped? Compressed files have a few issues:
1) they're treated as binary i.e. any change requires a new copy of
the entire file to be stored
2) they can't be diffed
3) git compresses its packfiles anyway, so there is little (if any)
space gain through compression

>>>> Most of the example meshes are not that big, but multiply 
>>>> that by 30 and then some when meshes are moved around or 
>>>> renamed.
>>> 
>>> I just question if it is worth it. Seems convenient to just 
>>> have the meshes there.
>> 
>> Keeping the meshes there will put a limit on which demos we can 
>> add. I think it would be good to allow for more complex demos 
>> requiring bigger meshes (not necessarily run on the buildbot 
>> every day).
> 
> Ok.
> 
>>> If we keep them out of the repo I think we should include some 
>>> automagic downloading when building the demos.
>> 
>>> Also should we rename the script to download-demo-meshes, or 
>>> something more descriptive, as this is what that script now 
>>> basically does?
>> 
>> It is not only meshes, but also markers and velocity fields. 
>> Perhaps it can be renamed download-demo-data?
> 
> Sounds good.
> 
> Johan

I did some more experimenting:

1) Repository size: there is quite some mileage repacking the repos with
the following steps:
$ git reflog expire --expire=now --all
$ git gc --aggressive --prune=now
$ git repack -ad
e.g. DOLFIN: 372MiB -> 94MiB

2) Stripping out the files suggested by Anders
(https://gist.github.com/alogg/5213171#file-files_to_strip-txt) brings
the repo size down to 172MiB and 24MiB after repacking.

3) I haven't yet found a reliable way to migrate feature branches to
the filtered repository. Filtering the repository rewrites its history
and therefore changes/invalidates all commit ids (SHA1s) and therefore
the marks files created when initially converting the repository.
There are 2 possible options for filtering the repository during
conversion:

a) bzr fast-import-filter: seems to be a pain to use with many files
(need to pass each path individually as an argument) and seems not to
support writing marks files, therefore haven't tried.

b) git_fast_filter: when using to filter the converted git repo, the
exported marks file in the last step contains 83932 marks instead of
the expected 14399 - I can't say why. Unfortunately I haven't been
able to use it directory in the conversion pipeline, it's not
compatible to a bzr fast-export stream. That's probably fixable, but I
can't estimate how much work it would be to fix it since I'm not
familiar enough with details of the fast-import format.

TL;DR: Repacking repos saves a lot of space already without stripping
large files. Stripping files is easy to do and saves even considerably
more space, but I haven't been able to reliably import feature
branches into a filtered repository.

Florian

---

From: "Garth N. Wells" <[email-masked]>
Date: Mon, 25 Mar 2013 18:45:25 +0800

>   https://gist.github.com/alogg/5213171

Looks ok to me. Does it remove all of the old sandbox/ code? It can all go.

How big is the repo after removing these files?

Garth

---

From: "Garth N. Wells" <[email-masked]>
Date: Mon, 25 Mar 2013 18:50:18 +0800

> have in the working tree.

Clever.

> On the note of storing gzipped meshes: Do they change frequently?

No.

> Why
> are they stored gzipped?

Habit. It's not good for version control.

> Compressed files have a few issues:

Wow. What do these commands do?

> the repo size down to 172MiB and 24MiB after repacking.

I like this. It will make cloning on slow connection much better.

> branches into a filtered repository.

How about we give everyone a periodic within which to merge code on
Launchpad, then we don't worry about features branches and marks in
the conversion? Small changes can always come later in the form of
patches.

Garth

---

From: Florian Rathgeber <[email-masked]>
Date: Mon, 25 Mar 2013 10:55:53 +0000

On 25/03/13 10:45, Garth N. Wells wrote:
>> of files that I suggest we strip from the history (once the new 

No, sandbox is not completely stripped, but that would be easy to do.

> How big is the repo after removing these files?

See my other reply: 172MiB (down from 372MiB), 24MiB after repacking.

Florian

> Garth
> 
> 
>> Btw, is there some equivalent of the Gist thing for Bitbucket? I
>> found it very convenient for sharing the above data. Actually, I
>> couldn't figure out any other convenient way to do it.
>> 
>> think about what this means for conversion of branches (like his
>> FFC branch), Johannes is about to make a new release and I need
>> feedback on the file removal. So the new message is to keep on
>> working on the Launchpad repositories (with focus on cleanup and
>> bug fixes) until this has settled, and I'll run the final script
>> for conversion of the repositories again when we have worked out
>> all the issues.
>> 
>> -- Anders
>>

---

From: Florian Rathgeber <[email-masked]>
Date: Mon, 25 Mar 2013 11:32:35 +0000

On 25/03/13 10:50, Garth N. Wells wrote:
>>>>>> They have already been removed and there's a good system
>>>>>> in place for handling them. Keeping the meshes elsewhere
>>>>>> will encourage use of the mesh gallery and keeping better
>>>>>> track of which meshes to use. There were lots of meshes
>>>>>> named 'mesh.xml' or 'mesh2d.xml' which were really copies
>>>>>> of other meshes used in other demos, some of them were
>>>>>> gzipped, some not etc. That's all very clean now. Take a
>>>>>> look at how it's done in trunk. I think it looks quite
>>>>>> nice.
>>>>> 
>>>>> Nice and clean, but it really is just 30 meshes.
>>>>> Duplications are mostly related to dolfin_fine.xml.gz,
>>>>> which there are 7 copies of, and that file is 86K.
>> 
>> If they're bit-by-bit identical git will only store a single copy
>> in the repository anyway, regardless of how many copies you
>> happen to have in the working tree.
> 
> Clever.
> 
>> Why are they stored gzipped?
> 

With a bit of trickery we might even be able to convert all those
gzipped meshes i.e. unzip them in each revision and add only keep the
xml in the repo (retrospectively for the entire history).

>> Compressed files have a few issues: 1) they're treated as binary
>> i.e. any change requires a new copy of the entire file to be
>> stored 2) they can't be diffed 3) git compresses its packfiles
>> anyway, so there is little (if any) space gain through
>> compression
>> 
>>>>>> Most of the example meshes are not that big, but
>>>>>> multiply that by 30 and then some when meshes are moved
>>>>>> around or renamed.
>>>>> 
>>>>> I just question if it is worth it. Seems convenient to
>>>>> just have the meshes there.
>>>> 
>>>> Keeping the meshes there will put a limit on which demos we
>>>> can add. I think it would be good to allow for more complex
>>>> demos requiring bigger meshes (not necessarily run on the
>>>> buildbot every day).
>>> 
>>> Ok.
>>> 
>>>>> If we keep them out of the repo I think we should include
>>>>> some automagic downloading when building the demos.
>>>> 
>>>> Yes, or at least a message stating: "You have not downloaded
>>>> demo data. Please run the script foo."
>>>> 
>>>>> Also should we rename the script to download-demo-meshes,
>>>>> or something more descriptive, as this is what that script
>>>>> now basically does?
>>>> 
>> 1) Repository size: there is quite some mileage repacking the
>> repos with the following steps: $ git reflog expire --expire=now
>> --all

git keeps track of how branch HEADs move and does not garbage collect
these revision. This information is kept for 90 days by default. Tell
git to clear this history and "release" if for garbage collection.

>> $ git gc --aggressive --prune=now

Invoke git's garbage collection and tell it to aggressively remove all
objects from packfiles which are no longer reachable in the DAG.

>> $ git repack -ad

Rewrite the packfiles and remove all redundant packs.

>> (https://gist.github.com/alogg/5213171#file-files_to_strip-txt)
>> brings the repo size down to 172MiB and 24MiB after repacking.
> 
>> 3) I haven't yet found a reliable way to migrate feature branches
>> to the filtered repository. Filtering the repository rewrites its
>> history and therefore changes/invalidates all commit ids (SHA1s)
>> and therefore the marks files created when initially converting
>> the repository. There are 2 possible options for filtering the
>> repository during conversion:
>> 
>> a) bzr fast-import-filter: seems to be a pain to use with many
>> files (need to pass each path individually as an argument) and
>> seems not to support writing marks files, therefore haven't
>> tried.
>> 
>> b) git_fast_filter: when using to filter the converted git repo,
>> the exported marks file in the last step contains 83932 marks
>> instead of the expected 14399 - I can't say why. Unfortunately I
>> haven't been able to use it directory in the conversion pipeline,
>> it's not compatible to a bzr fast-export stream. That's probably
>> fixable, but I can't estimate how much work it would be to fix it
>> since I'm not familiar enough with details of the fast-import
>> format.
>> 
>> TL;DR: Repacking repos saves a lot of space already without
>> stripping large files. Stripping files is easy to do and saves
>> even considerably more space, but I haven't been able to reliably
>> import feature branches into a filtered repository.
> 
> How about we give everyone a periodic within which to merge code
> on Launchpad, then we don't worry about features branches and marks
> in the conversion? Small changes can always come later in the form
> of patches.

Yes, that's an option. Git has very good support for importing patch
series, maybe bzr can export patch series in the git am format. The
other alternative is importing the feature branch into the
non-filtered git repository and transplant it to the filtered one via
interactive rebase. It's just a bit more work than what I would have
hoped for.

Florian

---

From: Anders Logg <[email-masked]>
Date: Mon, 25 Mar 2013 13:02:48 +0100

> Rewrite the packfiles and remove all redundant packs.

Wow. I didn't much of that but it sounds like a good thing...

> > I like this. It will make cloning on slow connection much better.

Trimming the repository down to 24 MB sounds very tempting.

> interactive rebase. It's just a bit more work than what I would have

I like Garth's suggestion. How about we set a deadline for Friday for
any pending merges in combination with being a bit more accomodating
with getting the merges in place? Then we freeze the repositories over
the weekend and do the conversions/stripping. After that, anyone
wishing to merge code in will need to clone a fresh copy of the new
git repository and patch manually.

--
Anders

---

From: Anders Logg <[email-masked]>
Date: Mon, 25 Mar 2013 13:03:57 +0100

> Looks ok to me. Does it remove all of the old sandbox/ code? It can all go.

I only filtered for files larger than 10k.

--
Anders