Subject: [Fenics] Project changes: moving from Launchpad

From: Anders Logg <[email-masked]>
Date: Mon, 18 Mar 2013 18:54:48 +0100

Dear FEniCS users and developers,

A couple of weeks back, a discussion got started on whether
Launchpad/bzr is the best platform for FEniCS development.

Sites like Bitbucket and GitHub are competing heavily to offer
superior services to coders, the development of bzr seems to have
stagnated and Git offers several attractive features over Bazaar.

We've discussed various options over the last week with several
developers, and done some testing and evaluation of our different
options. Based on these discussions, Garth and I would like to make
the following proposal:

1. Move all the code to Git

2. Move all the code to Bitbucket

3. Move the user questions to http://scicomp.stackexchange.com/

4. Move the mailing list(s) to fenicsproject.org

5. Move all the code into a single repository using Git subtress

Some comments:

- Bitbucket and GitHub seem very comparable. One small advantage of
  Bitbucket is that they allow free private repositories for academic
  use, which some of us already use heavily for papers and
  presentations.

- If we later want to switch to GitHub, it would be a very lightweight
  operation, since it would only involve moving a Git repository (not
  user questions, mailing lists, project teams etc etc). We can also
  mirror the Git repository on GitHub right from the start so that
  GitHub users can access it conveniently.

- I've asked the moderators on scicomp.stackexchange whether they
  would welcome FEniCS user questions and they are very welcoming.

  http://meta.scicomp.stackexchange.com/questions/338/can-scicomp-stackexchange-be-used-for-project-specific-questions

  Some work will be required to create instructions on which
  questions are appropriate to send to scicomp.stackexchange, and
  which might instead go to the mailing list.

- Let's start with a single mailing list [email-masked].
  If we need more, we can add it later.

- I haven't thought about the Launchpad PPA service, so that remains
  to be discussed.

What we do now is:

- Hear back from the community. I've had a chance to talk to most of
  you here at the FEniCS'13 conference in Cambridge, but not all.
  Please state any objections.

- Move the code over to Bitbucket as soon as we can. Then the rest.

PS: Thanks to Andy for being so obnoxious and bugging me about why we
are still on Launchpad, and for suggesting the use of
scicomp.stackexchange for the user questions.

--
Anders

---

From: "Marie E. Rognes" <[email-masked]>
Date: Mon, 18 Mar 2013 18:48:12 +0000

Two practical questions:

* Bitbucket issues look nice for tracking bugs. Bur how about the registered open bugs? Will 'someone' transfer these from Launchpad or will they be lost in the move? 

* Is there any work involved in maintaining the mailinglist? 

--
Marie

On 18. mars 2013, at 17:54, Anders Logg <[email-masked]> wrote:

> _______________________________________________
> Mailing list: https://launchpad.net/~fenics
> Post to     : [email-masked]
> Unsubscribe : https://launchpad.net/~fenics
> More help   : https://help.launchpad.net/ListHelp

---

From: Anders Logg <[email-masked]>
Date: Tue, 19 Mar 2013 00:39:13 +0100

> * Bitbucket issues look nice for tracking bugs. Bur how about the
> * registered open bugs? Will 'someone' transfer these from Launchpad
> * or will they be lost in the move?

Yes, we should move the bugs.

> * Is there any work involved in maintaining the mailinglist?

I hope not. I'll see how much work it is. Hopefully I'll just have to
sudo apt-get install mailman.

--
Anders

---

From: David Ham <[email-masked]>
Date: Mon, 18 Mar 2013 23:48:03 +0000

A third practical issue, we need to think about how we cleanly transition
branches.

David

On Monday, March 18, 2013, Marie E. Rognes wrote:

> registered open bugs? Will 'someone' transfer these from Launchpad or will
> they be lost in the move?
>
> On 18. mars 2013, at 17:54, Anders Logg <[email-masked] <javascript:;>>
> wrote:
>
> > - Let's start with a single mailing list [email-masked]<javascript:;>
> .
> > Post to     : [email-masked] <javascript:;>
> Post to     : [email-masked] <javascript:;>

-- 
Dr David Ham
Department of Computing
Imperial College London

http://www.imperial.ac.uk/people/david.ham

---

From: Anders Logg <[email-masked]>
Date: Tue, 19 Mar 2013 02:09:26 +0100

Yes. I assume Florian will tell us all about setting up the Git
subtrees. Then we just freeze the development for the few hours it
takes to set that up and just do the transition.

--
Anders

On Mon, Mar 18, 2013 at 11:48:03PM +0000, David Ham wrote:
>   A third practical issue, we need to think about how we cleanly
>   transition branches.
>   David
>     Two practical questions:
>     registered open bugs? Will 'someone' transfer these from Launchpad
>     or will they be lost in the move?

>     > 3. Move the user questions to [2]http://scicomp.stackexchange.com/
>     >
>     > 4. Move the mailing list(s) to [3]fenicsproject.org
>     >

>     > Â Bitbucket is that they allow free private repositories for
>     academic

>     > Â use, which some of us already use heavily for papers and
>     > Â presentations.
>     >

>     > Â operation, since it would only involve moving a Git repository
>     (not

>     > Â user questions, mailing lists, project teams etc etc). We can
>     also

>     > Â mirror the Git repository on GitHub right from the start so that
>     > Â GitHub users can access it conveniently.
>     >
>     > Â would welcome FEniCS user questions and they are very welcoming.
>     >
>     >
>     Â [4]http://meta.scicomp.stackexchange.com/questions/338/can-scicomp
>     -stackexchange-be-used-for-project-specific-questions

>     >
>     > Â Some work will be required to create instructions on which
>     > Â questions are appropriate to send to scicomp.stackexchange, and
>     > Â which might instead go to the mailing list.
>     >
>     > - Let's start with a single mailing list
>     [5][email-masked].

>     > Â If we need more, we can add it later.
>     >

>     > Â to be discussed.

>     > Â you here at the FEniCS'13 conference in Cambridge, but not all.
>     > Â Please state any objections.
>     >

>     > Mailing list: [6]https://launchpad.net/~fenics
>     > Post to Â  Â  : [7][email-masked]
>     > Unsubscribe : [8]https://launchpad.net/~fenics
>     > More help Â  : [9]https://help.launchpad.net/ListHelp
>     Mailing list: [10]https://launchpad.net/~fenics
>     Post to Â  Â  : [11][email-masked]
>     Unsubscribe : [12]https://launchpad.net/~fenics
>     More help Â  : [13]https://help.launchpad.net/ListHelp
> Referenser
>   1. javascript:;
>   2. http://scicomp.stackexchange.com/
>   3. http://fenicsproject.org/
>   4. http://meta.scicomp.stackexchange.com/questions/338/can-scicomp-stackexchange-be-used-for-project-specific-questions
>   5. javascript:;
>   6. https://launchpad.net/~fenics
>   7. javascript:;
>   8. https://launchpad.net/~fenics
>   9. https://help.launchpad.net/ListHelp
>  10. https://launchpad.net/~fenics
>  11. javascript:;
>  12. https://launchpad.net/~fenics
>  13. https://help.launchpad.net/ListHelp
>  14. http://www.imperial.ac.uk/people/david.ham

---

From: Anders Logg <[email-masked]>
Date: Tue, 19 Mar 2013 02:10:52 +0100

I tried Google groups as an alternativ to setting up a mailman list on
fenicsproject.org (which required some non-trivial configuration):

https://groups.google.com/forum/?fromgroups=#!forum/fenics-dev

Would that be a good alternative, or does it require everyone who
wants to post to register for a Google email account?

--
Anders

---

From: "Marie E. Rognes" <[email-masked]>
Date: Tue, 19 Mar 2013 11:05:45 +0000

> sudo apt-get install mailman.

Ok, thanks.

--
Marie

---

From: "Garth N. Wells" <[email-masked]>
Date: Tue, 19 Mar 2013 11:46:35 +0000

> wants to post to register for a Google email account?
It would be nice if we don't have to maintain a mailing list ourselves.

Garth

> >     [5][email-masked] <javascript:;>.
>
>
>
> >     > Post to Â  Â  : [7][email-masked] <javascript:;>
> >     Post to Â  Â  : [11][email-masked] <javascript:;>
> >   4.

-- 
Garth N. Wells
Department of Engineering, University of Cambridge
http://www.eng.cam.ac.uk/~gnw20

---

From: Anders Logg <[email-masked]>
Date: Tue, 19 Mar 2013 12:52:21 +0100

On Tue, Mar 19, 2013 at 11:46:35AM +0000, Garth N. Wells wrote:
>     [1]fenicsproject.org (which required some non-trivial
>     configuration):
>     [2]https://groups.google.com/forum/?fromgroups=#!forum/fenics-dev

I see a couple of advantages of having a mailing list hosted on fenicsproject.org:

- Not requiring users to create a Google account (assuming that is necessary)

- A more appealing addres: [email-masked]

The difficulty is not in setting up mailman itself, which is very
basic, but in getting the infrastructure for sending/receiving emails
on the server in place (using postfix). This is something we need to
do anyway, to get emails when things fail, break-in attempts etc.

I'll make an attempt to get the mail server in place, then we can try
out the mailman list and evaluate whether to use that or Google
groups.

--
Anders

---

From: Andy Ray Terrel <[email-masked]>
Date: Tue, 19 Mar 2013 09:26:37 -0500

> - Not requiring users to create a Google account (assuming that is necessary)

Not strictly necessary, but a good idea:
* https://sites.google.com/site/tomihasa/google-groups-faq#isgoogleaccountrequired

---

From: Andy Ray Terrel <[email-masked]>
Date: Tue, 19 Mar 2013 09:48:22 -0500

I have been playing around with the branches on bitbucket and I have
to say I'm less impressed.  With github, we would just give
instructions for each individual to fork the
github.com/fenics/package.git repo and push their branches to their
space.  Then a pull request (or merge request) can be done in the same
fashion as lp. The show up as issues and can be referenced easily plus
they have a nice listing.  We can also setup a buildbot for the pull
requests that show if pull request pass the tests, for example see
https://github.com/sympy/sympy-bot .

 On Bitbucket I haven't figured out how to do a pull request from two
separate branches (only to the main repo and its forks).  Jed tells me
to send a bug report,

-- Andy

---

From: Anders Logg <[email-masked]>
Date: Tue, 19 Mar 2013 18:13:08 +0100

For future reference, here are the commands for conversion of from bzr
to git (currently in progress):

  bzr branch lp:dolfin
  cd dolfin
  git init
  bzr fast-export `pwd` | git fast-import
  rm -r .bzr
  git reset HEAD

--
Anders

---

From: Anders Logg <[email-masked]>
Date: Tue, 19 Mar 2013 18:31:35 +0100

>  bzr fast-export `pwd` | git fast-import

Correction from Florian. Let's do this:

bzr fast-export --export-marks=marks.bzr `pwd` | git fast-import --export-marks=marks.git

The marks files can be used by anyone who wants to convert their
existing feature branches and then merge them on the git side. They
will be put on the web page for download. More info here:

http://dgleich.wordpress.com/2011/01/22/convert-bazaar-to-git/

Garth and I are currently converting the repositories for DOLFIN, UFC,
UFL, FFC, Instant and FIAT so please don't touch the Launchpad repos
now.

--
Anders

---

From: Anders Logg <[email-masked]>
Date: Tue, 19 Mar 2013 19:12:43 +0100

What does that mean? A pull request from two separate branches?

--
Anders

---

From: Andy Ray Terrel <[email-masked]>
Date: Tue, 19 Mar 2013 13:26:23 -0500

Person A and Person B fork from repo O (for original).

On Github A can put a pull request to B, but not bitbucket.

Okay so you can just email the patch, but you don't get the super
nifty code review interface.

-- Andy

---

From: Anders Logg <[email-masked]>
Date: Tue, 19 Mar 2013 19:30:33 +0100

That seems like a useful feature. Are you sure it can't be done on
Bitbucket?

--
Anders

---

From: Andy Ray Terrel <[email-masked]>
Date: Tue, 19 Mar 2013 13:32:23 -0500

Let me test it with a different repo.  Jed says they don't do it with
PETSc and it could be the private repo junk getting in the way.  I do
plan on submitting a bug (with a screen shot of me putting any repo in
Github's pull request field).

-- Andy

---

From: Florian Rathgeber <[email-masked]>
Date: Tue, 19 Mar 2013 23:43:04 +0000

On 19/03/13 17:31, Anders Logg wrote:
>> For future reference, here are the commands for conversion of 
>> from bzr to git (currently in progress): bzr branch lp:dolfin cd 
>> dolfin git init bzr fast-export `pwd` | git fast-import rm -r 
>> .bzr git reset HEAD
> 
> Garth and I are currently converting the repositories for DOLFIN, 
> UFC, UFL, FFC, Instant and FIAT so please don't touch the
> Launchpad repos now.

Anders, Garth and I sat down this evening to work out a migration
strategy and I think we've arrived at something workable.

Just a few words of explanation about the pitfalls: Converting the
trunk is relatively straightforward (and the conversion scripts should
have finished by now), but we need to be very careful when migrating
feature branches s.t. they keep the correct ancestry relationship.

Consider the following situation of a feature branch branching of
trunk after revision 1803:

o trunk (r 1806) --> d9e67c2
|
o  + feature-branch (r 1805) --> 16c0470
|  |
o  +
| /
|/
o base (r 1803) --> dafb344
|

After converting trunk to Git, bzr revision 1806 becomes git revision
d9e67c2 and 1803 becomes dafb344. When migrating feature-branch, it is
*critical* that revision 16c0470 is a descendant of dafb344.

For this to work, bzr fast-export needs to know which revisions it has
already exported and git fast-import needs to know which revisions it
has already seen (and both need to agree on that!). Therefore you need
to have access to the *same* marks file that was used when converting
trunk.

If it doesn't have access to this information when importing
feature-branch, fast-{im,ex}port will go ahead and do all the
conversion for revision 1803 and its ancestors *again* and you will
end up with *the entire history replicated* but *different* git
revision ids s.t. git has no way of knowing that feature-branch should
be a descendant of dafb344.

Anders will make all the marks files used for the conversions of the
FEniCS projects available and we'll send detailed migration
instructions for feature branches once all the infrastructure is in
place. I've already sketched out how that is going to work:
https://gist.github.com/kynan/41b7e7f405ce38fa250a

Florian

> -- Anders
> 
> _______________________________________________ Mailing list: 
> https://launchpad.net/~fenics Post to     : 
> [email-masked] Unsubscribe : 
> https://launchpad.net/~fenics More help   : 
> https://help.launchpad.net/ListHelp
>

---

From: Florian Rathgeber <[email-masked]>
Date: Wed, 20 Mar 2013 02:10:48 +0000

On 20/03/13 00:51, Anders Logg wrote:
> On Wed, Mar 20, 2013 at 12:16:09AM +0000, Florian Rathgeber wrote:
> 
>>> Anders, Garth and I sat down this evening to work out a
>>> migration strategy and I think we've arrived at something
>>> workable.
>>> 
>>> Just a few words of explanation about the pitfalls: Converting
>>> the trunk is relatively straightforward (and the conversion
>>> scripts should have finished by now), but we need to be very
>>> careful when migrating feature branches s.t. they keep the
>>> correct ancestry relationship.
>>> 
>>> Consider the following situation of a feature branch branching
>>> of trunk after revision 1803:
>>> 
>>> o trunk (r 1806) --> d9e67c2 | o  + feature-branch (r 1805)
>>> --> 16c0470 |  | o  + | / |/ o base (r 1803) --> dafb344 |
>>> 
>>> After converting trunk to Git, bzr revision 1806 becomes git 
>>> revision d9e67c2 and 1803 becomes dafb344. When migrating 
>>> feature-branch, it is *critical* that revision 16c0470 is a 
>>> descendant of dafb344.
>>> 
>>> For this to work, bzr fast-export needs to know which revisions
>>> it has already exported and git fast-import needs to know
>>> which revisions it has already seen (and both need to agree on
>>> that!). Therefore you need to have access to the *same* marks
>>> file that was used when converting trunk.
>>> 
>>> conversion for revision 1803 and its ancestors *again* and you 
>>> will end up with *the entire history replicated* but
>>> *different* git revision ids s.t. git has no way of knowing
>>> that feature-branch should be a descendant of dafb344.
>>> 
>>> Anders will make all the marks files used for the conversions
>>> of the FEniCS projects available and we'll send detailed
>>> migration instructions for feature branches once all the
>>> infrastructure is in place. I've already sketched out how that
>>> is going to work: 
>> We really need to make sure this is done properly or we'll
>> create ourselves a lot of work and headaches. I've had a chat
>> with Andy and I think he has a better idea of the differences
>> between BitBucket and GitHub than the rest of us. Is it worth for
>> the 4 of us have a (Skype/Google hangout) discussion before 
>> announcing it all publicly? There is also the question whether or
>> not you'd like to make/suggest some changes to the current
>> development workflow. Git differs from bzr in some way e.g.
>> branches are much more lightweight and therefore it encourages
>> branching much more often. Also git has very advanced history
>> rewriting capabilities so you could switch to a rebase instead of
>> merge workflow if you like.
> 
> Thanks to Florian for the great assistance.
> 
> The files needed to convert a branch which has a FEniCS component 
> trunk branch as an ancestor can be found here, together with the 
> script used to do the conversion:
> 
> http://fenicsproject.org/pub/software/git/bzr-to-git-conversion-2013-03-19/
>
>  We will store this data on the server for an eternity.
> 
> If you don't know what that means or don't care, you can ignore
> this message.
> 
> New Git branches of FEniCS components have been created here:
> 
> https://bitbucket.org/fenics-project
> 
> These should now be used as the main development branches.

I've written an automatic branch import script which should allow
anyone to import their bzr feature branches into the new git
repositories: https://gist.github.com/kynan/41b7e7f405ce38fa250a

To get it, in a directory on your path do:
$ wget
https://gist.github.com/kynan/41b7e7f405ce38fa250a/raw/import-branch.sh
$ chmod +x import-branch.sh

Let's take DOLFIN as an example (it works the same way for the other
projects) and assume you want to import the branch my-feature:

1) Clone the repository
$ git clone [email-masked]:fenics-project/dolfin.git
$ cd dolfin

2) Import your feature branch
$ import-branch.sh dolfin /path/to/my-feature my-feature

3) Check that everything worked and the ancestry relationship is correct
$ git log --graph --all --decorate --pretty=oneline --abbrev-commit

4) Importing further branches should work also incrementally and even
after you've worked with the git repository *as long as you keep the
marks files* in .git/info/fast-import/dolfin.marks.{git,bzr}.

This should allow you resume working in git right where you left off
in bzr! Please let us know if there are any migration issues.

Florian

---

From: Anders Logg <[email-masked]>
Date: Wed, 20 Mar 2013 10:17:17 +0100

These should now be used as the main development branches, but please
hang on for a day or as we work out some more details.

--
Anders

On Tue, Mar 19, 2013 at 11:43:04PM +0000, Florian Rathgeber wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> iEYEARECAAYFAlFI+AgACgkQ8Z6llsctAxa8CACfdFTlG3ngexR/cg2dmmfQvJVP
> NWwAoLUmXCJtNuA48L/IcJ3t4CdB7CIf
> =iTK9
> -----END PGP SIGNATURE-----