Subject: Re: [Fenics] UFR - The Unified Fenics Repository

From: Roland Siegbert <[email-masked]>
Date: Sat, 09 Feb 2013 12:49:49 +0100

Hello,

just reading through the discussion, my impression is, that you are 
looking for something like git submodule [1] to ease the packaging 
process, deducing from what Anders wrote. The pros and cons are laid out 
nicely in Martin's blueprint [2].

After a bit of searching, I found a corresponding feature in bzr, called 
"nested trees" [3]. Maybe it makes sense to wait until Canonical 
releases it?

In git, the corresponding workflow would be:

git clone repo-easing-packaging
git submodule init (tell repo-easing-packaging about dependency-repos)
git submodule update (fetch dependency-repos in the working state)

this ensures, that repo-easing-packaging will always work.

Imagine now, there are changes in any of the dependency-repos, than you 
can fetch them like so:

git submodule foreach git pull origin master

and test against the changes and adjust repo-easing-packaging accordingly.

At my workplace we currently have the problem that we ran into the issue 
of having one big repo containing many should-be-orthogonal software 
pieces and now we have to untangle them and integrate them using 
submodules to ease deployment because it's tangled, heavy to use and clumsy.

Hope I could help and excuse my probably cheeky jump into that 
discussion. Just discovered this project a few days ago and bought the 
fenicsbook this week. :-)

Best,

Roland

[1]: http://git-scm.com/book/en/Git-Tools-Submodules

tl;dr: Basically a "mother repo" can pull and track "associated children 
repos".

[2]: 
https://blueprints.launchpad.net/fenics/+spec/combine-projects-into-one-repository

[3]: 
http://doc.bazaar.canonical.com/migration/en/foreign/bzr-on-git-projects.html

---

From: Myles English <[email-masked]>
Date: Sat, 09 Feb 2013 13:23:37 +0000

Hi Roland,

> submodules to ease deployment because it's tangled, heavy to use and clumsy.

Just a note to say that if FEniCS is going to move to git (yes please)
and consider 'git submodule' then you would benefit from looking at 'git
subtree'.  I am just going to quote the man page:

       """
       Subtrees allow subprojects to be included within a subdirectory of the main
       project, optionally including the subproject’s entire history.

       For example, you could include the source code for a library as a subdirectory
       of your application.

       Subtrees are not to be confused with submodules, which are meant for the same
       task. Unlike submodules, subtrees do not need any special constructions (like
       .gitmodule files or gitlinks) be present in your repository, and do not force
       end-users of your repository to do anything special or to understand how
       subtrees work. A subtree is just a subdirectory that can be committed to,
       branched, and merged along with your project in any way you want.
       """

Myles

---

From: Roland Siegbert <[email-masked]>
Date: Sat, 9 Feb 2013 15:02:45 +0100

Myles,

thank you for this one. :-)

> subtree'.  I am just going to quote the man page:
It's even already included into git since git-1.7.11, though it seems
to be quite young! Workstation here: LM14 w/ git-1.7.10.4 - however,
there is a origin/hack: https://github.com/apenwarr/git-subtree.

I really like that this approach, as it seems to be free of any
overhead and it is even possible to "export" the commits done in the
monolithic repo to the subtree repo.

Roland

Further reading for anybody interested:

[1]: http://log.pardus.de/2012/08/modular-git-with-git-subtree.html

It seems they had a similar problem at the Horde Project.

---

From: Anders Logg <[email-masked]>
Date: Sun, 10 Feb 2013 21:22:43 +0100

Thanks for the pointers. This would solve one part of the problem -
having to commit in separate repositories and branching each one when
working on features that cut across projects.

However, it doesn't solve the overhead of making releases of 6
different packages and clicking the buttons in Launchpad, making
tarballs, release notes etc for each one.

--
Anders

> _______________________________________________
> Mailing list: https://launchpad.net/~fenics
> Post to     : [email-masked]
> Unsubscribe : https://launchpad.net/~fenics
> More help   : https://help.launchpad.net/ListHelp

---

From: Myles English <[email-masked]>
Date: Mon, 11 Feb 2013 03:25:10 +0000

Hi Anders,

> working on features that cut across projects.

One tool that appears to be popular for working in git and pushing to
bazaar is [1].

> tarballs, release notes etc for each one.

I don't really have an opinion on unification, but perhaps some scripts
using the Launchpad API[2] could solve the button clicking problem?  I
had a quick look and there are various scripts available, e.g. [3]
appears to upload release tarballs.

Other things people have done is move the code to github and use lp for
the packaging [4].

Myles

Footnotes: 
[1]  https://github.com/termie/git-bzr-ng

[2]  https://help.launchpad.net/API

[3]  http://bazaar.launchpad.net/~lazr-developers/launchpadlib/trunk/view/head:/contrib/upload_release_tarball.py

[4]  https://blueprints.launchpad.net/openstack-devel/+spec/move-to-github

---

From: Florian Rathgeber <[email-masked]>
Date: Mon, 18 Feb 2013 20:29:23 +0000

I'm a bit late to this discussion, apologies if I've missed any
further development off-list since. Still I wanted to comment on a few
of the suggestions from my experience:

It would be great to be able to have the best of both worlds i.e.
concerted versioning and releases for all core FEniCS components,
while at the same time being able to get and contribute to components
in isolation.

The last point I think hasn't come up in the discussion yet. I think
the UFR would (further) raise the barrier to entry for external FEniCS
contributors, since they would be faced with a huge codebase they will
have to learn to navigate first. Imagine you're not a FEM person, but
find instant useful for your work and want to contribute. How much
effect this might have is pure speculation and it may be a sacrifice
worth making to make the lives of the core contributors easier.

Back to composing of repositories with matching versions: Git supports
this by virtue of submodules or subtrees as already mentioned.
Unfortunately bzr doesn't have any comparable feature. There is bzr
externals [5] which mimics svn externals, but is far from mature.

I wouldn't wait for nested trees support to land in bzr. There was
virtually no progress on this feature since 2009 [6] other than an
update of the spec [7].

On 11/02/13 03:25, Myles English wrote:
> 
> Hi Anders,
> 
> Anders Logg writes:
> 
>> Thanks for the pointers. This would solve one part of the problem
>> - having to commit in separate repositories and branching each
>> one when working on features that cut across projects.
> 
> One tool that appears to be popular for working in git and pushing
> to bazaar is [1].

Note that using git-bzr-ng is a one-way operation, not a two-way
bridge, since it uses bzr-fast{ex,im}port. `bzr server --git` might
become a 2-way bridge, once [8] is resolved, which is unlikely to happen.

> I don't really have an opinion on unification, but perhaps some
> scripts using the Launchpad API[2] could solve the button clicking
> problem?  I had a quick look and there are various scripts
> available, e.g. [3] appears to upload release tarballs.
> 
> Other things people have done is move the code to github and use lp
> for the packaging [4].

Note that importing Git repositories containing submodules is
currently not supported due to a bug [9].

Florian

[5] https://launchpad.net/bzr-externals
[6] http://wiki.bazaar.canonical.com/NestedTreeProgress
[7] http://doc.bazaar.canonical.com/developers/nested-trees.html
[8] https://bugs.launchpad.net/bzr-git/+bug/544776
[9] https://bugs.launchpad.net/launchpad/+bug/402814

> Myles
> 
> 
> Footnotes: [1]  https://github.com/termie/git-bzr-ng
> 
> [3]
> http://bazaar.launchpad.net/~lazr-developers/launchpadlib/trunk/view/head:/contrib/upload_release_tarball.py
>
>  [4]
> https://blueprints.launchpad.net/openstack-devel/+spec/move-to-github
>
>  _______________________________________________ Mailing list:
> https://launchpad.net/~fenics Post to     :
> [email-masked] Unsubscribe :
> https://launchpad.net/~fenics More help   :
> https://help.launchpad.net/ListHelp

---

From: Anders Logg <[email-masked]>
Date: Tue, 19 Feb 2013 11:26:03 +0100

It seems to me there are essentially three groups that this suggestion
will affect in different ways:

1. Core developers who have to suffer the hassle of making multiple
releases and working every day in multiple repositories.

2. Advanced users who want to pick and choose specific components.

3. Users who don't really care how the code is organized as long as it
can be installed easily.

I think the suggestion will be a big relief for the group (1) of core
developers (5-10 people) and it will have a positive effect on the big
group of users (> 1000 people?) who don't need to worry about matching
version numbers. But it might be a small setback for the relatively
small group (2) of advanced users.

I might be wrong but I estimate the size of group (2) as somewhere
around 5-10 people and I think we've heard from all of them in this
discussion. :-) So my suggestion would be to go ahead with the
proposal and while doing so make sure to structure the new repo and
build system in such a way that it's easy to install single
components.

The extra hassle for users that only want UFL would then be to
download a bigger tarball, fenics-2.0.0.tar.gz instead of
ufl-2.0.0.tar.gz. But apart from that, it would just be something like

$ cmake -DFENICS_ENABLE_ONLY_UFL . # or similar...
$ make install

--
Anders

---

From: "Garth N. Wells" <[email-masked]>
Date: Wed, 20 Feb 2013 22:15:20 +0000

> small group (2) of advanced users.

I'm still not fully convinced either way. I find the concept of nested
repositories appealing. How convenient it is in practice I don't know.
Would someone who has spoken in favour of nested repositories be
wiling to make an example somewhere that we could test? I'm open to
changing version control system if something else suits our needs.

Garth

---

From: Anders Logg <[email-masked]>
Date: Thu, 21 Feb 2013 00:35:42 +0100

> changing version control system if something else suits our needs.

Same here, but we're fairly committed to Launchpad and Launchpad does
not do git.

And will a nested repository solve the problem of making multiple
releases?

--
Anders

---

From: "Garth N. Wells" <[email-masked]>
Date: Thu, 21 Feb 2013 08:39:32 +0000

> Same here, but we're fairly committed to Launchpad and Launchpad does

Hopefully there is a volunteer to show us how it would work with
nested repos. It would have to be pretty compelling to contemplate
moving from bzr (Launchpad supports automated import from other
version control systems).

I'd also like to see how bzr handles a very large unified repository.
Recently the speed of bzr on larger repositories has been bugging me.

> And will a nested repository solve the problem of making multiple

We can choose to make just one release.

Garth

---

From: Anders Logg <[email-masked]>
Date: Thu, 21 Feb 2013 12:05:17 +0100

> Recently the speed of bzr on larger repositories has been bugging me.

Agree.

> We can choose to make just one release.

Yes, but only if it can be made from a single source directory and
there is a single page to upload it to, and a single set of release
notes to write.

--
Anders

---

From: Myles English <[email-masked]>
Date: Thu, 21 Feb 2013 12:11:29 +0000

Hi Garth,

Having piped up about 'git subtree' I feel implicated in this discussion
but can't commit any more time to it for the next couple of weeks,
sorry.  Some caveats I should point out, 1) I am not a fenics developer,
2) git is familiar to me and bazaar is not, 3) most of what follows is
theory.

One solution, is that the project is kept unchanged on launchpad (lp)
and mirror the repositories on github.  The OpenStack project has done
this, they have 35 components, and they have a simple script to sync
git<-->lp, I have no idea if they like that setup or not.  Maybe the
increased complexity is worth it for 35 components but would not be for
FEniCS.

Garth N. Wells writes:

>>>> releases

Scripts that work with lp API should solve this, as mentioned.

>>>> and working every day in multiple repositories.

Git subtree should solve this.  You could work everyday in one repo and
split the changes back out to the individual components before making a
release.  One change in workflow is that it is better not to mix changes
to more than one component in the same commit.

>>>> 2. Advanced users who want to pick and choose specific components.

The individual components would still be available in lp.

>>>> 3. Users who don't really care how the code is organized as long as it

Ubuntu packages would still be available from lp, source from github or
lp.

>>>> small group (2) of advanced users.

Core developers may not be happy about being forced to use git, or the
slightly increased complexity of syncing and maintenance.

>>> repositories appealing.

Florian said that nesting bazaar repos is not feasible.  The git subtree
approach is not quite the same concept in that you just work with one
repo, and push to the component repos when ready.

>>> How convenient it is in practice I don't know.  Would someone who
>>> has spoken in favour of nested repositories be wiling to make an
>>> example somewhere that we could test? I'm open to changing version
>>> control system if something else suits our needs.

>> Same here, but we're fairly committed to Launchpad and Launchpad does

Launchpad would continue to be used under this scheme.

> version control systems).

I have made a start, this is how to set up a local UFR using
git-subtree:

Install git-bzr-ng and git.

# make git repos, this is quite slow
mkdir ~/tmp/fex
cd ~/tmp/fex
${BZR_REPOS}=~/path_to_bzr_repos
for i in dolfin ufc ufl ffc viper instant; do
  git bzr clone ${BZR_REPOS}/${i} ${i}
done

# make UFR
mkdir ~/tmp/fenics
cd ~/tmp/fenics
git init
echo "Needs an initial commit" > README
git cim "Added README"

# import all the components to the UFR as subtrees
for i in ufc ufl ffc viper instant dolfin; do
    git subtree add --prefix=${i} --squash \
                 ~/tmp/fex/${i} master
  done

Then do 'man git-subtree' look at push pull etc, which is the tricky
bit, I don't have time now to go any further.  I use git-subtree for
including a bibliography file and LaTeX style file that I use in lots of
different LaTeX projects and it works well.  I don't push the changes
back to the upstream repository that often, which is probably normal for
a library, and the most difficult bit is remembering the right command.
(Hmm, why is it that in seeking to simplify something it ends up looking
more complicated?)

Myles

---

From: Anders Logg <[email-masked]>
Date: Thu, 21 Feb 2013 18:47:39 +0100

> to more than one component in the same commit.

I'm not sure that's what we need. The point of the proposed change is
to make it easy for developers to work on 'fenics' without having to
worry about syncing and merging with multiple sources.

> Launchpad would continue to be used under this scheme.

I wouldn't mind changing from bzr to git but the point of a change
would have to be to make things simpler, not more complex.

> (Hmm, why is it that in seeking to simplify something it ends up looking

Thanks for the demo.

--
Anders

---

From: Martin Sandve Alnæs <[email-masked]>
Date: Mon, 25 Feb 2013 10:13:44 +0100

Some thoughts I have now on the different aspects:

- A workflow that mixes two revision control systems is not sustainable.
It's bound to cause more problems than it solves. In particular considering
that most fenics developers prefer not having to learn the details of a
single system.

- Releases are not a big problem either way. Releasing more often will
reduce the per-release cost of blueprint and bug management, and allow
Johannes to keep the release process automated with lp scripting. If we
want one release for users, we can have that without repository change by
just checking out all current projects into a fenics/ directory and making
a tarball of that.

- If a good workflow with the unified repository requires not mixing
changes across components in one commit, I don't see any gain at all,
that's how it always is today. What we can not do without a unified
repository is commits that span multiple projects, for interface changes
that do affect multiple projects. If that's inconvenient to do, a unified
repository is worthless from my point of view.

- I think the two-way split (keeping dolfin separate, joining at least
ufc-ffc-ufl) sounds most compelling and carries less risk.

- The way I see it now, the main backside of the unification (assuming a
two-way split) is the complexity of backporting bug fixes. Worst case
scenario there is having to do manual patching, but it may be easier and we
don't do that much backporting.

Martin

---

From: Myles English <[email-masked]>
Date: Mon, 25 Feb 2013 13:15:12 +0000

Mainly just to clarify something I said earlier, as I am approaching the
limit of my knowledge on this subject.  Also, unless someone is willing
to try out the git subtree idea and enjoy the experience it won't go any
further anyway.

Martin Sandve Alnæs writes:

> It's bound to cause more problems than it solves.

Git is commonly used to work conveniently with centralised Subversion
repositories.

> In particular considering that most fenics developers prefer not
> having to learn the details of a single system.
>
> that's how it always is today.

The only reason for not mixing changes across components in one commit
is that the same commit message ends up in both repos (unless --squash
is used) where it may not make sense in that context.

> What we can not do without a unified repository is commits that span
> multiple projects, for interface changes that do affect multiple
> projects. If that's inconvenient to do, a unified repository is
> worthless from my point of view.
>

Myles

---

From: Anders Logg <[email-masked]>
Date: Tue, 26 Feb 2013 02:16:04 +0100

Agree.

> just checking out all current projects into a fenics/ directory and making

Depends on whether Johannes can script everything. It's a big relief
for the rest of us if Johannes makes the releases, but if it takes him
a full day, then that's still many wasted hours.

> repository is worthless from my point of view.

Agree, we should definitely be able to commit for multiple components
at once. Otherwise there's no point.

> ufc-ffc-ufl) sounds most compelling and carries less risk.

I'm still tempted by having one big repo.

> don't do that much backporting.

Why is that more difficult with one repository? You mean to versions
that existed before the unification? In that case, it's a problem that
will go away as soon as we stop supporting those old versions...

-- 
Anders

---

From: "Garth N. Wells" <[email-masked]>
Date: Tue, 26 Feb 2013 09:07:21 +0000

> a full day, then that's still many wasted hours.

I expect the more frequent releases will reduce the burden because
less script updating will be required. The longer it's left, the more
that changes.

> at once. Otherwise there's no point.

Agree (with some painful recent debugging in mind around changesets
where changes were made in different repositories).

>> ufc-ffc-ufl) sounds most compelling and carries less risk.

Even more granular would be ufc-ffc. That way, FFC would contain all
the code formatting.

> I'm still tempted by having one big repo.

I'm inclined to stay close to the status quo. If a big repo is
contemplated, someone should make one and we can test if it's workable
with bzr. It may just be too big and bzr too slow.

>> don't do that much backporting.

I'd advocate more

    frequent releases = less need for backporting

Garth

---

From: "Marie E. Rognes" <[email-masked]>
Date: Tue, 26 Feb 2013 10:56:52 +0100

On 02/26/2013 10:07 AM, Garth N. Wells wrote:
> with bzr.

I am also not completely convinced and agree with the above.

--
Marie

---

From: Martin Sandve Alnæs <[email-masked]>
Date: Tue, 26 Feb 2013 10:57:12 +0100

> with bzr. It may just be too big and bzr too slow.

The best way to do such things is usually gradually. The first steps could
be:
1) Move ufc into the ffc repo.
2) Move dolfin wrapper generation back from dolfin to ffc?
In these cases there are no big history and patching issues,
so this can be done soon with no issues whatsoever (but
preferably after we merge the work in progress by Anders and I).

Martin

---

From: Martin Sandve Alnæs <[email-masked]>
Date: Tue, 26 Feb 2013 10:59:20 +0100

Already did, from the initial post:

Here's a summary of pros and cons:
https://blueprints.launchpad.net/fenics/+spec/combine-projects-into-one-repository
including a link to an already merged repository prototype and a script to
create it.

The repo is here (not the most recent versions anymore):
 lp:~martinal/+junk/unified-fenics-repo<https://blueprints.launchpad.net/+branch/~martinal/+junk/unified-fenics-repo>

Martin

On 26 February 2013 10:56, Marie E. Rognes <[email-masked]> wrote:

> ______________________________**_________________
> More help   : https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp>
>

---

From: Roland Siegbert <[email-masked]>
Date: Tue, 26 Feb 2013 11:06:08 +0100

> version control systems).
You can have a look at my dotfiles, they use git submodule:

git clone http://github.com/rawland/dotfiles
<a have a look at the folders here - especially dotfiles/vim/bundle/*>
git submodule init
git submodule update
<have a look again>

To update the "unified main" repo to the most recent version of the
children, write:
git submodule foreach git pull origin master

Though I would advocate for git subtree, as it makes it possible to
commit to each specific repo, as far as I understood.

Best,

Roland

---

From: Anders Logg <[email-masked]>
Date: Tue, 26 Feb 2013 17:10:41 +0100

> preferably after we merge the work in progress by Anders and I).

Sounds like a good start. Any objections to this?

Does this need we need to use CMake for FFC?

-- 
Anders

---

From: "Garth N. Wells" <[email-masked]>
Date: Wed, 27 Feb 2013 20:54:16 +0000

On Tuesday, 26 February 2013, Anders Logg wrote:

> > On 26 February 2013 10:07, Garth N. Wells <[email-masked]<javascript:;>>
> wrote:
> >
> > > On 26 February 2013 01:16, Anders Logg <[email-masked] <javascript:;>>
> wrote:

No.

> Does this need we need to use CMake for FFC?

I do like the simplicity of the Python install for FFC over CMake.

Garth

-- 
Garth N. Wells
Department of Engineering, University of Cambridge
http://www.eng.cam.ac.uk/~gnw20

---

From: Johan Hake <[email-masked]>
Date: Wed, 27 Feb 2013 22:13:17 +0100

On 02/27/2013 09:54 PM, Garth N. Wells wrote:
>     > On 26 February 2013 10:07, Garth N. Wells <[email-masked]
>     <javascript:;>> wrote:
>     >
>     > > On 26 February 2013 01:16, Anders Logg <[email-masked]
>     <javascript:;>> wrote:

If we are going to keep each project in different sub directories we can
keep distutils for ffc.

 ffc/
     ffc/
        setup.py
        ...
     ufc/
        CMakeList.txt
        ...

Johan

---

From: Anders Logg <[email-masked]>
Date: Wed, 27 Feb 2013 22:46:32 +0100

> If we are going to keep each project in different sub directories we can

Sounds like a simple solution, but we also need a top level
installation method. Should that be CMake or distutils?

-- 
Anders

---

From: Florian Rathgeber <[email-masked]>
Date: Tue, 05 Mar 2013 22:32:38 +0000

On 27/02/13 21:46, Anders Logg wrote:
>>>>>> On Mon, Feb 25, 2013 at 10:13:44AM +0100, Martin Sandve
>>>>>> Alnæs
>>> wrote:
>>>>> 
>>>>>> - I think the two-way split (keeping dolfin separate,
>>>>>> joining
>>> at least
>>>>> Even more granular would be ufc-ffc. That way, FFC would
>>>>> contain all the code formatting.
>>>>> 
>>>>> I'm inclined to stay close to the status quo. If a big repo
>>>>> is contemplated, someone should make one and we can test if
>>>>> it's
>>> workable
>>>> The best way to do such things is usually gradually. The
>>>> first
>>> steps could
>>>> be: 1) Move ufc into the ffc repo. 2) Move dolfin wrapper
>>>> generation back from dolfin to ffc? In these cases there are
>>>> no big history and patching issues, so this can be done soon
>>>> with no issues whatsoever (but preferably after we merge the
>>>> work in progress by Anders and I).
>>> 
>> If we are going to keep each project in different sub directories
>> we can keep distutils for ffc.
>> 
>> ffc/ ffc/ setup.py ... ufc/ CMakeList.txt ...
> 

Why not support both (as PETSc does)? A top level CMakeLists.txt
descends in both the ffc subdirectory (calling distutils as e.g.
described in this blog post:
http://bloerg.net/2012/11/10/cmake-and-distutils.html) and ufc
subdirectory.

I think it's really important that FEniCS components are also
distutils/pip-installable so you can require them as dependencies in
other Python packages. That's what we would need for PyOP2, but it
currently falls over because UFC has no distutils installation path.

Florian

---

From: Johan Hake <[email-masked]>
Date: Wed, 06 Mar 2013 08:17:36 +0100

On 03/05/2013 11:32 PM, Florian Rathgeber wrote:
>>>>>>> On Mon, Feb 25, 2013 at 10:13:44AM +0100, Martin
>>>>>>> Sandve Alnæs
>>>> wrote:
>>>>>> 
>>>>>> I'm inclined to stay close to the status quo. If a big
>>>>>> repo is contemplated, someone should make one and we can
>>>>>> test if it's
>>>> workable
>>>>> generation back from dolfin to ffc? In these cases there
>>>>> are no big history and patching issues, so this can be done
>>>>> soon with no issues whatsoever (but preferably after we
>>>>> merge the work in progress by Anders and I).
>>>> 
>>> If we are going to keep each project in different sub
>>> directories we can keep distutils for ffc.
>>> 

This could be done, but we need the CMake for the the UFC extention
module configuration, which generates a Python module. Your example
above does not cover that case. As I understand it it is rather the
other way around: distutils need to call CMake.

> distutils/pip-installable so you can require them as dependencies
> in other Python packages. That's what we would need for PyOP2, but
> it currently falls over because UFC has no distutils installation
> path.

Correct me if I am wrong, for pip-install to work we need a top level
distutils script? If so that script can install the ffc and python
components of ufc (when we have merged fcc and ufc). That same script
need to trigger the configure part of CMake and either wait for it to
finish (some popen logic maybe?). Then either let CMake install
everything or just the .h files or let distutils install everything
once the extension module is generated.

Johan

---

From: Anders Logg <[email-masked]>
Date: Wed, 6 Mar 2013 08:31:35 +0100

> other way around: distutils need to call CMake.

Could the extension module utility be moved to DOLFIN? Or does it not
fit in there?

--
Anders

---

From: Johan Hake <[email-masked]>
Date: Wed, 06 Mar 2013 08:35:45 +0100

> Could the extension module utility be moved to DOLFIN? Or does it not

I do not think it fits in there. It is pure ufc stuff. Anyone who
creates a ufc class can wrap it to Python using the ufc extension module
stuff.

Johan

---

From: Martin Sandve Alnæs <[email-masked]>
Date: Wed, 6 Mar 2013 08:40:24 +0100

No, it is used by ffc.

Martin

---

From: Florian Rathgeber <[email-masked]>
Date: Wed, 06 Mar 2013 10:44:07 +0000

On 06/03/13 07:17, Johan Hake wrote:
>>>>>>> Even more granular would be ufc-ffc. That way, FFC
>>>>>>> would contain all the code formatting.
>>>>>>> 
>>>>>>> repo is contemplated, someone should make one and we
>>>>>>> can test if it's
>>>>> workable
>>>>>>  generation back from dolfin to ffc? In these cases
>>>>>> there are no big history and patching issues, so this can
>>>>>> be done soon with no issues whatsoever (but preferably
>>>>>> after we merge the work in progress by Anders and I).
>>>>> 
> This could be done, but we need the CMake for the the UFC
> extention module configuration, which generates a Python module.
> Your example above does not cover that case. As I understand it it
> is rather the other way around: distutils need to call CMake.

For using CMake as the top-level build method I don't think there are
many changes needed for the UFC build: CMake descends into the
subdirectory and calls the UFC CMakeLists.txt.

>> distutils/pip-installable so you can require them as
>> dependencies in other Python packages. That's what we would need
>> for PyOP2, but it currently falls over because UFC has no
>> distutils installation path.
> 
> Correct me if I am wrong, for pip-install to work we need a top
> level distutils script? If so that script can install the ffc and
> python components of ufc (when we have merged fcc and ufc). That
> same script need to trigger the configure part of CMake and either
> wait for it to finish (some popen logic maybe?). Then either let
> CMake install everything or just the .h files or let distutils
> install everything once the extension module is generated.

Yes, distutils requires a top level setup.py/setup.cfg. This could
simply depend on FFC and UFC and they could provide their own setup.py.

For UFC I think both options you describe would work. The only tricky
part I think is passing options given to setup.py to CMake in a
sensible way i.e. --prefix should become CMAKE_INSTALL_PREFIX. Maybe
also read an environment variable like UFC_CONFIGURE_OPTIONS and pass
this to CMake as a raw argument string.

I've looked for examples other than PETSc doing this but haven't
really found any so far. And PETSc's way is too elaborate for this use
case.

Florian

---

From: Johan Hake <[email-masked]>
Date: Thu, 07 Mar 2013 09:26:05 +0100

On 03/06/2013 11:44 AM, Florian Rathgeber wrote:
>>>>>>>>> - I think the two-way split (keeping dolfin
>>>>>>>>> separate, joining
>>>>>> at least
>>>>>>>> I'm inclined to stay close to the status quo. If a
>>>>>>>> big repo is contemplated, someone should make one and
>>>>>>>> we can test if it's
>>>>>> workable
>>>>>>> The best way to do such things is usually gradually.
>>>>>>> The first
>>>>>> steps could
>>>>>>> be: 1) Move ufc into the ffc repo. 2) Move dolfin
>>>>>>> wrapper generation back from dolfin to ffc? In these
>>>>>>> cases there are no big history and patching issues, so
>>>>>>> this can be done soon with no issues whatsoever (but
>>> Why not support both (as PETSc does)? A top level
>>> CMakeLists.txt descends in both the ffc subdirectory (calling
>>> distutils as e.g. described in this blog post: 
>> Your example above does not cover that case. As I understand it
>> it is rather the other way around: distutils need to call CMake.
> 
> For using CMake as the top-level build method I don't think there
> are many changes needed for the UFC build: CMake descends into the 

Agree. This is not a problem.

>>> dependencies in other Python packages. That's what we would
>>> need for PyOP2, but it currently falls over because UFC has no 
>> level distutils script? If so that script can install the ffc
>> and python components of ufc (when we have merged fcc and ufc).
>> That same script need to trigger the configure part of CMake and
>> either wait for it to finish (some popen logic maybe?). Then
>> either let CMake install everything or just the .h files or let
>> distutils install everything once the extension module is
>> generated.
> 

Letting the toplevel setup.py call ffc's and ufc's setup.py?

Would that be necessary? My impression is that ffc and ufc will be
merged as one package. We could add some logic for not installing, let
say ufc, but that can be included as options in the toplevel script
without adding two setup.py script for each of ufc and ffc.

> For UFC I think both options you describe would work. The only
> tricky part I think is passing options given to setup.py to CMake
> in a sensible way i.e. --prefix should become CMAKE_INSTALL_PREFIX.
> Maybe also read an environment variable like UFC_CONFIGURE_OPTIONS
> and pass this to CMake as a raw argument string.

Yes that would need to be figured out, but it should not be that hard.

> really found any so far. And PETSc's way is too elaborate for this
> use case.

Yes, I think so too.

Johan

---

From: Anders Logg <[email-masked]>
Date: Thu, 7 Mar 2013 10:33:35 +0100

I don't see a reason for separating the installs of FFC and UFC. One cannot
be used without the other.

--
Anders

On 7 March 2013 09:26, Johan Hake <[email-masked]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJROE8XAAoJEEzjkVbXgH+MOpQH/i8/YY0fWs6iC0GLpYqHcrG0
> nUB+yUFbWkfQvePNH0yE9v/QC5if9X5w0drkmeYbF6mVsC+VUWmoN21Jc2tCWulA
> BYZ3/eUvFuk89BkPZmwD/mSo3yOiOHW0J3jog1x45gwUdVuKjoksFZO4p/B699gd
> biE6xZfz2aFyxY9B7NYPscKO1u/6ANE/kshkMPu/4sjYW5pzBSAtallKLx7pygHl
> 2GsoQNaGHxrhdo2FAGL+k2G87/+QYuI8lwiAb1E1wqKjqi3JbHDoJv+uoM6N5+Rz
> 23H9Rf2PTsux/8vh/6Dg7hDQ8DaaVNoAzxfKY3HuR2ZSB2CnkTL3h0EIdNLJCXs=
> =nw6q
> -----END PGP SIGNATURE-----
>