building graph-tool under Mac OS X

Hi,

I dont know if graph-tool is fully supported under macosx, but I have
encountered a few problems with building it (on osx 10.6). The problems seem
to stem from the configure/libtool scripts- about which I know very little-
but I have figured out enough to get it to build. I noticed the same problem
was reported on this list back in October (was it ever resolved?) - the
errors were along the lines of "configure: error: No usable boost::python
found". This problem would seem to be solved by modifying the configure
script, which tries to link against the boost libraries with
libboost-python*.[so,a] extensions- on osx I have compiled boost as a shared
library and so it should look for libboost-python*.[dylib]. There is a patch
for the configure script below. I also found some problems with the libtool
script, so I replaced it with libtool (from the macports libtool package)
(cp /opt/local/bin/glibtool ~/src/graph-tool/libtool) and the build
proceeded to completion. This patch just gets the build working- I dont
really know where the configure script gets the [so,a] information from, and
its old version of libtool- the proper fix would address these issues.

There was also a problem I encountered with numpy_bind.hh when compiling
with gcc44 (from macports). The fix was to add another element to the
numpy_types map- the patch is also below.

regards,

tcb

--- configure~ 2010-01-21 05:30:22.000000000 +0000 +++ configure 2010-01-21
05:41:37.000000000 +0000 @@ -16695,7 +16695,7 @@ BOOSTLIBDIR=`echo
$BOOST_LDFLAGS | sed -e 's/[^\/]*//'` if test
"x$ax_boost_user_iostreams_lib" = "x"; then - for libextension in `ls
BOOSTLIBDIR/libboost\_iostreams\*\.\{so,a\}\* 2>/dev/null | sort \-r | sed 's,\.\*/,,' | sed \-e 's;^lib\\\(boost\_iostreams\.\*\\\)\\\.so\.\*;\1;' -e
's;^lib\(boost_iostreams.*\)\.a*$;\1;'` ; do + for libextension in `ls
BOOSTLIBDIR/libboost\_iostreams\*\.\{dylib,a\}\* 2>/dev/null | sort \-r | sed 's,\.\*/,,' | sed \-e 's;^lib\\\(boost\_iostreams\.\*\\\)\\\.dylib\.\*;\1;' -e
's;^lib\(boost_iostreams.*\)\.a*;\\1;'\` ; do ax\_lib={libextension}
as_ac_Lib=`$as_echo "ac_cv_lib_$ax_lib''_exit" | $as_tr_sh` { $as_echo
"as\_me:{as_lineno-$LINENO}: checking for exit in -l$ax_lib" >&5 @@
-16926,7 +16926,7 @@ BOOSTLIBDIR=`echo $BOOST_LDFLAGS | sed -e 's/[^\/]*//'`
if test "x$ax_boost_user_regex_lib" = "x"; then - for libextension in `ls
BOOSTLIBDIR/libboost\_regex\*\.\{so,a\}\* 2>/dev/null | sort \-r | sed 's,\.\*/,,' | sed \-e 's;^lib\\\(boost\_regex\.\*\\\)\\\.so\.\*;\1;' -e
's;^lib\(boost_regex.*\)\.a*$;\1;'` ; do + for libextension in `ls
BOOSTLIBDIR/libboost\_regex\*\.\{dylib,a\}\* 2>/dev/null | sort \-r | sed 's,\.\*/,,' | sed \-e 's;^lib\\\(boost\_regex\.\*\\\)\\\.dylib\.\*;\1;' -e
's;^lib\(boost_regex.*\)\.a*;\\1;'\` ; do ax\_lib={libextension}
as_ac_Lib=`$as_echo "ac_cv_lib_$ax_lib''_exit" | $as_tr_sh` { $as_echo
"as\_me:{as_lineno-$LINENO}: checking for exit in -l$ax_lib" >&5

--- a/src/graph/numpy_bind.hh +++ b/src/graph/numpy_bind.hh @@ -50,13 +50,16
@@ typedef mpl::map< mpl::pair<int64_t, mpl::int_<NPY_INT64> >,
mpl::pair<uint64_t, mpl::int_<NPY_UINT64> >, mpl::pair<double,
mpl::int_<NPY_DOUBLE> >, - mpl::pair<long double, mpl::int_<NPY_LONGDOUBLE>

+ mpl::pair<long double, mpl::int_<NPY_LONGDOUBLE> >, + mpl::pair<unsigned

long int, mpl::int_<NPY_ULONG> > > numpy_types;

attachment.html (6.48 KB)

Thanks for the patch! I was the one taking out the issue. I also
managed to solve the libtool problem but i got stuck to the numpy
issue. I'll check out your second patch. Thanks again!
BTW i think we should create a macport of this lib!

Hi,

That's a great idea to take it into macports. I am only just checking
out graph-tool and it looks really nice, and a macports would
certainly be very useful. It also doesnt seem like it would be too
much work- I'm sure the macports guys would help out with the
autotools configuration- and other than the numpy patch it seems to
compile and run just fine on the mac. I have only compiled it 32-bit
with gcc44- I have not tried a 64-bit compile and I dont know if its
written to be 64bit safe (this would be very nice if it was). There
don't seem to be any tests which would be helpful in verifying that
the port was working- but a 32bit macport would be a start at least.

I initially found this project when I was looking for a working
version of the python bindings for the boost graph library-
development seems to be discontinued for some time. Can anyone confirm
if graph-tool is effectively a customised python binding for the bgl?
What would be involved in updating the python bgl bindings?

regards,

tcb

Hi tcb,

Thanks a lot for taking the time to do this! I don't have macos myself,
so I could not fix those issues. I planned to install it on a VM on my
laptop, but never got around to it due to lack of time.

I'll include your patches, but first I have to understand this libtool
thing. I generated the tarballs with libtool version 2.2.6b, which as
far as I know is the newest one, so what you are using must be a
specific non-official modification for macos. The libtool script
(ltmain.sh) is not included in graph-tool itself, and is generated
(copied from the system) when the tarball is created. I don't know hot
to include it, other than generating the tarball in a macos
environment...

Could you guys check if you run ./autogen.sh (with the git version of
graph-tool) before building the package solves the libtool issue without
copying it by hand? Could you also send me the ltmain.sh/libtool script
you use?

There are some more answers below.

That's a great idea to take it into macports. I am only just checking
out graph-tool and it looks really nice, and a macports would
certainly be very useful. It also doesnt seem like it would be too
much work- I'm sure the macports guys would help out with the
autotools configuration- and other than the numpy patch it seems to
compile and run just fine on the mac. I have only compiled it 32-bit
with gcc44- I have not tried a 64-bit compile and I dont know if its
written to be 64bit safe (this would be very nice if it was). There
don't seem to be any tests which would be helpful in verifying that
the port was working- but a 32bit macport would be a start at least.

64 bit should not be a problem at all, and I use it on 64 bit machines
all the time.

I initially found this project when I was looking for a working
version of the python bindings for the boost graph library-
development seems to be discontinued for some time. Can anyone confirm
if graph-tool is effectively a customised python binding for the bgl?
What would be involved in updating the python bgl bindings?

I think graph-tool works as an effective python binding for the bgl,
even though there are some algorithms there which are not (at this time)
in the BGL, such as random graph generation, rewiring, community
detection, triangulation and others. There are also some algorithms in
the BGL which are not reflected in graph-tool, but I'm in the gradual
process of incorporating them.

Regarding python-bgl, I believe all it needs is a maintainer, since no
one seems to be interested. The actual implementation is quite different
from graph-tool, since they opt for a list-based adjacency list (which
can be a performance killer) and try to be very close to the C++ API. In
graph-tool I kept a certain distance from it to make things more
comfortable in the python side (for instance, property map handling is
way easier in graph-tool IMHO), and included other nice things such as
graph filtering.

Thanks again for helping out!

Cheers,
Tiago

Hi,

The macports libtool is 2.2.6b and seems to work fine, so it was a bit
puzzling why there was any problem at all. After looking through the
configure script, at the end you have some lines to correct a problem
with linking the stdc++ library- it simply reads the libtool script,
pipes it through sed and writes it back again. This is what causes the
problems with libtool- the shell script is doing some interpretation
of the libtool script as it reads it and when you write it back again,
and some of the elaborate quoting of strings is changed.

You might try this patch on osx which removes the sed replacements in
the configure.ac file. I think things should work ok on osx without
this- I have no idea how you enable this conditionally on osx...

It is nice that the 64bit support is there. At the moment I have only
built it with 32bit on osx. I had thought that things were working ok,
but I had compiled the graph-tool library with a different compiler
version to my python libraries- so I have to compile over again... I
have encountered another few problems with the default compiler on osx
(gcc-4.2) which does not like the tr1::random functions- I got round
this by converting tr1::random to boost::random which seems to work
just fine (I'm using boost 1.41).

A major problem with trying to work on graph-tool is that the whole
library seems to take an extraordinary amount of time (and memory) to
compile (given the amount of code). I assume this is because of the
sophisticated use of template meta-programming. Is there any way to
reduce the compile times? Has anyone used precompiled headers? Do they
help?

regards,

tcb

--- configure.ac.bak 2010-01-25 08:05:51.000000000 +0000
+++ configure.ac 2010-01-25 08:06:00.000000000 +0000
@@ -325,6 +325,6 @@
# __cxa_allocate_exception() see:
# http://www.nabble.com/preventing-boost::python::throw_error_already_set-td20958723.html
# http://wiki.fifengine.de/Segfault_in_cxa_allocate_exception
-AC_MSG_NOTICE([Modifying libtool to correctly link libstdc++ before
anything else...])
-libtool=`cat libtool | sed "s/-nostdlib/-nostdlib -lstdc++/g"`
-echo "$libtool" > libtool
+#AC_MSG_NOTICE([Modifying libtool to correctly link libstdc++ before
anything else...])
+#libtool=`cat libtool | sed "s/-nostdlib/-nostdlib -lstdc++/g"`
+#echo "$libtool" > libtool

Hi,

The macports libtool is 2.2.6b and seems to work fine, so it was a bit
puzzling why there was any problem at all. After looking through the
configure script, at the end you have some lines to correct a problem
with linking the stdc++ library- it simply reads the libtool script,
pipes it through sed and writes it back again. This is what causes the
problems with libtool- the shell script is doing some interpretation
of the libtool script as it reads it and when you write it back again,
and some of the elaborate quoting of strings is changed.

You might try this patch on osx which removes the sed replacements in
the configure.ac file. I think things should work ok on osx without
this- I have no idea how you enable this conditionally on osx...

This is unfortunately necessary (at least on GNU/Linux) since there is
currently a problem with libtool/gcc when binaries are linked with
libstdc++ in a random order, which causes segfaults. The modification
ensures that libstdc++ gets linked first. What can be done, is to check
whether the corresponding line is there in the file, and only do the
modification if it looks exactly as expected, otherwise leave it alone.

It is nice that the 64bit support is there. At the moment I have only
built it with 32bit on osx. I had thought that things were working ok,
but I had compiled the graph-tool library with a different compiler
version to my python libraries- so I have to compile over again... I
have encountered another few problems with the default compiler on osx
(gcc-4.2) which does not like the tr1::random functions- I got round
this by converting tr1::random to boost::random which seems to work
just fine (I'm using boost 1.41).

Be careful, there are some serious bugs with the boost random library!
Take a look at: Boost users' mailing page: Re: [Boost-users] [random] Bias in boost::random generators?

I switched to tr1, since the boost::random stuff seems a bit
abandoned. However the tr1 has not entirely stabilized yet, and you need
gcc 4.4 or newer to use it.

A major problem with trying to work on graph-tool is that the whole
library seems to take an extraordinary amount of time (and memory) to
compile (given the amount of code). I assume this is because of the
sophisticated use of template meta-programming. Is there any way to
reduce the compile times? Has anyone used precompiled headers? Do they
help?

Precompiled headers do not really help in this case, but I recommend
using ccache (http://ccache.samba.org), which works very well.

Otherwise you can also turn off graph filtering to save some time and
memory.

I hear GCC 4.5 will have improved template performance, since they
switched to O(1) instantiation, instead of O(N).

Cheers,
Tiago

Hi,

Hi,

The macports libtool is 2.2.6b and seems to work fine, so it was a bit
puzzling why there was any problem at all. After looking through the
configure script, at the end you have some lines to correct a problem
with linking the stdc++ library- it simply reads the libtool script,
pipes it through sed and writes it back again. This is what causes the
problems with libtool- the shell script is doing some interpretation
of the libtool script as it reads it and when you write it back again,
and some of the elaborate quoting of strings is changed.

You might try this patch on osx which removes the sed replacements in
the configure.ac file. I think things should work ok on osx without
this- I have no idea how you enable this conditionally on osx...

This is unfortunately necessary (at least on GNU/Linux) since there is
currently a problem with libtool/gcc when binaries are linked with
libstdc++ in a random order, which causes segfaults. The modification
ensures that libstdc++ gets linked first. What can be done, is to check
whether the corresponding line is there in the file, and only do the
modification if it looks exactly as expected, otherwise leave it alone.

Ok, you could use sed to edit the file in place, instead of reading it
as a string- something like:

  eval sed -i -e "s/-nostdlib/-nostdlib -lstdc++/g" libtool

It is nice that the 64bit support is there. At the moment I have only
built it with 32bit on osx. I had thought that things were working ok,
but I had compiled the graph-tool library with a different compiler
version to my python libraries- so I have to compile over again... I
have encountered another few problems with the default compiler on osx
(gcc-4.2) which does not like the tr1::random functions- I got round
this by converting tr1::random to boost::random which seems to work
just fine (I'm using boost 1.41).

Be careful, there are some serious bugs with the boost random library!
Take a look at: Boost users' mailing page: Re: [Boost-users] [random] Bias in boost::random generators?

I switched to tr1, since the boost::random stuff seems a bit
abandoned. However the tr1 has not entirely stabilized yet, and you need
gcc 4.4 or newer to use it.

Oops, I hadn't seen that- I saw no indication that there were any such
problems with the boost random library (although this follow up mail
seems to suggest the problems were fixed as of 2009-05-30
Boost users' mailing page: Re: [Boost-users] [random] Bias in boost::random generators?). Unfortunately
there is very little information on what works in the tr1 libraries
across different compiler versions.

A major problem with trying to work on graph-tool is that the whole
library seems to take an extraordinary amount of time (and memory) to
compile (given the amount of code). I assume this is because of the
sophisticated use of template meta-programming. Is there any way to
reduce the compile times? Has anyone used precompiled headers? Do they
help?

Precompiled headers do not really help in this case, but I recommend
using ccache (http://ccache.samba.org), which works very well.

Otherwise you can also turn off graph filtering to save some time and
memory.

I hear GCC 4.5 will have improved template performance, since they
switched to O(1) instantiation, instead of O(N).

Ok, that's good to hear. If I recompile with gcc-4.5 then I also need
to recompile my whole python stack with 4.5 too (including numpy and
scipy etc...)- so that is my motivation to get something working with
gcc-4.2.

Cheers,
Tiago

regards

Hi

Ok, you could use sed to edit the file in place, instead of reading it
as a string- something like:

  eval sed -i -e "s/-nostdlib/-nostdlib -lstdc++/g" libtool

Indeed, this will probably solve the problem.

I committed all the necessary modifications to the git repository. Could
you please get the new git version, and see if it works for you?

Be careful, there are some serious bugs with the boost random library!
Take a look at: Boost users' mailing page: Re: [Boost-users] [random] Bias in boost::random generators?

I switched to tr1, since the boost::random stuff seems a bit
abandoned. However the tr1 has not entirely stabilized yet, and you need
gcc 4.4 or newer to use it.

Oops, I hadn't seen that- I saw no indication that there were any such
problems with the boost random library (although this follow up mail
seems to suggest the problems were fixed as of 2009-05-30
Boost users' mailing page: Re: [Boost-users] [random] Bias in boost::random generators?). Unfortunately
there is very little information on what works in the tr1 libraries
across different compiler versions.

The tr1 should eventually stabilize, and things should change less and
less in the next versions... Boost::random on the other hand does not
seem to have he necessary maintenance a RNG library needs to have. Even
though the problem in question was fixed, it took quite some time for it
to be detected, and according to other people, it is still has some
other serious problems left.

I hear GCC 4.5 will have improved template performance, since they
switched to O(1) instantiation, instead of O(N).

Ok, that's good to hear. If I recompile with gcc-4.5 then I also need
to recompile my whole python stack with 4.5 too (including numpy and
scipy etc...)- so that is my motivation to get something working with
gcc-4.2.

No, it is not so drastic. There was a C++ ABI change after 4.2 (IIRC),
but the interface to C libraries (such as python, numpy, etc) should be
stable. You should only have to recompile the boost libraries. And if
you do it now with gcc 4.4, it should still be compatible with 4.5.

Cheers,
Tiago

Hi,

Those changes look good, and fix the configure problems. It seems to
be compiling away just fine (I'll let you know in a day or so if it
finishes :slight_smile: )

It is unfortunate about the boost random libraries- they seemed to me
very well designed, and they should be more carefully maintainted. I
am not so sure the tr1 random libraries are better maintained. Anyway,
it would only be an issue if we wanted to make a macport using the
gcc-4.2 compiler (which chokes on tr1::random).

Hi,

Hi,

Those changes look good, and fix the configure problems. It seems to
be compiling away just fine (I'll let you know in a day or so if it
finishes :slight_smile: )

Good! I'll make a new release once we know it is working.

It is unfortunate about the boost random libraries- they seemed to me
very well designed, and they should be more carefully maintainted. I
am not so sure the tr1 random libraries are better maintained. Anyway,
it would only be an issue if we wanted to make a macport using the
gcc-4.2 compiler (which chokes on tr1::random).

The tr1::random interface was reviewed by the tr1 committee, and will be
part of the upcoming C++ standard. It is now in GCC, and I assume there
as a great deal of responsibility (and interest) in getting it to work
right. So far I haven't seen any problems, and I'm using it constantly
for a variety of projects. Anyway, as soon as tr1 becomes the standard,
there will be no point in using boost::random, or many other boost
libraries (tuple, type_traits, etc), since boost is largely a playground
for libraries that one day might make it to the standard.

Using gcc 4.4 should not be a problem in macports... Apparently you can
choose your own compiler, and it is there on the list:

http://trac.macports.org/wiki/UsingTheRightCompiler

The only problem is that boost (and CGAL) would have to be compiled with
it as well.

Cheers,
Tiago