Porting to Windows

Hi,

I don't know if it is right to pollute this list with my problems in trying to
compile graph-tool for Win64. I'll try to make it brief.
Using MSYS2/MinGW64, I managed to compile the whole src/graph modules
and install them as *.pyd modules in the site-packages/graph_tool directory,
along with the Python files.
Unfortunately, I get a segfault when importing graph_tool.libgraph_tool_core .
The segfault happens deep in libboost_python3.
Before posting a long backtrace, maybe someone may offer some clue on what is
happening. I have posted some details about what I did below.

If someone is interested and/or has ideas on how o go on, I'll be glad to share
and work on this problem.

(And I must say that I would love to get this tool to work under Windows,
because it seems to be one of the best of its kind!)

Best regards,

Fabrice

The background is: python 3.4, boost 1.57, clang 3.5.1 (g++ 4.9.2 as provided
with mingw64 couldn't compile a few objects without segfaulting).
Is it possible that there is a conflict among those libs/tools (of thread model
for example)?

Currently I have configured like this:
./configure --enable-debug --disable-optimize CFLAGS="-I/mingw64/include"
LDFLAGS="-L/mingw64/lib" CC=clang CXX='clang -std=c++11 -m64 -DMS_WIN64=1'
CXXCPP='clang -E -std=c++11 -m64 -DMS_WIN64=1' PYTHON=python3
Then I run 'make -k -j 8' to compile all the objects. Linking the various dlls
fails, because under Windows, you can't link graph_tool_centrality.dll without
referencing grap_tool_core.dll.a. However the build process is not prepared to
do that and in the right order.
So I manually replicated what libtool does, ie :

clang -std=c++11 -shared -nostdlib C:/Local/MSys64/mingw64/x86_64-w64-
mingw32/lib/dllcrt2.o C:/Local/MSys64/mingw64/x86_64-w64-mingw32/lib/crtbegin.o
.libs/*.o C:/Local/MSys64/mingw64/x86_64-w64-mingw32/lib/crtend.o -
L/mingw64/lib -O3 -
Wl,--export-all-symbols -Wl,--as-needed -o .libs/libgraph_tool_core.dll -Wl,--
enable-auto-image-base -Xlinker --out-implib -Xlinker
.libs/libgraph_tool_core.dll.a -lpython3.4m -lboost_iostreams -lboost_python3 -
lboost_regex -lboost_system -lCGAL -lgmp -lcairomm-1.0 -lcairo -lexpat -lbz2 -
lstdc++ -lpthread -lgcc_s -lgcc -lmoldname -lmingwex -lmingw32 -lmsvcrt -
ladvapi32 -lshell32 -luser32 -lkernel32

for s in ${subdirs}; do
(cd $s ; clang -std=c++11 -shared -nostdlib C:/Local/MSys64/mingw64/x86_64-w64-
mingw32/lib/dllcrt2.o C:/Local/MSys64/mingw64/x86_64-w64-mingw32/lib/crtbegin.o
.libs/*.o C:/Local/MSys64/mingw64/x86_64-w64-mingw32/lib/crtend.o
../.libs/libgraph_tool_core.dll.a -L/mingw64/lib -O3 -Wl,--export-all-
symbols -Wl,--as-needed -o .libs/libgraph_tool_$s.dll -Wl,--enable-auto-image-
base -Xlinker --out-implib -Xlinker .libs/libgraph_tool_$s.dll.a -lpython3.4m -
lboost_iostreams -lboost_python3 -lboost_regex -lboost_system -lCGAL -lgmp -
lcairomm-1.0 -lcairo -lexpat -lbz2 -lstdc++ -lpthread -lgcc_s -lgcc -lmoldname
-lmingwex -lmingw32 -lmsvcrt -ladvapi32 -lshell32 -luser32 -lkernel32)
done

for i in `find . -name \*.dll` ; do cp $i /mingw64/lib/python3.4/site-
packages/graph_tool/`basename $i .dll`.pyd; done

At this point, the modules are installed.
The only thing to change on the Python side is to remove
the calls to getdlopenflags() and setdlopenflags() which are not available to
Windows.

Hi,

I don't know if it is right to pollute this list with my problems in trying to
compile graph-tool for Win64. I'll try to make it brief.
Using MSYS2/MinGW64, I managed to compile the whole src/graph modules
and install them as *.pyd modules in the site-packages/graph_tool directory,
along with the Python files.
Unfortunately, I get a segfault when importing graph_tool.libgraph_tool_core .
The segfault happens deep in libboost_python3.

Before posting a long backtrace, maybe someone may offer some clue on what is
happening. I have posted some details about what I did below.

The backtrace would actually be helpful.

The background is: python 3.4, boost 1.57, clang 3.5.1 (g++ 4.9.2 as provided
with mingw64 couldn't compile a few objects without segfaulting).
Is it possible that there is a conflict among those libs/tools (of thread model
for example)?

Did you guarantee that Boost.Python was compiled with exactly the same
Python version that you used to compile graph-tool? It could also help
if you compile everything with the same compiler.

Currently I have configured like this:
./configure --enable-debug --disable-optimize CFLAGS="-I/mingw64/include"
LDFLAGS="-L/mingw64/lib" CC=clang CXX='clang -std=c++11 -m64 -DMS_WIN64=1'
CXXCPP='clang -E -std=c++11 -m64 -DMS_WIN64=1' PYTHON=python3
Then I run 'make -k -j 8' to compile all the objects. Linking the various dlls
fails, because under Windows, you can't link graph_tool_centrality.dll without
referencing grap_tool_core.dll.a. However the build process is not prepared to
do that and in the right order.
So I manually replicated what libtool does, ie :

clang -std=c++11 -shared -nostdlib C:/Local/MSys64/mingw64/x86_64-w64-
mingw32/lib/dllcrt2.o C:/Local/MSys64/mingw64/x86_64-w64-mingw32/lib/crtbegin.o
.libs/*.o C:/Local/MSys64/mingw64/x86_64-w64-mingw32/lib/crtend.o -
L/mingw64/lib -O3 -
Wl,--export-all-symbols -Wl,--as-needed -o .libs/libgraph_tool_core.dll -Wl,--
enable-auto-image-base -Xlinker --out-implib -Xlinker
.libs/libgraph_tool_core.dll.a -lpython3.4m -lboost_iostreams -lboost_python3 -
lboost_regex -lboost_system -lCGAL -lgmp -lcairomm-1.0 -lcairo -lexpat -lbz2 -
lstdc++ -lpthread -lgcc_s -lgcc -lmoldname -lmingwex -lmingw32 -lmsvcrt -
ladvapi32 -lshell32 -luser32 -lkernel32

for s in ${subdirs}; do
(cd $s ; clang -std=c++11 -shared -nostdlib C:/Local/MSys64/mingw64/x86_64-w64-
mingw32/lib/dllcrt2.o C:/Local/MSys64/mingw64/x86_64-w64-mingw32/lib/crtbegin.o
.libs/*.o C:/Local/MSys64/mingw64/x86_64-w64-mingw32/lib/crtend.o
../.libs/libgraph_tool_core.dll.a -L/mingw64/lib -O3 -Wl,--export-all-
symbols -Wl,--as-needed -o .libs/libgraph_tool_$s.dll -Wl,--enable-auto-image-
base -Xlinker --out-implib -Xlinker .libs/libgraph_tool_$s.dll.a -lpython3.4m -
lboost_iostreams -lboost_python3 -lboost_regex -lboost_system -lCGAL -lgmp -
lcairomm-1.0 -lcairo -lexpat -lbz2 -lstdc++ -lpthread -lgcc_s -lgcc -lmoldname
-lmingwex -lmingw32 -lmsvcrt -ladvapi32 -lshell32 -luser32 -lkernel32)
done

for i in `find . -name \*.dll` ; do cp $i /mingw64/lib/python3.4/site-
packages/graph_tool/`basename $i .dll`.pyd; done

At this point, the modules are installed.

I can't vouch for this way of compiling things, since I'm very
unfamiliar with the windows build environment. But doing things by hand
like this seems very invasive. A better approach would be to investigate
why autotools is not building things right, and fix that. A quick google
search reveals things like: http://gnuwin32.sourceforge.net/packages/autoconf.htm

The only thing to change on the Python side is to remove the calls to
getdlopenflags() and setdlopenflags() which are not available to
Windows.

This may be fatal. In GNU/Linux, for instance, this is necessary to
guarantee lazy evaluation of symbols when importing. I have no idea if
this is necessary in windows, and if so, what are the alternative
approaches to dlopen.

Tiago de Paula Peixoto <tiago <at> skewed.de> writes:

I can't vouch for this way of compiling things, since I'm very
unfamiliar with the windows build environment. But doing things by hand
like this seems very invasive. A better approach would be to investigate
why autotools is not building things right, and fix that. A quick google
search reveals things like:

http://gnuwin32.sourceforge.net/packages/autoconf.htm

Actually, I know quite well this gnuwin32. I'm building with MSYS2/MinGW64 for
x86_64.

DLLs do not work quite the same under Windows and under Linux.
Libtool is of not much help here.

The way you build your set of modules can't work like this under Windows.
You need to link every other dll with libgraph_tool_core.dll.a
There is no automatic import of symbols. (Actually, it is possible to write an
init function for the dll, but nothing like this is provided automatically)

The segfault I got was because g++ (actually cc1plus.exe) ran out of resources.
It used about 4Gb and crashed. However, I managed to overcome the problem and I
have the whole set of dlls (renamed *.pyd) modules that I can import in Python.
Given the g++ crash, I tried to compile them with clang, but the layout of
bjects must be different. The dlls compiled fine, but I couldn't import them
into python.

Anyway ...

> The only thing to change on the Python side is to remove the calls to
> getdlopenflags() and setdlopenflags() which are not available to
> Windows.

This may be fatal. In GNU/Linux, for instance, this is necessary to
guarantee lazy evaluation of symbols when importing. I have no idea if
this is necessary in windows, and if so, what are the alternative
approaches to dlopen.

... I may have a problem here. I don't know yet. I have exported all symbols by
default in all *.pyd modules. Currently, I can't yet import all from graph_tool
because of a bug in pycairo. Slowly progressing.

Fabrice

Fabrice Popineau <fabrice.popineau <at> gmail.com> writes:

... I may have a problem here. I don't know yet.
I have exported all symbols by
default in all *.pyd modules. Currently, I can't yet
import all from graph_tool
because of a bug in pycairo. Slowly progressing.

So after fixing pycairo, I tried the small example
"Building a price network" from
the quickstart doc.
Everything runs well till the point the price.xml.gz file is saved.
It segfaults afterwards in the stats module at this line

# # Let's plot its in-degree distribution
in_hist = vertex_hist(g, "in")

I must add that I have used the git version of graph_tool.
I hope it is stable enough.

Fabrice

Everything runs well till the point the price.xml.gz file is saved.
It segfaults afterwards in the stats module at this line

It is hard to say what it could be without a backtrace.

I must add that I have used the git version of graph_tool.
I hope it is stable enough.

I get no segfaults with the git version.

Best,
Tiago