Compiling graph-tool for pypy?

Hi!

Thanks for graph-tool, really a great piece of software!

I've been experimenting with graph-tool and pypy3 (by making $PYTHON point
to the pypy interpreter), and I couldn't manage to successfully run
configure, even while setting --with-python-module-path. I don't know if
there's a correct value to put for LIBS here--maybe the directory
structures for lib directories of cpython and pypy are different?

Do you think there's a way to make these two pieces of software cooperate?

thanks a lot!

Here, the error that I get:

configure: error:
  Could not link test program to Python. Maybe the main Python library has
been
  installed in some non-standard library path. If so, pass it to configure,
  via the LIBS environment variable.
  Example: ./configure LIBS="-L/usr/non-standard-path/python/lib"

attachment.html (1.53 KB)

Unfortunately, this is not possible. Graph-tool uses the CPython C API
(via boost-python) which is not implemented by pypy.

Note that if you are hoping to gain some speed improvement by using
pypy, this would not be achieved for graph-tool, since it is implemented
in C++.

Best,
Tiago

Well, since some years things have changed :slight_smile: Pypy does have support,
sometimes with performance problems, for most of the Cpython APIs, and
libraries using those APIs like numpy/scipy/pandas for example work out of
the box.

My performance bottleneck is not in the graph-tool code, it's actually in
an algorithmic part unrelated with graph-tool, so I understand your comment
but it's not about that :slight_smile:

thanks a lot for your help!

attachment.html (1.82 KB)

Unfortunately, this is not possible. Graph-tool uses the CPython C API
(via boost-python) which is not implemented by pypy.

Pypy does have support, sometimes with performance problems, for most of the Cpython APIs

Indeed, the situation may not be hopeless. IIUC, With PyPy’s C API compatibility layer (cpyext), many compiled extensions can be built for PyPy with minimal changes (or perhaps none at all?). The conda-forge project has been methodically adding PyPy support to many of its packages, including numpy, scipy, matplotlib, etc. Just recently, their package for boost itself was updated for PyPy support[1]. On my machine, I can build a “hello world” extension with boost-python and run it with PyPy, without any special magic. (See attached.)

In the coming months, *maybe* all of graph-tool’s dependencies will be supported in conda-forge. But that won’t happen on its own. If this is important to you, please consider assisting with the following PRs, which currently fail:

https://github.com/conda-forge/pycairo-feedstock/pull/21
https://github.com/conda-forge/pygobject-feedstock/pull/27

I think those are the only remaining obstacles to at least attempting to build a graph-tool PyPy package. (I’m not sure about the glib feedstock.)

I’m not a PyPy user (yet?) so I can’t make any promises about my own commitment to this effort. And I certainly can’t comment on what will happen if it turns out patches are required in the graph-tool source code itself. Nonetheless, I think there’s *hope* that a sufficiently motivated PyPy user could make progress on this. If you decide to try it, feel free to ping me on relevant github threads (@stuarteberg).

Best,
Stuart

[1]: https://github.com/conda-forge/boost-feedstock/pull/110#issuecomment-763940556

attachment.html (4.04 KB)

hello.cpp (569 Bytes)

>>Unfortunately, this is not possible. Graph-tool uses the CPython C API
>>(via boost-python) which is not implemented by pypy.

>Pypy does have support, sometimes with performance problems, for most
of the Cpython APIs

Indeed, the situation may not be hopeless. IIUC, With PyPy’s C API
compatibility layer (cpyext), many compiled extensions can be built for
PyPy with minimal changes (or perhaps none at all?). The conda-forge
project has been methodically adding PyPy support to many of its
packages, including numpy, scipy, matplotlib, etc. Just recently, their
package for boost itself was updated for PyPy support[1]. On my machine,
I can build a “hello world” extension with boost-python and run it with
PyPy, without any special magic. (See attached.)

This is indeed very promising! I wasn't aware pypy was so ahead with the
C API.

In the coming months, **maybe** all of graph-tool’s dependencies will be
supported in conda-forge. But that won’t happen on its own. If this is
important to you, please consider assisting with the following PRs,
which currently fail:

https://github.com/conda-forge/pycairo-feedstock/pull/21
<https://github.com/conda-forge/pycairo-feedstock/pull/21&gt;

https://github.com/conda-forge/pygobject-feedstock/pull/27
<https://github.com/conda-forge/pygobject-feedstock/pull/27&gt;

I think those are the only remaining obstacles to at least attempting to
build a graph-tool PyPy package. (I’m not sure about the glib feedstock.)

I’m not a PyPy user (yet?) so I can’t make any promises about my own
commitment to this effort. And I certainly can’t comment on what will
happen if it turns out patches are required in the graph-tool source
code itself. Nonetheless, I think there’s **hope** that a sufficiently
motivated PyPy user could make progress on this. If you decide to try
it, feel free to ping me on relevant github threads (@stuarteberg).

Ideally we would need boost-python + pypy support in Arch, and other
distros, since it also needs a shared library component. Otherwise, at
the moment it would be a significant effort for me to try this myself...

Best,
Tiago