mcmc_equilibrate - Segmentation fault (core dumped)

graph-tool
1

I am trying to run the following snippet of code:

    import graph_tool.all as gt
    import timeit, os
    import cPickle as pickle
    import numpy as np

    g = gt.load_graph('graph_no_multi_reac_type.gt')
    gt.remove_parallel_edges(g)

    print 'minimising state'
    start = timeit.default_timer()
    state = gt.minimize_nested_blockmodel_dl(g, deg_corr=True,verbose=False)
    duration = timeit.default_timer()-start
    print 'duration: ', duration

    state = state.copy(sampling=True)
    with open('state_n_dc.pkl','wb') as state_pkl:
        pickle.dump(state,state_pkl,-1)
    print 'equilibrating Markov chain'
    start = timeit.default_timer()
    mcmc_args = {}
    mcmc_args=dict(parallel = True,niter=10)
    history = gt.mcmc_equilibrate(state, wait=1000,
history=True,mcmc_args=mcmc_args)
    duration = timeit.default_timer()-start
    print 'duration: ', duration
    with open('history1.pkl','wb') as his1_pkl:
        pickle.dump(history,his1_pkl,-1)

On the attached graph ( graph_no_multi_reac_type.gt
<http://main-discussion-list-for-the-graph-tool-project.982480.n3.nabble.com/file/n4027034/graph_no_multi_reac_type.gt&gt;
). I, however, get an error message stating "Segmentation fault (core
dumped)". To be precise my screen output is:

    minimising state
    duration: 991.349531889
    equilibrating Markov chain
    Segmentation fault (core dumped)

So the error seems to arise during `history = gt.mcmc_equilibrate(state,
wait=1000, history=True,mcmc_args=mcmc_args)`. Why might that be?

I have observed the issue both with `2.20dev (commit 01432d60, Sun Jan 15
19:54:52 2017 +0000)` and `2.20 (commit f6ef9990, Fri Jan 27 16:40:08 2017
+0000)`.

Best,

Philipp

2

Please supply the actual graph you used, so this can be investigated.

The git version you tried is more than a month old; do you see this problem
with a current git version?

Since you tried with 2.20dev, I assume you compiled it yourself. What OS are
you using, what GCC version do you have, and are you using sparsehash or not?

Best,
Tiago

3

Hi Tiago,

Tiago Peixoto wrote

Please supply the actual graph you used, so this can be investigated.

The graph should be online (at least it shows up via the Nabble interface).
Or what do you mean by "actual"?

Tiago Peixoto wrote

The git version you tried is more than a month old; do you see this
problem
with a current git version?

Good question, I am not sure, shall check that as soon as I can.

Tiago Peixoto wrote

Since you tried with 2.20dev, I assume you compiled it yourself. What OS
are
you using, what GCC version do you have, and are you using sparsehash or
not?

I compiled it myself indeed. I am using Ubuntu 16.04 LTS and "gcc version
5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)". `libsparsehash-dev` is
installed on the machine. I have not provided any specific compiler flags
for it but I would expect that it will have picked it up automatically.

Best,

Philipp

4

The graph should be online (at least it shows up via the Nabble interface).
Or what do you mean by "actual"?

I'm sorry, I missed this in the original email.

The git version you tried is more than a month old; do you see this
problem
with a current git version?

Good question, I am not sure, shall check that as soon as I can.

I was able to reproduce the bug with the newest git. It is caused by using
mcmc_args=dict(parallel=True). I will fix this soon, but in the meantime you
can just use parallel=False. Note that, as the documentation warns,
parallel=True triggers an approximated MCMC which will _not_ lead to the
correct posterior distribution.

Best,
Tiago

5

Tiago Peixoto wrote

Note that, as the documentation warns,
parallel=True triggers an approximated MCMC which will _not_ lead to the
correct posterior distribution.

I know, the documentation states that assymptotic exactness cannot be
guaranteed if using parallel=True. What I wasn't quite clear on is how
severe the consequences of this are. Is this a case of potentially
introducing a small error or is this a case of "we have no idea what the
answer will say"? If the latter is the case what would a sensible use case
be for switching parallel to True? (I am trying to use it to calculate
missing edges which even with parallel takes very long for the network so I
fear without it it would just not be feasible.)

Best,

Philipp

6

Hi Tiago,

Do you have any thoughts on this?

Best,

Philipp

7

The posterior will be wrong, but it is not possible to say how wrong in
general. This is useful only when, e.g. using it is a heuristic to find the
minimum description length, since the shape of the posterior does not
matter, only its maximum.

8

Thank you for that clarification. I presume that I should not be using it to
calculate the likelihood of missing edges then?

From what I gather implementing parallel MCDM is not straightforward (though

my knowledge of this is hazy). Is it on your list of future developments at
all? Or will this section of graph-tool stay large non-parallel for now?

Best,

Philipp

9

Thank you for that clarification. I presume that I should not be using it to
calculate the likelihood of missing edges then?

Not if you want the probabilities to be correct; though they might still be
approximately correct.

From what I gather implementing parallel MCDM is not straightforward (though
my knowledge of this is hazy). Is it on your list of future developments at
all? Or will this section of graph-tool stay large non-parallel for now?

I don't think it is trivial to write a fully parallel code, since the model
likelihood depends on the global state. So, threads that read it and modify
it need to synchronize.