Problems with minimise_blockmodel

P-M · March 22, 2017, 12:58pm

Hello,

I am experiencing some problems with minimize_blockmodel_dl. In particular
if I run:

for deg_corr in [True, False]:
        for overlap in [True, False]:
            # Initialize the Markov chain from the "ground state"
            state = gt.minimize_blockmodel_dl(g,
deg_corr=deg_corr,overlap=overlap)

I get the error message:
Traceback (most recent call last):
  File "model_class_selection.py", line 23, in <module>
    state = gt.minimize_blockmodel_dl(g, deg_corr=deg_corr,overlap=overlap)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/minimize.py",
line 298, in minimize_blockmodel_dl
    mcmc_multilevel_args=mcmc_multilevel_args)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/minimize.py",
line 100, in get_states
    max_state = mcmc_multilevel(max_state, B=B_max, **mcmc_multilevel_args)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/mcmc.py",
line 364, in mcmc_multilevel
    state = state.shrink(B=B_next, **shrink_args)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/overlap_blockmodel.py",
line 652, in shrink
    (bstate.get_nonempty_B(), B, self.B))
ValueError: cannot shrink state to a smaller number of groups: 15643 ->
136997 (total: 178096)

If I run:

for deg_corr in [True, False]:
        for overlap in [True, False]:
            for layers in [True, False]:
                # Initialize the Markov chain from the "ground state"
                state = gt.minimize_blockmodel_dl(g,
deg_corr=deg_corr,overlap=overlap,layers=layers)

I get:

Traceback (most recent call last):
  File "model_class_selection.py", line 20, in <module>
    state = gt.minimize_blockmodel_dl(g,
deg_corr=deg_corr,overlap=overlap,layers=layers)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/minimize.py",
line 298, in minimize_blockmodel_dl
    mcmc_multilevel_args=mcmc_multilevel_args)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/minimize.py",
line 94, in get_states
    **dmask(state_args, ["B", "b", "deg_corr", "clabel"]))
TypeError: __init__() takes at least 3 arguments (6 given)

Does anybody know what this is being caused by?

Best wishes,

Philipp

tiago · March 22, 2017, 2:03pm

Hello,

I am experiencing some problems with minimize_blockmodel_dl. In particular
if I run:

for deg_corr in [True, False]:
        for overlap in [True, False]:
            # Initialize the Markov chain from the "ground state"
            state = gt.minimize_blockmodel_dl(g,
deg_corr=deg_corr,overlap=overlap)

I get the error message:
Traceback (most recent call last):
  File "model_class_selection.py", line 23, in <module>
    state = gt.minimize_blockmodel_dl(g, deg_corr=deg_corr,overlap=overlap)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/minimize.py",
line 298, in minimize_blockmodel_dl
    mcmc_multilevel_args=mcmc_multilevel_args)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/minimize.py",
line 100, in get_states
    max_state = mcmc_multilevel(max_state, B=B_max, **mcmc_multilevel_args)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/mcmc.py",
line 364, in mcmc_multilevel
    state = state.shrink(B=B_next, **shrink_args)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/overlap_blockmodel.py",
line 652, in shrink
    (bstate.get_nonempty_B(), B, self.B))
ValueError: cannot shrink state to a smaller number of groups: 15643 ->
136997 (total: 178096)

I cannot reproduce this.

As you should have learned by now, we can only make any progress
understanding problems like this if you provide at least some _minimial_
information, such as the version of graph-tool that is being used, as
well as a complete example that shows the problem (in this case, we need
the actual network used.)

If I run:

for deg_corr in [True, False]:
        for overlap in [True, False]:
            for layers in [True, False]:
                # Initialize the Markov chain from the "ground state"
                state = gt.minimize_blockmodel_dl(g,
deg_corr=deg_corr,overlap=overlap,layers=layers)

I get:

Traceback (most recent call last):
  File "model_class_selection.py", line 20, in <module>
    state = gt.minimize_blockmodel_dl(g,
deg_corr=deg_corr,overlap=overlap,layers=layers)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/minimize.py",
line 298, in minimize_blockmodel_dl
    mcmc_multilevel_args=mcmc_multilevel_args)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/minimize.py",
line 94, in get_states
    **dmask(state_args, ["B", "b", "deg_corr", "clabel"]))
TypeError: __init__() takes at least 3 arguments (6 given)

Does anybody know what this is being caused by?

Did you read the documentation?

If you want to use the layered model, you have to supply the edge
covariates, i.e.

    state = gt.minimize_blockmodel_dl(g, deg_corr=deg_corr,
                                      overlap=overlap,
                                      layers=True,
                                      state_args=dict(ec=eprop))

Best,
Tiago

P-M · March 22, 2017, 2:42pm

Hi Tiago,

Tiago Peixoto wrote

I cannot reproduce this.

As you should have learned by now, we can only make any progress
understanding problems like this if you provide at least some _minimial_
information, such as the version of graph-tool that is being used, as
well as a complete example that shows the problem (in this case, we need
the actual network used.)

Of course, the network is attached ( graph_1950_clean.zip
<http://main-discussion-list-for-the-graph-tool-project.982480.n3.nabble.com/file/n4027139/graph_1950_clean.zip>
) and here is the script that I ran ( model_class_selection_test.py
<http://main-discussion-list-for-the-graph-tool-project.982480.n3.nabble.com/file/n4027139/model_class_selection_test.py>
)

Tiago Peixoto wrote

Did you read the documentation?

If you want to use the layered model, you have to supply the edge
covariates, i.e.

    state = gt.minimize_blockmodel_dl(g, deg_corr=deg_corr,
                                      overlap=overlap,
                                      layers=True,
                                      state_args=dict(ec=eprop))

You are right, this is in the cookbook, I could have spotted that myself. I
was looking only at the documentation. Thanks for the explanation though.

Best,

Philipp

P-M · March 22, 2017, 2:42pm

Also, I am running version 2.23dev (commit 04dabc1c, Wed Mar 8 13:35:11 2017
+0000) of graph-tool on Ubuntu 16.04 LTS.

tiago · March 22, 2017, 3:47pm

With the help of your script, I was able to reproduce the problem, which
only occurs with filtered graphs. I have already fixed it in git.

Best,
Tiago

P-M · March 23, 2017, 4:54pm

Thank you very much Tiago, it seems to work using unfiltered graphs!

Best wishes,

Philipp

P-M · March 24, 2017, 5:45pm

Hi Tiago,

A new error message:

Running:

import graph_tool.all as gt
import numpy as np
import cPickle as pickle
import timeit

g = gt.load_graph('graph_no_multi_1950.gt')

with open('model_selection_results_1950.dat','w') as output:
    deg_corr = False
    overlap = True
    nL = 10

    # Initialize the Markov chain from the "ground state"
    state = gt.minimize_nested_blockmodel_dl(g,
deg_corr=deg_corr,overlap=overlap)
    print 'minimised state'
    bs = state.get_bs() # Get hierarchical partition.
    bs += [np.zeros(1)] * (nL - len(bs)) # Augment it to L = 10 with
                                            # single-group levels.

state = state.copy(bs=bs, sampling=True)

Returns the following error message:

Traceback (most recent call last):
  File "model_class_selection_nested_ndc_o.py", line 27, in <module>
    state = state.copy(bs=bs, sampling=True)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/nested_blockmodel.py",
line 146, in copy
    **kwargs)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/nested_blockmodel.py",
line 90, in __init__
    nb[:len(b)] = b
ValueError: could not broadcast input array from shape (63754) into shape
(15643)

Here is the graph ( graph_no_multi_1950_clean.zip
<http://main-discussion-list-for-the-graph-tool-project.982480.n3.nabble.com/file/n4027153/graph_no_multi_1950_clean.zip>
) and the script ( model_class_selection_nested_ndc_o.py
<http://main-discussion-list-for-the-graph-tool-project.982480.n3.nabble.com/file/n4027153/model_class_selection_nested_ndc_o.py>
). The graph-tool version is 2.23dev (commit 8c8fa901, Mon Mar 6 16:13:20
2017 +0100).

Do you know what this might be caused by?

Best,

Philipp

tiago · March 24, 2017, 6:12pm

I can't seem to reproduce this.

It is always a good idea to upgrade to the newest git version before
reporting bugs.

Best,
Tiago

P-M · March 24, 2017, 6:16pm

OK, will investigate further and get back to you if it persists. Thanks for
looking into it though.

Best,

Philipp

P-M · March 27, 2017, 4:13pm

Running the newest git version (2.23dev (commit 0b766917, Mon Mar 27 12:31:02
2017 +0100)) I now get the following screen output:

minimised state
/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/overlap_blockmodel.py:253:
UserWarning: unrecognized keyword arguments: ['vweight', 'eweight']
  str(list(kwargs.keys())))
/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/blockmodel.py:410:
UserWarning: unrecognized keyword arguments: ['copy_bg']
  str(list(kwargs.keys())))
equilibrating
Traceback (most recent call last):
  File "model_class_selection_nested_ndc_o.py", line 44, in <module>
    callback=collect_marginals)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/mcmc.py",
line 125, in mcmc_equilibrate
    delta, nmoves = state.mcmc_sweep(**mcmc_args)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/nested_blockmodel.py",
line 588, in mcmc_sweep
    return self._h_sweep(lambda s, **a: s.mcmc_sweep(**a), c=c, **kwargs)
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/nested_blockmodel.py",
line 505, in _h_sweep
    get_entropy_args(eargs))
  File
"/home/pmj27/anaconda2/lib/python2.7/site-packages/graph_tool/inference/blockmodel.py",
line 715, in _couple_state
    self._state.couple_state(state._state, entropy_args)
AttributeError: 'graph_tool::OverlapBlockState<boost::adj_list<unsi' object
has no attribute 'couple_state'

Seeing as it ran error free for you previously I presume you can't reproduce
this error either? What could we do to pinpoint it?

tiago · March 27, 2017, 4:18pm

Do I have to ask you again to send a complete example?

There is zero I can do with error messages without any context.

Best,
Tiago

P-M · March 27, 2017, 4:26pm

Hi Tiago,

graph and script are entirely unchanged from my previous message. Here they
are again just in case ( graph_no_multi_1950_clean.zip
<http://main-discussion-list-for-the-graph-tool-project.982480.n3.nabble.com/file/n4027160/graph_no_multi_1950_clean.zip>
, model_class_selection_nested_ndc_o.py
<http://main-discussion-list-for-the-graph-tool-project.982480.n3.nabble.com/file/n4027160/model_class_selection_nested_ndc_o.py>
)

Best,

Philipp

tiago · March 28, 2017, 11:41am

This time, I can in fact reproduce the problem.

I have already fixed it in git.

Best,
Tiago

P-M · March 28, 2017, 4:09pm

Thank you! I shall update my version of graph-tool then and rerun.

Best,

Philipp