For the first version I needed 400-500GB RAM, so that wouldn't be overkill. The threads however seem pretty much useless, indeed.

Hoiweveriwever, as the latter version seems so much more memory efficient, I am wondering why it's not implemented as the default in form of a function call with the same prominence in the documentation. Would love to hear Tiago or anyone else chime in on this. What's the advantage of the boiler-plate minimize_nested_blockmodel? Is there any except less lines of code?

Cheers,

Felix



On Wednesday, Feb 26, 2020 at 9:28 PM, Deklan Webster <deklanw@gmail.com> wrote:
I had success with the latter method. Although, it seems the hardware you have is a bit overkill (I did 3m~ edge graph with 16GB RAM laptop with enough RAM to spare). I'm not sure if it's possible to utilize multithreading for this, but if not those threads you have won't help much. Still hoping to get an answer about multithreading from someone more knowledgeable

On Wed, Feb 26, 2020 at 10:35 AM James Ruffle <j.ruffle@qmul.ac.uk> wrote:
Dear community,

I have been following this thread as have a similar predicament - trying to run the nested block model on a graph of 1mil nodes and 12 mil edges.
Since I have read of the computational cost from others, I have got access to a server with 500Gb RAM with ~120 CPU threads.
Based upon this graph complexity and your prior experience, which approach you would aim for:

1) minimize_nested_blockmodel_dl(g)

or

2) prime an empty object and MCMC equilibrate with multi-flip:
bs = [np.zeros(1)]*6
state=NestedBlockState(g=g,bs=bs,sampling=True)
gt.mcmc_equilibrate(state,multiflip=True)

Am very grateful for the guidance…!
(I wonder how long this will take to run…??)

James

> On 24 Feb 2020, at 18:40, Tiago de Paula Peixoto <tiago@skewed.de> wrote:
>
> Am 24.02.20 um 19:32 schrieb Davide Cittaro:
>> In the thread linked (https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmain-discussion-list-for-the-graph-tool-project.982480.n3.nabble.com%2FSBM-on-Dense-Graphs-td4027902.html%23a4027907&amp;data=02%7C01%7C%7C3b3a0d8d77904e2eb56e08d7b9591fc8%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637181664763877491&amp;sdata=0dTsu3ywNL1WL7FZrvc%2BHg1lEZXEjAaM2hkUMTq640k%3D&amp;reserved=0) is written
>>
>> (Remember to replace mcmc_sweep() with multiflip_mcmc_sweep() in the code
>> above. I had forgotten to make this change the last release...)
>>
>> Does this mean that one should also set "multiflip=True" in mcmc_equilbrate?
>
> Yes.
>
> --
> Tiago de Paula Peixoto <tiago@skewed.de>
> _______________________________________________
> graph-tool mailing list
> graph-tool@skewed.de
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.skewed.de%2Fmailman%2Flistinfo%2Fgraph-tool&amp;data=02%7C01%7C%7C3b3a0d8d77904e2eb56e08d7b9591fc8%7C569df091b01340e386eebd9cb9e25814%7C0%7C0%7C637181664763877491&amp;sdata=izkn35h55P2quQwzqI4AaSS7Zk%2BO%2FY6Hhc9Fdmo38D8%3D&amp;reserved=0

_______________________________________________
graph-tool mailing list
graph-tool@skewed.de
https://lists.skewed.de/mailman/listinfo/graph-tool
_______________________________________________
graph-tool mailing list
graph-tool@skewed.de
https://lists.skewed.de/mailman/listinfo/graph-tool