Hi all, I am running multiple minimizations using joblib (spawning threads, not processes). I just want to let you know that if you ever tried something similar and everything crashed, a working solution would be to disable memmapping in parallel_config. More in detail:
having this function for the minimization:
>>> def fast_min(state, beta, n_sweep, fast_tol, max_iter=max_iter, seed=None):
>>> if seed:
>>> gt.seed_rng(seed)
>>> dS = 1e9
>>> n = 0
>>> while (np.abs(dS) > fast_tol) and (n < max_iter):
>>> dS, _, _ = state.multiflip_mcmc_sweep(beta=beta, niter=n_sweep, c=0.5)
>>> n += 1
>>> return state
having this list of blockstates (simplified for readibility):
>>> n_init = 100
>>> states = [gt.NestedBlockState(g=g) for n in range(n_init)]
This used to crash in my current configuration
>>> states = Parallel(n_jobs=-1, prefer='threads')(
delayed(fast_min)(states[x], beta, n_sweep, tolerance, seeds[x]) for x in range(n_init)
whereas this works fine (and pretty fast, actually)
>>> with parallel_config(backend='threading',
max_nbytes=None,
n_jobs=-1):
>>> states = Parallel()(
>>> delayed(fast_min)(states[x], beta, n_sweep, tolerance, seeds[x]) for x in range(n_init)
>>> )