It seems like your right, it does need a lot of memory, always around 3gb.... and then it starts swapping.
What do you mean exactly ba parallel build. I didn't start make -j, just regular make

Is there anaything I can do to fix my issue?

Here is one example of the make -d, what takes up a lot of time:

Invoking recipe from Makefile:505 to update target `graph_closeness.lo'.
Putting child 0xf43910 (graph_closeness.lo) PID 11481 on the chain.
Live child 0xf43910 (graph_closeness.lo) PID 11481
  CXX      graph_closeness.lo

Reaping winning child 0xf43910 PID 11481
Live child 0xf43910 (graph_closeness.lo) PID 11532
Reaping winning child 0xf43910 PID 11532
Removing child 0xf43910 PID 11532 from chain.