Have you checked whether MATLAB has memory consumption limits? You can turn off java with the -nojvm flag when starting up matlab from a terminal prompt to reduce memory consumption by MATLAB.
More info here: https://www.mathworks.com/help/matlab/matlab_prog/resolving-out-of-memory-errors.html
More info here: https://www.mathworks.com/help/matlab/matlab_prog/resolving-out-of-memory-errors.html