Problem in compiling gap_fit MPI with meson

[lormio]

I am encountering the same (or at least similar) issue with the gap_fit executable compiled with meson, with MPI turned on:

SYSTEM ABORT: proc=0 Traceback (most recent call last)
File "../src/libAtoms/linearalgebra.F90", line 2348 kind unspecified
LA_Matrix_Factorise: cannot factorise, error: 8

However, when building gap_fit using QUIP_ARCH=linux_x86_64_gfortran_openmpi and make config (using -lopenblas -lscalapack and no extra link options), the produced gap_fit does not have any problem running the fit to the end, using the same training file and config_file.

I am compiling quip inside a conda environment with openblas, scalapack and compilers downloaded from conda itself.

Originally posted by @lormio in #715

I add to the issue quoted the packages installed in the conda environment and some information

conda packages: scalapack, openblas, gxx, gcc, gfortran, openmpi, meson, ninja.

The issue seems to be the following: the old build system finds and links Scalapack correctly, while Meson doesn't.
OLD BUILD
$ ldd build/linux_x86_64_gfortran_openmpi/gap_fit | grep scalapack
libscalapack.so => /home/miolalor/.conda/envs/quip_comp/lib/libscalapack.so (0x0000742212600000)
MESON
$ ldd builddir/src/Programs/gap_fit | grep scalapack
No output

Moreover, just watching these symbols produced by ldd, I noticed that the symbols for MPI are different, with the old build having an extra 2 of them(I don't know if it's related or not, but I don't know why it should differ):
OLD BUILD
$ ldd build/linux_x86_64_gfortran_openmpi/gap_fit | grep mpi
libmpi_usempif08.so.40 => /home/miolalor/.conda/envs/quip_comp/lib/libmpi_usempif08.so.40 (0x00007747a44ad000)
libmpi_usempi_ignore_tkr.so.40 => /home/miolalor/.conda/envs/quip_comp/lib/libmpi_usempi_ignore_tkr.so.40 (0x00007747a4499000)
libmpi_mpifh.so.40 => /home/miolalor/.conda/envs/quip_comp/lib/libmpi_mpifh.so.40 (0x00007747a4422000)
libmpi.so.40 => /home/miolalor/.conda/envs/quip_comp/lib/libmpi.so.40 (0x0000774799400000)
MESON
ldd builddir/src/Programs/gap_fit | grep mpi
libmpi_mpifh.so.40 => /home/miolalor/.conda/envs/quip_comp/lib/libmpi_mpifh.so.40 (0x00007f57742b9000)
libmpi.so.40 => /home/miolalor/.conda/envs/quip_comp/lib/libmpi.so.40 (0x00007f5769e00000)

However, I do not understand why meson is failing in linking the scalapack library when using the old build system I just specify the -lscalapack, without having to specify the -I or -L paths.

The executable produced with meson will run when used without mpiexec until the error as reported in the previous issue, and complain about undefined symbols when run in parallel (blacs_gridinit_). The old build system gap_fit works without issue instead, with or without multiple MPI processes.

[jameskermode]

Thanks for clarifying the cause of this and #715. I'll take a look and see if I can fix the Meson build.

[jameskermode]

Please try the fix in #717. I've checked it works on macOS but not Linux.

[lormio]

The fix does indeed make Scalapack linked to gap_fit.

$ ldd /home/miolalor/Desktop/QUIP/build_mpi/src/Programs/gap_fit|grep scalapack
libscalapack.so => /home/miolalor/.conda/envs/quip_comp/lib/libscalapack.so (0x000077a596a00000)

However, the problem of fitting persists. The meson compiled program fails with:

SYSTEM ABORT: proc=0 Traceback (most recent call last)
File "../src/libAtoms/linearalgebra.F90", line 2348 kind unspecified
LA_Matrix_Factorise: cannot factorise, error: 10

MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
Proc: [[59966,0],0]
Errorcode: 1

when run without mpiexec.

With multiple processes instead, it fails with:

$ mpirun -np 2 /home/miolalor/Desktop/QUIP/build_mpi/src/Programs/gap_fit config_file=test

...

Finished sparse covariance matrix calculation of coordinate 3
TIMER: gpFull_covarianceMatrix_sparse_Coordinate3_sparse done in .23300000000001653E-002 cpu secs, .23296379949897528E-002 wall clock secs, .23292189999999380E-002 mpi wall secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate3 done in .23490000000001565E-002 cpu secs, .23472380125895143E-002 wall clock secs, .23468200000005268E-002 mpi wall secs.
Started sparse covariance matrix calculation of coordinate 4
SYSTEM ABORT: proc=1 Traceback (most recent call last)
File "../src/libAtoms/linearalgebra.F90", line 2520 kind unspecified
LA_Matrix_Solve_Vector: length of b or x is not n

The old build system gap_fit works just fine and produces a gap potential, even using a dummy dataset and hyperparameters (with or without mpiexec). I attached the zip file of the test directory with the files I am using if you want to try and reproduce the behaviour.

testsQUIP.zip

[jameskermode]

Think I've figured it out. In the old Makefile build, HAVE_QR=1 is the default (you have to explicitly say 'n' to disable it). But the meson build never defines -DHAVE_QR.

Without HAVE_QR, gap_fit falls into the Cholesky path which:

1. Doesn't support MPI/ScaLAPACK parallelism at all
2. Uses LA_Matrix_Factorise (dpotrf Cholesky) instead of QR decomposition
3. Is numerically less stable (Cholesky requires positive definiteness; QR doesn't)

This explains both errors:

• Without mpiexec: Cholesky fails because the matrix isn't positive definite(error 10)
• With mpiexec: The non-QR path doesn't support distributed matrices, causing dimension mismatches

The fix is to add -DHAVE_QR to the meson build. It should always be on when GAP is enabled (it was default 'y' in the old build). I'll push a fix to do this to the same PR shortly.

[jameskermode]

I checked serial compile, MPI compile run in serial and MPI compile run in parallel with your dataset, all using meson and all work correctly now. Thanks for reporting and helping to debug!

[jameskermode]

I've added MPI tests that would have caught this problem

[lormio]

Thank you for the help, it works!