Bonjour,
I have a package that we compile once on the head node, and then we use Slurm to run it on the different nodes. Before the recent shutdown, it was working very well. But after that, it stopped working. The error was: “illegal instruction”. I googled it and it turned out that the problem usually occurs when a program or library is compiled on a newer CPU but executed on an older one. The usual solution is to use #SBATCH --constraint=AVX2 in the bash file. But when I checked the features that were activated on the cluster (by sinfo --help) it seems these constraints are not activated. I tried to go to one of the nodes and compile the code there (to use the same instructions), but it seems we don’t have access to the nodes. Finally, I used a trick and submit the compilation as a job to compile the package on the nodes. I managed to solve the problem, but I think there should be a better solution. I also wanted to post this topic for others that may get the same error.
Bonne journée,
Amir