Large-Scale Population Synthesis on HPC Facilities 🚀

This tutorial will cover how to setup and run a large, multi-metallicity population run on a HPC with slurm. We will dive deeper into the population_params.ini file and how the Population class works for multi-metallicity populations.

Initialization File for multi-metallicity runs

Let’s copy the default population synthesis ini file to your working directory. Make sure you’re on a cluster to be able to run and submit the jobs!

import os
import shutil
from posydon.config import PATH_TO_POSYDON

path_to_params = os.path.join(PATH_TO_POSYDON, "posydon/popsyn/population_params_default.ini")
shutil.copyfile(path_to_params, './population_params.ini')

Open the population_params.ini file and you should see all 8 currently supported metallicities present.

metallicity = [2., 1., 0.45, 0.2, 0.1, 0.01, 0.001, 0.0001]

For our test run, we do not require a large population, so set the number of binaries to 1000. You might also want to make sure the dump_rate is set to 100 binaries for this example.

dump_rate = 100
...
number_of_binaries = 1000

Setup the Population Synthesis Run

POSYDON provides posydon-setup-popsyn as a command line script that you can use to setup a (multi-metallicity) population run on a HPC facility. This will split each metallicity into a separate slurm job-array and a dependent job which will automatically merge the output of the separate jobs. This removes the need to manually setup each slurm job. Below are two examples, but you might require to adjust the inputs for your email, cluster and available partitions.

Older POSYDON installations

If the posydon-setup-popsyn command is not available, you might have to install POSYDON.

1. Go to the directory: cd $PATH_TO_POSYDON

2. Uninstall using pip pip uninstall posydon

3. Reinstall: if you’re using the development version: pip install .

See the Installation instructions if any issues occur.

# example for yggdrasil
posydon-setup-popsyn population_params.ini --job_array=10 \
                                            --walltime=00:14:00 \
                                            --mem_per_cpu=5G  \
                                            --partition=debug-cpu \
                                            --email=max.briel@unige.ch \
                                            --account=fragkos

# example for quest cluster
posydon-setup-popsyn population_params.ini --job-array=10 \
                                           --walltime=00:14:00 \
                                           --partition=posydon-std \
                                           --email=max.briel@unige.ch \
                                           --mem_per_cpu=5G

setup-popsyn --help should provide a complete list of possible input parameters.

Walltime and job_array number fine-tuning

The above examples will setup the population run with an array of 10 jobs for each metallicity. As such, each job will run 100 binaries of the 1000 binaries per metallicity.

Fine-tuning the dump_rate, mem_per_cpu, number of job-array’s, and walltime compared to the total number of binaries can be helpful in optimizing your cluster usage. See the FAQ on more detailed guidance on the memory footprint of a population synthesis run in POSYDON. By default, 5GB of RAM is requested per job, which fits well with the default dump_rate of 2000.

As a rule of thumb for the walltime, a single binary takes about 1-2 seconds to run. In the example run, each job of 100 binaries should take around 2 to 3 minutes. However, some additional setup time for loading the grids is required! See the FAQ for another example of walltime estimation.

After you’ve ran the above setup, you should now have several files in your work directory.

You can submit all job arrays and merged jobs using slurm_submit.sh

sh slurm_submit.sh

If one of your runs fails, you can manually submit them again after fixing the issue. The *_logs folder will contain the logs of each job in the job_array.

Populations inspection

Once the runs and the mergers have finished. You should have 8 files named MET_Zsun_population.h5, where MET are the metallicities.

We will inspect two to check the number of binaries in the population.

from posydon.popsyn.synthetic_population import Population

file1 = '1e-01_Zsun_population.h5'
file2 = '1e-02_Zsun_population.h5'

pop1 = Population(file1)
print(pop1.mass_per_metallicity)

pop2 = Population(file2)
print(pop2.mass_per_metallicity)

In the next tutorial, we will look into selecting specific events and combining the different metallicity runs into a single population and selecting the binary black hole mergers in them! BBH analysis