Large-Scale Population Synthesis on HPC Facilities πŸš€οƒ

Tutorial goal

This tutorial will cover how to setup and run a large, multi-metallicity population run on a HPC with slurm.

We will dive deeper into the population_params.ini file and the Population class for multi-metallicity populations.

If you haven’t done so yet, export the path POSYDON environment variables. For example:

[ ]:

%env PATH_TO_POSYDON=/Users/simone/Google Drive/github/POSYDON-public/
%env PATH_TO_POSYDON_DATA=/Volumes/T7/

Creating the Initialization File for multi-metallicity runs

Let’s copy the default population synthesis ini file to your working directory. Make sure you’re on a cluster to be able to run and submit the jobs.

[ ]:

import os
import shutil
from posydon.config import PATH_TO_POSYDON

path_to_params = os.path.join(PATH_TO_POSYDON, "posydon/popsyn/population_params_default.ini")
shutil.copyfile(path_to_params, './population_params.ini')

Open the population_params.ini file and do the following edits to run a large model at 8 differet metallicities:

  • set metallicity = [2., 1., 0.45, 0.2, 0.1, 0.01, 0.001, 0.0001]

  • set number_of_binaries = 100

You might also want to make sure the dump_rate is set to 10 binaries. dump_rate = 10

Setting-up the Population Synthesis Model

POSYDON provides setup-popsyn that you can use to setup a (multi-)metallicity population run on a HPC facility.

This will split each metallicity into a separate slurm job-array and a dependent job which will automatically merge the output of the separate jobs.

Below are two examples, but you might require to adjust the inputs for your email, cluster and available partitions.

Older POSYDON installations

If the setup-popsyn command is not available, you might have to install POSYDON. 1. Go to the directory: cd $PATH_TO_POSYDON 2. Uninstall using pip pip uninstall posydon 3. Reinstall: if you’re using the development version: pip install .

See the `Installation instructions <>`__ if any issues occur.

[ ]:

# example for yggdrasil
posydon-setup-popsyn population_params.ini --job_array=10 --walltime=00:14:00 --partition=debug-cpu --email=max.briel@unige.ch --account=fragkos

[ ]:

# example for quest cluster
posydon-setup-popsyn population_params.ini --job-array=10 --walltime=00:14:00

setup-popsyn --help should provide a complete list of possible input parameters.

walltime and job_array number fine-tuning

The above examples will setup the population run with an array of 10 jobs for each metallicity. As such, each job will run 10 binaries of the 100 binaries per metallicity.

Fine-tuning the dump_rate compared to the number of binaries each job runs can be helpful. However, the larger the dump_rate the higher the memory footprint of each job. As a default, 4Gb of RAM is requested per job.

Similarly, the walltime and job_array can be fine-tuned. A single binary takes about 1-2 seconds to run. Depending on the number of binaries each job does, you might want to raise or lower the walltime for optimal perfomance.

For example, with 100.000 binaries split over 100 jobs (per metallicity), means that every job runs 1.000 binaries. dump_rate=1000 is a good amount of binaries to keep in memory, when you’re doing an initial-final run (see here for more details on different run types). This will take around 33 minutes per job. So a walltime of 00:45:00 is reasonable.

Instead of setting a dump_rate, it’s also possible to set a ram_per_cpu. The code will try to stay below 90% of this limit, but this is not always guaranteed due to additional python overhead.

After you’ve ran the above setup, you should now have several files in your work directory.

You can submit all job arrays and merged jobs using slurm_submit.sh

[ ]:

sh slurm_submit.sh

If one of your runs fails, you can manually submit them again after fixing the issue. The *_logs folder will contain the logs of each job in the job_array.

Populations inspection

Once the runs and the mergers have finished. You should have 8 files named MET_Zsun_population.h5, where MET are the metallicities.

We will inspect two to check the number of binaries in the population.

[ ]:

from posydon.popsyn.synthetic_population import Population

file1 = '1e-01_Zsun_population.h5'
file2 = '1e-02_Zsun_population.h5'

pop1 = Population(file1)
print(pop1.mass_per_metallicity)

pop2 = Population(file2)
print(pop2.mass_per_metallicity)

In the next tutorial, we will look into selecting specific events and combining the different metallicity runs into a single population!