{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Large-Scale Population Synthesis on HPC Facilities 🚀\n", "\n", "\n", "This tutorial will cover how to set up and run a large, multi-metallicity population run on an HPC with slurm.\n", "We will dive deeper into the `population_params.ini` file and how the `Population` class works for multi-metallicity populations." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Initialization File for multi-metallicity runs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's copy the default population synthesis `ini` file to your working directory.\n", "Make sure you're on a cluster to be able to run and submit the jobs!\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "import shutil\n", "from posydon.config import PATH_TO_POSYDON\n", "\n", "path_to_params = os.path.join(PATH_TO_POSYDON, \"posydon/popsyn/population_params_default.ini\")\n", "shutil.copyfile(path_to_params, './population_params.ini')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Open the `population_params.ini` file and you should see that the metallicity is currently set to solar (represented by a value of ``1.``), but in the comment, you can see all 8 currently supported metallicities. \n", "\n", "```\n", "metallicity = [1.] #[2., 1., 0.45, 0.2, 0.1, 0.01, 0.001, 0.0001]\n", "```\n", "\n", "For our test run, we do not require a large population, so set the number of binaries to 1000. \n", "You can also leave the metallicity at just solar for now, but for multi-metallicity runs, \n", "you would simply add whatever available metallicities you desire to the list.\n", "You might also want to make sure the `dump_rate` is set to 100 binaries for this example. What\n", "a `dump_rate = x` says is basically every time the POSYDON runs `x` binaries, dump their \n", "data into a file and remove them from RAM to free up some space. So, in this example, after \n", "each batch of 100 binaries is run we are going to save their data to a file and combine them \n", "all in the end. (If you are curious about guidelines on memory usage when running larger \n", "populations, please have a look at our [Code Usage FAQ](../../troubleshooting-faqs/code-questions.rst).)\n", "\n", "\n", "```\n", "dump_rate = 100\n", "...\n", "number_of_binaries = 1000\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup the Population Synthesis Run" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "POSYDON provides `posydon-popsyn setup` as a command line script that you can use to set up a (multi-metallicity) population run on an HPC facility.\n", "This will split each metallicity into a separate slurm job array and a dependent job which will automatically merge the output of the separate jobs.\n", "This removes the need to manually set up each slurm job.\n", "Below are two examples, but you might need to adjust the inputs for your email, cluster and available partitions.\n", "\n", "
\n", "\n", "**Older POSYDON installations**\n", "\n", "If the `posydon-popsyn setup` command is not available, you might have to install POSYDON.\n", "1. Go to the directory: `cd $PATH_TO_POSYDON`\n", "2. Uninstall using pip `pip uninstall posydon`\n", "3. Reinstall: if you're using the development version: `pip install .`\n", " \n", "See the [Installation instructions](../../getting-started/installation-guide.rst) if any issues occur.\n", "
\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "vscode": { "languageId": "shellscript" } }, "outputs": [], "source": [ "# example for yggdrasil\n", "posydon-popsyn setup population_params.ini --job_array=10 \\\n", " --walltime=00:14:00 \\\n", " --mem_per_cpu=5G \\\n", " --partition=debug-cpu \\\n", " --email=max.briel@unige.ch \\\n", " --account=fragkos" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "vscode": { "languageId": "shellscript" } }, "outputs": [], "source": [ "# example for quest cluster\n", "posydon-popsyn setup population_params.ini --job_array=10 \\\n", " --walltime=00:14:00 \\\n", " --partition=posydon-std \\\n", " --account=b1119 \\\n", " --email=max.briel@unige.ch \\\n", " --mem_per_cpu=5G\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`posydon-popsyn setup --help` should provide a complete list of possible input parameters.\n", "\n", "\n", "### Walltime and job_array number fine-tuning\n", "\n", "The above examples will set up the population run with an array of 10 jobs for each metallicity. As such, each job will run 100 binaries of the 1000 binaries per metallicity.\n", "\n", "Fine-tuning the `dump_rate`, `mem_per_cpu`, number of `job-array`'s, and `walltime` compared to the total number of binaries can be helpful in optimizing your cluster usage. See [the FAQ](../../troubleshooting-faqs/code-questions.rst) for more detailed guidance on the memory footprint of a population synthesis run in POSYDON.\n", "By default, 5GB of RAM is requested per job, which fits well with the default `dump_rate` of 2000.\n", "\n", "As a rule of thumb for the `walltime`, a single binary takes about 1-2 seconds to run. In the example run, each job of 100 binaries should take around 2 to 3 minutes. However, some additional set up time for loading the grids is required! See [the FAQ](../../troubleshooting-faqs/code-questions.rst) for another example of `walltime` estimation." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After you've run the above setup, you should now have several files in your working directory.\n", "\n", "You can submit all job arrays and merged jobs using `slurm_submit.sh`\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "vscode": { "languageId": "shellscript" } }, "outputs": [], "source": [ "sh slurm_submit.sh" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After your job has finished, you can use the following command to verify that your run was successful.\n", "Please replace `{run_folder}` with the folder, where you ran the population.\n", "```\n", "posydon-popsyn check {run_folder}\n", "```\n", "\n", "Additionally, the `*_logs` folder will contain the logs of each job in the job_array.\n", "\n", "If one of your runs failed, you can use the `posydon-popsyn rescue {run_folder}` command to try to rescue it.\n", "This allows failed job in a job array to be resubmitted. You can provide the same commands as `posydon-popsyn setup`, if you want to change slurm parameters, such as walltime and mem-per-cpu.\n", "\n", "\n", "## Population inspection\n", "\n", "Once the runs and the concatenation have finished. You should have 8 files named `MET_Zsun_population.h5`, where `MET` are the metallicities.\n", "\n", "We will inspect two to check the number of binaries in the population." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from posydon.popsyn.synthetic_population import Population\n", "\n", "file1 = '1e-01_Zsun_population.h5'\n", "file2 = '1e-02_Zsun_population.h5'\n", "\n", "pop1 = Population(file1)\n", "print(pop1.mass_per_metallicity)\n", "\n", "pop2 = Population(file2)\n", "print(pop2.mass_per_metallicity)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the next tutorial ([BBH analysis](bbh_analysis.ipynb)), we will look into selecting specific events and combining the different metallicity runs into a single population and selecting the binary black hole mergers in them!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] } ], "metadata": { "kernelspec": { "display_name": "posydon_env", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 4 }