Processing MESA grids with the POSYDON pipeline API
If you haven’t done it already, export the environemnt variables.
[1]:
%env PATH_TO_POSYDON=/srv/beegfs/scratch/shares/astro/posydon/simone/documentation/POSYDON/
%env PATH_TO_POSYDON_DATA=/srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/
env: PATH_TO_POSYDON=/srv/beegfs/scratch/shares/astro/posydon/simone/documentation/POSYDON/
env: PATH_TO_POSYDON_DATA=/srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/
Configurating the Initialisation File for the Pipeline
Let’s copy the pipeline ini file template for the UNIGE HPC cluster.
[2]:
import os
import shutil
from posydon.config import PATH_TO_POSYDON
path_to_ini = os.path.join(PATH_TO_POSYDON, 'grid_params/pipeline_yggdrasil.ini')
shutil.copyfile(path_to_ini, './pipeline.ini')
[2]:
'./pipeline.ini'
We now edit the pipeline ini file to point to the MESA grid directory test_grid/ containing a set of 100 MESA models of the HMS-HMS grid at 0.1Zsun for the mass ratio q=0.7, see the running MESa grid getting started tutorial.
In order for the pipeline to be able to process the data we need to follow the following directory naming convention: /HMS-HMS/1e-01_Zsun/test_grid/.
Here we just want to run the first step of the pipeline, in order to create the h5 files containing the MESA models. Hence, after setting up the HPC account options and PATH_TO_GRIDS the value
[9]:
from posydon.config import PATH_TO_POSYDON_DATA
PATH_TO_GRIDS = os.path.join(PATH_TO_POSYDON_DATA, 'POSYDON_data/tutorials/processing-pipeline')
os.listdir(PATH_TO_GRIDS)
[9]:
['HMS-HMS']
We set:
CREATE_GRID_SLICES = True
COMBINE_GRID_SLICES = False
CALCULATE_EXTRA_VALUES = False
TRAIN_INTERPOLATORS = False
EXPORT_DATASET = False
RERUN = False
And edit the [step_1] section of the file to
GRID_TYPES = ['HMS-HMS']
METALLICITIES = [['1e-01_Zsun']]
GRID_SLICES = [['test_grid']]
COMPRESSIONS = [['LITE', 'ORIGINAL']]
DROP_MISSING_FILES = True
STOP_BEFORE_CARBON_DEPLETION = 1
CREATE_PLOTS = []
DO_CHECKS = []
Also remember to set
CREATE_PLOTS = []
DO_CHECKS = []
for all other steps, otherwise the pipeline will try to create plots and do checks on the data, which is not possible since we are not running the full pipeline.
Setting-up and Running the Post Processing Pipeline
We are now rady to setup the grid pipeline with the setup-pipeline command, as follows:
[12]:
!setup-pipeline pipeline.ini
/home/bavera/.conda/envs/posydon_env/bin/setup-pipeline:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
__import__('pkg_resources').require('posydon==1.0.0+194.g3953a14')
+++++++++++++++++++ACCOUNT+++++++++++++++++++
{ 'ACCOUNT': 'meynet',
'EMAIL': 'simone.bavera@unige.ch',
'MAILTYPE': 'ALL',
'PARTITION': 'private-astro-cpu',
'WALLTIME': '24:00:00'}
++++++++++++++++++++SETUP++++++++++++++++++++
{ 'CALCULATE_EXTRA_VALUES': False,
'COMBINE_GRID_SLICES': False,
'CREATE_GRID_SLICES': True,
'EXPORT_DATASET': False,
'PATH': '.',
'PATH_TO_GRIDS': '/srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/',
'RERUN': False,
'TRAIN_INTERPOLATORS': False,
'VERBOSE': True,
'VERSION': ''}
-------------CREATE_GRID_SLICES-------------- step_1 : True
{ 'COMPRESSIONS': [['LITE', 'ORIGINAL']],
'CREATE_PLOTS': [],
'DO_CHECKS': [],
'DROP_MISSING_FILES': True,
'GRID_SLICES': [['test_grid']],
'GRID_TYPES': ['HMS-HMS'],
'METALLICITIES': [['1e-01_Zsun']],
'STOP_BEFORE_CARBON_DEPLETION': 1}
-------------COMBINE_GRID_SLICES------------- step_2 :False
-----------CALCULATE_EXTRA_VALUES------------ step_3 :False
-------------TRAIN_INTERPOLATORS------------- step_4 :False
---------------EXPORT_DATASET---------------- step_9 :False
--------------------RERUN-------------------- rerun :False
[13]:
!ls
logs processing.ipynb step_1.csv step_2_plots.csv
pipeline.ini run_pipeline.sh step_1.slurm step_2_plots.slurm
Let’s sumit the job to the cluster with the run_pipeline command:
[14]:
!./run_pipeline.sh
step_1.slurm submitted as 28467595
[16]:
!squeue -u bavera -j 28467595
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
28467595_1 private-a psygrid1 bavera R 0:12 1 cpu144
28467595_0 private-a psygrid1 bavera R 0:12 1 cpu144
We are not generating a LITE and and ORIGINAL version of the grid. You can inspect the output of the pipeline in the working directory ./logs/.
[22]:
!tail -n 20 ./logs/step_1/grid_slice_0.out
Processing /srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/HMS-HMS/1e-01_Zsun/test_grid/Zbase_0.0014_m1_32.5279_m2_22.7695_initial_z_1.4200e-03_initial_period_in_days_4.5574e+01_grid_index_45
Processing /srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/HMS-HMS/1e-01_Zsun/test_grid/Zbase_0.0014_m1_20.8585_m2_14.6009_initial_z_1.4200e-03_initial_period_in_days_1.1574e+00_grid_index_32
Processing /srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/HMS-HMS/1e-01_Zsun/test_grid/Zbase_0.0014_m1_123.3602_m2_86.3521_initial_z_1.4200e-03_initial_period_in_days_3.9376e+00_grid_index_73
Processing /srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/HMS-HMS/1e-01_Zsun/test_grid/Zbase_0.0014_m1_8.5770_m2_6.0039_initial_z_1.4200e-03_initial_period_in_days_1.1574e+00_grid_index_12
100%|██████████| 100/100 [02:06<00:00, 1.27s/it]
Storing initial/final values and metadata to HDF5...
/srv/beegfs/scratch/shares/astro/posydon/simone/documentation/POSYDON/posydon/grids/psygrid.py:654: UserWarning: Ignored MESA run because of missing binary history in: /srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/HMS-HMS/1e-01_Zsun/test_grid/Zbase_0.0014_m1_123.3602_m2_86.3521_initial_z_1.4200e-03_initial_period_in_days_1.0000e-01_grid_index_70
/srv/beegfs/scratch/shares/astro/posydon/simone/documentation/POSYDON/posydon/grids/psygrid.py:654: UserWarning: Ignored MESA run because of missing binary history in: /srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/HMS-HMS/1e-01_Zsun/test_grid/Zbase_0.0014_m1_123.3602_m2_86.3521_initial_z_1.4200e-03_initial_period_in_days_3.4021e-01_grid_index_71
/srv/beegfs/scratch/shares/astro/posydon/simone/documentation/POSYDON/posydon/grids/psygrid.py:654: UserWarning: Ignored MESA run because of missing binary history in: /srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/HMS-HMS/1e-01_Zsun/test_grid/Zbase_0.0014_m1_300.0000_m2_210.0000_initial_z_1.4200e-03_initial_period_in_days_1.0000e-01_grid_index_90
/srv/beegfs/scratch/shares/astro/posydon/simone/documentation/POSYDON/posydon/grids/psygrid.py:654: UserWarning: Ignored MESA run because of missing binary history in: /srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/HMS-HMS/1e-01_Zsun/test_grid/Zbase_0.0014_m1_20.8585_m2_14.6009_initial_z_1.4200e-03_initial_period_in_days_1.0000e-01_grid_index_30
Loading HDF5 grid...
Loading initial/final values...
Acquiring paths to MESA directories...
Getting configuration metadata...
Enumerating runs and checking integrity of grid...
Done.
Great, the processing happened successuflly. Let’s load the grid.
Exploring the PSyGrid Object and Exporting Failed Simulation to Be Re-Run
[24]:
from posydon.grids.psygrid import PSyGrid
path_to_lite_grid = os.path.join(PATH_TO_GRIDS, 'HMS-HMS/1e-01_Zsun/LITE/test_grid.h5')
grid = PSyGrid(path_to_lite_grid)
grid.load()
Here is how to access the grid data, please refer to the PSyGrid documentation for more details.
[34]:
# initial final values
# grid.initial_values
# grid.final_values
# track 0 values
# grid[0].history1
# grid[0].history2
# grid[0].binary_history
# list name of columns
# grid.final_values.dtype.names
# show unique interpolation classes
import numpy as np
np.unique(grid.final_values['interpolation_class'])
[34]:
array(['initial_MT', 'no_MT', 'not_converged', 'stable_MT', 'unstable_MT'],
dtype='<U70')
As you can see we have not_coverged models, which are not suitable for population synthesis. They can be exported to be resimulated with a different set of parameters and reprocessed with the pipleline. See the PSyGrid documentation for more details.
[43]:
from collections import Counter
Counter(grid.final_values['interpolation_class'])
[43]:
Counter({'stable_MT': 37,
'no_MT': 21,
'initial_MT': 19,
'unstable_MT': 12,
'not_converged': 11})
Let’s export the models to be rerun, and apply a different set of paramter, e.g. let’s limit the max opacity to 0.5 (opacity_max = 0.5 in inlist). The rerun method allows to export a grid.csv file with points having a given termination_flag_1 or a given index.
[45]:
ids = np.squeeze(np.argwhere(grid.final_values['interpolation_class'] == 'not_converged'))
ids
[45]:
array([ 7, 19, 28, 35, 49, 63, 64, 70, 72, 77, 86])
[46]:
grid.rerun(runs_to_rerun=ids, new_mesa_flag={'opacity_max' : 0.5})
[47]:
!head -n 20 ./grid.csv
m1,m2,initial_period_in_days,initial_z,Zbase,opacity_max
20.8584585754,14.6009210028,0.3402066919,0.00142,0.00142,0.5
79.1045989347,55.3732192543,13.3958849241,0.00142,0.00142,0.5
8.577008953,6.0039062671,45.5736969568,0.00142,0.00142,0.5
8.577008953,6.0039062671,13.3958849241,0.00142,0.00142,0.5
5.5,3.85,13.3958849241,0.00142,0.00142,0.5
13.3754695598,9.3628286919,45.5736969568,0.00142,0.00142,0.5
79.1045989347,55.3732192543,45.5736969568,0.00142,0.00142,0.5
5.5,3.85,3.937572435,0.00142,0.00142,0.5
50.725759589,35.5080317123,0.3402066919,0.00142,0.00142,0.5
79.1045989347,55.3732192543,155.0447668126,0.00142,0.00142,0.5
32.5278519901,22.7694963931,0.3402066919,0.00142,0.00142,0.5
Congratulation, you have exported the grid to a csv file to be rerun! If you do not know how to run a MESA grid with POSYDON, follow the tutorial to run it.
For production grids the processing pipeline allows to export preset configurations of params.ini and grid.csv already cofigured to extract the subsample of models to be rerun and the correct MESA inlist commit. See the advanced tutorial for an example.