The fullt POSYDON Processing Pipeline experince
From A to Z: in this tutorial you learn to run all steps of the POSYDON post processing pipeline. This includes: - step 1: create the h5 file of each MESA grid, this include the base grid and any expantion or rerun - step 2: stack the grids togeather by concatenating them in the corret order - step 2 plot: plot 2D grid splices to visaulize termination flags, final values and other properties - step 2 check failure rate: check the failure rate of the grid - step 3: compute post processing values on the ORIGINAL grid (these values are stored as additional final values columns) - step 4: train the initial final interpolators and classifiers - step 9: export the entire POSYDON dataset in the working directory
All these steps are linked togeather and, when possible, they will be executed in parrallel, else sequencially.
We support two types of compression LITE and ORIGINAL, the former resamples the stellar histories, the binary history and the fial stellar profile. Plase refer to the POSYDON v1.0.0 paper for more details.
We illustrate this example using the 100 HMS-MHS MESA models which we refer to as the test_grid and the opacity_max rerun layer, rerun_opacity_max_text_grid, we generated by rerunning failed models in the test_grid with a lower hier opacity max limit as illustrative fix.
Preparing the Pipeline initialization file
The following cell export the entire pipeline.ini, the purpuse of this tutorial is to get you acquinted with the pipeline and not to teach you how to export the entire POSYDON v2.0.0 dataset.
EXTRA: If you are a POSYDON developer or just a curius person, you can find the full POSYDON v2.0.0 pipeline.ini file in $PATH_TO_POSYDON/grid_params/pipeline_quest.ini and $PATH_TO_POSYDON/grid_params/pipeline_yggdrasil.ini. At the time of writing this tutorial we support 5 layers of reruns and store the raw MESA data of v2.0.0 across two HPC facility. Exporting the entire v2.0.0 dataset from start to finish might require more than 24 hours computing time. !!! WARNING !!! before
exportig the entire POSYDON v2.0.0 dataset, make sure you know what you are doing!
Now sit back relax and enjoy the POSYDON experince!
[67]:
%%writefile pipeline.ini
[account]
ACCOUNT = 'meynet'
PARTITION = 'public-cpu'
WALLTIME = '06:00:00'
MAILTYPE = 'ALL'
EMAIL = 'simone.bavera@unige.ch'
[pipeline setup]
PATH_TO_GRIDS = '/srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/'
VERSION = '' # 'v2' in quest and '' in yggdrasil
PATH = '.' # working dir
VERBOSE = True
# steps
CREATE_GRID_SLICES = True
COMBINE_GRID_SLICES = True
CALCULATE_EXTRA_VALUES = True
TRAIN_INTERPOLATORS = True
EXPORT_DATASET = True
# rerun step
RERUN = False
#CREATE_GRID_SLICES
[step_1]
# e.g. ['CO-HMS_RLO','CO-HeMS','HMS-HMS']
GRID_TYPES = ['HMS-HMS']
# e.g. ['2e+00_Zsun', '1e+00_Zsun', '4.5e-01_Zsun', '2e-01_Zsun', '1e-01_Zsun', '1e-02_Zsun', '1e-03_Zsun', '1e-04_Zsun']
METALLICITIES = [
# HMS-HMS
['1e-01_Zsun']
]
GRID_SLICES = [
# HMS-HMS
['test_grid', 'rerun_opacity_max_test_grid']
]
COMPRESSIONS = [
# HMS-HMS
['LITE', 'ORIGINAL']
]
DROP_MISSING_FILES = True
# EXTRA PARAMETERS
# only applied to HMS grids
STOP_BEFORE_CARBON_DEPLETION = 1
# supported plots: e.g. 'combined_TF12', 'termination_flag_1', 'termination_flag_2', 'termination_flag_3', 'termination_flag_4', and any quantity valid for a Z-plotting
CREATE_PLOTS = []
# supported checks: e.g. 'failure_rate'
DO_CHECKS = []
#COMBINE_GRID_SLICES
[step_2]
GRID_TYPES = ['HMS-HMS']
METALLICITIES = [# HMS-HMS
['1e-01_Zsun']
]
GRID_SLICES = [
# HMS-HMS
[['test_grid', 'rerun_opacity_max_test_grid']]
]
GRIDS_COMBINED = [
# HMS-HMS
['grid_test_combined']
]
COMPRESSIONS = [
# HMS-HMS
['LITE', 'ORIGINAL']
]
DROP_MISSING_FILES = True
# supported plots: e.g. 'combined_TF12', 'termination_flag_1', 'termination_flag_2', 'termination_flag_3', 'termination_flag_4', and any quantity valid for a Z-plotting
CREATE_PLOTS = ['PLOT_AFTER_COMBINE']
# supported checks: e.g. 'failure_rate'
DO_CHECKS = ['CHECK_AFTER_COMBINE']
#CALCULATE_EXTRA_VALUES
[step_3]
GRID_TYPES = ['HMS-HMS']
METALLICITIES = [# HMS-HMS
['1e-01_Zsun']
]
GRID_SLICES = [# HMS-HMS
['grid_test_combined']
]
COMPRESSIONS = [
# HMS-HMS
['LITE']
]
DROP_MISSING_FILES = True
# supported plots: e.g. 'combined_TF12', 'termination_flag_1', 'termination_flag_2', 'termination_flag_3', 'termination_flag_4', and any quantity valid for a Z-plotting
CREATE_PLOTS = ['PLOT_AFTER_EXTRA']
# supported checks: e.g. 'failure_rate', 'CO_TYPE', 'SN_TYPE'
DO_CHECKS = ['CHECK_AFTER_EXTRA']
#TRAIN_INTERPOLATORS
[step_4]
GRID_TYPES = ['HMS-HMS']
METALLICITIES = [# HMS-HMS
['1e-01_Zsun']
]
GRID_SLICES = [# HMS-HMS
['grid_test_combined_processed']
]
INTERPOLATION_METHODS = ["linear","1NN"]
COMPRESSIONS = [
# HMS-HMS
['LITE']
]
CONTROL_GRIDS = [
# HMS-HMS
['grid_test_combined'] # if available this should be a randomly sampled grid
]
DROP_MISSING_FILES = True
# supported plots: e.g. 'combined_TF12', 'termination_flag_1', 'termination_flag_2', 'termination_flag_3', 'termination_flag_4', and any quantity valid for a Z-plotting
CREATE_PLOTS = ['PLOT_AFTER_TRAINING']
# supported checks: e.g. 'failure_rate'
DO_CHECKS = ['CHECK_AFTER_TRAINING']
#EXPORT_DATASET
[step_9]
GRID_TYPES = ['HMS-HMS']
METALLICITIES = [# HMS-HMS
['1e-01_Zsun']
]
GRID_SLICES = [
# HMS-HMS
['grid_test_combined_processed']
]
COMPRESSIONS = [
# HMS-HMS
['LITE']
]
DROP_MISSING_FILES = True
#EXPORT_RERUNS
[rerun]
GRID_TYPES = ['HMS-HMS']
METALLICITIES = [# HMS-HMS
[ '1e-01_Zsun']
]
GRID_SLICES = [
# HMS-HMS
['grid_test_combined'],
]
COMPRESSIONS = [
# HMS-HMS
['LITE']
]
DROP_MISSING_FILES = True
# example reruns are 'PISN', 'reverse_MT', 'opacity_max', 'TPAGBwind'
RERUN_TYPE = None
Overwriting pipeline.ini
Setting-up and Running the Post Processing Pipeline
We are all set, let’s setup the pipeline and run it!
[68]:
!setup-pipeline pipeline.ini
/home/bavera/.conda/envs/posydon_env/bin/setup-pipeline:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
__import__('pkg_resources').require('posydon==1.0.0+194.g3953a14')
+++++++++++++++++++ACCOUNT+++++++++++++++++++
{ 'ACCOUNT': 'meynet',
'EMAIL': 'simone.bavera@unige.ch',
'MAILTYPE': 'ALL',
'PARTITION': 'public-cpu',
'WALLTIME': '06:00:00'}
++++++++++++++++++++SETUP++++++++++++++++++++
{ 'CALCULATE_EXTRA_VALUES': True,
'COMBINE_GRID_SLICES': True,
'CREATE_GRID_SLICES': True,
'EXPORT_DATASET': True,
'PATH': '.',
'PATH_TO_GRIDS': '/srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/',
'RERUN': False,
'TRAIN_INTERPOLATORS': True,
'VERBOSE': True,
'VERSION': ''}
-------------CREATE_GRID_SLICES-------------- step_1 : True
{ 'COMPRESSIONS': [['LITE', 'ORIGINAL']],
'CREATE_PLOTS': [],
'DO_CHECKS': [],
'DROP_MISSING_FILES': True,
'GRID_SLICES': [['test_grid', 'rerun_opacity_max_test_grid']],
'GRID_TYPES': ['HMS-HMS'],
'METALLICITIES': [['1e-01_Zsun']],
'STOP_BEFORE_CARBON_DEPLETION': 1}
-------------COMBINE_GRID_SLICES------------- step_2 : True
{ 'COMPRESSIONS': [['LITE', 'ORIGINAL']],
'CREATE_PLOTS': [ 'combined_TF12',
'termination_flag_1',
'termination_flag_2',
'termination_flag_3',
'termination_flag_4',
'rl_relative_overflow_1',
'rl_relative_overflow_2',
'lg_mtransfer_rate'],
'DO_CHECKS': ['failure_rate'],
'DROP_MISSING_FILES': True,
'GRIDS_COMBINED': [['grid_test_combined']],
'GRID_SLICES': [[['test_grid', 'rerun_opacity_max_test_grid']]],
'GRID_TYPES': ['HMS-HMS'],
'METALLICITIES': [['1e-01_Zsun']]}
-----------CALCULATE_EXTRA_VALUES------------ step_3 : True
{ 'COMPRESSIONS': [['LITE']],
'CREATE_PLOTS': [ 'S1_MODEL01_CO_type',
'S1_MODEL01_SN_type',
'S1_MODEL01_mass',
'S1_MODEL01_spin',
'S1_MODEL01_m_disk_radiated',
'S1_MODEL02_CO_type',
'S1_MODEL02_SN_type',
'S1_MODEL02_mass',
'S1_MODEL02_spin',
'S1_MODEL02_m_disk_radiated',
'S1_MODEL03_CO_type',
'S1_MODEL03_SN_type',
'S1_MODEL03_mass',
'S1_MODEL03_spin',
'S1_MODEL03_m_disk_radiated',
'S1_MODEL04_CO_type',
'S1_MODEL04_SN_type',
'S1_MODEL04_mass',
'S1_MODEL04_spin',
'S1_MODEL04_m_disk_radiated',
'S1_MODEL05_CO_type',
'S1_MODEL05_SN_type',
'S1_MODEL05_mass',
'S1_MODEL05_spin',
'S1_MODEL05_m_disk_radiated',
'S1_MODEL06_CO_type',
'S1_MODEL06_SN_type',
'S1_MODEL06_mass',
'S1_MODEL06_spin',
'S1_MODEL06_m_disk_radiated',
'S1_MODEL07_CO_type',
'S1_MODEL07_SN_type',
'S1_MODEL07_mass',
'S1_MODEL07_spin',
'S1_MODEL07_m_disk_radiated',
'S1_MODEL08_CO_type',
'S1_MODEL08_SN_type',
'S1_MODEL08_mass',
'S1_MODEL08_spin',
'S1_MODEL08_m_disk_radiated',
'S1_MODEL09_CO_type',
'S1_MODEL09_SN_type',
'S1_MODEL09_mass',
'S1_MODEL09_spin',
'S1_MODEL09_m_disk_radiated',
'S1_MODEL10_CO_type',
'S1_MODEL10_SN_type',
'S1_MODEL10_mass',
'S1_MODEL10_spin',
'S1_MODEL10_m_disk_radiated'],
'DO_CHECKS': ['CO_type', 'SN_type'],
'DROP_MISSING_FILES': True,
'GRID_SLICES': [['grid_test_combined']],
'GRID_TYPES': ['HMS-HMS'],
'METALLICITIES': [['1e-01_Zsun']]}
-------------TRAIN_INTERPOLATORS------------- step_4 : True
{ 'COMPRESSIONS': [['LITE']],
'CONTROL_GRIDS': [['grid_test_combined']],
'CREATE_PLOTS': [ 'INTERP_ERROR_age',
'INTERP_ERROR_star_1_mass',
'INTERP_ERROR_star_2_mass',
'INTERP_ERROR_period_days',
'INTERP_ERROR_S1_co_core_mass',
'INTERP_ERROR_S1_co_core_radius',
'INTERP_ERROR_S1_he_core_mass',
'INTERP_ERROR_S1_he_core_radius',
'INTERP_ERROR_S1_center_h1',
'INTERP_ERROR_S1_center_he4',
'INTERP_ERROR_S1_surface_h1',
'INTERP_ERROR_S1_surface_he4',
'INTERP_ERROR_S1_surf_avg_omega_div_omega_crit',
'INTERP_ERROR_S1_log_Teff',
'INTERP_ERROR_S1_log_L',
'INTERP_ERROR_S1_log_R',
'INTERP_ERROR_S1_spin_parameter',
'INTERP_ERROR_S1_lambda_CE_10cent',
'INTERP_ERROR_S2_co_core_mass',
'INTERP_ERROR_S2_co_core_radius',
'INTERP_ERROR_S2_he_core_mass',
'INTERP_ERROR_S2_he_core_radius',
'INTERP_ERROR_S2_center_h1',
'INTERP_ERROR_S2_center_he4',
'INTERP_ERROR_S2_surface_h1',
'INTERP_ERROR_S2_surface_he4',
'INTERP_ERROR_S2_surf_avg_omega_div_omega_crit',
'INTERP_ERROR_S2_log_Teff',
'INTERP_ERROR_S2_log_L',
'INTERP_ERROR_S2_log_R',
'INTERP_ERROR_S2_spin_parameter',
'INTERP_ERROR_S2_lambda_CE_10cent',
'INTERP_ERROR_S1_MODEL01_mass',
'INTERP_ERROR_S1_MODEL01_spin',
'INTERP_ERROR_S1_MODEL01_m_disk_radiated',
'INTERP_ERROR_S1_MODEL05_mass',
'INTERP_ERROR_S1_MODEL05_spin',
'INTERP_ERROR_S1_MODEL05_m_disk_radiated',
'INTERP_ERROR_S1_MODEL06_mass',
'INTERP_ERROR_S1_MODEL06_spin',
'INTERP_ERROR_S1_MODEL06_m_disk_radiated',
'INTERP_ERROR_S1_MODEL10_mass',
'INTERP_ERROR_S1_MODEL10_spin',
'INTERP_ERROR_S1_MODEL10_m_disk_radiated'],
'DO_CHECKS': [],
'DROP_MISSING_FILES': True,
'GRID_SLICES': [['grid_test_combined_processed']],
'GRID_TYPES': ['HMS-HMS'],
'INTERPOLATION_METHODS': ['linear', '1NN'],
'METALLICITIES': [['1e-01_Zsun']]}
---------------EXPORT_DATASET---------------- step_9 : True
{ 'COMPRESSIONS': [['LITE']],
'DROP_MISSING_FILES': True,
'GRID_SLICES': [['grid_test_combined_processed']],
'GRID_TYPES': ['HMS-HMS'],
'METALLICITIES': [['1e-01_Zsun']]}
--------------------RERUN-------------------- rerun :False
[29]:
!ls
logs step_2.csv step_3.slurm
pipeline.ini step_2_plots.csv step_4.csv
POSYDON_data step_2_plots.slurm step_4_plots.csv
run_full_piepeline.ipynb step_2.slurm step_4_plots.slurm
run_pipeline.sh step_3_checks.csv step_4.slurm
step_1.csv step_3_checks.slurm step_9.csv
step_1.slurm step_3.csv step_9.slurm
step_2_checks.csv step_3_plots.csv
step_2_checks.slurm step_3_plots.slurm
Data will be exported in the POSYDON_data/ directory once the pipeline is done. You can keep track of the progress by checking the log files of each step in the logs/ directory. Let’s submit the entire pipeline with the shell script.
[30]:
!./run_pipeline.sh
step_1.slurm submitted as 28474240
step_2.slurm submitted as 28474241
step_2_plots.slurm submitted as 28474242
step_2_checks.slurm submitted as 28474243
step_3.slurm submitted as 28474244
step_3_plots.slurm submitted as 28474245
step_3_checks.slurm submitted as 28474246
step_4.slurm submitted as 28474247
step_4_plots.slurm submitted as 28474248
step_9.slurm submitted as 28474249
[33]:
!squeue -u bavera
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
28473967_1 private-a mesa_gri bavera R 5:13:52 1 cpu144
28473967_4 private-a mesa_gri bavera R 5:13:52 1 cpu144
28473967_5 private-a mesa_gri bavera R 5:13:52 1 cpu144
28473967_7 private-a mesa_gri bavera R 5:13:52 1 cpu144
28474249_[0-2] public-cp psygrid9 bavera PD 0:00 1 (Dependency)
28474248_[0-1] public-cp psygrid4 bavera PD 0:00 1 (Dependency)
28474247_[0-1] public-cp psygrid4 bavera PD 0:00 1 (Dependency)
28474246_[0] public-cp psygrid3 bavera PD 0:00 1 (Dependency)
28474245_[0] public-cp psygrid3 bavera PD 0:00 1 (Dependency)
28474244_[0] public-cp psygrid3 bavera PD 0:00 1 (Dependency)
28474243_[0-1] public-cp psygrid2 bavera PD 0:00 1 (Dependency)
28474242_[0] public-cp psygrid2 bavera PD 0:00 1 (Dependency)
28474241_[0-1] public-cp psygrid2 bavera PD 0:00 1 (Dependency)
28474240_0 public-cp psygrid1 bavera R 0:04 1 cpu024
28474240_1 public-cp psygrid1 bavera R 0:04 1 cpu024
28474240_2 public-cp psygrid1 bavera R 0:04 1 cpu024
28474240_3 public-cp psygrid1 bavera R 0:04 1 cpu024
Exploring the visualization outputs
While we wait for the entire pipeline to finish we can explore the plots which we find in PATH_TO_GRIDS/HMS-HMS/1e-01_Zsun/plots/GRID_NAME.
[61]:
!ls /srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/HMS-HMS/1e-01_Zsun/plots/grid_test_combined_processed/S1_MODEL01/
CO_type mass m_disk_radiated SN_type spin
[63]:
from IPython.display import display, Image
path = '/srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/HMS-HMS/1e-01_Zsun/plots/'
# termination flags
img = Image(path+'grid_test_combined/TF12/grid_q_0.70.png')
display(img)
img = Image(path+'grid_test_combined/TF1/grid_q_0.70.png')
display(img)
img = Image(path+'grid_test_combined/TF2/grid_q_0.70.png')
display(img)
img = Image(path+'grid_test_combined/TF3/grid_q_0.70.png')
display(img)
img = Image(path+'grid_test_combined/TF4/grid_q_0.70.png')
display(img)
# processed values
img = Image(path+'grid_test_combined_processed/S1_MODEL01/SN_type/grid_q_0.70.png')
display(img)
img = Image(path+'grid_test_combined_processed/S1_MODEL01/CO_type/grid_q_0.70.png')
display(img)
img = Image(path+'grid_test_combined_processed/S1_MODEL01/mass/grid_q_0.70.png')
display(img)
img = Image(path+'grid_test_combined_processed/S1_MODEL01/spin/grid_q_0.70.png')
display(img)
In this tutorial we did not create a randomly sampled grid to test the accuracy of the interolators. If you create a radomly sampled grid, then the pipeline will also export 2D error maps evaluated on the randomly sampled gird. This is left as an exercise to the reader.
TODO: add the automatic export of violin plots and cofiusion matrices to the plotting.
Check that you obtained the dataset.
[80]:
!ls POSYDON_data/HMS-HMS/
1e-01_Zsun.h5 interpolators
Congratulations you have completed the POSYDON v2.0.0 pipeline tutorial!