{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Export Rerun Grids with POSYDON Post Processing Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes the POSYDON MESA engineer pratitioner might be challenged with exporting grid points to be rerun systematically for multiple MESA grids with the aim of fixing the \"islands of unhappines\" (cit. Aaron Dotter). These islands are portions of the parameter space where the MESA simulations did not converge to a solution. The POSYDON post processing pipeline can be used to export the grid points to be rerun `grid.csv` (give any predefined logic) and the `pipeline.ini` file already preconfigured with the correct `MESA-INLIST` submodule branch and commit. The `params.ini` file can be used to run the grid points with the `mesa` command line tool.\n",
    "\n",
    "POSYDON MESA grids associated with v2.0.0 made use of the following predefined rerun options in the post processing pipeline:\n",
    "- 'PISN': \"matthias_PISN-d68228338b91fd487ef5d55c9b6ebb8cc5f0e668\"\n",
    "- 'reverse_MT':  \"zepei_fix_implicit-afa1860ddf9894aa1d82742ee2a73e8e92acd4a9\"\n",
    "- 'opacity_max': \"matthias_PISN-d68228338b91fd487ef5d55c9b6ebb8cc5f0e668\"\n",
    "- 'opacity_max_hms-hms':  \"zepei_fix_implicit-afa1860ddf9894aa1d82742ee2a73e8e92acd4a9\"\n",
    "- 'TPAGBwind': \"development-22c1bb9e730343558c3e70984a99b3fc1f3c346e\"\n",
    "- 'thermohaline_mixing':  \"development-22c1bb9e730343558c3e70984a99b3fc1f3c346e\"\n",
    "Please refer to the POSYDON v2.0.0 paper for more details.\n",
    "\n",
    "Here we show how to export the `opacity_max` reruns associated with the \"getting started\" tutorial using the post-processing pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you haven't done it already, export the environemnt variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: PATH_TO_POSYDON=/srv/beegfs/scratch/shares/astro/posydon/simone/documentation/POSYDON/\n",
      "env: PATH_TO_POSYDON_DATA=/srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/\n"
     ]
    }
   ],
   "source": [
    "%env PATH_TO_POSYDON=/srv/beegfs/scratch/shares/astro/posydon/simone/documentation/POSYDON/\n",
    "%env PATH_TO_POSYDON_DATA=/srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the Pipeline initialization file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's copy the pipeline ini file template for the UNIGE HPC cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./pipeline.ini'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import shutil\n",
    "from posydon.config import PATH_TO_POSYDON\n",
    "\n",
    "path_to_ini = os.path.join(PATH_TO_POSYDON, 'grid_params/pipeline_yggdrasil.ini')\n",
    "shutil.copyfile(path_to_ini, './pipeline.ini')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now edit the pipeline ini file to point to the MESA grid directory `test_grid/` containing a set of 100 MESA models of the HMS-HMS grid at 0.1Zsun for the mass ratio q=0.7, see the running MESa grid getting started tutorial.\n",
    "\n",
    "In order for the pipeline to be able to process the data we need to follow the following directory naming convention:\n",
    "`/HMS-HMS/1e-01_Zsun/test_grid/`.\n",
    "\n",
    "Here we just want to run the rerun step of the pipeline, in order to create the `grid.csv` file and the `pipeline.ini` file to be used to run the subsample of MESA simulations with the new configuration. Notice that the rerun step can be also used after the second step to export reruns of concatenate MESA grid (PSyGrid object `h5` file). \n",
    "\n",
    "After setting up the HPC account options and `PATH_TO_GRIDS` the value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from posydon.config import PATH_TO_POSYDON_DATA\n",
    "\n",
    "PATH_TO_GRIDS = os.path.join(PATH_TO_POSYDON_DATA, 'POSYDON_data/tutorials/processing-pipeline')\n",
    "PATH_TO_GRIDS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set:\n",
    "```ini\n",
    "    CREATE_GRID_SLICES = False\n",
    "    COMBINE_GRID_SLICES = False\n",
    "    CALCULATE_EXTRA_VALUES = False\n",
    "    TRAIN_INTERPOLATORS = False\n",
    "    EXPORT_DATASET = False\n",
    "    RERUN = True\n",
    "```\n",
    "\n",
    "And edit the `[rerun]` section of the file to\n",
    "```ini\n",
    "    GRID_TYPES = ['HMS-HMS']\n",
    "    METALLICITIES = [['1e-01_Zsun']\n",
    "                    ]\n",
    "    GRID_SLICES = [['test_grid']]\n",
    "    COMPRESSIONS = [['LITE']]\n",
    "    DROP_MISSING_FILES = True\n",
    "    RERUN_TYPE = 'opacity_max' \n",
    "```\n",
    "\n",
    "Also remember to set\n",
    "```ini\n",
    "    CREATE_PLOTS = []\n",
    "    DO_CHECKS = []\n",
    "```\n",
    "for all other steps, otherwise the pipeline will try to create plots and do checks on the data, which is not possible since we are not running the full pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting-up and Running the Post Processing Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now rady to setup the grid pipeline with the `posydon-setup-pipeline` command, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/bavera/.conda/envs/posydon_env/bin/posydon-setup-pipeline:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html\n",
      "  __import__('pkg_resources').require('posydon==1.0.0+194.g3953a14')\n",
      "\n",
      "+++++++++++++++++++ACCOUNT+++++++++++++++++++ \n",
      "{ 'ACCOUNT': 'meynet',\n",
      "  'EMAIL': 'simone.bavera@unige.ch',\n",
      "  'MAILTYPE': 'ALL',\n",
      "  'PARTITION': 'public-cpu',\n",
      "  'WALLTIME': '24:00:00'}\n",
      "\n",
      "++++++++++++++++++++SETUP++++++++++++++++++++\n",
      "{ 'CALCULATE_EXTRA_VALUES': False,\n",
      "  'COMBINE_GRID_SLICES': False,\n",
      "  'CREATE_GRID_SLICES': False,\n",
      "  'EXPORT_DATASET': False,\n",
      "  'PATH': '.',\n",
      "  'PATH_TO_GRIDS': '/srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/',\n",
      "  'RERUN': True,\n",
      "  'TRAIN_INTERPOLATORS': False,\n",
      "  'VERBOSE': True,\n",
      "  'VERSION': ''}\n",
      "\n",
      "\n",
      "-------------CREATE_GRID_SLICES--------------  step_1 :False \n",
      "\n",
      "\n",
      "-------------COMBINE_GRID_SLICES-------------  step_2 :False \n",
      "\n",
      "\n",
      "-----------CALCULATE_EXTRA_VALUES------------  step_3 :False \n",
      "\n",
      "\n",
      "-------------TRAIN_INTERPOLATORS-------------  step_4 :False \n",
      "\n",
      "\n",
      "---------------EXPORT_DATASET----------------  step_F :False \n",
      "\n",
      "\n",
      "--------------------RERUN--------------------  rerun  : True \n",
      "{ 'COMPRESSIONS': [['LITE']],\n",
      "  'DROP_MISSING_FILES': True,\n",
      "  'GRID_SLICES': [['test_grid']],\n",
      "  'GRID_TYPES': ['HMS-HMS'],\n",
      "  'METALLICITIES': [['1e-01_Zsun']],\n",
      "  'RERUN_TYPE': 'opacity_max'}\n"
     ]
    }
   ],
   "source": [
    "!posydon-setup-pipeline pipeline.ini"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great let's run the pipepline with the shell command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rerun.slurm submitted as 28473947\n"
     ]
    }
   ],
   "source": [
    "!./run_pipeline.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the job is done, you can check that the files were created in the `PATH_TO_GRIDS/HMS-HMS/1e-01_Zsun/` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "grid.csv  HMS-HMS_yggdrasil.ini\n"
     ]
    }
   ],
   "source": [
    "!ls /srv/beegfs/scratch/shares/astro/posydon/POSYDON_GRIDS_v2/POSYDON_data/230914/POSYDON_data/tutorials/processing-pipeline/HMS-HMS/1e-01_Zsun/rerun_opacity_max_test_grid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! You can now submit the simulation. See the tutorial on how to run the grid with the POSYDON MESA submission API tool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "posydon_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}