{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": [
    "# eDisGo full workflow walkthrough\n",
    "\n",
    "Rebuilds the complete eDisGo workflow **step by step** on a small example grid (fast to run).\n",
    "\n",
    "> **Kernel:** select a Python environment that has eDisGo installed. See the\n",
    "> [installation guide](https://edisgo.readthedocs.io/en/dev/installation.html)\n",
    "> for how to set one up.\n",
    "\n",
    "**Roadmap:** Stages 0–2 run **without a database**. From Stage 3 on an OEDB token is\n",
    "required (read automatically from the package config, or from the `OEP_TOKEN`\n",
    "environment variable). Stage 6 (OPF) additionally needs Julia + Gurobi.\n",
    "\n",
    "For background on connecting to the open energy database, see the\n",
    "[data sources documentation](https://edisgo.readthedocs.io/en/dev/userguide/data_sources.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1",
   "metadata": {},
   "source": [
    "## Stage 0 — Setup & sanity check\n",
    "\n",
    "Imports, central paths/parameters and the OEDB database engine. `engine(...)` reads the\n",
    "token automatically from the package config (or from the `OEP_TOKEN` environment variable)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import annotations\n",
    "\n",
    "import os\n",
    "\n",
    "from pathlib import Path\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import requests\n",
    "\n",
    "from IPython.display import display\n",
    "\n",
    "from edisgo import EDisGo\n",
    "from edisgo.io.db import engine as egon_engine\n",
    "\n",
    "# ---- central parameters -------------------------------------------------\n",
    "# Directory the example grid is downloaded into. Override via the EDISGO_DATA_DIR\n",
    "# environment variable to point at your own data location.\n",
    "DATA_DIR = Path(os.environ.get(\"EDISGO_DATA_DIR\", Path.home() / \".edisgo\"))\n",
    "GRID_ID = \"ding0_test_network_3\"  # small eDisGo example grid (egon-data format)\n",
    "SCENARIO = \"eGon2035\"\n",
    "\n",
    "grid_path = DATA_DIR / GRID_ID\n",
    "\n",
    "\n",
    "def download_example_grid(target: Path) -> None:\n",
    "    \"\"\"Download the eDisGo example ding0 grid (egon-data format) from GitHub.\"\"\"\n",
    "    target.mkdir(parents=True, exist_ok=True)\n",
    "    base = (\n",
    "        \"https://raw.githubusercontent.com/openego/eDisGo/dev/\"\n",
    "        \"tests/data/ding0_test_network_3\"\n",
    "    )\n",
    "    filenames = [\n",
    "        \"buses\",\n",
    "        \"generators\",\n",
    "        \"lines\",\n",
    "        \"loads\",\n",
    "        \"network\",\n",
    "        \"switches\",\n",
    "        \"transformers\",\n",
    "        \"transformers_hvmv\",\n",
    "    ]\n",
    "    for name in filenames:\n",
    "        resp = requests.get(f\"{base}/{name}.csv\")\n",
    "        resp.raise_for_status()\n",
    "        (target / f\"{name}.csv\").write_bytes(resp.content)\n",
    "\n",
    "\n",
    "# Fetch the example grid on first run. Replace ``grid_path`` with your own ding0\n",
    "# grid directory (egon-data format) to analyse a different grid.\n",
    "if not grid_path.is_dir():\n",
    "    download_example_grid(grid_path)\n",
    "\n",
    "assert grid_path.is_dir(), f\"Grid folder missing: {grid_path}\"\n",
    "print(\"Grid path:\", grid_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the OEDB engine (reads the token automatically from the package config).\n",
    "# ssh=False -> public OEDB via token. Only actually needed from Stage 3 on.\n",
    "engine = egon_engine(path=None, ssh=False, token=None)\n",
    "# Print only the host, NOT engine.url (which would contain the token in clear text):\n",
    "print(\"OEDB engine connected to:\", engine.url.host)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4",
   "metadata": {},
   "source": [
    "## Stage 1 — Load & inspect the grid *(no DB)*\n",
    "\n",
    "`EDisGo(...)` loads the ding0 CSV topology. `legacy_ding0_grids=False` selects the new\n",
    "egon-data format. We then look into the topology DataFrames and plot the MV grid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5",
   "metadata": {},
   "outputs": [],
   "source": [
    "edisgo = EDisGo(\n",
    "    ding0_grid=str(grid_path),\n",
    "    legacy_ding0_grids=False,\n",
    ")\n",
    "print(edisgo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect the topology – everything is a pandas DataFrame\n",
    "print(\"Buses:        \", len(edisgo.topology.buses_df))\n",
    "print(\"Lines:        \", len(edisgo.topology.lines_df))\n",
    "print(\"Loads:        \", len(edisgo.topology.loads_df))\n",
    "print(\"Generators:   \", len(edisgo.topology.generators_df))\n",
    "print(\"Transformers: \", len(edisgo.topology.transformers_df))\n",
    "\n",
    "display(edisgo.topology.loads_df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the MV grid (static matplotlib version)\n",
    "edisgo.plot_mv_grid_topology(technologies=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8",
   "metadata": {},
   "source": [
    "## Stage 2 — Worst-case & pre-reinforcement *(no DB)*\n",
    "\n",
    "Before real time series come in, a first grid reinforcement is computed using\n",
    "**worst-case assumptions** (simultaneous maximum load resp. maximum feed-in). The\n",
    "result — reinforcement measures plus costs — lands in `edisgo.results`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9",
   "metadata": {},
   "outputs": [],
   "source": [
    "edisgo.set_time_series_worst_case_analysis()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10",
   "metadata": {},
   "outputs": [],
   "source": [
    "edisgo.reinforce()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at the result of the pre-reinforcement\n",
    "costs = edisgo.results.grid_expansion_costs\n",
    "print(\"Total grid expansion costs: {:.1f} kEUR\".format(costs[\"total_costs\"].sum()))\n",
    "display(costs.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12",
   "metadata": {},
   "source": [
    "## Stage 3 — Real time series + power flow *(DB needed from here on)*\n",
    "\n",
    "Now we set a **real time index** (24 h in July 2035), import the scenario's generator\n",
    "fleet and fetch the active-power time series from the OEDB (`oedb`). Reactive power\n",
    "comes **last**, then the power flow (`analyze`).\n",
    "\n",
    "> **Ordering rule:** set the time index *before* the `oedb` imports;\n",
    "> call `set_time_series_reactive_power_control()` at the very end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13",
   "metadata": {},
   "outputs": [],
   "source": [
    "timeindex = pd.date_range(start=\"2035-07-15\", periods=24, freq=\"h\")\n",
    "edisgo.set_timeindex(timeindex)\n",
    "edisgo.timeseries.timeindex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Integrate the scenario's generator fleet\n",
    "edisgo.import_generators(generator_scenario=SCENARIO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Active-power time series from the OEDB; dispatchable \"other\" held constant at 70 %\n",
    "dispatchable_ts = pd.DataFrame(0.7, index=timeindex, columns=[\"other\"])\n",
    "edisgo.set_time_series_active_power_predefined(\n",
    "    fluctuating_generators_ts=\"oedb\",\n",
    "    conventional_loads_ts=\"oedb\",\n",
    "    dispatchable_generators_ts=dispatchable_ts,\n",
    "    scenario=SCENARIO,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16",
   "metadata": {},
   "source": [
    "## Stage 4 — Flexible assets from the OEDB *(DB needed)*\n",
    "\n",
    "Import home batteries, heat pumps, DSM potentials and electromobility (charging\n",
    "processes) for the scenario."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17",
   "metadata": {},
   "outputs": [],
   "source": [
    "edisgo.import_home_batteries(scenario=SCENARIO, engine=engine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18",
   "metadata": {},
   "outputs": [],
   "source": [
    "edisgo.import_heat_pumps(scenario=SCENARIO, engine=engine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19",
   "metadata": {},
   "outputs": [],
   "source": [
    "edisgo.import_dsm(scenario=SCENARIO, engine=engine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20",
   "metadata": {},
   "outputs": [],
   "source": [
    "edisgo.import_electromobility(data_source=\"oedb\", scenario=SCENARIO, engine=engine)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21",
   "metadata": {},
   "source": [
    "## Stage 5 — Strategies, reactive power & flexibility bands\n",
    "\n",
    "Apply the reference charging strategy (`\"dumb\"` = charge immediately) and the heat-pump\n",
    "operating strategy, set storage to 0, then compute **reactive power** (last!) and the\n",
    "EV flexibility bands. Afterwards we collect the flexible components for the optimisation\n",
    "in Stage 6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22",
   "metadata": {},
   "outputs": [],
   "source": [
    "edisgo.apply_charging_strategy(strategy=\"dumb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23",
   "metadata": {},
   "outputs": [],
   "source": [
    "edisgo.apply_heat_pump_operating_strategy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialise storage active power to 0 (the optimisation handles dispatch later)\n",
    "edisgo.timeseries.storage_units_active_power = pd.DataFrame(\n",
    "    data=0.0,\n",
    "    index=edisgo.timeseries.timeindex,\n",
    "    columns=edisgo.topology.storage_units_df.index,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ALWAYS set reactive power last\n",
    "edisgo.set_time_series_reactive_power_control()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute the EV flexibility bands (charging flexibility per use case)\n",
    "edisgo.electromobility.get_flexibility_bands(\n",
    "    edisgo,\n",
    "    use_case=[\"home\", \"work\", \"public\", \"hpc\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Collect the flexible components for the optimisation\n",
    "flexible_cps = edisgo.topology.charging_points_df[\n",
    "    edisgo.topology.charging_points_df.sector.isin([\"home\", \"work\"])\n",
    "].index.values\n",
    "flexible_hps = edisgo.topology.loads_df[\n",
    "    edisgo.topology.loads_df.type == \"heat_pump\"\n",
    "].index.values\n",
    "flexible_loads = edisgo.dsm.p_min.columns.values\n",
    "flexible_storage = edisgo.topology.storage_units_df.index.values\n",
    "\n",
    "print(\n",
    "    \"flex CPs:\",\n",
    "    len(flexible_cps),\n",
    "    \"| flex HPs:\",\n",
    "    len(flexible_hps),\n",
    "    \"| flex DSM:\",\n",
    "    len(flexible_loads),\n",
    "    \"| flex storage:\",\n",
    "    len(flexible_storage),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28",
   "metadata": {},
   "source": [
    "## Stage 6 — OPF (`pm_optimize`) *(Julia + Gurobi needed)*\n",
    "\n",
    "The flexibility optimisation runs via **PowerModels.jl** (Julia) + **Gurobi**. It is\n",
    "computationally intensive. If Julia or Gurobi are missing, **only this cell** fails —\n",
    "then skip it and continue with Stage 7 (final reinforcement without an optimised flex\n",
    "dispatch)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29",
   "metadata": {},
   "outputs": [],
   "source": [
    "edisgo.pm_optimize(\n",
    "    flexible_cps=flexible_cps,\n",
    "    flexible_hps=flexible_hps,\n",
    "    flexible_loads=flexible_loads,\n",
    "    flexible_storage_units=flexible_storage,\n",
    "    opf_version=2,\n",
    "    method=\"soc\",\n",
    "    warm_start=False,\n",
    "    s_base=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30",
   "metadata": {},
   "source": [
    "## Stage 7 — Final reinforcement, save & compare\n",
    "\n",
    "With the (optimised) time series the **final grid reinforcement** is performed. Then we\n",
    "save the result and look at the costs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31",
   "metadata": {},
   "outputs": [],
   "source": [
    "edisgo.reinforce()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32",
   "metadata": {},
   "outputs": [],
   "source": [
    "final_costs = edisgo.results.grid_expansion_costs\n",
    "print(\n",
    "    \"Total final grid expansion costs: {:.1f} kEUR\".format(\n",
    "        final_costs[\"total_costs\"].sum()\n",
    "    )\n",
    ")\n",
    "display(final_costs.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the result (topology + time series + results)\n",
    "out = DATA_DIR / \"results\" / GRID_ID\n",
    "edisgo.save(\n",
    "    directory=str(out),\n",
    "    save_topology=True,\n",
    "    save_timeseries=True,\n",
    "    save_results=True,\n",
    ")\n",
    "print(\"Saved to:\", out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34",
   "metadata": {},
   "source": [
    "---\n",
    "### Done! 🎉\n",
    "\n",
    "You have run through the full eDisGo workflow. Suggested next steps: try another grid,\n",
    "use a longer time window, and compare grid expansion costs with and without the OPF\n",
    "flexibility dispatch."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (eDisGo)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}