{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "0",
      "metadata": {},
      "source": [
        "# Reading datasets into BDF\n",
        "\n",
        "BDF supports a few ways to read source data:\n",
        "\n",
        "- a URL for a file hosted on the web\n",
        "- a path for a local file\n",
        "- an identifier from the BDF data registry\n",
        "\n",
        "In this notebook, we will show some examples for how to read data from these different sources. We will also demonstrate some helpful features including:\n",
        "\n",
        "- viewing the raw data in the format supplied from the vendor / original source\n",
        "- keeping only the required columns or including all columns"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Import the package\n",
        "import bdf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Read from a local file path\n",
        "filepath = \"../../data/SINTEF__LiGrR2032__2024-04-30__25degC__Landt.csv\"\n",
        "\n",
        "# read() returns (frame, metadata)\n",
        "df, meta = bdf.read(filepath)\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Peek at the raw vendor file without normalization to BDF\n",
        "vendor_df, _ = bdf.read(filepath, normalize=False, validate=False)\n",
        "vendor_df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Keep all columns unknown to BDF\n",
        "df_req, _ = bdf.read(filepath, include_unknown=True)\n",
        "print(df_req.columns)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Read data on the Web from a URL\n",
        "df, meta = bdf.read(\"https://zenodo.org/records/17289383/files/SINTEF__NaCR32140-MP10-04__2025-08-25__GITT_0p05C_25degC__BioLogic.mpt\")\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6",
      "metadata": {},
      "source": [
        "## Read from the dataset registry\n",
        "\n",
        "The dataset registry is a JSON file (datasets.json) that maps ids to URLs and\n",
        "optional plugin hints. This is separate from the metadata registry built by\n",
        "`bdf.build_registry`. Resolve an id to its entry with `bdf.load_registry` /\n",
        "`bdf.get_entry`, then pass the entry's URL to `bdf.read`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Read from the BDF data registry\n",
        "# This registry is a local JSON file (datasets.json), separate from the metadata registry.\n",
        "import json\n",
        "from pathlib import Path\n",
        "\n",
        "registry_path = Path(\"out/datasets.json\")\n",
        "registry_path.parent.mkdir(parents=True, exist_ok=True)\n",
        "registry_path.write_text(json.dumps({\n",
        "    \"schema_version\": \"0.3\",\n",
        "    \"entries\": [\n",
        "        {\n",
        "            \"id\": \"sintef-biologic-demo\",\n",
        "            \"url\": \"https://zenodo.org/records/17289383/files/SINTEF__NaCR32140-MP10-04__2025-08-25__GITT_0p05C_25degC__BioLogic.mpt\",\n",
        "            \"plugin\": \"biologic_mpt\",\n",
        "        }\n",
        "    ],\n",
        "}, indent=2), encoding=\"utf-8\")\n",
        "\n",
        "reg = bdf.load_registry(registry_path)\n",
        "entry = bdf.get_entry(reg, \"sintef-biologic-demo\")\n",
        "df, meta = bdf.read(entry[\"url\"])\n",
        "df.head()"
      ]
    }
  ],
  "metadata": {
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.10"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}