{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "0",
      "metadata": {},
      "source": [
        "# The `TableNormalizer` Class\n",
        "This example describes the functionality of the `TableNormalizer` class, and is aimed at those wishing to create a custom normalizer or contribute to the project."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1",
      "metadata": {},
      "outputs": [],
      "source": [
        "import bdf\n",
        "import polars as pl\n",
        "from bdf.table_normalizers import TableNormalizer, Syn, DateTimeSyn, ResolvedColumn"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2",
      "metadata": {},
      "source": [
        "A `TableNormalizer` class defines the mapping between bdf-defined columns and names that may appear as column headings for your data. You can initialise a `TableNormalizer` by providing synonyms or mappings. We will load a simple example with time, voltage and current:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3",
      "metadata": {},
      "outputs": [],
      "source": [
        "df = pl.DataFrame(\n",
        "    {\n",
        "        \"Test-Time\": [\"00:00:00.00\", \"00:00:01.00\", \"00:00:02.00\"],\n",
        "        \"Voltage-V\": [3.2, 3.3, 3.4],\n",
        "        \"Current-mA\": [10.0, 10.0, 10.0],\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4",
      "metadata": {},
      "source": [
        "To define synonyms, use the `Syn` and `DateTimeSyn` classes. `Syn` provides a simple mapping for numeric columns. Its sole argument is a string that will be used to match column names in the data. the `{unit}` placeholder identifies the position of the unit in the column name, and will match any valid unit that the pint library can parse.\n",
        "\n",
        "`DateTimeSyn` allows you to define a set of candidate formats that can be used to parse datetime string columns."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5",
      "metadata": {},
      "outputs": [],
      "source": [
        "normalizer = TableNormalizer(\n",
        "    test_time_second=[DateTimeSyn(syn=Syn(hdr=\"Test-Time\"), fmts=(\"%H:%M:%S.%f\",))],\n",
        "    voltage_volt=[Syn(hdr=\"Voltage-{unit}\")],\n",
        "    current_ampere=[Syn(hdr=\"Current-{unit}\")],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6",
      "metadata": {},
      "source": [
        "## Scoring against the normalizer\n",
        "The `TableNormalizer.score_columns()` is used to evaluate how well a defined TableNormalizer matches the data. This is used for auto-detection of sources of data. Every column parsable by the synonyms will add to the score_columns."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7",
      "metadata": {},
      "outputs": [],
      "source": [
        "score_columns = normalizer.score_columns(df.columns)\n",
        "print(\"Headers:\", df.columns)\n",
        "print(\"Score:\", score_columns)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8",
      "metadata": {},
      "source": [
        "The score_columns will drop when we provide columns that do not match the synonyms in the TableNormalizer. For example, making the unit unparseable:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9",
      "metadata": {},
      "outputs": [],
      "source": [
        "import copy\n",
        "\n",
        "cols = copy.deepcopy(df.columns)\n",
        "cols.remove(\"Current-mA\")\n",
        "cols.append(\"Current-notamps\")\n",
        "print(\"Headers:\", cols)\n",
        "print(\"Score:\", normalizer.score_columns(cols))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "10",
      "metadata": {},
      "source": [
        "Or using a unit that is unreachable from the bdf base unit (e.g. amperes from volts):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "11",
      "metadata": {},
      "outputs": [],
      "source": [
        "cols = copy.deepcopy(df.columns)\n",
        "cols.remove(\"Current-mA\")\n",
        "cols.append(\"Current-V\")\n",
        "print(\"Headers:\", cols)\n",
        "print(\"Score:\", normalizer.score_columns(cols))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "12",
      "metadata": {},
      "source": [
        "Or having a different variant of the quantity in the data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "13",
      "metadata": {},
      "outputs": [],
      "source": [
        "cols = copy.deepcopy(df.columns)\n",
        "cols.remove(\"Current-mA\")\n",
        "cols.append(\"Amps-mA\")\n",
        "print(\"Headers:\", cols)\n",
        "print(\"Score:\", normalizer.score_columns(cols))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "14",
      "metadata": {},
      "source": [
        "However, multiple synonyms can be provided to a single field in the `TableNormalizer`, allowing the same normalizer to match a range of data even if their headings differ slightly:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "15",
      "metadata": {},
      "outputs": [],
      "source": [
        "normalizer = TableNormalizer(\n",
        "    test_time_second=[DateTimeSyn(syn=Syn(hdr=\"Test-Time\"), fmts=(\"%H:%M:%S.%f\",))],\n",
        "    voltage_volt=[Syn(hdr=\"Voltage-{unit}\")],\n",
        "    current_ampere=[Syn(hdr=\"Current-{unit}\"), Syn(hdr=\"Amps-{unit}\")],\n",
        ")\n",
        "print(\"Headers:\", cols)\n",
        "print(\"Score:\", normalizer.score_columns(cols))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "16",
      "metadata": {},
      "source": [
        "## Resolving column names and unit conversions\n",
        "When you define a synonym, the `TableNormalizer.resolve()` method performs the name matching and any unit conversion. It returns a dictionary of the matched bdf columns and defines the operations needed to perform the normalization in a `bdf.ResolvedColumn` instance. You can see for the current column that the `scale` attribute reflects the mA unit."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "17",
      "metadata": {},
      "outputs": [],
      "source": [
        "import pprint\n",
        "\n",
        "pprint.pprint(normalizer.resolve(df.columns))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "18",
      "metadata": {},
      "source": [
        "You can avoid auto-detection entirely by using the `ResolvedColumn` class in place of synonyms in your `TableNormalizer`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "19",
      "metadata": {},
      "outputs": [],
      "source": [
        "resolved_normalizer = TableNormalizer(\n",
        "    test_time_second=[DateTimeSyn(syn=Syn(hdr=\"Test-Time\"), fmts=(\"%H:%M:%S.%f\",))],\n",
        "    voltage_volt=[Syn(hdr=\"Voltage-{unit}\")],\n",
        "    current_ampere=ResolvedColumn(source_header=\"Current-mA\", scale=0.001),\n",
        ")\n",
        "print(\"Headers:\", df.columns)\n",
        "print(\"Score:\", resolved_normalizer.score_columns(df.columns))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "20",
      "metadata": {},
      "source": [
        "## Returning the normalized data\n",
        "The `TableNormalizer.normalize()` method will normalize the columns into the correct unit, returning a bdf-compliant dataframe:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "21",
      "metadata": {},
      "outputs": [],
      "source": [
        "normalizer.normalize(df)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "22",
      "metadata": {},
      "source": [
        "## Initialising a TableNormalizer from a schema file\n",
        "Since `TableNormalizer` is a Pydantic model, it can be constructed from a plain dict (e.g. loaded from a JSON config file). Each field accepts either a list of synonym strings / `DateTimeSyn` dicts, or a `ResolvedColumn` dict."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "23",
      "metadata": {},
      "outputs": [],
      "source": [
        "json_normalizer = TableNormalizer.model_validate(\n",
        "    {\n",
        "        \"test_time_second\": [{\"syn\": \"Test-Time\", \"fmts\": [\"%H:%M:%S.%f\"]}],\n",
        "        \"voltage_volt\": [\"Voltage-{unit}\"],\n",
        "        \"current_ampere\": [\"Current-{unit}\"],\n",
        "    }\n",
        ")\n",
        "print(\"Score:\", json_normalizer.score_columns(df.columns))"
      ]
    }
  ],
  "metadata": {
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}