{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "0",
      "metadata": {},
      "source": [
        "# Ingest raw data into BDF\n",
        "\n",
        "This notebook demonstrates `bdf.ingest`, which:\n",
        "\n",
        "- converts raw vendor files into BDF artifacts\n",
        "- validates existing BDF files\n",
        "- writes outputs to parquet by default (csv optional)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1",
      "metadata": {},
      "outputs": [],
      "source": [
        "from pathlib import Path\n",
        "import bdf"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2",
      "metadata": {},
      "source": [
        "## Pick a source directory\n",
        "\n",
        "By default this uses the local `data/` folder in the repo."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3",
      "metadata": {},
      "outputs": [],
      "source": [
        "source_dir = Path(\"./in/ingest/example_raw_data_collection/g20m7\")\n",
        "source_dir.exists()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4",
      "metadata": {},
      "source": [
        "## Run ingest\n",
        "\n",
        "The ingest function converts all the raw files in a directory and serializes them as .bdf in a subdirectory data/ . It also reads the metadata files to serialize RDF linked data descriptions. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5",
      "metadata": {},
      "outputs": [],
      "source": [
        "summary = bdf.ingest(\n",
        "    source_dir,\n",
        "    format=\"parquet\",\n",
        "    recursive=True,\n",
        "    validate_existing=True,\n",
        "    validate_converted=True,\n",
        "    raise_on_error=False,\n",
        "    layout=\"nested\"\n",
        ")\n",
        "summary\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6",
      "metadata": {},
      "source": [
        "## Inspect results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7",
      "metadata": {},
      "outputs": [],
      "source": [
        "print(\"converted:\", len(summary.get(\"converted\", [])))\n",
        "print(\"validated:\", len(summary.get(\"validated\", [])))\n",
        "print(\"failed:\", len(summary.get(\"failed\", [])))\n",
        "print(\"skipped:\", len(summary.get(\"skipped\", [])))\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8",
      "metadata": {},
      "outputs": [],
      "source": [
        "summary.get(\"failed\", [])[:5]\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9",
      "metadata": {},
      "source": [
        "## Ingest a repository\n",
        "\n",
        "Use `bdf.ingest` with `discover_collections=True` to ingest every folder\n",
        "that contains `contribution.json`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "10",
      "metadata": {},
      "outputs": [],
      "source": [
        "repo_root = Path(\"./in/ingest/example_raw_data_collection\")\n",
        "repo_summary = bdf.ingest(repo_root, layout=\"nested\", format=\"parquet\", discover_collections=True)\n",
        "repo_summary\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "11",
      "metadata": {},
      "source": "## Notes\n\n- Use `format=\"csv\"` if you need `.bdf.csv` output instead of parquet.\n- CSV ingest writes machine headers (`skos:notation`) by default. Pass `labels=\"preferred\"` to write `prefLabel` headers.\n"
    }
  ],
  "metadata": {
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.10"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}