{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0",
   "metadata": {},
   "source": [
    "# Simple example: data engineering\n",
    "\n",
    "Train a scientific domain classifier with an explainable Naive Bayes model.\n",
    "Run `data.ipynb`, `train.ipynb`, then `deploy.ipynb` in that order.\n",
    "\n",
    "The original example downloaded large Semantic Scholar corpus shards.\n",
    "For a reproducible local run, we use the bundled MAG sentences instead,\n",
    "wrapping each domain in a list to keep the same multilabel data interface.\n",
    "By default we take 60 examples per class; `GREAT_AI_NOTEBOOK_FULL=1` uses\n",
    "the entire bundled training split. These results are not the original\n",
    "Semantic Scholar experiment's metrics.\n",
    "\n",
    "### Extract"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0df47c23",
   "metadata": {},
   "source": [
    "## Local setup\n",
    "\n",
    "Use Python 3.12 or 3.13 and install this checkout with `uv pip install --editable '.[notebooks]'`.\n",
    "See **Run notebooks locally** in the repository README for the kernel, execution order,\n",
    "and full-data options. Run related notebooks from the same working directory\n",
    "to share the default `.cache/` model store and `tracing_database.json` database.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a800dc4d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-28T22:47:31.221616Z",
     "iopub.status.busy": "2026-10-28T22:47:31.221419Z",
     "iopub.status.idle": "2026-10-28T22:47:31.227624Z",
     "shell.execute_reply": "2026-10-28T22:47:31.226922Z"
    }
   },
   "outputs": [],
   "source": [
    "from docs.notebook_support import DATA_DIR, FULL_RUN, mag_lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cell-3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-28T22:47:31.229286Z",
     "iopub.status.busy": "2026-10-28T22:47:31.229129Z",
     "iopub.status.idle": "2026-10-28T22:47:31.231400Z",
     "shell.execute_reply": "2026-10-28T22:47:31.230864Z"
    }
   },
   "outputs": [],
   "source": [
    "PER_LABEL = None if FULL_RUN else 60"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cell-4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-28T22:47:31.232556Z",
     "iopub.status.busy": "2026-10-28T22:47:31.232454Z",
     "iopub.status.idle": "2026-10-28T22:47:31.318875Z",
     "shell.execute_reply": "2026-10-28T22:47:31.318472Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 420 bundled MAG sentences\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "records = [json.loads(line) for line in mag_lines(DATA_DIR / \"train.txt\", PER_LABEL)]\n",
    "print(f\"Loaded {len(records)} bundled MAG sentences\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-5",
   "metadata": {},
   "source": [
    "### Transform\n",
    "\n",
    "Clean the text with `great_ai.utilities.clean` and keep a list of accepted\n",
    "labels for each publication. Duplicate texts are combined before splitting\n",
    "to keep every copy of a paper in the same split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cell-6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-28T22:47:31.319705Z",
     "iopub.status.busy": "2026-10-28T22:47:31.319649Z",
     "iopub.status.idle": "2026-10-28T22:47:32.655835Z",
     "shell.execute_reply": "2026-10-28T22:47:32.655342Z"
    }
   },
   "outputs": [],
   "source": [
    "from great_ai.utilities import clean\n",
    "\n",
    "labels_by_text = {}\n",
    "for record in records:\n",
    "    text = clean(record[\"text\"], convert_to_ascii=True)\n",
    "    labels_by_text.setdefault(text, set()).add(record[\"label\"])\n",
    "preprocessed_data = [(text, sorted(labels)) for text, labels in labels_by_text.items()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cell-7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-28T22:47:32.656875Z",
     "iopub.status.busy": "2026-10-28T22:47:32.656706Z",
     "iopub.status.idle": "2026-10-28T22:47:32.658407Z",
     "shell.execute_reply": "2026-10-28T22:47:32.657982Z"
    }
   },
   "outputs": [],
   "source": [
    "X, y = zip(*preprocessed_data)  # X is the input, y is the expected output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-8",
   "metadata": {},
   "source": [
    "### Load\n",
    "\n",
    "Store the dataset in the default local TinyDB tracing database,\n",
    "`tracing_database.json`. Run the data, training and deployment notebooks\n",
    "from the same working directory to share it. A dataset tag makes rerunning\n",
    "this cell replace just this example's ground truth instead of duplicating it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cell-9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-10-28T22:47:32.659171Z",
     "iopub.status.busy": "2026-10-28T22:47:32.659099Z",
     "iopub.status.idle": "2026-10-28T22:47:32.670028Z",
     "shell.execute_reply": "2026-10-28T22:47:32.669511Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;226mEnvironment variable ENVIRONMENT is not set, defaulting to development mode ‼️\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;226mCannot find credentials files, defaulting to using ParallelTinyDbDriver\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;226mThe selected tracing database (ParallelTinyDbDriver) is not recommended for production\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;226mCannot find credentials files, defaulting to using LargeFileLocal\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;39mGreatAI (v0.1.11): configured ✅\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;39m  🔩 tracing_database: ParallelTinyDbDriver\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;39m  🔩 large_file_implementation: LargeFileLocal\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;39m  🔩 is_production: False\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;39m  🔩 should_log_exception_stack: True\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;39m  🔩 prediction_cache_size: 512\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;39m  🔩 dashboard_table_size: 50\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;226mYou still need to check whether you follow all best practices before trusting your deployment.\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;226m> Find out more at https://se-ml.github.io/practices\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "from great_ai import add_ground_truth, delete_ground_truth\n",
    "\n",
    "random.seed(42)\n",
    "delete_ground_truth(\"local-domain-example\")\n",
    "add_ground_truth(\n",
    "    X, y, tags=\"local-domain-example\", train_split_ratio=0.8, test_split_ratio=0.2\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-10",
   "metadata": {},
   "source": [
    "### Next: [Part 2](https://great-ai.schmelczer.dev/examples/simple/train)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv (3.13.5)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}