Simple example: data engineering¶

Train a scientific domain classifier with an explainable Naive Bayes model. Run data.ipynb, train.ipynb, then deploy.ipynb in that order.

The original example downloaded large Semantic Scholar corpus shards. For a reproducible local run, we use the bundled MAG sentences instead, wrapping each domain in a list to keep the same multilabel data interface. By default we take 60 examples per class; GREAT_AI_NOTEBOOK_FULL=1 uses the entire bundled training split. These results are not the original Semantic Scholar experiment's metrics.

Extract¶

Local setup¶

Use Python 3.12 or 3.13 and install this checkout with uv pip install --editable '.[notebooks]'. See Run notebooks locally in the repository README for the kernel, execution order, and full-data options. Run related notebooks from the same working directory to share the default .cache/ model store and tracing_database.json database.

In [1]:

from docs.notebook_support import DATA_DIR, FULL_RUN, mag_lines

from docs.notebook_support import DATA_DIR, FULL_RUN, mag_lines

In [2]:

PER_LABEL = None if FULL_RUN else 60

PER_LABEL = None if FULL_RUN else 60

In [3]:

import json

records = [json.loads(line) for line in mag_lines(DATA_DIR / "train.txt", PER_LABEL)]
print(f"Loaded {len(records)} bundled MAG sentences")

import json records = [json.loads(line) for line in mag_lines(DATA_DIR / "train.txt", PER_LABEL)] print(f"Loaded {len(records)} bundled MAG sentences")

Loaded 420 bundled MAG sentences

Transform¶

Clean the text with great_ai.utilities.clean and keep a list of accepted labels for each publication. Duplicate texts are combined before splitting to keep every copy of a paper in the same split.

In [4]:

from great_ai.utilities import clean

labels_by_text = {}
for record in records:
    text = clean(record["text"], convert_to_ascii=True)
    labels_by_text.setdefault(text, set()).add(record["label"])
preprocessed_data = [(text, sorted(labels)) for text, labels in labels_by_text.items()]

from great_ai.utilities import clean labels_by_text = {} for record in records: text = clean(record["text"], convert_to_ascii=True) labels_by_text.setdefault(text, set()).add(record["label"]) preprocessed_data = [(text, sorted(labels)) for text, labels in labels_by_text.items()]

In [5]:

X, y = zip(*preprocessed_data)  # X is the input, y is the expected output

X, y = zip(*preprocessed_data) # X is the input, y is the expected output

Load¶

Store the dataset in the default local TinyDB tracing database, tracing_database.json. Run the data, training and deployment notebooks from the same working directory to share it. A dataset tag makes rerunning this cell replace just this example's ground truth instead of duplicating it.

In [6]:

import random
from great_ai import add_ground_truth, delete_ground_truth

random.seed(42)
delete_ground_truth("local-domain-example")
add_ground_truth(
    X, y, tags="local-domain-example", train_split_ratio=0.8, test_split_ratio=0.2
)

import random from great_ai import add_ground_truth, delete_ground_truth random.seed(42) delete_ground_truth("local-domain-example") add_ground_truth( X, y, tags="local-domain-example", train_split_ratio=0.8, test_split_ratio=0.2 )

Environment variable ENVIRONMENT is not set, defaulting to development mode ‼️

Cannot find credentials files, defaulting to using ParallelTinyDbDriver

The selected tracing database (ParallelTinyDbDriver) is not recommended for production

Cannot find credentials files, defaulting to using LargeFileLocal

GreatAI (v0.1.11): configured ✅

  🔩 tracing_database: ParallelTinyDbDriver

  🔩 large_file_implementation: LargeFileLocal

  🔩 is_production: False

  🔩 should_log_exception_stack: True

  🔩 prediction_cache_size: 512

  🔩 dashboard_table_size: 50

You still need to check whether you follow all best practices before trusting your deployment.

> Find out more at https://se-ml.github.io/practices

Next: Part 2¶