Explore data and feasibility of approach¶

The original annotations are proprietary. This notebook defaults to 60 clearly synthetic sentences with two simulated annotators so the entire training/deployment workflow can run locally without credentials. These data and their scores demonstrate execution only.

To use your own JSON annotation files (each maps sentences to binary labels), set GREAT_AI_ANNOTATIONS_DIR to their directory before starting Jupyter. No private data or cloud configuration is included in this repo.

Local setup¶

Use Python 3.12 or 3.13 and install this checkout with uv pip install --editable '.[notebooks]'. See Run notebooks locally in the repository README for the kernel, execution order, and full-data options. Run related notebooks from the same working directory to share the default .cache/ model store and tracing_database.json database.

In [1]:

import json
import os
from pathlib import Path

from docs.notebook_support import demo_annotations

annotation_dir = os.environ.get("GREAT_AI_ANNOTATIONS_DIR")
if annotation_dir:
    annotations = [
        json.loads(p.read_text(encoding="utf-8"))
        for p in sorted(Path(annotation_dir).expanduser().glob("*.json"))
    ]
    if not annotations:
        raise ValueError(f"No annotation JSON files found in {annotation_dir}")
else:
    annotations = demo_annotations()
    print("Using synthetic demo annotations; scores are not research results.")

evaluations = {
    sentence: [
        annotation[sentence] for annotation in annotations if sentence in annotation
    ]
    for sentence in sorted(
        {sentence for annotation in annotations for sentence in annotation}
    )
}
X = list(evaluations)
y = [int(sum(e) > 0) for e in evaluations.values()]

import json import os from pathlib import Path from docs.notebook_support import demo_annotations annotation_dir = os.environ.get("GREAT_AI_ANNOTATIONS_DIR") if annotation_dir: annotations = [ json.loads(p.read_text(encoding="utf-8")) for p in sorted(Path(annotation_dir).expanduser().glob("*.json")) ] if not annotations: raise ValueError(f"No annotation JSON files found in {annotation_dir}") else: annotations = demo_annotations() print("Using synthetic demo annotations; scores are not research results.") evaluations = { sentence: [ annotation[sentence] for annotation in annotations if sentence in annotation ] for sentence in sorted( {sentence for annotation in annotations for sentence in annotation} ) } X = list(evaluations) y = [int(sum(e) > 0) for e in evaluations.values()]

Using synthetic demo annotations; scores are not research results.

Save the processed dataset using local versioned storage for the training notebook.

In [2]:

from great_ai.large_file import LargeFileLocal

with LargeFileLocal("summary-train-dataset-small", "w", encoding="utf-8") as f:
    json.dump((X, y), f)

from great_ai.large_file import LargeFileLocal with LargeFileLocal("summary-train-dataset-small", "w", encoding="utf-8") as f: json.dump((X, y), f)

Fetching cached versions of summary-train-dataset-small

Fetching cached versions of summary-train-dataset-small

Fetching cached versions of summary-train-dataset-small

Filter out sentences which don't have enough annotations.

In [3]:

y1 = [e[0] for e in evaluations.values() if len(e) == 2]
y2 = [e[1] for e in evaluations.values() if len(e) == 2]

y1 = [e[0] for e in evaluations.values() if len(e) == 2] y2 = [e[1] for e in evaluations.values() if len(e) == 2]

Calculate Cohen's kappa.

It's a bit low but the task itself is pretty subjective so it's not all that surprising.

In [4]:

import sklearn.metrics

if y1:
    print(sklearn.metrics.cohen_kappa_score(y1, y2))
else:
    print("No sentences have exactly two annotations; kappa is unavailable.")

import sklearn.metrics if y1: print(sklearn.metrics.cohen_kappa_score(y1, y2)) else: print("No sentences have exactly two annotations; kappa is unavailable.")

0.8

Can we train anything on this data?

Let's try with a trivial SVM.

In [5]:

X = [s for s in evaluations.keys()]
y = [int(sum(e) > 0) for e in evaluations.values()]

X = [s for s in evaluations.keys()] y = [int(sum(e) > 0) for e in evaluations.values()]

In [6]:

from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import cross_val_score

model = Pipeline(
    steps=[
        ("vectorizer", TfidfVectorizer(sublinear_tf=True, min_df=3, max_df=0.3)),
        ("classifier", LinearSVC(random_state=42)),
    ]
)  # baseline model

cross_val_score(model, X, y, cv=5)

from sklearn.model_selection import train_test_split from sklearn.svm import LinearSVC from sklearn.pipeline import Pipeline from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.model_selection import cross_val_score model = Pipeline( steps=[ ("vectorizer", TfidfVectorizer(sublinear_tf=True, min_df=3, max_df=0.3)), ("classifier", LinearSVC(random_state=42)), ] ) # baseline model cross_val_score(model, X, y, cv=5)

Out[6]:

array([0.41666667, 0.25      , 0.16666667, 0.41666667, 0.5       ])

The cross-validation shows promising accuracies. But accuracy isn't everything, therefore, we should investigate the accuracy metrics.

In [7]:

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
model.fit(X_train, y_train)

X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y ) model.fit(X_train, y_train)

Out[7]:

Pipeline(steps=[('vectorizer',
                 TfidfVectorizer(max_df=0.3, min_df=3, sublinear_tf=True)),
                ('classifier', LinearSVC(random_state=42))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

In [8]:

y_predicted = model.predict(X_test)
print(
    sklearn.metrics.classification_report(
        [y > 0 for y in y_test], [y > 0 for y in y_predicted], zero_division=0
    )
)
sklearn.metrics.ConfusionMatrixDisplay.from_predictions(
    [y > 0 for y in y_test],
    [y > 0 for y in y_predicted],
    xticks_rotation="vertical",
    values_format=".2f",
)
None

y_predicted = model.predict(X_test) print( sklearn.metrics.classification_report( [y > 0 for y in y_test], [y > 0 for y in y_predicted], zero_division=0 ) ) sklearn.metrics.ConfusionMatrixDisplay.from_predictions( [y > 0 for y in y_test], [y > 0 for y in y_predicted], xticks_rotation="vertical", values_format=".2f", ) None

              precision    recall  f1-score   support

       False       0.00      0.00      0.00         5
        True       0.58      1.00      0.74         7

    accuracy                           0.58        12
   macro avg       0.29      0.50      0.37        12
weighted avg       0.34      0.58      0.43        12

The report above describes the active dataset. The synthetic demo cannot establish the feasibility or quality of a real summarisation system.

Next: Part 2