Skip to content

How to perform prediction with GreatAI#

After creating a GreatAI service by wrapping your prediction function, and optionally configuring it, it's time to do some prediction.

Let's take the following example:

greeter.py
from great_ai import GreatAI

@GreatAI.create
def greeter(your_name: str) -> str:
    return f'Hi {your_name}!'

One-off prediction#

Even though greeter is now an instance of GreatAI, you can continue using it as a regular function.

>>> greeter('Bob')
Trace[str]({'created': '2022-07-11T14:31:46.183764',
  'exception': None,
  'feedback': None,
  'logged_values': {'arg:your_name:length': 3, 'arg:your_name:value': 'Bob'},
  'models': [],
  'original_execution_time_ms': 0.0381,
  'output': 'Hi Bob!',
  'tags': ['greeter', 'online', 'development'],
  'trace_id': '7c284fd7-7f0d-4464-b5f8-3ef126df34af'})

As you can see, the original return value is wrapped in a Trace object (which is also persisted in your database of choice). You can access the original value under the output property.

Repeated inputs are cached by default and reuse their trace ID. Each call returns an independent copy, so changing a returned output does not change later results. Feedback updates are reflected in subsequent calls; deleting a persisted trace causes the next prediction for that input to run again and receive a new trace ID. Set configure(prediction_cache_size=0) before creating the service to disable caching.

NumPy values#

The Python API accepts NumPy arrays and scalars. Array cache keys include their contents, shape, and dtype. Outputs retain their NumPy types in Python; traces stored in the database and HTTP responses serialize them as JSON arrays and scalars. Unsupported objects must be converted to JSON-compatible values before logging or returning them.

An np.ndarray annotation exposes a JSON array in the HTTP API and converts the incoming list to an array before calling your function:

import numpy as np
from great_ai import GreatAI

@GreatAI.create
def double(values: np.ndarray) -> np.ndarray:
    return values * 2

assert double(np.array([1, 2])).output.tolist() == [2, 4]

For this service, send {"values": [1, 2]} to /predict. Return annotations, including Pydantic models, are included in the prediction response's OpenAPI schema. Custom validation failures from @parameter return HTTP 422.

Online prediction#

Likely, the main way you would like to expose your model is through an HTTP API. @GreatAI.create scaffolds many REST API endpoints for your model and creates a FastAPI app available under GreatAI.app. This can be served using uvicorn or any other ASGI server.

Since most ML code lives in Jupyter notebooks, therefore, deploying a notebook containing the inference function is supported. To achieve this, uvicorn is wrapped by the great-ai command-line utility, which — among others — takes care of feeding a notebook into uvicorn. It also supports auto-reloading.

In development#

great-ai greeter.py

Success

Your model is accessible at localhost:6060.

Some configuration options are also supported.

great-ai greeter.py --port 8000 --host 127.0.0.1 --timeout_keep_alive 10
More options

For more options (but no Notebook support), simply use uvicorn for starting your app (available at greeter.app).

In production#

There are three main approaches for deploying a GreatAI service.

Manual deployment#

The app is run in production-mode if the value of the ENVIRONMENT environment variable is set to production.

ENVIRONMENT=production great-ai greeter.py

Simply run ENVIRONMENT=production great-ai deploy.ipynb in the command-line of a production machine.

This is the crudest approach; however, it might be fitting for some contexts.

Containerised deployment#

Run the notebook directly in a container or create a service for it using your favourite container orchestrator.

docker run -p 6060:6060 --volume `pwd`:/app --rm \
  git.schmelczer.dev/andras/great-ai:latest deploy.ipynb

You can replace pwd with the path to your code's folder.

Use a Platform-as-a-Service#

Similar to the previous approach, your code will run in a container. However, instead of manually managing it, you can just choose from a plethora of PaaS providers (such as AWS ECS, DO App platform, MLEM, Streamlit) that take a Docker image as a source and handle the rest of the deployment.

To this end, you can also create a custom Docker image. It is especially useful if you have third-party dependencies, such as PyTorch or TensorFlow.

# syntax=docker/dockerfile:1
FROM git.schmelczer.dev/andras/great-ai:latest

COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

# Remove this block if you don't have a requirements.txt
COPY requirements.txt ./   
RUN uv pip install --system --no-cache --requirement requirements.txt

# Cache an S3 model using credentials available only during this RUN.
# Remove this step if your application does not use a remote model.
RUN --mount=type=secret,id=s3,target=/run/secrets/s3.ini,required=true \
    large-file --backend s3 --secrets /run/secrets/s3.ini --cache my-domain-predictor

# Add your application code to the image
COPY . .

# The default ENTRYPOINT is great-ai; specify its argument using CMD
CMD ["deploy.ipynb"]

Create an s3.ini with the S3 keys described in the configuration guide. Use concrete values in this build configuration: ENV: references are resolved inside the build container, which does not inherit your shell's environment. Replace my-domain-predictor with the key used when saving your model.

Keep credentials and any local model cache out of COPY . . by adding these entries to your project's .dockerignore:

s3.ini
mongo.ini
.env
.cache

Build with Docker BuildKit secrets. This example disables Docker's layer cache so that the large-file --cache step runs and downloads the model into the image:

DOCKER_BUILDKIT=1 docker build --no-cache --progress=plain \
  --secret id=s3,src=s3.ini --tag my-great-ai-service .

The secret mount is removed after the RUN instruction; the downloaded model cache remains in the image. Supply any configuration needed by the application separately at runtime. For example, an application that auto-configures S3 from s3.ini can use a read-only bind mount:

docker run --rm -p 6060:6060 \
  --mount type=bind,source="$(pwd)/s3.ini",target=/app/s3.ini,readonly \
  my-great-ai-service

Batch prediction#

Processing larger amounts of data on a single machine is made easy by the GreatAI's process_batch method. This relies on multiprocessing (parallel_map) to take full advantage of all available CPU-cores.

>>> greeter.process_batch(['Alice', 'Bob'])
[Trace[str]({'created': '2022-07-11T14:36:37.119183',
   'exception': None,
   'feedback': None,
   'logged_values': {'arg:your_name:length': 5, 'arg:your_name:value': 'Alice'},
   'models': [],
   'original_execution_time_ms':  0.1251,
   'output': 'Hi Alice!',
   'tags': ['greeter', 'online', 'development'],
   'trace_id': '90ffa15f-e839-41c4-8e7a-3211168bc138'}),
 Trace[str]({'created': '2022-07-11T14:36:37.166659',
   'exception': None,
   'feedback': None,
   'logged_values': {'arg:your_name:length': 3, 'arg:your_name:value': 'Bob'},
   'models': [],
   'original_execution_time_ms':  0.0571,
   'output': 'Hi Bob!',
   'tags': ['greeter', 'online', 'development'],
   'trace_id': 'f48e94c7-0815-48b3-a864-41349d3dae84'})]