Tutorial: train in PyTorch, ship in Talario

Goal: take a model you trained in PyTorch and serve it through Talario — portable across GPUs, reproducible CPU==GPU, in-process, with no model rewrite and no PHP. You stay in Python the whole way; Talario is the serving engine behind an HTTP endpoint.

Time: ~10 minutes. You need: Python + PyTorch on your machine, and a Talario build (./build.sh → modules/talario.so). See [Install(https://github.com/DMJ-CV-91913/talario/blob/main/docs/operations/building.md).

Step 1 — a trained PyTorch model#

Any nn.Module works. Here's a tiny MLP so the tutorial is self-contained:

# train.py
import torch, torch.nn as nn

model = nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10))
# ... your training loop here ...   (pretend it's trained)
model.eval()
torch.save(model.state_dict(), "mlp.pt")

Step 2 — export to ONNX#

This is the whole handoff. Talario imports ONNX and matches PyTorch to ~1e-7 on the supported op set.

# export.py
import torch, torch.nn as nn

model = nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10))
model.load_state_dict(torch.load("mlp.pt")); model.eval()

example = torch.randn(1, 784)
torch.onnx.export(
    model, example, "mlp.onnx",
    input_names=["x"], output_names=["logits"],
    opset_version=17, dynamic_axes=None,
)
print("wrote mlp.onnx")

python export.py        # -> mlp.onnx

Supported models today: dense/MLP and transformer-encoder graphs import cleanly (MatMul/Gemm, activations, LayerNorm, Softmax, Transpose, Reshape, Gather, …). CNN/RNN operators (Conv, MaxPool, BatchNorm, GRU/LSTM) are being added to the importer — if an export fails to import, that op isn't wired yet; see troubleshooting.

Step 3 — serve the ONNX with Talario#

One command. The model is imported once and stays warm; every request is a single submission.

php -d extension=modules/talario.so php/onnx_serve.php mlp.onnx cpu 127.0.0.1:5055 784
#                                                       ^onnx   ^dev ^addr          ^in-dim
# use 'gpu' to serve on any Vulkan GPU — same result, to the bit

Check it's up:

curl -s http://127.0.0.1:5055/health
# {"status":"ok","device":"Talario CPU (SIMD / Tessero)","in":784}

Step 4 — call it from Python#

Plain HTTP — call it like a local function:

# infer.py
import json, urllib.request

def infer(features):
    req = urllib.request.Request(
        "http://127.0.0.1:5055/infer",
        data=json.dumps(features).encode(),
        headers={"Content-Type": "application/json"},
    )
    return json.loads(urllib.request.urlopen(req).read())["output"]

x = [0.0] * 784                       # your real feature vector
logits = infer(x)
print(len(logits), logits[:5])

(When the talario pip client ships, this becomes TalarioClient("http://127.0.0.1:5055").infer(x).)

Step 5 — verify it matches PyTorch#

Reproducibility is the point — confirm Talario agrees with PyTorch, and that CPU and GPU agree:

# verify.py
import torch, torch.nn as nn, numpy as np
from infer import infer

model = nn.Sequential(nn.Linear(784,128), nn.ReLU(), nn.Linear(128,10))
model.load_state_dict(torch.load("mlp.pt")); model.eval()

x = np.random.randn(784).astype("float32")
torch_out = model(torch.from_numpy(x)[None]).detach().numpy()[0]
talario_out = np.array(infer(x.tolist()))
print("max|Δ| PyTorch vs Talario:", float(np.max(np.abs(torch_out - talario_out))))
# expect ~1e-6 or better

Re-run onnx_serve.php with gpu instead of cpu and call again — the logits are identical (bit-for-bit for exact ops), which is Talario's core guarantee: the model you validated on a CPU behaves identically on any GPU.

What you just did#

Trained in PyTorch → exported ONNX → served it through Talario on any GPU, reproducibly, in-process, callable from Python — with zero model rewriting and no PHP. That's the "ship" half of your stack replaced by a portable, deterministic engine; training stays exactly where it is.

No-PyTorch smoke (optional)#

To see the whole path without PyTorch installed, Talario can emit a real ONNX MLP itself:

php -d extension=modules/talario.so php/onnx_demo.php cpu     # emits + imports + checks an ONNX MLP

Troubleshooting#

Next: [serve in production / behind an endpoint(https://github.com/DMJ-CV-91913/talario/blob/main/docs/operations/serving.md) · [drive it from an LLM CLI via MCP(https://github.com/DMJ-CV-91913/talario/blob/main/docs/tutorials/mcp-and-llm-clis.md).

Edit this page on GitHub