What are onnxruntime-web and transformers.js

Our website runs on two important open source libraries, onnxruntime-web and transformers.js. They are the key to run models in browser.

ONNX model format

ONNX is an open format for representing machine-learning models as computation graphs. A graph is made of nodes connected by data flow: each node applies an operator (for example, a matrix multiplication or activation function) to input tensors and produces output tensors for later nodes. The graph also describes its inputs, outputs, data types, and other metadata. Model weights are stored as tensor data, either inside the ONNX file or in external data files. The graph is therefore more than a list of calculations: its connections define the order and dependencies of the work. See the ONNX project overview¹ and ONNX IR specification².

onnxruntime-web

ONNX Runtime Web is a JavaScript runtime that loads ONNX graphs, creates an inference session, and executes the graph in a browser. It provides execution providers (EPs) that run supported operators on different browser APIs: WebAssembly (WASM) runs on the CPU, while WebGPU can use the GPU. WebGL is also available but is in maintenance mode. WASM can use multiple threads when browser security and runtime configuration allow it; this is separate from choosing a GPU provider. The model still needs compatible operators, tensor types, and shapes for the selected provider. See the ONNX Runtime Web guide³ and WebGPU tutorial⁴.

Choosing a backend and handling fallback

There is no single browser-independent provider order that an application should assume. The app can inspect capabilities such as navigator.gpu, then request an available provider in InferenceSession.create(). For example, executionProviders: ['webgpu', 'wasm'] expresses a preference list; it does not mean every WebGPU failure will transparently retry the whole session on WASM. Provider availability and operator support vary by browser, operating system, device, and model. An application that needs a reliable CPU path should handle session-creation or inference failures and explicitly retry with a WASM session where appropriate. A graph may also be partitioned so supported operators use an accelerated provider while other operators use a supported fallback, but this depends on the providers and model. Consult the session options guide⁵ and current Web provider support matrix⁶.

transformers.js

Transformers.js is a higher-level JavaScript library inspired by Hugging Face's Python transformers package. It uses ONNX Runtime for inference in supported browser and Node.js setups; it does not use PyTorch as its browser inference backend. It adds task-oriented APIs such as pipeline(), model and tokenizer loading, and common input preprocessing and output postprocessing. A pipeline can make a pretrained model easier to use, while direct ONNX Runtime Web gives the application more control over session creation, tensors, and model-specific processing. See the Transformers.js overview⁷ and pipeline API⁸.

Comparison	Python transformers	Transformers.js
Runtime environments	Python; servers and desktop environments	Browsers, Node.js, and Electron
Main inference runtime	Primarily PyTorch	ONNX Runtime in supported configurations
Runs directly in a browser	❌	✅
WebGPU	Not its primary execution path	Supported for accelerated inference where available
WASM	Not usually used as its main backend	Available through ONNX Runtime Web
CUDA	Supported through compatible PyTorch setups	Depends on the runtime and environment; browser inference does not use CUDA directly
Model training	Full training ecosystem	Primarily an inference library
Fine-tuning	Supported by the Python ecosystem	Not a general-purpose fine-tuning framework
Trainer	Available	Not included
Model coverage	Broad ecosystem; new architectures often appear here first	A subset of models, with compatible ONNX weights and JavaScript-side support required
Custom model code	Flexible Python model implementations	More constrained by supported architectures and ONNX export/runtime support
Model formats	PyTorch, Safetensors, and others	Commonly ONNX for inference
Deployment	Requires a suitable Python runtime and dependencies	Can run in a statically hosted web app when assets and runtime are configured for it
Privacy	Depends on the application architecture	Can run locally in the browser; the app must still avoid sending user data elsewhere

An example of using onnxruntime-web directly

This example uses the real SqueezeNet 1.1 ONNX model from the ONNX Model Zoo. It classifies a browser image into one of ImageNet's 1,000 classes. The model expects an RGB tensor shaped [1, 3, 224, 224], with values scaled to [0, 1] and normalized using ImageNet channel means and standard deviations.⁹

import * as ort from 'onnxruntime-web';

const modelUrl = 'https://huggingface.co/onnxmodelzoo/squeezenet1.1-7/resolve/61e525224ad479521059f4586bcacf50ad3627ca/squeezenet1.1-7.onnx';
const session = await ort.InferenceSession.create(modelUrl, {
  executionProviders: ['webgpu', 'wasm'],
});

// `image` is a loaded HTMLImageElement. Center-crop it to square, then resize to 224px.
const canvas = document.createElement('canvas');
canvas.width = canvas.height = 224;
const ctx = canvas.getContext('2d', { willReadFrequently: true });
const side = Math.min(image.naturalWidth, image.naturalHeight);
ctx.drawImage(image,
  (image.naturalWidth - side) / 2, (image.naturalHeight - side) / 2, side, side,
  0, 0, 224, 224,
);
const { data: pixels } = ctx.getImageData(0, 0, 224, 224);
const mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225];
const values = new Float32Array(3 * 224 * 224);
for (let i = 0; i < 224 * 224; i++) {
  for (let c = 0; c < 3; c++) {
    values[c * 224 * 224 + i] = (pixels[i * 4 + c] / 255 - mean[c]) / std[c];
  }
}

const input = new ort.Tensor('float32', values, [1, 3, 224, 224]);
const outputs = await session.run({ [session.inputNames[0]]: input });
const scores = outputs[session.outputNames[0]].data;
const bestClass = scores.indexOf(Math.max(...scores));
console.log(`ImageNet class index: ${bestClass}, score: ${scores[bestClass]}`);

await session.release();

This reports the ImageNet class index; displaying a human-readable class name requires the matching 1,000-label list.

An example of using transformers.js

This example uses the real Xenova/vit-base-patch16-224 ONNX model for image classification. Transformers.js loads the model and its image processor, prepares the image, runs inference through ONNX Runtime, and returns human-readable labels with scores.¹⁰¹¹

import { pipeline } from '@huggingface/transformers';

const classifier = await pipeline(
  'image-classification',
  'Xenova/vit-base-patch16-224',
);
const results = await classifier(
  'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/tiger.jpg',
  { top_k: 3 },
);
console.log(results);
// [{ label: 'tiger, Panthera tigris', score: ... }, ...]

The first run downloads the model assets and caches them in the browser. The image URL is an example input; a local image can also be passed to the pipeline.

Choosing between onnxruntime-web and transformers.js。

Use onnxruntime-web directly when the application needs low-level control: custom preprocessing, exact tensor shapes, session options, provider selection, or a model-specific pipeline that the library does not provide. The app is responsible for preparing inputs, loading the correct ONNX files, decoding outputs, and managing the session.

Use Transformers.js when its supported task pipelines and model utilities fit the job. It can handle common model and tokenizer loading, preprocessing, and postprocessing behind a simpler API. That convenience comes with the library's supported model formats and task implementations, and does not remove the need to check model compatibility, asset downloads, or available browser backends.

References

Footnotes

1. ONNX, About ONNX. ↩

2. ONNX, IR (Intermediate Representation) Specification. ↩

3. ONNX Runtime, Get started with ONNX Runtime Web. ↩

4. ONNX Runtime, Using the WebGPU Execution Provider. ↩

5. ONNX Runtime, Environment Flags and Session Options. ↩

6. ONNX Runtime, ONNX Runtime Web supported versions and browsers. ↩

7. Hugging Face, Transformers.js documentation. ↩

8. Hugging Face, The pipeline API. ↩

9. ONNX Model Zoo, SqueezeNet 1.1 model card (model file, input shape, preprocessing, and output details). ↩

10. Hugging Face, Xenova/vit-base-patch16-224 model card. ↩

11. Hugging Face, Transformers.js image-classification pipeline. ↩