Models · Semantic segmentation

OneFormer

OneFormer (SHI Labs): one network for several segmentation tasks, used here for ADE20K scene segmentation. Commercially usable.

Commercial use: yes.

variantinputbackendtime (ms)peak memory (MiB, allocated / reserved)
oneformer-swinl-ade20ksynthetic (1024x2048)original (fp32)
original
portable
kernels
portable + torch.compile
TensorRT fp16
307
205
139
100.0
fails: Inductor: Triton kernel does not compile (torch bug)
83.3
6853 / 7808
7395 / 7792
2054 / 2476
2054 / 2476
–
3258 (engine)
Details
  • Original code: https://github.com/huggingface/transformers at c4c4605f9618
  • Code licence: Apache-2.0 (permissive); files:
  • Weights licence: MIT
  • Licence evidence: https://huggingface.co/shi-labs/oneformer_ade20k_swin_large/blob/4a5bac8e64f82681a12db2e151a4c2f4ce6092b2/README.md
  • Training data: ImageNet-21k / ImageNet-22k (full ImageNet, fall 2011 release), ADE20K (SceneParsing, 150 classes)
  • Kernel libraries: fvm_seg, seg_up
  • Variants: oneformer-swinl-ade20k
  • Outputs: logits; label_set: ade20k-150
  • Input size: any H x W >= 1, resized or padded inside the model
  • Tested up to (portable, fp16, RTX 5060 Ti 16 GB): oneformer-swinl-ade20k 1988x2880 (8.4 GiB peak allocated, 14.9 reserved)
  • oneformer-swinl-ade20k 1024x2048, first call: TensorRT fp16 140 s (ONNX + engine).
  • oneformer-swinl-ade20k original 1024x2048: torch.autocast(fp16) around the forward
  • oneformer-swinl-ade20k load / first call: original 5.08 s / 0.805 s (512x512), portable 4.1 s / 0.147† s, kernels 3.27† s / 0.109† s
  • † measured in the speed sweep after the portable load (or an earlier size) in the same process: file cache and kernels are warm, so the time is lower than a fresh start.
  • Protocol, first call, 3 warm-up, peak reset, median of 3 rounds x median of 5 (tools/speed_check.py), synthetic CUDA input (bench_inputs); peak = max over the first call and the timed calls

Measured

Each image at its own size, argmax of the logits. Original = the original code named under Details, same checkpoint and protocol.

variantdatasetoriginal fp32original fp16portablekernelsnote
oneformer-swinl-ade20kADE20K val, every 4th image, mIoU / aAcc %54.49 / 85.0654.26 / 85.0554.37 / 85.0554.38 / 85.14

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 939adfe8, 2026-11-06.

Parity

CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.001 and p99 <= 0.01 (absolute error of the output logits).

variantkernels error vs original, mean / p99GPU, torchkernels used
oneformer-swinl-ade20k4e-05 / 0.00197RTX 5060 Ti, torch 2.14.1+cu130fvm_seg@739693dff3b4:sm_120, seg_up@3e44e71200cd:sm_120