Foundation models for medical imaging are improving fast, but scoring well on a benchmark and understanding what a model actually learned are two different things. We built a single notebook that loads Jolia's embeddings into FiftyOne and shows that gap closing in practice, from a UMAP of whole-scan embeddings down to organ-level similarity search and attention maps.

Key takeaways

• Jolia produces 102 named per-organ embeddings plus one global CT embedding from a single forward pass, with no segmentation masks or spatial supervision required.
• FiftyOne visualizes Jolia's embeddings in an interactive UMAP, runs per-organ similarity search on them, and renders them as attention heatmaps on the CT scan.
• On the same chest CT scan, searching by the lung embedding versus the liver embedding returns two different sets of neighbor scans. This shows Jolia's per-organ embeddings are independent views of the scan, not copies of one global vector.
• The Jolia and FiftyOne demo runs entirely on a laptop CPU or Apple Silicon MPS, since Jolia has only about 22M parameters.

Jolia: Per-Organ Understanding, No Segmentation Masks Required.

Jolia is a 3D CT foundation model trained on paired chest and abdominal scans and their radiology reports. Where a standard CLIP-style model crushes an entire scan and an entire report into one global vector each — and loses the liver in the process — Jolia does something cleverer.

Its training method, ConQuer (Concept Queries), augments the global alignment with a set of per-concept alignments: one learnable query per anatomical region. An LLM splits each report into organ-by-organ sections, and on the image side a small set of cross-attention queries learns to pool the matching features — with no segmentation masks and no spatial supervision at all. Each query simply learns where to look through a per-organ contrastive loss.

The payoff: from a single forward pass, Jolia emits a global [CLS] embedding and 102 named per-organ embeddings. It sets a new state of the art on findings classification, cross-center transfer, and abdominal report generation — and, as a free byproduct, the queries produce anatomically coherent attention maps that show where the model looked for each organ. It's a research preview, not a medical device, but as an object of study it's a gift.

FiftyOne: From Metrics to Embeddings You Can Explore.

FiftyOne handles the embedding-to-insight pipeline (UMAP/t-SNE projections, similarity indexes) with a few lines of code, and “saved views” let you bookmark any slice of your data so it's one click away in the app. It's the difference between "the model gets 84 AUROC" and "here are the exact scans it gets wrong, and here's the report text explaining why."

How FiftyOne Turns Jolia's Embeddings Into a Map.

Jolia doesn't diagnose; it produces embeddings. FiftyOne is built to explore embeddings. Use the model exactly as intended and let the tool do what it's best at. But the real magic is the per-organ structure. Most CT foundation models can only answer "which whole scans look alike?" Jolia can answer "which scans have a lung like this one?" — and FiftyOne's similarity search turns that into a button.

The most convincing part is what happens when you swap the organ. Index a chest scan by its lung embedding and the nearest neighbors share lung pathology. Now re-sort the same scan by its liver embedding — and the neighbor set scatters, because on a chest CT the liver is barely in the field of view. That scatter is the proof: each organ query is a genuinely independent view of the scan, not a relabeling of one global vector. You can see the concept queries doing their job — including where they don't.

Layer the organ saliency maps on as FiftyOne heatmap overlays and you get honest interpretability — including the cases where the pancreas query smears into its neighbors, small and low-contrast as it is. Nothing cherry-picked. Together, they turn a strong-but-opaque model into something you can actually reason about.

Note: the demo runs on a laptop (CPU or Apple Silicon MPS) since Jolia is only ~22M parameters.

Demo: Exploring Jolia's Embeddings in FiftyOne.

Embedding Atlas — a UMAP of Jolia's global [CLS] embedding, colored by finding. Lasso a cluster and watch the matching scans light up. Does the representation organize by pathology? (Spoiler: yes.)

Per-Organ Similarity (lungs) — index scans by a single organ's embedding and search "find me lungs like this one." Segmentation-free, and unique to Jolia. Because CT-RATE is a chest dataset, we use the lungs as the hero organ: they dominate the scan, so the returned neighbors both rank right and look right in the thumbnail.

Organ Contrast (lungs vs liver) — the same probe scan, re-sorted by its liver embedding. The neighbors scatter, because the liver is barely in a chest scan's frame. Same scan, different organ, different neighbors — the independence of the concept queries, made visible. (The true liver-similarity demo belongs on abdominal CT, where the liver fills the frame; the notebook shows the one-line swap to Merlin-Abd-CT.)

Attention Overlays (Concept-Query Saliency Maps). — render an organ saliency map on the CT for each concept query. Show the clean lung map, then the deliberately-tricky pancreas one. (If your Jolia build exposes the raw concept-query attention, the notebook uses it directly; otherwise it falls back to an occlusion-saliency estimate — where masking a region and measuring how much the organ embedding shifts stands in for "where the query looks.")

Findings Probe Explorer — train a quick linear probe on the frozen [CLS] + lungs feature, write predictions back, and filter to the false positives and false negatives — reading each miss's report text to ask: model error, or noisy label?

Try Jolia and FiftyOne Yourself.

Grab the pieces and try it yourself:

Where to take it from here: scale past the default subset toward CT-RATE's full validation set; bring in Merlin-Abd-CT to exercise Jolia's 172-finding abdominal taxonomy and abdominal organ queries; wire in a second clinical center to reproduce the paper's cross-center transfer story; or swap the occlusion-fallback attention for the model's true concept-query weights if your build exposes them.

Jolia is a research preview and not for clinical use. CT-RATE labels are model-extracted from reports, not clinical ground truth — which is exactly why an inspection tool like FiftyOne earns its keep.

FAQ

What is Jolia?

Jolia is a 3D CT vision-language foundation model built by Raidium. It's trained on paired chest and abdominal CT scans and their radiology reports using ConQuer (Concept Queries), a method that learns one query per anatomical region so the model can align each organ with the matching part of the report, without segmentation masks.

What is FiftyOne?

FiftyOne is Voxel51's open source tool for building datasets and understanding visual AI models. In this project, it loads Jolia's CT embeddings into an interactive app for UMAP visualization, per-organ similarity search, and attention heatmap overlays.

How many embeddings does Jolia produce per CT scan?

Jolia produces one global [CLS] embedding plus 102 named per-organ embeddings from a single forward pass.

What dataset was used in the Jolia and FiftyOne demo?

The demo uses real chest CT scans from the public CT-RATE dataset on Hugging Face, which is gated and requires accepting its terms before download.

Can Jolia be used for clinical diagnosis?

No. Jolia is a research preview, not a medical device. The CT-RATE labels used in this demo are also model-extracted from radiology reports, not clinical ground truth.