Zero-Shot Learning Techniques: A Comprehensive Guide

A classifier trained on 1,000 product categories breaks the day the catalog adds category 1,001. Retraining needs labeled examples that do not exist yet, and labeling them takes weeks the business does not have. Zero-shot learning (ZSL) closes that gap. The model recognizes classes it never saw during training by reasoning over a description of each class instead of labeled examples. This guide covers the core zero-shot learning techniques, where teams use them today, and the infrastructure question most guides skip. Where do the embeddings live once the model leaves the notebook, and how do you query them as new classes keep arriving?

What Is Zero-Shot Learning?

Zero-shot learning is a machine learning approach in which a model classifies data from categories it never encountered during training, without any new labeled examples. Traditional supervised learning needs labeled samples for every class it predicts. Zero-shot learning replaces those samples with semantic information, such as attribute lists, text descriptions, or word embeddings, that relates unseen classes to seen ones.

A model trained on images of horses, tigers, and pandas has never seen a zebra. Given the description “horse-shaped, striped, black and white,” it maps the new image to a point in a shared semantic space near that description and labels it a zebra. This attribute-based approach traces to Lampert, Nickisch, and Harmeling’s CVPR 2009 paper, which introduced the Animals with Attributes benchmark.

How Zero-Shot Learning Works

Every zero-shot learning technique solves the same problem. It builds a bridge between classes the model has seen and classes it has not. Three families of techniques build that bridge: semantic embeddings, generative models, and transfer learning.

Semantic and visual-semantic embeddings

Semantic embedding methods map seen and unseen classes into one shared vector space. Class representations come from human-defined attributes (“has stripes,” “has hooves”) or from word vectors such as Word2Vec and GloVe. Related concepts land near each other, so the model can place an unseen class by its relationships to classes it already knows.

The DeViSE (Deep Visual-Semantic Embedding) model, published at NIPS 2013, is the reference example. DeViSE aligns visual features from an image network with word embeddings of class labels. It labels an image with a class it never saw during training by finding the nearest label vector. Visual-semantic embeddings generalize the pattern. A convolutional neural network (CNN) extracts image features, and a projection such as canonical correlation analysis (CCA) maps them into the text space.

Generative models: GANs and VAEs

Generative approaches flip the problem. Instead of projecting inputs into semantic space, they synthesize training examples for unseen classes from their descriptions, then train a standard classifier on real and synthetic data together.

• Generative adversarial networks (GANs). A generator creates synthetic samples and a discriminator judges whether they are real. Trained against each other, the pair learns to produce realistic features for a class from its description alone. Ian Goodfellow and colleagues introduced GANs in 2014.
• Variational autoencoders (VAEs). A VAE encodes inputs into a latent space and decodes them back, which forces generated samples to follow the learned distribution. VAEs train more stably than GANs and produce diverse samples that help the classifier cover the full range of an unseen class.

Transfer learning: domain adaptation and domain generalization

Transfer learning reuses a model pretrained on a large dataset and applies its knowledge to classes or domains it was not trained on.

• Domain adaptation. This approach transfers knowledge from a source domain to a related target domain with a different data distribution.
• Domain generalization. This approach trains on many diverse domains so the model performs on new ones without per-domain adaptation. It matters in production, where input data drifts.

How to evaluate zero-shot learning models

Accuracy on unseen classes alone overstates real-world performance. The generalized ZSL setting mixes seen and unseen classes at test time and exposes a known bias toward predicting seen classes. The standard metric is the harmonic mean of per-class accuracy on seen and unseen classes, which drops sharply if either side fails.

In 2018, researchers from Zhejiang University and Alibaba Group proposed quasi-fully supervised learning (QFSL) to reduce that bias. On the AwA2, CUB, and SUN benchmarks, QFSL outperformed prior methods by margins of 9.3% to 24.5% in the generalized ZSL setting (CVPR 2018).

Where Zero-Shot Learning Is Used Today

Zero-shot learning runs in production across computer vision, natural language processing, medical imaging, and autonomous driving. The common thread is a long tail of classes too rare or too new to label. The zero-shot learning examples below are named systems, not hypotheticals.

Image and video recognition. OpenAI’s CLIP, trained on 400 million image-text pairs, matched the accuracy of a fully supervised ResNet-50 on ImageNet zero-shot, without using any of ImageNet’s 1.28 million labeled training images.

Natural language processing. Zero-shot text classification frames each candidate label as a hypothesis for a natural language inference (NLI) model. Yin, Hay, and Roth formalized this approach in 2019, and NLI-trained models such as facebook/bart-large-mnli on Hugging Face now let teams sort support tickets or social posts into labels defined at query time. Large language models extend the pattern to classification and extraction from a task description alone.

Healthcare. CheXzero, published in Nature Biomedical Engineering in 2022, detected pathologies in chest X-rays with accuracy comparable to radiologists without training on explicitly labeled images. It learned instead from the free-text radiology reports paired with each X-ray.

Autonomous vehicles. Road scenes contain objects no training set anticipated, such as fallen cargo, unusual construction equipment, or animals. Open-vocabulary detectors such as Google’s OWL-ViT (ECCV 2022) localize objects from free-text queries rather than a fixed label set. Perception teams use them to flag categories the base model never saw.

How to Serve Zero-Shot Learning Embeddings in Production

Serving zero-shot learning in production means storing class embeddings and answering nearest-neighbor queries against them at low latency, while new classes arrive continuously. The model does the reasoning. The database decides whether that reasoning is fast, current, and filterable.

In a notebook, class embeddings sit in a NumPy array. In production, teams add and retire categories constantly, and every new class needs its embedding written and indexed before the next request. A common pattern bolts a standalone vector database onto the primary database, which creates a sync pipeline between two systems. When that pipeline lags, the classifier returns labels for categories that no longer exist or misses categories added an hour ago.

티DB removes the second system. TiDB’s vector search stores embeddings in a native VECTOR column inside the same MySQL-compatible SQL engine that holds the rest of the application’s data. An HNSW (hierarchical navigable small world) index, a graph-based structure for approximate nearest-neighbor search, accelerates K-nearest-neighbor (KNN) queries. Registering a new class is a single INSERT in the same transaction as its metadata, with no separate system to keep in sync.

The vector index lives on TiFlash, TiDB’s columnar storage engine and the analytical half of its HTAP (hybrid transactional/analytical processing) architecture. When you define a vector index at table creation, TiDB creates the TiFlash replica automatically. One SQL query can then combine similarity search with relational metadata, such as restricting candidates to active categories.

The example below stores 512-dimension class embeddings (the output size of CLIP ViT-B/32), registers an unseen class, and classifies an input. Vector literals are truncated.

CREATE TABLE zsl_classes (
  id BIGINT PRIMARY KEY AUTO_RANDOM,
  label VARCHAR(255) NOT NULL,
  description TEXT,
  status ENUM('active', 'retired') NOT NULL DEFAULT 'active',
  embedding VECTOR(512),
  VECTOR INDEX idx_embedding ((VEC_COSINE_DISTANCE(embedding)))
);

-- Register a new, unseen class from its text-encoder embedding
INSERT INTO zsl_classes (label, description, embedding)
VALUES ('zebra', 'horse-shaped, black and white stripes', '[0.012, -0.087, ...]');

-- Classify an input: nearest class embeddings first, then filter
SELECT label, distance FROM (
  SELECT label, status,
         VEC_COSINE_DISTANCE(embedding, '[0.031, -0.054, ...]') AS distance
  FROM zsl_classes
  ORDER BY distance
  LIMIT 10
) AS candidates
WHERE status = 'active'
ORDER BY distance
LIMIT 3;

The inner query uses the HNSW index to find the 10 nearest class embeddings. The outer query applies the metadata filter. TiDB does not use the vector index when a WHERE filter sits in the same query block as the KNN search, so post-filtering in an outer query keeps the search on the index. Size the inner LIMIT larger than the final result so the filter does not starve the output. See the TiDB vector search documentation for distance functions and index options.

How Zero-Shot Learning Maps to the AI Agent Cold-Start Problem

An AI agent that meets a new tool, schema, or user intent with zero prior examples faces the same problem zero-shot learning solves for classification. It has to act correctly on something it was never trained on, using only a description and what it already knows.

A ZSL classifier places an unseen class by its description. An agent reads a tool’s schema and reasons about how to use it from that description alone. Large language models make the first attempt possible. Improving after the first attempt depends on memory. An agent that stores what it tried, what worked, and what failed can retrieve that context the next time a similar tool or intent appears. The second encounter becomes a few-shot problem instead of another cold start.

That memory has to be durable, queryable, and isolated per agent. Persistent agent memory on TiDB keeps structured state in SQL tables and semantic recall in vector columns side by side. An agent can retrieve past episodes by similarity and filter them by tool, user, or outcome in one query.

Manus, a general-purpose agentic AI platform, shows what this looks like at scale. After launching in March 2025, Manus drew a waitlist of more than two million users within weeks and migrated to TiDB Cloud in two weeks to keep up. TiDB now handles context persistence behind Manus’s agent swarms. TiDB Cloud branching gives each agent an isolated, copy-on-write copy of the database to experiment in and commit results from.

What Limits Zero-Shot Learning Techniques Today

Two limitations decide whether zero-shot learning techniques hold up in production. The first is the quality of the semantic information. The second is the difficulty of explaining predictions.

Semantic quality sets the ceiling. ZSL removes the need for labeled examples of unseen classes. It does not remove the need for good descriptions. A model can only place an unseen class as accurately as its attributes, text description, or embedding represent it. Vague or overlapping descriptions produce confident misclassifications.

Interpretability limits high-stakes use. When a ZSL model labels a chest X-ray or a road obstacle, reviewers need to know why. Distance in an embedding space is not an explanation a clinician or safety auditor can act on. Explainable AI (XAI) techniques help. Teams in regulated domains still pair ZSL with human review.

Zero-shot learning moves the hard part of classification from labeling to representation. Once a model can place an unseen class by its description, production success depends on where those embeddings live and how fast you can query and update them.

Try vector search on your own zero-shot embeddings. Spin up a free TiDB Cloud 스타터 cluster, load your class embeddings into a VECTOR column, and run the query above against the relational data your application already uses.

Frequently Asked Questions About Zero-Shot Learning