In the last year, 18 million people have visited The Met online collection. That’s more than the population of most nations. These individuals access our website from dozens of countries, each with their own goal in mind for using the collection: a jeweler seeking inspiration for her latest collection in the Egyptian holdings, a scholar tracking a provenance trail, a student learning about Impressionism for the first time, a craftsman who spends his spare time studying and recreating medieval locks. (Yes, these are all real people we have met in the course of making improvements to our online collection.) Within the past hour their queries have included “snake,” “egyptian pots,” “charles le roux,” “chinese propaganda posters,” “fashion,” “turkish comb,” and the perennial favorite: “flowers.” This selection reflects the variety of ways people think about a collection, from specific object types and cultural artifacts to materials, places, artists, and subject matter.
A single search for "snake" pulls coiled serpents, faience gods, and gold bracelets from centuries apart.
These people visit The Met’s website specifically because they are all but guaranteed to find an example of what they are looking for within an online collection of more than 500,000 objects. And for many kinds of queries, we meet that expectation. When people look for specific, known objects with exact titles or artist names we surface usable results. If you need to know the year Sargent’s Madame X was painted or the dimensions of Bruegel’s The Harvesters, we deliver. This is because until now, our search relied on matching the user’s query to words found on each object’s individual record.
This approach becomes more limiting when people want to find works of art based on subject matter or themes, like surfacing all works depicting “cats,” “tulips,” or “fun shoes.” Only about 32% of metmuseum.org users search with a specific artwork in mind and a clear expectation of what the “right” result looks like. The more common type of search query is more abstract, oriented toward discovery and exploration across several works of art without a predetermined destination or a single “right” answer in mind. For these types of searches—like “year of the horse” or “people eating together”—a good batch of results doesn’t depend on finding an exact text match to a title or description but rather on understanding both the meaning of the search term and what each artwork depicts, then surfacing results accordingly. This is a search experience we hadn’t yet been able to offer, until now.
How does semantic search work?
To offer this experience, we added a new layer of data about what each artwork depicts on top of our existing metadata. Consider Degas’s The Dance Class. The title tells us the setting, but not what’s in the room: the ballerinas in their tutus, the elderly instructor, the sheet music, and the mirror in front of which they rehearse. Our metadata keyword search is weak for queries about objects like “ballerinas” or compositional themes. To fill the gap, we need some way to search the visual content.
Semantic search allows for results that reflect a deeper understanding of an artwork’s meaning as well as its content.
The photo libraries on our phones already have this feature: we can search for “spring flowers” and find relevant photos, even ones we never labeled. The phone software turns every photo into a set of coordinates, but instead of just two numbers, like the latitude and longitude of a location on a map, a photo gets hundreds or even thousands of numbers, placing it in a vast, multidimensional space. In that space, pictures of similar things land near each other. A query like “blue dress” gets turned into coordinates in the same space, and the search returns the photos nearest to it. These coordinates are called embeddings. We can also make embeddings of artwork images, and it works fairly well for common subjects that the embedding model has been trained on, such as portraits, landscapes, and still lifes. But the more complex and detailed the composition, the less reliable the results become.
Rather than directly turning an artwork image into an embedding, we can take a different approach by using AI to describe the image in words. This kind of generation isn't new. Screen readers and accessibility tools have been producing alternative (or “alt”) text for years, and modern vision-language models (VLMs) can now produce long descriptions that capture the subject, composition, and fine details—even a cat hiding under a table.
A search for "teapots shaped like produce" yields a citron, a cauliflower, and a pineapple—all rendered in clay.
Once we have these artwork descriptions, we can search them in two ways. The simplest is an ordinary keyword search: a visitor searching for “cats” now finds paintings that depict cats by searching both the text in the object record as well as the object image visual description. We can go further by embedding the descriptions in the same way we embed images—by turning each one into coordinates in a space where similar wording lands close together. A search for “cats” also surfaces felines, kittens, and tabbies. And because proximity in that space reflects semantic meaning rather than exact spelling, it crosses languages, too: a search for "猫" or "gato" returns the same artworks as "cat."
Supporting multiple ways of searching
The approach we’ve adopted introduces semantic search as a supplement to keyword search, not a replacement. Exact-match queries for titles, artists, or dates continue to yield strong results, while thematic and descriptive queries now produce good matches, too.
The new as-you-type suggestions feature provides additional avenues for discovery to curious users.
With this improvement came a challenge: search queries now return more results than before. While this means fewer dead ends and more opportunities for discovery, there is also more content to sift through. For users in an exploratory mode, that abundance is a boon, but for users searching for something very specific in mind, it can feel overwhelming. To serve both types of users, we offer as-you-type suggestions that provide shortcuts to applying filters for artists and makers, object types and materials, geographic locations, and departments. Applying a filter surfaces a narrower set of results with that attribute directly tagged to those results, creating a faster path to a specific destination. The suggestions also serve our users by surfacing potential avenues of inquiry they may not have thought to explore. In this way, the same feature does different work for different seekers, prioritizing for precision for one group and discovery for the other.
At its best, semantic search reveals kinships between objects from across different time periods, geographies, and mediums that are more difficult to notice onsite, simply because these objects may live in different wings of the museum. Search technology develops at a rapid pace, as do users’ expectations. We will continue to update and make improvements to this resource. We invite you to explore for yourself: search for “mythical beasts” or “people dancing in a circle,” and discover what emerges.
