Build with AI at The Met

There are so many ways we experiment with emerging technologies at The Met. We conduct internal tests, launch pilot projects, and, when it makes sense, we integrate them thoughtfully into the many ways we engage our audiences through exhibitions, online features, digital content, and programming. At the same time, it can be just as important to look outward for inspiration, collaborating with other institutions, artists, makers, and sectors outside of museums.

Generative artificial intelligence, or GenAI, is one such technology. There are many thoughtful and valid debates about the merits and ethical concerns surrounding this broad category of tech—and these same discussions are happening at The Met. Still, we see our audiences engaging with GenAI daily: while they’re inside the Museum, as they seek highlights and recommendations, and while they're at home to conduct research, find inspiration, and plan their vacations to New York. We feel it’s important to cautiously explore this frontier with hands-on testing, in conversation with our visitors and in coordination with other institutions and practitioners.

On April 16, we invited over forty technologists, makers, students, and other creators to the museum for a one-day hackathon called “Build with AI at The Met.” We asked participants to focus on our Asian art collection, encouraging them to rapidly develop experiments that encourage our audiences to connect with the collection and with each other. We didn’t place limits on whether ideas needed to be on-site or at home, opting to keep the prompt open-ended. The resulting breadth of ideas was fantastic, from games to tools to speculative provocations.

Following introductory remarks from Mike Hearn, Douglas Dillon Chair of Asian Art, participants visited the galleries to view the works in person and to forge their own connections. From there, they had about five hours to build. What we found fascinating is that GenAI has not only opened new ways for visitors to engage with collections; it has also completely transformed how quickly one can develop working prototypes.

Participants developed ideas that solved real logistical issues such as wayfinding; that connected the collection with contemporary fashion; that transformed objects into playable 3D games; and that invited interaction such as dancing, drawing, and even exercise. We have selected three projects to share below.

Instrument Explorer

Aaron Zhao created a tool that transformed depictions of instruments in the collection into playable experiences. It used AI to understand the historical context, categorize by instrument type, and generate both educational content and interactive, digital string, percussion, or wind instruments.

Fish The Met

Glub Glub Labs (Josh Zhong and Tres Pimentel) made a digital device that lets visitors “catch” fish in the collection. They used AI to analyze the artworks, attempt to identify the species of fish, and research historical context for each. For speculative visitors, this becomes an interactive scavenger hunt—with learning packaged inside a video game-like experience.

Semantic Explorer

Morry Kolman built a highly detailed, multipart tool for visitors to rapidly explore the collection using natural language. It allows for free-form semantic search, lateral exploration (“show me more like X”), favoriting, and sharing. It was a Swiss Army knife of useful and meaningful features, with a novel swiping mechanism to discover interconnected works.

All three of these are completely different: poetic, playful, educational, practical, fun, and meditative. And there were twenty additional projects! For us, the event reaffirmed our belief in the ways GenAI can fuel creativity, and that it will unlock new ways for our audiences to find themselves in our collection. We can’t wait to see what the community makes next.

This program is made possible by Anthropic, the team behind Claude.