The Flow Project: Connecting Colors and Movement

The Flow Project's logo. Image courtesy of the author

From Don Undeen, Senior Manager of MediaLab:



The MediaLab's creative-technology interns are encouraged to adopt a very broad interpretation of what it means to explore the intersection of art, technology, and the museum experience. Sometimes our projects are practical or academically rigorous, and other times parts of the Met's collection bring out the child in all of us, encouraging us to dance, play, and paint with the motion of our bodies, like in The Flow Project.

«The Metropolitan Museum of Art has been a great space for me to work with things I'm passionate about while helping to create something innovative for the Museum. Don Undeen made me feel comfortable in my new environment, and I was able to meet and learn from the staff of many different departments. My goal was to translate Impressionist techniques into a live experience for visitors because it is an artistic style that has profoundly affected my work.»

Why Impressionism?

As a pianist, I think of colorful, bright splashes of paint layered with watery, flowing brushstrokes every time I hear pieces by Impressionist composers, like Maurice Ravel and Claude Debussy. Their imaginative and lively forms made Impressionism my favorite art movement. Impressionism's unfinished, sketch-like appearance and bright colors are still as beautiful today as they were when the movement began. This revolutionary style reminds me of myself as a unique digital artist, so I decided to create a series of interactive art experiments to share my interpretation of Impressionism.

Translating Impressionism into a Digital Experience

My favorite Impressionist piece, Claude Monet's Impression: Sunrise, gave the Impressionist movement its name when critic Louis Leroy accused it of being a sketch of"impression." Impression: Sunrise demonstrates techniques that many independent artists of the time adopted: short, broken brushstrokes that barely convey form; pure unblended colors; and an emphasis on the effects of light.

Claude Monet (French, 1840–1926). Impression: Sunrise, 1873. Musée Marmottan Monet, Paris

Impressionists often emphasized shadows and highlights, and their loose brushwork created a spontaneous and effortless effect that masked how carefully they constructed their compositions. In order to make these characteristics an interactive live experience, I created tools for visitors to experiment with analyzing colors, simulating brushstrokes, and seeing dynamic movement.

The Project

The Flow Project is a series of interactive digital experiments that demonstrate the characteristics of Impressionism in real time. The experiments are broken down into three simple apps—Dynamic Colors, Monet Mirror, and Flow Brushes—each focused on demonstrating a different aspect of Impressionism, including toned-down colors and light, wet brushes, and natural flow. I created these apps with motion tracking, GL Shader Language, and computer vision using openFrameworks, an open-source C++ toolkit for creative coding.

"Art is not nature . . . There was a lot more to be got out of colour."—Pierre Bonnard

Dynamic Colors

Left: Vincent van Gogh (1853–1890). Country road in Provence by night, circa 1890. Oil on canvas; 90, 6 x 72 cm. Kröller-Müller Museum, The Netherlands. Right: A screenshot of a 3D mesh produced with Dynamic Colors. Image courtesy of the author

Dynamic Colors is an app that turns flat paintings into 3D, volumetric forms by assigning depth to the colors, and using the assigned depth to generate 3D meshes of the paintings.

A demo video of another 3D mesh created with Dynamic Colors. Video courtesy of the author

The brightness of the colors is connected to a depth value, the saturation is connected to the distance between each vertex, and the threshold of brightness and saturation is adjustable in real time. I added Perlin noise to smooth the activity.

Monet Mirror

Left: Flowers in Central Park in Monet Mirror. Right: A screenshot from a video of a user using Monet Mirror. Images courtesy of the author

Monet Mirror is an app that turns live video into a Monet painting by adding a wet-brush filter to the output video. I used shader language to simulate liquefied pixels and layered the output video with a toned-down background.

A demo video of users using Monet Mirror. Video courtesy of the author

"There is always a delightful sense of movement, vibration, and life."—Theodore Robinson

Flow Brushes

Screenshots of watercolor paintings based on each user's movements that were produced with Flow Brushes. Images courtesy of the author

Flow Brushes is an app that produces watercolor paintings based on user movements in real time, and I turned it into an interactive installation. The installation projected live video of the app in action on a wall in the Uris Center for Education. I created this app and installation with the Optical Flow algorithm, generative art, and motion-capture technology, using Kinect, GLSL, blending modes, ofxGui (user interface add-on for openFrameworks), and midi controller for live performance.

Video of dancer Sun Kim experimenting with Flow Brushes. Video courtesy of the author

Installation in Carson Family Hall

With the help of Paul Caro, Senior Manager of Media Production and Technology Services, and Don, I installed Flow Brushes in Carson Family Hall for the fall 2014 MediaLab Expo. It was powered by a hidden Mac Mini and a Kinect, both of which were placed on a small table facing the audience. Carson Family Hall is very spacious and visually striking, and it's also easily accessible from the Met's 81st Street entrance, so it immediately became a popular attraction, especially for kids.

Video of visitors having fun experimenting with Flow Brushes. Video courtesy of the author

The MediaLab Expo was very successful, and the project received a lot of constructive feedback and potential collaborators. Don said that Flow Brushes was "the first large-scale interactive installation by the MediaLab" and that it has inspired later intern projects.

Conclusion

The Metropolitan Museum is an inspiring place that provides unlimited creative possibilities, and I believe this has been an invaluable experience for my future career. I want to thank Don for being supportive, cheerful, and for remaining dedicated to pushing us to make our projects bigger and better. I'm also grateful for the MediaLab challenging me to create engaging connections between art, technology, and the public, and I am really happy to see my ideas come to life.

Flow Brushes generating watercolor paintings of a visitor's movements in the fall 2014 MediaLab expo. Image courtesy of the author