From Pixels to Paint
In The Studio with artist Joanne Hastie by Sandy Liu
When first thinking about robots making art, common sentiments probably centre around curiosity or fear.
HOW DO ROBOTS CREATE ART? WHAT DOES IT MEAN FOR ART?
IF ROBOTS CAN EVEN REPLACE HUMANS IN A FIELD OF CREATIVITY, DOES IT MEAN HUMANS CAN ENTIRELY BE REPLACED?
Walking into Joanne’s studio, I was welcomed by the gentle buzzing sound of the robot arm. There was really nothing intimidating about the robot, it was just quietly, diligently waving its paint brush, dipping paint as if a studious art student refining its craft, one brush stroke at a time.
Most of the walls were covered with distinct schools of artwork from abstract to impressionism, from shapes to landscape, from human to robot-made. Joanne ran through her folders of robot art experiments as I went through the array of questions I meditated on, curious to get her insights on this fascinating way to create artwork.
WHAT INSPIRED YOU TO CHOOSE ROBOTS AND MACHINE LEARNING TO CREATE?
Joanne: I have been painting for as long as I can remember, but coming from a family of engineers, it was just natural to become an engineer as well. I chose mechanical engineering because of the drawing and design work involved. My first few work terms were based in automation. I had a 2 week opportunity to work with an industrial KUKA robot (KUKA is a German manufacturer of industrial robots and solutions for factory automation), the whole time I was working with it I wanted to have it hold a paint brush. Years later, I still had this idea so I eventually bought myself a desktop robot to see if I could learn to program it to paint. It was a solid year of learning to code in Python, after that I started experimenting with machine learning.
Joanne first started automating her painting process when she entered the 2018 international RobotArt competition. The challenge was to paint a picture with no human intervention. She won 5th place in the competition and gained international recognition. Read about Joanne’s experience: https://joannehastie.com/project/2018-robotart-competition/
GAN - GENERATIVE ADVERSARIAL NETWORK
Some of Joanne’s latest tech art uses GANs. To understand Joanne’s process, it would be useful to define GAN - Generative Adversarial Networks. GANs are deep neural net architectures comprised of two nets, pitting one against the other (thus the “adversarial”). They can learn to mimic any distribution of data. That is, GANs can be taught to create worlds eerily similar to our own in any domain: images, music, speech, prose. (Source: https://pathmind.com/wiki/generative-adversarial-network-gana)
In this sense, with the two networks, a generator and a discriminator, GANs can take in random inputs and learn from a training set to produce new combinations of output that could mimic a human’s work and then have the ability to classify it as good or bad.
An example of GAN is the project This Person Does Not Exist, where every image here although looks like a real person, is actually a dreamed up image by a GAN based on the inputs it was trained by. With this in mind, let’s talk about Joanne’s process.
Joanne uses GANs in 3 of her non-robotic, hand-painted series including:
generating style filters for her cityscape paintings and
Throughout these image generation processes she uses another algorithm called a classifier to sort through large amounts of images her code generates. She uses both trained and untrained classifiers in both her GAN processed art and her robotic abstract paintings.
WHAT DOES THE PROCESS LOOK LIKE?
Joanne: In order to automate something, you have to know something to its very basic motions. The reason we practice something is so we don’t have to think about how to do it - we can do it automatically. I practice painting so I can focus on creating the painting and not on the details such as how to mix colors, clean the brush or what brush to use.
SAME AS GREAT ATHLETES ACTUALLY DON’T THINK WHEN THEY PERFORM, THEY KEEP A BLANK MIND.
Joanne: That’s right, but in setting up the automation, it forces me to think of each discrete step so I can program the robot (ex. Dipping the paint, apply paint in a ‘painterly’ fashion, cleaning the brush, etc.). For my current robotic series, I follow the steps:
I have python code that generates brush stroke sets. There are three different styles of brush strokes generated that are layered with different colors. Each brush stroke set is digitally layered onto backgrounds that I have painted by hand.
I train a classifier on my preferences of what I deem to be good or bad based on my own preferences by labelling earlier generated paintings. The newly generated paintings are then classified as either good or bad - the top ranked ones will be considered for painting.
I then place each of the backgrounds into the robot stand and paint the code-selected painting for that background
WERE THERE A LOT OF TRIAL AND ERROR IN THE BEGINNING? WHAT WERE SOME OF YOUR FAVORITE FAILS?
Joanne: I have stacks of failed paintings. Often leading to new ideas. A great example is once I turned the robot on to paint and then went outside on a run, when I returned, I realized that I had forgotten to toggle on the action to clean the brush between colors. So the robot mixed all the colors together and the painting came out almost black with thick brush strokes. I had an open studio last year and people loved this example and actually preferred the one I created by “mistake”.
The one on the left is the intended painting, compared to the right is where Joanne forgot to include the command to clean the paint brush between colors. Joanne’s machine learning abstracts where she uses two classifiers - one to select the composition and an untrained one to suggest what object they could be. More on this project here.
WHAT DO YOU THINK CONSTITUTES GOOD ROBOT ART?
Joanne: At the beginning of this work I used my Instagram account (@paintingvariables) where I posted all the robot paintings and did not disclose that the art is done by a robot to see the audience’s reaction. I had an idea that I could train classifier to understand what “good or bad” art was based on the number of likes I get for each picture. I quickly realized this is not a proper experiment because the platform is more in control of what people see and the more I posted the more likes I got. Reverting back to traditional standards, I became the critic and started to categorize the paintings myself with my own preferences. Since the code generates so many good paintings I often focus on the question, “Would I use paint and time on this?”
WHAT ABOUT PRICING? DO YOU PRICE THE ART DIFFERENT FROM HOW YOU WOULD PRICE THE ORIGINALS?
Joanne: I actually don’t think about robot art as different from human art because it’s really just another tool that I use to create, I value the art similarly. Therefore, I usually price by square inch like any other artwork I create, but the bigger challenge is getting validation on what’s considered good.
BEYOND PRICING, WHAT IS THE VALUE OF COMPUTER OR ROBOT ART IN YOUR OPINION?
Joanne: To the artist I think it’s super valuable because in order to teach the robot, the artist has to understand how she paints down to a granular level, each decision is deliberate. It also allows artists to iterate on more ideas and test out color palettes much faster.
I do have a fear though that people would think it’s less valuable if the art is made by a robot, but to me it’s not personified, it’s more around the question, “Why does the artist have to hold the brush?”
SO EVEN THOUGH YOU’VE PROGRAMMED A ROBOT THAT CAN DO SUCH GREAT WORK, IT SOUNDS LIKE YOU’D AGREE WITH THE PEOPLE THAT THINK CREATIVITY AND ART IS PROBABLY THE ONE THING THAT ROBOTS CAN’T REPLACE HUMANS IN DOING?
Joanne: That’s right, I don’t think robots can replace artists if people just understood how much work is required to get the robot to paint. The creativity is all based on my inputs, so the work that a robot creates is very artist dependant.
I think it’s unfortunate how society has traditionally split arts and technology to two separate domains. More artists would benefit from learning about technology if they aren’t intimidated by it. It is hard for art and art galleries to survive if they don’t embrace technology.
I COMPLETELY AGREE THAT A TREND THAT WE CAN’T IGNORE IS THAT DISCIPLINES CAN NO LONGER BE INNOVATIVE BY STAYING IN THEIR OWN LANES. WE SEE MORE AND MORE INTERDISCIPLINARY COLLABORATIONS OR ARTISTS WHO ARE ALSO TECHNOLOGISTS.
LOOKING AHEAD TO THE FUTURE, WHAT’S NEXT FOR YOU?
WHERE DO YOU SEE THE TECHNOLOGY GOING AND WHAT IS THE WILDEST DREAM THAT YOU CAN IMAGINE NOW WITH THE WORK YOU DO?
Joanne: It has definitely been an iterative process, I now have a new 7DOF robot (seven degrees of freedom) that allows for more flexibility of movement and more positions. I am interested in getting the robot to mix its own colors as I currently do it manually. If it can mix its own colors this can open up a lot more possibilities to painting representational.
Where else can I replace myself in the process? Can I get it to swap brushes (I currently do this manually) or make judgement calls as it is painting with a vision system? I frequently get invited to show my robot at conferences and events, so I wonder can the viewer interact with the robot with a set of parameters? I have other ideas such as can an artist paint in the moment on location with a more portable tablet that informs a robot that is painting in the studio? What if the robot can be mobile in the studio to have even more reach…
I have often said I’d love to have it help me paint large wall murals.
In December 2014, Joanne attended a three week Art Residency in Graniti Sicily. During her stay she was to paint a mural in the town.
A new robotic arm Joanne just acquired that allows more flexibility in how the paint brush is held.
Joanne and I with the adorable parrot Larry in the studio.
Thank you Joanne for sharing your art with us
This interview was conducted by
Marketing & Business Consultant focused on merging tech with arts and culture. Harvard MBA. Sandy writes about business, technology, the arts & life.