Computers are changing how art is made
Software can inspire and even guide the creativity of artists
Increasingly, artists are teaming up with computers. Some look for inspiration. Others might ask a computer to create a melody for their lyrics. Several computer programs even design paintings from scratch. Such programs are raising questions about whether computer are only assistants or might be considered true equals.
damircudic/E+/Getty Images
Maya Ackerman just wanted to write a song.
She tried for years — song after song. In the end, she didn’t like any of the tunes she wrote. “I didn’t have the gift, if you will,” she says. “All the melodies that came into my mind were so boring that I couldn’t imagine wasting time performing them.”
Maybe, she thought, a computer could help. Computer programs already are useful for recording songs that people come up with. Ackerman now wondered if a computer could be more — a songwriting partner.
It was a flash of inspiration. “I knew in an instant that it would be possible for a machine to give me ideas,” she says. That inspiration led to the creation of ALYSIA. This computer program can generate brand-new melodies, based on a user’s lyrics.
As a computer scientist at Santa Clara University in California, Ackerman has a lot of experience using algorithms (AL-goh-rith-ums). These are step-by-step mathematical recipes to solve problems and make predictions. Algorithms are useful in programming computers. They also can be useful for everyday tasks. Online movie and music servers use algorithms to recommend films and songs. Self-driving cars need algorithms to safely navigate roads. Some grocery stores track the freshness of produce using algorithms that are connected to cameras or sensors,
When a computer runs software, it’s completing tasks by following algorithms written as computer code. Computer scientists like Ackerman analyze, study and write algorithms to solve a wide range of problems. Some of them use algorithms in the field of artificial intelligence, or AI. This emerging technology teaches computers to mimic tasks or activities that the human brain usually handles. In ALYSIA’s case, that’s songwriting.
Ackerman isn’t the only one using AI for songwriting. Some programs build entire orchestral scores around small bits of melody. Others generate music for many instruments. AI also is finding its way into other arts. Painters, sculptors, dance choreographers and photographers have found new ways to collaborate with AI algorithms.
And those efforts are paying off. In October 2018, an art auction in New York City became the first to sell an AI-generated work. A group of computer scientists and artists in France used AI algorithms to create the work. This portrait of an imaginary man made a splash: The painting sold for $432,500.
Ahmed Elgammal runs a computer-science lab that focuses on using AI to influence art. It’s at Rutgers University in Piscataway, N.J. “AI is a creative tool that will be accepted as an art form,” he says. Eventually, he adds, “It will affect the way art is made, and what art will be.”
Virtual art school
Artists and computer scientists began looking for new ways to create art with computers back in the 1950s and 1960s. They built computer-controlled robotic arms holding pencils or paint brushes. In the 1970s, an abstract painter named Harold Cohen introduced the world to the first artistic AI system, called AARON. Over the decades, Cohen added new forms and figures to AARON’s abilities. Its art often depicted plants or other living things.
A recent experiment from Elgammal’s group at Rutgers now suggests that algorithms can create works that could be considered fine art. For this study, 18 people viewed hundreds of images. Each image showed a painting or other work of visual art. Some had been created by people. An AI algorithm had created the rest. Each participant ranked the images based on aspects such as their “novelty” and “complexity.” The final question: Did a human or AI create this work of art?
Elgammal and his collaborators had assumed that art made by people would rank higher in categories such as novelty and complexity. But they were wrong. The recruits they invited to review the works often judged AI-created art to be better than those by people. And the participants had concluded that human artists had created most of the AI art.
In 1950, a British computer-science pioneer named Alan Turing introduced the Turing Test. A computer program that can pass the Turing Test is one that can convince a person that it (the program), is human. Elgammal’s experiment functioned as a kind of Turing Test.
“From a viewer point of view, these works passed the Turing Test of art,” he now argues.
His group’s AI algorithm uses an approach known as machine learning. First, the researchers feed tens of thousands of images of art into the algorithm. This is to train it. Explains Elgammal, “It learns by itself the rules of what makes art.”
It then uses those rules and patterns to generate new art — something it hasn’t seen before. This is the same approach used by algorithms that can recommend movies or music. They collect data on someone’s choices, then predict what might be similar to those choices.
Since its Turing Test experiment, Elgammal’s group has invited hundreds of artists to use their software. The goal isn’t to show that AI can replace artists. Instead, it seeks to use them as one source of inspiration. The researchers have created a web-based tool, called Playform. It lets artists upload their own sources of inspiration. Then Playform creates something new.
“We want to show an artist that an AI can be a collaborator,” Elgammal says.
More than 500 artists have used it. Some use Playform to generate images. Then they use those visuals in new ways for their own works. Others find ways to combine the AI-generated images. An exhibit last year at the largest art museum in Beijing, China, included more than 100 works shaped by AI. Many had been created using Playform. (You can use it, too: Playform.io.)
Bringing art and AI together is Elgammal’s passion. He grew up in Alexandria, Egypt, where he loved studying art history and architecture. He also enjoyed math and computer science. In college, he had to choose — and he chose computer science.
Still, he says, “I never abandoned my love for art and art history.”
Rise of the cybersongs
Ackerman, in California, has a similar story. Although she listens to pop music, she really likes opera. She studied piano as a child and even performed on national television in Israel, where she grew up. When she was 12, her family moved to Canada. They couldn’t afford a piano or the lessons to continue her training. So by high school, she said, she felt lost.
Her father, a computer programmer, suggested she try coding. “I was really good at it,” she says. “I loved the sense of creation.”
“When I wrote my first computer program,” she says, “I was so amazed that I could make a computer do something. I was creating.”
In graduate school she took up singing lessons and music came back into her life. She sang in staged operas. Those lessons and performances made her want to sing her own songs. And that led to her songwriting dilemma — and ALYSIA.
The first version of it came together in a few months. In the three years since then, Ackerman and her team have made it easier to use. Other improvements have led it to also turn out better music.
Like Elgammal’s algorithm, the algorithm that runs ALYSIA teaches itself the rules. But instead of analyzing art, ALYSIA trains by identifying patterns in tens of thousands of successful melodies. It then uses those patterns to create new tunes.
When users type in lyrics, ALYSIA generates a pop melody to match the words. The program can also generate lyrics based on a topic from the user. Most of ALYSIA’s users are first-time songwriters. “They come in without any experience,” Ackerman says. “And they write songs about very beautiful and touching things.” In November 2019, the French magazine Liberation named a song written with ALYSIA — “Is this Real?” — as its song of the day.
Ackerman thinks ALYSIA offers a glimpse of how computers will continue to change art. “Human-machine collaboration is the future,” she believes. That collaboration can take many forms. In some cases, an artist can do all the work. A painter might scan a painting, for instance, or a musician might record a song. In other cases, a computer does all the creative work. With no knowledge about art or coding, someone simply pushes a button and the computer creates something.
Those two situations are extremes. Ackerman is looking for “the sweet spot” — where the computer can keep the process moving, but the human artist remains in control.
But is it creative?
Paul Brown says AI makes it possible for more people to engage with art. “It enables a whole new community to get involved,” he says — one that lacks drawing or other skills that one normally links with creative artistic behavior.
Brown is a digital artist. Throughout his 50-year career, he has been exploring the use of algorithms in art. After training as a visual artist in the 1960s, he began exploring how to use machines to create something new. By the 1990s, he was designing and teaching classes in Australia on using computers in art. Now, he has a studio at the University of Essex in England.
The rise in AI’s popularity also has triggered a debate, Brown says. Are the computers themselves creative? It depends on whom you ask, and how you ask it. “I’ve got younger colleagues who believe that artists working with computers are doing something new that doesn’t relate to traditional art,” he says. “But new technologies are always adopted very quickly. This isn’t a particularly new branch of anything, but it does allow them to do new things.”
Brown says artists who can write code are at the forefront of this new movement. But at the same time, he also sees AI as but one more tool in an artist’s toolbox. Michelangelo used a stonemason’s tools to create many of his most famous works. The introduction of paint in tubes, in the mid-19th century, allowed artists like Monet to work outdoors. Similarly, he thinks computers enable artists to do new things.
Elgammal says it’s not that simple. There is a way in which AI algorithms themselves are creative, he argues. Computer scientists design the algorithm and choose the data used to train it. “But when I push that button,” he points out, “I have no choice over what subject matter is going to be created. What genre, or color or composition. Everything comes through the machine by itself.”
In that way, the computer is like an art student: It trains, then creates. But at the same time, Elgammal says, these creations would not be possible without the people setting up the system. As computer scientists continue to refine and improve their algorithms, they’ll continue to blur the line between creativity and computation.
Ackerman agrees. “Computers can do creative stuff in ways that are different than humans,” she says. “And it’s so exciting to see that.” Now, she says, “How far can we push the creativity of a computer if a human is not involved?”
