Lessons in Choreorobotics

Here's how I might make a dance.

I'll start with a move.

Maybe I'll mirror that move to the other side.

How about I repeat that move twice.

Maybe I'll change the scale.

Maybe I'll change the speed.

How about orientation?

Maybe the energetic emphasis.

These phrases are choreographic tools

that I use to modify an existing dance motion.

They were mirroring, repetition, scale, speed,

orientation, and energetic emphasis.

These types of tools, it's just a short list

and a much longer list of ways

that you can modify dance motions.

Choreographers have a very long practice in history

of modifying dance motions in these ways,

and you could see how I could immediately take one motion

and turn it into this long extended, creative,

compelling, and original dance.

But one very critical thing to note

is that these tools are not specific to a human body.

They could be done on any moving body, including a robot.

Shared tools between roboticists and choreographers

are just one thing that these two domains have in common.

And choreo-robotics is the emerging field

that captures all of these intersections

between dance and robotics.

I'm a robot choreographer.

At Stanford, I use artificial intelligence

to teach robots how to do all kinds of new tasks.

Like open doors, much harder than you think it is,

respond to gestures, play music, and dance.

During my PhD, I spent three years

as the inaugural artist in residence

at Everyday Robots at Google X.

I've spent the last several years traveling the world,

dancing with robots,

and choreographing over a dozen different kinds of robots

for art installations and performances.

So people often ask me like, Why I dance with robots?

And as my former colleague and friend, Benjie Holson put it,

Well, the main difference between a robot and a computer

is that a robot moves.

How a robot moves is a vital part of the way

that a robot is perceived?

And choreographers are experts

at making meaning out of movement.

In fact, art and culture is one of the most important ways

that we incorporate new technologies

into our society and our public discourse.

One fundamental part of our species

is our relationships with our tools that we build.

And so if we aren't incorporating them

in art and culture,

then we will have missed this opportunity

to kind of codify them into the public dialogue.

Dance for me is that art and culture.

So we used to dance around fires, we dance for the rain,

we dance at proms and weddings, we dance on TikTok,

and now we dance with robots.

Robots are increasingly becoming part of our everyday lives,

and as this transition continues,

they're going to be recognized for more than just

the utilitarian value that they provide.

They're going to be recognized for

how they show up in the world, how they move.

Movement elicits emotions.

If we aren't intelligent about the way that a robot moves

relative to people in our everyday lives,

we will have missed this incredibly important opportunity

for robots to not only be safe,

but for them to create harmonious, dynamic interactions

with other humans as a part of our daily lives.

We've seen a very similar process before.

When computers moved from military and research applications

into personal computing,

all of these other domains became super relevant,

like psychology, semiotics, animation, sound design.

We observed that process in personal computing,

and we're experiencing something very similar in robotics.

As we move from research labs and factories

into office buildings and sidewalks,

we need domains like dance to help us create these

legible, compatible and safe relationships

between humans and robots.

So towards this end, all of my art,

research, and business endeavors ask this question,

what kinds of novel experiences can we create

that leverage a robot's features as both

a utilitarian tool and an evocative, moving, social agent?

When I was an artist in residence at Everyday Robots,

there were dozens of these kinds of robots running around.

They were wiping tables and sorting trash

and doing all kinds of useful things.

Well, I'm an artist in residence,

so I have these ideas about how can we imbue these robots

with more personality

and give a sense of wonder and delight

in sides of these offices.

So my first idea was that we would have the robots

play these bespoke, sculptural musical instruments.

So the robots are going down the hall, they wipe the table,

they play a little cello, and then they keep going.

And that was my first idea.

So I was brainstorming on this with my friend Tom Emberson,

who's a software engineer at Everyday Robots.

And Tom had this incredible provocation.

He said, What if the robot was the instrument?

So after a few months working with Tom

and a couple of software engineers, Adrian Lee,

Daniel Lamb, Emery Fisher, Kyle Jeffrey,

and our marvelous composer, Peter Van Stratton,

we turned Tom's provocation

into a new software called Music Mode.

In Music Mode, we take every single one of the joint,

the robot's, joint velocities,

and we map them to different sounds

so that the robot makes music while it moves.

This is so different

from how many choreographers are used to working

where you pick a piece of music

and you choreograph something to it.

In this case, we literally took those encoder values

and we translated them into sound, we sonified them.

So you have this robot that's wiggling around

and it's creating music.

This was very artistically challenging

because we needed to have all of these different joints

be able to play in combination with each other

and sound great in any permutation at any time.

And we also wanted the joint itself

to sort of match up with that kind of music.

So if you had a big heavy torso joint,

you wanted a low sound

or a tiny, tiny gripper joint, you needed a high bell sound.

So we decided to go with analog instruments

because we thought it add this warmth and depth

and richness to the robots

that we weren't getting otherwise.

And here's what it looks like just on the robot's elbow,

so you can hear the woodwinds and listen for it.

[gentle music]

Here's what it looked like to have many joints

moving in combination with each other.

For all the musicians out there,

you'll see that the head is going

with these kind of pulsating brass sounds.

We've got the bell of the gripper, the strings on the base.

And here's what it sounds like.

[soft music]

[soft music continues]

This feature became so popular

that we put it on all 180 robots,

and we allowed the software to be turned on at any time

because it didn't interfere with the robot's motions.

So the robots could be doing anything,

and you would hear the music like wipe a table

and you hear the symphony.

[soft music]

[audience clapping]

I got a number of cold emails from different engineers

who were sort of breaking into tears

in the middle of the day

because they were feeling themselves having this moment

of human connection with a robot.

And again, it was this sort of symbiotic nature

between art and human robot interaction and technology

where it just elevated this task

from a table wiping task

into this beautiful part of everyday life.

And made me think a lot about,

wow, we could live that way if we wanted to.

So now that we had music

that you could dynamically generate with movement,

I wanted to create this project that would get the robots

to improvisationally move around

so we could have many robots

who were all playing this at once.

And this is now called Project Starling.

So Project Starling started from this question,

what would it be like for humans and robots to move together

as if they were in a canonical,

sort of beautiful flock together,

as if it was as natural

as all of us moving through this building?

And we started with this algorithm called Boids Algorithm,

it's a very well-known flocking algorithm

and then we added a ton of stuff on top of it.

So one of the features that we added

was this circling feature.

We could get the robot to sort of swirl in these whirlpools

around the middle of the atrium.

We also started working on a following feature.

So you can imagine kind of being a camp counselor

and having your kids follow you around.

That's a little bit what this felt like.

So this was our following feature that we incorporated.

We added these parallel lines so you could have this sort of

very beautiful wave-like action

that you were experiencing with the robots

as they were going by.

And then I trained a machine learning model

to select between the gains

for all of these different features,

much like I myself, a choreographer would.

What does that mean?

It means that every single time these robots would flock,

the experience was completely different.

It was improvisational, it was surprising,

it was organic, it felt different, it was compelling.

So every single time that this experience happened,

it was never the same.

And this is exactly what we were trying to capture

about this sort of symbiotic nature of art and technology.

We then added some gesture recognition on top of it

so I could put my hands together

and the robots would all look it up at the sky.

I put my left hand in the air, they would all turn,

put my right hand in the air,

they'd all open and close their grippers.

So again, we kind of, little by little,

we were layering on this artistry

and it allowed you to have a new novel experience

every single time.

So this is what everything looks like all together.

The music is music mode.

The flocking behavior, of course, is from Project Starling.

[gentle music]

[audience clapping]

Kids love this, by the way,

and they're a very tough audience.

So I thought that was a great endorsement.

And we actually ran a research experiment,

both with music mode and with Project Starling.

And we found that robots that were running music mode

while they were doing a variety of different kinds of tasks

were considered more anthropomorphic,

animate, likable, safe, and intelligent.

This was what we learned from music mode.

And so then we ran a study with this flocking behavior

with Project Starling,

and we asked people what they experienced

as they did this experiment.

And these are some of the responses that we got.

I feel like I'm in heaven.

Am I ascending?

They were so cute out there singing and dancing.

I have no idea what they were actually supposed to be doing,

but it made me very happy.

And This feels poetic.

I'm having a visceral emotional reaction.

We are all spending quite a bit of time these days

talking about artificial intelligence and robotics.

I invite you to consider all the ways

in which you can use artificial intelligence and robotics

to unlock deep, human feeling

to create businesses, research and art

that are profound, joyful, and meaningful.

Maybe even something that feels like you're ascending.

My work is about making robots more friendly,

interactive, easy to teach, and easy to navigate around

and to take robots a step further,

to elevate them from a tool

to a whimsical part of everyday life.

And again, I get asked all the time, why should we do this?

I've got a lot of great answers about robot safety,

robot legibility, human robot collaborative effectiveness.

Yet my tendency is to ask why not?

What kind of world do we wanna live in?

One where our experiences with robots and other technologies

make us feel fear, boredom, confusion, less human?

Or one where our proximity to and collaborations with robots

give us a sense of curiosity, joy, and deep humanness?

Let's keep exploring,

building businesses, research and art

that creates the world we want to live in.

[audience clapping]