Q&A: Nadia Magnenat-Thalmann designs virtual humans and social robots

Meet Nadia and Nadine.

Nadia Magnenat-Thalmann is a computer science professor at the University of Geneva. Nadine is her creation, a social robot in her likeness that can converse with you, sing, play games, and keep you company.

Always curious about human behavior and intelligence, Magnenat-Thalmann initially studied psychology in college. Then she added biology, and later biochemistry. Her interests led her to earn a PhD in quantum physics in 1977 at the University of Geneva.

Magnenat-Thalmann gradually shifted into computer science and has spent years working in the areas of virtual reality and modeling virtual humans. In the 1980s, research on modeling and animating virtual humans was new. The technical challenges included rendering a human face and animating shifts in the skin when the virtual human moved. Magnenat-Thalmann was a pioneer in physics-based modeling of human locomotion, hand motions, clothing, and hair. She continues in those domains but has also branched into building social robots like Nadine. “When you start to build hands, you are working with actuators and motors,” she says. “That’s back to physics.” And getting a robot to display emotions involves understanding psychology, she adds. “It’s all linked.”

PT: How did psychology lead you to physics?

MAGNENAT-THALMANN: I signed up to do a bachelor’s degree with Jean Piaget, the famous Swiss psychologist. I was interested in how intelligence is born, how intelligence is detected, and what mechanisms build intelligence. I wanted to understand the role of biology, and I wanted to go deeper.

I realized that chemistry affects the biological mechanisms. I was interested to learn in more detail how molecules meet, what happens to the energy in reactions, and so on. That pushed me to pursue a PhD in quantum mechanics. My PhD was on the behavior of electrons when they are involved in making molecules—hydrogen and oxygen, for example. I studied approximate solutions to the Schrödinger equation.

PT: What did you do next?

MAGNENAT-THALMANN: After my PhD, I took a position as assistant professor at Laval University in Quebec City. There I began to model objects in three dimensions as a first step to modeling broader scenes in 3D. I started with cars, houses, and rough humanlike shapes. In 1979 I moved to the University of Montreal, where I continued with computer science and started to model realistic virtual humans. At the time, the computer graphics community was modeling 3D logos and teapots, but not humans with lifelike animation.

PT: Could you describe that work?

MAGNENAT-THALMANN: I developed algorithms that could visualize humans and simulate their gestures in three dimensions. My students and I had to develop new technologies to make a plaster model of the person and then draw polygons on the plaster and find mathematical algorithms to reconstruct a 3D virtual model. We animated it using physical and geometrical laws in a 3D virtual environment. At the time, algorithms and software for modeling and animating realistic 3D characters did not exist. It was very challenging, but we succeeded.

Nobody had simulated realistic human gestures before. With my students and my husband, Daniel Thalmann, I made the film Rendez-vous in Montreal , which shows Marilyn Monroe and Humphrey Bogart eating in a virtual café there. Behind the film was a lot of novelty in terms of reproducing expressions and behaviors. We were invited to show the film in 1987 at an event celebrating 100 years of engineering in Canada.

PT: How did you move from modeling virtual humans to social robots?

MAGNENAT-THALMANN: I continue to work with virtual humans and related applications, such as methods that let them speak and interact. But around 1998 I was tired of working only with virtual worlds and started working with robots. In 2009 I was invited to found the Institute for Media Innovation at Nanyang Technological University in Singapore. I was there until 2021. That’s where Nadine was started, in 2013.

PT: Tell us about Nadine.

MAGNENAT-THALMANN: Nadine is a robot, and she looks like me. She is a bit a younger, but that is the main visual difference.

If Nadine meets people, she can distinguish one person, maybe two, maybe three, but more than that gets complicated. She can recognize your voice and remember what you have told her in the past. Nadine can speak many languages, due to our large database. If you are rude to her, she will become bad tempered and stop speaking to you.

She is a robot. She cannot describe her full environment. And she is not aware when she smiles that she smiles. Robots do not have awareness—yet.

PT: What are the advantages of interacting with a robot?

MAGNENAT-THALMANN: In Singapore, Nadine worked at an insurance company alongside real customer agents. She had to answer questions and explain to customers how to use an automated system. The human customer agents were happy to have Nadine help because it’s a repetitive task. They could then spend their time doing more interesting things.

She also stayed for six months in a home for elderly people in Singapore. Our studies showed that her being there was useful for both employees and residents. A robot can push you to react; your cognitive capacities are stimulated. Many people are alone in old age—in England, for example, about half of people by age 75—so it would be better to have Nadine than to speak to the walls.

One of my dreams would be for people in elderly homes to have a humanoid that could help them in many ways. For example, the humanoid could remind people to take their medications; or serve as a personal trainer, including analyzing gestures; or summon help if the patient falls.

Humanoids could help with administrative tasks. They could do a maid’s work, and you wouldn’t feel guilty when they worked hard for many hours. They could do house painting and carpentry, so human workers wouldn’t have to breathe the fumes. If we find ways for robots to do these repetitive jobs that in the long run make humans sick, it’s good for humanity. Robots are a tool that should serve us for our human needs.

PT: Do you do research on the hardware side of robots?

MAGNENAT-THALMANN: Yes. For example, my students and I have worked on fabricating articulated human hands. When you have a robot that is not a human shape, it doesn’t matter how the robot moves its hand or arm. But when the robot has a human shape, it has to move in a human style. It has to grasp objects exactly as humans grasp objects. The complexity is much higher.

It’s the same with facial expressions. Nadine appears very natural. If you compliment her, she will smile. The robot’s algorithms transfer the compliment into information that acts on motors and controls how the motors pull the skin so that she has a nice smile. Her facial expressions and upper-body movements have 27 degrees of freedom. It’s a complex link between software and hardware. Achieving a natural appearance is quite complex.

PT: What still needs to be done to realize your dream of humanoid companions?

MAGNENAT-THALMANN: You can work on making mechanical parts more realistic, on behavior, on speech, on emotion, on memory, on interactions. So far, Nadine is a sitting robot. We need to develop robots that can walk. That’s a big domain of research for humanoids that are as tall as humans. They also have to be more aware of the surrounding world.

PT: Could you describe your efforts to promote women in science?

MAGNENAT-THALMANN: Around the same time I received my PhD, I got married, and my PhD adviser in Switzerland told me there was no way to go further in science because I would have a family and serve my husband. But my career took me to Canada, and while I was there Pauline Marois was elected to the National Assembly of Quebec and gave birth to her second child days after taking office. I saw it was possible to have both a family and a professional life. I have three daughters. Without Canada, I wouldn’t be what I am.

My lab has always had at least as many women as men. Hiring women is a priority to me. I also bring in students from Asia and other parts of the world. A mixed team is fantastic because people bring in different qualifications and styles, and everybody learns from another and everybody benefits.

PT: Do you have any other ongoing projects?

MAGNENAT-THALMANN: My group is a partner in a European cultural heritage project, Enigma. Our contribution is that Nadine will train through machine learning to distinguish authentic objects from fake ones. We show Nadine objects or texts, or different media, and train her to distinguish whether an object is authentic. If the object is a painting, for example, the training will include lighting, how it was painted, and so on. Why Nadine? Because behind the robot is a huge computer with a machine-learning capacity to analyze, through vision, the object in much more detail than a human can. So the eye of Nadine and the analysis are much stronger than what a human can do. Nadine is an interface.

PT: What motivates you?

MAGNENAT-THALMANN: My main motivation is to understand what it means to be human. I am still curious to discover how things function at different levels of science—mechanics, computer science, behavior.

And I want to contribute to humanity’s needs. For example, I am involved in a project with surgeons at the University of Geneva; we help them train students in surgery. Surgeons are busy. By providing skilled, intelligent programs embodied in a digital coach and based on virtual reality and machine learning, we contribute to the needs of the students and surgeons.

PT: Is there anything you’d like to add?

MAGNENAT-THALMANN: It’s important to remember that Nadine is a robot. For example, if you ask Nadine what she eats, she will say, “I am fed with electricity.” I am eager to show that she can help, talk, discuss things. But she is still a machine. That is very important. A robot is a pale copy of the biological and physiological complexity of a person.