RAPP Store | DEMO

ladia, an elderly person, is requesting robot assistance by pressing a button on her tablet.

TIAGo, the robot assistant, receives the request and moves to the location of the tablet. Gladia issues the voice command to be brought a cup of tea.

TIAGo confirms the task it is about to execute and moves to the kitchen. In the kitchen, the robot issues the voice command to Adam, the caregiver. Adam prepares a cup of tea and places it on TIAGo’s tray, then issues the voice command that the order can be delivered. TIAGo moves to Gladia’s location and asks for confirmation. Gladia issues the voice command that TIAGo rotates. TIAGo executes the command. Gladia takes the cup of tea and places it on the coffee table in front of her, then confirms. After the confirmation, TIAGo moves to the initial position.

Tiago operates as an Agent in a smart environment organized as a multi-agent system with multiple smart devices that provide continuous information, connected through wireless network. The PAL robotic software is directly available to the Core Agent and is made available to the ecosystem through task requests.

The control system is based on the RAPP architecture extended by a Task Harmoniser Agent and a Task Requester Agent. The main nodes are:

• The Core Agent – drives the hardware of the robot and provides the services to be called by other Agents, such as navigation, control of torso and head, text-to-speech and the translation of voice commands in intents.
• The Task Harmoniser Agent – receives task requests from the Task Requester Agents and schedules Dynamic Agents basing on a specified algorithm (e.g. priorities).
• Task Requester Agents – request the Task Harmoniser Agent to perform tasks based on the data from robot’s or additional sensors.

The multi-agent system is implemented using SMACH, a ROS-compatible agent framework.

In this scenario, Tiago is in rest mode in the living room. In the kitchen, Michael falls. The INCARE fall and indoor localization module in its bracelet identifies a hazardous event and sends it to the server. From the server, the signal is received by Tiago, which immediately starts moving towards the signal location. When Tiago arrives at Michael, it detects him in an unusual pose and establishes audio and video connection with a remote caregiver, streaming its camera and microphone input. Tiago checks on Michael, asking him if he is feeling well.

This scenario exemplifies Tiago’s object manipulation capabilities by grasping objects on a table and lifting them a few centimeters above. Execution of dexterity tasks, combined with scene understanding, extends the usability of the robot in a new range of scenarios, such as cleaning a room by picking out-of-place objects and placing them in a basket or bringing objects to the user at the request.

The core of the mechanism is based on machine learning, the grasping poses being identified using a Generative Adversarial Network (GAN) trained on the Cornell Grasp Dataset in a task of distinction between correct and incorrect grasps for a multitude of household objects in different poses. The input of the GAN are 4-channel images containing colour and depth information, preprocessed to a fixed size of (256, 256, 4), randomly rotated or scaled for data augmentation and pseudo-random alteration of pixel intensities in the colour channel to make the learning process invariant of lighting conditions. The best grasp candidates are identified as centers in the 3D space of the robot, opening of the gripper and rotation angle of the wrist. The learning process is conducted using mini-batches over multiple epochs until convergence, employs CUDA tensors for computational speed and penalizes a channel-wise L1 loss between target and predictions. The optimal motion to reach the target pose is computed using the KDL Inverse Kinematics solver.

This scenario proves Tiago’s ability of understanding the surrounding environment and answering simple questions about it. Tiago navigates through the house and answers several questions about the rooms through which it passes. Tiago can:

• recognize basic objects from the house;
• count instances;
• recognize colors;
• answer yes/no questions.

As Gladia is in another room, she can send Tiago to check things for her. For example, she sends Tiago to the bathroom to check how many towels are in there and whether the bathtub contains water. Then, she sends Tiago to the bedroom, to check whether her husband, Michael, is sitting on the bed. Finally, she wants to know the color of the bed sheets, to make sure that she hasn’t forgotten to change them.

Tiago’s reasoning mechanism is based on 3 components:

1. a model for visual question answering that combines:
   1. a VGG-16 architecture with inputs of size 224×224 pre-trained on ImageNet (for understanding images);
   2. time series vectors with each word transformed into a 300 dimensional representation calculated using Global Vectors (for understanding questions);
2. a color detector that extracts the dominant color from a given region;
3. an external knowledge base (Visual Genome) with logical correlations between nouns and verbs.

Tiago still has difficulties in answering questions that involve details. For example, even if it recognizes a clock on the wall, it is not yet capable of reading it, to tell the exact hour. It is also hard for Tiago to count partially occluded objects—given a couch with several cushions, it does not count them all.

The following videos exemplifies the use of Tiago and TurtleBot robots for supporting the elderly.