live.shamelfahmi.com

The Dynamic Legged Systems (DLS) Lab

  • Part of the Italian Institute of Technology, Genoa, Italy

  • 14 years of expertise in legged systems research

  • Robot Design, Locomotion, (Hydraulic) Control, Perception, etc.

  • Focus on heavy duty robots over rough terrain

Lab Head:
Claudio Semini

The Dynamic Legged Systems (DLS) Lab

HyQ: Early Work (2008-2013)

  • The first steps of HyQ

  • Mainly open loop with pre-generated trajectories 

C. Semini et al., Design of HyQ - a Hydraulically and Electrically Actuated Quadruped Robot JSCE, 2011.

HyQ: Early Work (2008-2013)

  • Focus on low level torque control (joint, leg and torso)

  • Blind locomotion

V. Barasuol et al.,A Reactive Controller Framework for Quadrupedal Locomotion on Challenging Terrain, ICRA, 2013.
C. Semini et al., Towards versatile legged robots through active impedance control, IJRR, 2015.
T. Boaventura et al., Model-Based Hydraulic Impedance Control for Dynamic Robots Robotics, T-RO, 2015.

HyQ2Max (2014-2016)

  • 2nd generation of the HyQ flagship robots

  • Challenges due to HyQ's workspace

  • HyQ2Max's workspace allows it to self-right

C. Semini et al, Design of the Hydraulically-Actuated, Torque-Controlled Quadruped Robot HyQ2Max, T-Mech, 2016.

HyQ2Max (2014-2016)

HyQReal (2019-)

  • What did HyQ2Max lack?

  • Power Autonomy, Robustness, and Reliability

    • MOOG Actuators

    • HPU (on-board)

    • Power Supply (on-board)

  • 3rd generation of the HyQ flagship robots

  • Quadrupedal Robots for Heavy Duty Operations

Piaggio P180 Avanti
Weight: 3300kg
Length: 14.4m

Wingspan: 14m

C. Semini et al, Brief introduction to the quadruped robot HyQReal, I-RIM, 2019.

HyQReal (2019-)

HyQReal (2019-)

  • First steps of HyQReal

  • Before pulling the plane

Tele-manipulation with HyQReal (2021)

  • Robot Teleoperativo (Partners: IIT + INAIL*)

  • Separate humans from hazardous situations

  • Teleoperating the robot

    • Hand exoskeleton and a VR interface (pilot station)
    • Teleoperate a robot with an arm (field robot)

*National Institute for Insurance Against Accidents at Work

Tele-manipulation with HyQReal (2021)

INAIL-IIT Robotic Arm

  • ARM:

    • Custom designed to fit the project's requirements

    • 5 DOF robotic arm

    • Leight weight: 5kg excluding end-effector

    • Payload: 5kg excluding end-effector

    • Torque/Impedance Control

    • Optimized Workspace

      • Pick up

      • Operate

      • Stow (for better distributed weight when not used)​

  • HERI-II hand

    • 4-fingered

    • Integrated pressure sensing

    • Integrated visual feedback

E. Barrett et al., A Compliant Mechanism with Progressive Stiffness for Robotic Actuation, AIM, 2021.
Z. Ren et al., HERI II: A robust and flexible robotic hand based on modular finger design and under actuation principles, IROS, 2018.

REMOTArm + HEXOTrAc-Plus

I. Sarakoglou et al., HEXOTRAC: A highly Under-Actuated Hand Exoskeleton for Finger Tracking and Force Feedback, IROS, 2016.
A. Brygo et al., Synergy-Based Interface for Bilateral Tele-manipulations of a Master-Slave System with Large Asymmetries, ICRA 2016.
G. Li et al., A Novel Orientability Index and the Kinematic Design of the RemoT-ARM: A Haptic Master with Large and Dexterous Workspace, 2020.

REMOTArm + HEXOTrAc-Plus

Robot Teleoperation

Overview of the DLS lab's Robots

Hip Adduction-Abduction (HAA)

Hip Flexion-Extension (HFE)

Knee Flexion-Extension (KFE)

Legged Robots are moving out from research labs

into the real world.

Is there anything common in these applications?

The terrains that legged robots traverse are never the same.

  • Dynamic: They can change or move.

  • Unexplored: The robot has not traversed it before.

  • Uncertain: In scenarios where vision is denied, or when the robot's perception is noisy.

The terrains introduce a large amount of uncertainty.

How to deal with this uncertainty?

Legged Robots Have to be Terrain Aware.

Terrain Aware Locomotion (TAL) means that the robot is able to take decisions based on the surrounding terrain using its perception.

Terrain Aware Locomotion (TAL)

  • Proprioceptive TAL (PTAL)

  • Exteroceptive TAL (ETAL)

Relies on the internal robot measurements to acquire the terrain information.

​Relies on directly acquiring the terrain information mainly using the robot's vision.

load-cells, torque sensors, encoders, etc.

lidar, radar, cameras, temperature sensors, etc.

Proprioceptive TAL (PTAL)

Barasuol 2019

Focchi 2013

M. Focchi et al., Local Reflex Generation for Obstacle Negotiation in Quadrupedal Locomotion, CLAWAR, 2013.
V. Barasuol et al., On the detection and localization of shin collisions and reactive actions in quadruped robots, CLAWAR, 2019.

Exteroceptive TAL (ETAL)

Barasuol 2015

Villarreal 2019

V. Barasuol et al., Reactive Trotting with Foot Placement Corrections through Visual Pattern Classification, IROS, 2015.
O. Villarreal et al., Fast and Continuous Foothold Adaptation for Dynamic Locomotion through CNNs, RA-L, 2019.

Proprioceptive TAL vs. Exteroceptive TAL

PTAL is crucial when

  • Visual feedback is denied (smoky areas, or areas with thick vegetation). 

  • Terrain map is unreliable.

 

 

But

  • Limited to corrective actions (step reflexes).

  • Do not act on what is ahead of the robot.

Let legged robots take decisions based on the surrounding terrain using their perception.

decisions

terrain

perception

Let legged robots take decisions based on the surrounding terrain using their perception.

Perception

  • Control

  • State Estimation

  • Planning

Terrain

  • Proprioception

  • Vision

Decisions

  • Physical Properties

    • Friction

    • Impedance

  • Geometry

Using Optimization and Machine Learning

Passive Whole-Body Control (pWBC)

STANCE: Locomotion Adaption over Soft Terrain

ViTAL: Vision-based Terrain-Aware Locomotion

Outline

  • Part I: Proprioceptive Terrain-Aware Locomotion

    • Passive Whole-Body Control (pWBC)

    • STANCE: Locomotion Adaption over Soft Terrain

 

  • Part II: Exteroceptive Terrain-Aware Locomotion

    • ViTAL: Vision-base Terrain-Aware Locomotion

S. Fahmi, C. Mastalli, M. Focchi, and C. Semini, “Passive Whole-Body Control for Quadruped Robots: Experimental Validation over Challenging Terrain,” IEEE Robot. Automat. Lett. (RA-L), vol. 4, no. 3, pp. 2553–2560, Jul. 2019.

 

S. Fahmi, M. Focchi, A. Radulescu, G. Fink, V. Barasuol, and C. Semini, “STANCE: Locomotion Adaptation over Soft Terrain,” IEEE Trans. Robot. (T-RO), vol. 36, no. 2, pp. 443–457, Apr. 2020.

 

S. Fahmi, “On Terrain-Aware Locomotion for Legged Robots,”  Ph.D. disseration, Apr. 2021.

Whole-Body Control (WBC)

  • To achieve dynamic locomotion, the robot should:

    • Track the desired reference trajectories.

    • Maintain balance.

    • Reason about the robot's (full) dynamics and limits.

 

  • To achieve TAL, the robot should reason about the contact interaction such as the terrain's inclination and frictional properties, etc.

 

  • How to deal with all these objectives in an optimal way?

 

  • A Whole-Body Control (WBC) Framework.

Whole-Body Control (WBC) Framework

  • Locomotion Planner [1]-[3]

  • State Estimator [4]

  • WBC

  • Low-level Control [5]

[1] B. Aceituno-Cabezas et al., Simultaneous Contact, Gait and Motion Planning for Robust Multi-Legged Locomotion via Mixed-Integer Convex Optimization, RA-L 2017.
[2] C. Mastalli et al., Trajectory and Foothold Optimization using Low-Dimensional Models for Rough Terrain Locomotion, ICRA, 2017.
[3] M. Focchi et al., Heuristic planning for rough terrain locomotion in presence of external disturbances and variable perception quality, STAR, 2020.
[4] S. Nobili et al., Heterogeneous sensor fusion for accurate state estimation of dynamic legged robots, RSS, 2017.
[5] T. Boaventura et al., Modelbased hydraulic impedance control for dynamic robots," T-RO, 2015.

  • Execute the planned trajectories of the body and swinging leg (control tasks).

  • Keep the robot balanced.

  • Respect the un-actuated part of the dynamics. 

  • Respect the robot's actuated part of the dynamics, and the joint & torque limits.

  • Respect the contact constraints due to the interaction with the terrain.

  • Maintain contact consistency.

Whole-Body Control (WBC) Objectives

  • Cast these objectives as an optimization problem via Quadratic Programming (QP).

  • Map the solution to joint torques using inverse dynamics.

Objectives:

1. Control Tasks Tracking

2. Physical Consistency

3. Joint & Torque Limits

4. Friction Constraints

5. Unilaterality Constraints

6. Stance Task

Results: pWBC

B. Aceituno-Cabezas et al., Simultaneous Contact, Gait and Motion Planning for Robust Multi-Legged Locomotion via Mixed-Integer Convex Optimization, RA-L 2017.
C. Mastalli et al., Trajectory and Foothold Optimization using Low-Dimensional Models for Rough Terrain Locomotion, ICRA, 2017.

Recap: pWBC

  • The pWBC is aware of the terrain's geometry and friction.

 

  • The pWBC can achieve dynamic locomotion while compliantly balancing the robot's trunk.

 

  • We showed the capabilities of the WBC under different gaits and different rigid terrain conditions.

 

  • Now, is the WBC terrain aware?

S. Fahmi et al., Passive Whole-Body Control for Quadruped Robots: Experimental Validation over Challenging Terrain, RA-L, 2019.

  • Most of WBCs are not terrain-aware. They fail to generalize beyond rigid terrain.

  • Locomotion over soft terrain by itself is challenging because of the induced contact dynamics.

What is the problem?

  • Mismatch between what the WBC assumes vs. what is actually happening.

  • Drop the assumption of the rigid terrain. Use a more generic model that accounts for the terrain impedance.

  • But that’s not enough, how do you know the impedance parameters of the terrain. 

  • Can we, estimate it?

How to solve it?

sWBC

Passive WBC (pWBC) = Standard WBC (sWBC)

C3WBC

Reformulate the sWBC to account for the contact dynamics

C3 = compliant contact consistent

C3WBC

How will the new WBC know about the current terrain impedance parameters?

C3 = compliant contact consistent

?

C3WBC

Feedback the terrain impedance parameters using a Terrain Compliance Estimator (TCE)

C3 = compliant contact consistent

TCE

Simultaneously, at every control loop

C3WBC

TCE

C3WBC

TCE

Soft Terrain Adaptation aNd Compliance Estimation
(STANCE)

Results: STANCE

Results: STANCE

Results: STANCE

Results: STANCE

  • Unlike the state of the art WBC, we do not assume that the ground is rigid.

 

  • Adapt online to any type of terrain compliance.

 

  • With STANCE, HyQ can traverse and transition between multiple terrains with different compliance without pre-tuning.

 

  • Each leg independently senses and adapts to changes in terrain compliance.

 

  • STANCE makes the locomotion strategy C3. It allowed HyQ to remain consistent with ground, and thus more robust in challenging and scenarios.

Recap: STANCE

S. Fahmi et al., STANCE: Locomotion Adaptation over Soft Terrain, T-RO, 2021.

Outline

  • Part I: Proprioceptive Terrain-Aware Locomotion

    • Passive Whole-Body Control (pWBC)

    • STANCE: Locomotion Adaption over Soft Terrain

 

  • Part II: Exteroceptive Terrain-Aware Locomotion

    • ViTAL: Vision-base Terrain-Aware Locomotion

Locomotion Planning

Two ways to deal with locomotion planning:

1. Coupled Planning:

  • Plan the feet & base states simultaneously.

2. Decoupled planning:

  • Separate the plan into feet & base plans.

  • Planning the feet motion ⇒ foothold selection

  • Planning the body motion ⇒ pose adaptation

  • Planning the feet motion ⇒ foothold selection

Vision-Based Foothold Adaptation (VFA)

Esteban 2020

  • Better Evaluation Criteria

  • Improved CNNs

Villarreal 2019

  • Better Evaluation Criteria

  • Self-supervised via CNNs

Barasuol 2015

  • Simple Evaluation Criteria

  • Expert demonstrations

Foothold Selection

V. Barasuol et al., Reactive Trotting with Foot Placement Corrections through Visual Pattern Classification, IROS, 2015.
O. Villarreal et al., Fast and Continuous Foothold Adaptation for Dynamic Locomotion through CNNs, RA-L, 2019.
D. Esteban et al., On the Influence of Body Velocity in Foothold Adaptations for Dynamic Legged Locomotion via CNNs, CLAWAR, 2020.

Goal: Select footholds based on the terrain information and the robot's capabilities. 

  • Despite the work done on vision-based strategies for foothold selection, pose adaptation strategies lag behind.

 

 

  • Pose adaptation is important because to reach a certain foothold, the robot's pose has to move simultaneously with the legs.

What is the problem?

Pose Adaptation

M. Kalakrishnan et al., Learning, planning, and control for quadruped locomotion over challenging terrain, IJRR 2011.
P. Fankhauser et al., Robust Rough-Terrain Locomotion with a Quadrupedal Robot, IROS, 2018.

Kalakrishnan 2011

Fankhauser 2018

Goal: Adapt the robot's pose based on the terrain information and the robot's capabilities. 

The Problem with Pose Adaptation

  • The problem with current pose adaptation strategies is that they focus on finding one optimal solution based on given selected footholds.

 

  • First, the footholds are selected.

  • Then, the pose is optimized based on that.

 

  • What if the given footholds are not reached (if the robot gets disturbed)?

 

  • The robot may end up in a state with no reachable safe footholds. 

  • Propose a different paradigm for pose adaptation.

 

  • Instead of finding body poses that are optimal w.r.t given footholds,

 

  • we find body poses that will maximize the chances of the robot to reach safe footholds.

 

  • Put the robot in a state that if it gets disturbed, it remains around a set of solutions that are still safe & reachable.

How to Solve This Problem?

Vision-Based Terrain Aware Locomotion (ViTAL)

  • ViTAL is an online vision-based locomotion planning strategy.

 

  • ViTAL selects footholds based on the robot's skills

 

  • Also, find the robot pose that maximizes the chances of success in reaching these footholds.

 

  • The notion of safety & success emerges from skills that characterize the robot's capabilities.

Vision-Based Foothold Adaptation (VFA)

Vision-Based Pose Adaptation (VPA)

Foothold Evaluation Criteria (FEC)

What the robot is capable of doing?

  • Avoid edges, corners, gaps, etc.
  • Remain with the workspace of the legs.
  • Avoid colliding with the terrain.

Vision-Based Foothold Adaptation (VFA)

Vision-Based Pose Adaptation (VPA)

Results: Performance with Different Velocities

Preliminary Experiments

Recap: ViTAL

  • ViTAL allows HyQ and HyQReal to traverse a wide variety of terrains under various forward velocities.

    • These terrains included stairs, gaps, etc.

    • Forward velocities varied from 0.2m/s to 0.75 m/s.

 

  • We showed that the VPA resulted in body poses that are aware of the terrain, and of what the robot and its legs can do.

 

  • The VPA puts the robot in a pose that provides the feet with higher number of safe footholds which allows the robot to succeed.

Claudio Semini

Victor Barasuol

Geoff Fink

Michele Focchi

Andreea Radulescu

Octavio Villarreal

Domingo Esteban

Carlos Mastalli

Pointers: things I did not talk about

  • Model Predictive Control

    • N. Rathod et al., Mobility-enhanced MPC for Legged Locomotion on Rough Terrain, under review, 2021.

    • O. Villarreal et al., MPC-based Controller with Terrain Insight for Dynamic Legged Locomotion, ICRA, 2020.

 

  • Feasible Region

    • A. Abdalla et al., An Efficient Paradigm for Feasibility Guarantees in Legged Locomotion, under review, 2021.
    • R. Orsolino et al., Feasible Region: an Actuation-Aware Extension of the Support Region T-RO, 2020.

 

  • State Estimation

    • S. Fahmi et al., On State Estimation for Legged Locomotion over Soft Terrain IEEE Sensors Letters, Jan. 2021.

    • G. Fink and C. Semini, Proprioceptive Sensor Fusion for Quadruped Robot State Estimation, IROS, 2020.

 

  • Robot Balancing

    • C. Gonzalez et al., Line Walking and Balancing for Legged Robots with Point Feet, IROS, 2020.