User Guide
v2025.03
v2025.03
  • Extend Robotics User Guide
  • introduction
    • Release Notes
      • AMAS v2025.03
      • AMAS V10.3
      • AMAS V9.3
      • AMAS V8.3
      • AMAS V7.6
    • AMAS Overview
    • Compatible XR Systems
    • Supported Robots
      • Robots
        • Ufactory
        • Universal Robots
        • Dobot
        • Pal Robotics
        • Mitsubishi Electric
        • AGILE-X
      • Accessories
        • Robotiq
        • ROBOTIS
        • Ufactory
        • On Robot
    • Choosing 3D Sensors
      • Limitations
      • Sensor Types
      • Support Matrix
      • Visual Comparison
    • Specifications
    • PC Requirements
    • Typical Network Connection
    • Network Requirements
  • getting started
    • Getting Started with AMAS
    • Navigating Through the User Guide
  • Cortex
    • Introduction
    • Robot Setup Instructions
      • UFactory xArm
      • UR e Series
      • UR CB Series
      • Dobot CR-A Series
      • AgileX
      • Dexterous Hands
        • Robotera XHAND1
        • Inspire Robots Dexterous Hand
      • Extend Avatar
        • Electronics Assembly
        • Mechanical assembly
          • Avatar Component Parts list
          • Assembly Instructions
        • xArm Adjustments
    • Sensor Setup Instructions
      • Sensor Positioning
    • Cortex Firmware Configuration
    • Instructions for Starting the Cortex
    • Cortex IP Configuration
    • Shutdown Instructions
    • Cortex Safety Protocols
  • AMAS VR Application
    • Download AMAS
      • Setup Quest Application
      • Running the Application
      • Home Scene Arrival
    • Control Panel
      • Interactions Modes
        • View Adjust Mode
        • Robot Control Mode
        • Interaction Utilities
      • Floating Keyboard
      • Error popup
      • Web Browser
    • Hand Tracking
    • Hand Gestures
    • Upper Body Tracking
    • Passthrough Mode
    • Managing Cortex
      • Adding Cortex Connection
      • Loading Cortex Connection
      • Modifying Robot Settings
        • Safety Parameter Update
      • Manual Robot Positioning
      • Modifying Sensor Settings
        • 2D - 3D fusion
      • Sensor Position Calibration
        • Calibration Prerequisite
          • Calibration Board
        • Sensor to Robot Calibration
        • Sensor to Sensor Calibration
        • Manual Calibration
    • Accessories
      • SenseGlove Nova 2
      • bHaptics TactGloves
    • Record and Replay
      • Creating a Recording
      • Load a Recorded Motion
      • Replay a Recorded Motion
    • Data Collection
      • Recording Data
      • Data Visualisation
        • ROS Bag
        • HDF5
        • LeRobot
    • Tutorial
    • Visual Haptic
      • Force Torque Calibration
    • Speech Recognition
    • Utilities
      • Stationary Headset
      • Exhibition Scene
    • UR Program Change
    • Mobile Robot Movement
      • Virtual Joystick
      • Quest series controller
    • Shutdown Instructions
    • Safety Utilities
      • Safety Utilities
      • Joint Limit Recovery Visualisation
      • Robot Rescue
    • Avatar Additional VR Functionalities
      • Avatar Digital Twin Assembly
      • Pose Sync Avatar Camera Setup
      • Torso Initialisation
      • Embody Control Mode
  • Command Console
    • Over-the-Air Firmware Updates
    • Updating Robot/Sensor Configurations
      • Robot Specific Firmware Settings
      • Sensor Specific Firmware Settings
  • Cortex API
    • Robots
      • Current Robot Publishers, Subscribers and Services
      • Gripper Customizations
      • Message and Service Definitions
    • Sensors
  • Other Information
    • Troubleshooting
    • Customized Hardware Integrations
    • Remote Demonstration Requirements
    • Guides
      • PAL TIAGo
        • Mobile Base, Arm and Torso Control
        • 3D Sensor
        • Additional Sensor Data Visualization
        • First Person View Mode
        • Take Home Functionality
        • Audio
        • Bundle
      • Robot - Sensor Sync
Powered by GitBook
On this page
  • Prerequisites
  • Overview
  • Details
  • Enabling pose sync in SenseKit firmware settings
  • Performing eye-in-hand calibration (with pose)

Was this helpful?

  1. Other Information
  2. Guides

Robot - Sensor Sync

Setup robot - sensor pose synchronisation for arm mounted sensor.

To see correctly synchronised 3D data from moving sensor user needs to setup synchronisation.

Prerequisites

Pose synchronisation is currently supported only in setups where the robot controller (RoboKit) and sensor controller (SenseKit) are running on the same embedded computer.

Overview

To setup pose synchronisation user needs to:

  1. Enable pose synchronisation using SenseKit firmware settings in AMAS.

  2. Perform eye-in-hand calibration (with pose) in AMAS.

The first step adds pose retrieved from the robot to the 3D data sent to AMAS. For example, we want the pose of the end-effector for the sensor mounted on the end-effector joint.

The second step enables the rendering of the 3D data with respect to the pose and calibrates the sensor position with respect to where it is mounted.

Details

Enabling pose sync in SenseKit firmware settings

In a typical scenario, the sensor is mounted on the last robot's joint with the end-effector.

We want the pose of this joint with respect to the robot base to be sent with the 3D data.

Setup the SenseKit firmware settings Pose Sync Settings as described in Modifying Sensor Settings with:

  • Reference Frame set to your robot base (typically link_base)

  • Relative Frame set to joint where sensor is mounted on robot (typically link5, link6 or link7)

Example: for arm with 7 joints like xArm 7 and sensor mounted with end effector, reference frame link_base and relative frame link7 should be setup.

Reminder: reboot the SenseKit after changing firmware settings

Performing eye-in-hand calibration (with pose)

In the previous step, we ensured that the pose of the robot joint that the camera is mounted on will be sent with 3D data. We still need to calibrate where the sensor is with respect to this joint.

Follow the Sensor to Robot Calibrationwith:

  • Calibration board at some fixed position near the robot arm

  • Mount type End Effector with Pose

Perform the calibration.

At the beginning of calibration, it may be helpful to enable 2D video feed from the camera. To enable 2D video feed, see Modifying Sensor Settings Set the mode to Point Cloud + Video

After taking 4 samples with the calibration board, initial calibration should be available and further calibration becomes simpler as 3D data will be aligned.

PreviousBundle

Was this helpful?