SenseKit Addition and Configuration

Guide to add a new SenseKit and configure it.

Adding a New SenseKit

Following are the steps required to add a new SenseKit:

1. Open the Configuration tab in the wrist panel.

2. Press on the SenseKits button, which will open the Active SenseKits panel.

3. Press on Add New button, which will open the panel with input fields.

4. Input the IP and port (default port is 9091) of the SenseKit.

   
5. Press Connect this will try to connect to the SenseKit device and find any configurations saved on the SenseKit device.

6. Previously stored configurations on the SenseKit device will be shown in the panel, and can be selected to load the configuration.

7. If no configurations are available, click on Create New Config button on the panel and define new configuration

SenseKit Configuration Elements

• Name : This field is name of the SenseKit configuration

• Mode : This field is used to the view option of the RGBD streaming

  • 3D (default) : show 3D point cloud

  • 2D : show 2D video

  • 3D + 2D: show 3D point cloud and 2D video

• Depth Codec :

  • zdepth (default) : lossless depth compression with sensor based accuracy

  • yuvdepth : lossy low bandwidth depth compression, not supported on Jetson

• Color Codec :

  • MJPEG : color compression with high bandwidth requirement

  • H.264 (default) : recommended color compression codec to use

• Fusion Offset (m unit) : when fusing multiple sensors treat sensor as if data was reported with larger distance to avoid noisy data fighting for view, only small values recommended (0.01 - 0.1) as it may have side effects with VR scene

• Cutoff Distance (m unit) : define to remove from 3D view data with depth larger than value, 0 to disable

• Cutoff Replacement (m unit) : define to replace data at depth larger then cutoff distance and missing data with the value, typically set to 0 (remove) or matches Cutoff Distance to perform 2D-3D Fusion

1. If the SenseKit Firmware Specific Settings are not configured during the OTA Firmware Update Process or require an update, click on the Sensekit Firmware Specific Settings button. Sensekit Firmware Specific Settings panel contains the following fields:

SenseKit Firmware Specific Settings

• ROS Port (default 9091) : the port used for connecting the AMAS VR app to the SenseKit

• Data Port (default 12345) : the port used for connecting the AMAS VR app to the RGBD streaming pipeline of the Sensekit

• Color Compression Config : hardware acceleration to use. In typical scenarios default is set per sensor when installing firmware and should not be changed. When installing firmware on non Extend Robotoics hardware choose method supported by hardware.

  • jetson_nvmpi.yaml : NVidia Jetson NVMPI (Jetson only)

  • nvenc.yaml : NVidia NVENC (x86-64)

  • nvenc_18_04.yaml : NVidia NVENC for Ubuntu 18.04 Bionic firmware (x86-64)

  • nvenc_motioncam.yaml : NVidia NVENC optimized for Photoneo MotionCam-3D (x86-64)

  • nvenc_zed_2i.yaml : NVidia NVENC optimized for StereoLabs ZED 2i (x86-64)

  • vaapi.yaml : Intel VAAPI (x86-64)

  • vaapi_amd.yaml : AMD VAAPI (x86-64)

  • vaapi_low_power.yaml : alternative Intel VAAPI low power encoding path (x86-64)

  • vaapi_motioncam.yaml : Intel VAAPI optimized for Photoneo MotionCam-3D (x86-64), may fail with non-standard MotionCam resolutions with firmware side iHD VAAPI driver and will work with i965 VAAPI driver (if hardware supports it)

• Enable Audio : enable/disable audio support on SenseKit, this setting should be only enabled for SenseKits with audio hardware. When enabled, please follow the instructions at AudioKit Addition and Configuration to configure audio support.
• Pose Sync Settings : toggle this option to send moving camera pose from reference to relative frame with the data (only available in setups with robot and sensor on single embedded computer). Do not enable this option if you have no intention to use pose data as waiting for it may add small delay.

  • Reference Frame : typically set to base of the arm link_base

  • Relative Frame : set to camera mounting point, for example end-effector camera mounted on arm with 7 joints would use link7

• FPS : desired framerate of the sensor, currently supported only for Kinect4A, FemtoBolt, MotionCam-3D, ZED-X and ZED-XM. Note that typically sensors support only limited set of framerates and may fail, fallback to default or not reach non-standard framerate. Decreasing framerate may help if bandwidth is limited. Increasing framerate may improve real-time performance at the cost of computational resources and bandwidth.

Some sensors have more unique firmware settings. For description seeSensor Specific Firmware Settings.

Changing firmware settings requires reboot of SenseKit to take effect.

1. Press Save Local or Save Local and Device to finish the configuration and connect to the SenseKit.

2. The Active SenseKits panel will now show this SenseKit.

Adding a New SenseKit

Modifying a Currently Active SenseKit

Follow the steps given below to modify a currently active SenseKit:

1. Open the Configuration tab in the wrist panel.

2. Press on the SenseKits button.

3. Press on the SenseKit name you would like to modify.

4. Edit the configuration of the SenseKit accordingly.

5. Press Save. This will update the SenseKit, and will update the stored configuration