Femto Bolt Documentation

Access AKDK Application Software with Femto Bolt

 

1. Overview

Orbbec SDK K4A Wrapper is designed and encapsulated based on Orbbec SDK, converting Orbbec SDK to Azure Kinect Sensor SDK interfaces. It mainly implements data stream reception, color parameter setting, D2C and point cloud functions, recording and playback, consistent APIs with Azure Kinect Sensor SDK, allowing users to quickly switch to Orbbec Femto Bolt and Orbbec Femto Mega cameras without modifying the application code.

Orbbec SDK K4A Wrapper open source link: https://github.com/orbbec/OrbbecSDK-K4A-Wrapper

The implementation principle of Orbbec SDK K4A Wrapper is as follows:

The characteristics of Orbbec SDK K4A Wrapper are as follows:

  • Maintain the original interfaces of Azure Kinect Sensor SDK unchanged.
  • Modify the implementation (impl) of Azure Kinect Sensor SDK C API, call Orbbec SDK internally to get video frames and control Femto Bolt and Femto Mega cameras.
  • Coordinate transformation, D2C, C2D, point cloud reuse Azure Kinect Sensor SDK.

Orbbec SDK K4A Wrapper currently supports the following cameras:

Orbbec Femto Bolt: Win10 x86/x64, Ubuntu18.04 x64, Ubuntu20.04 x64

Orbbec Femto Mega: Win10 x86/x64, Ubuntu20.04 x64

 

 

2.How AKDK applications switch to Orbbec SDK K4A Wrapper

This chapter mainly shows how to implement code project switching and adaptation to Femto Bolt cameras by replacing Azure Kinect Sensor SDK library files and header files with Orbbec SDK K4A Wrapper under the user application code project.

For compiled application executables, you can also directly switch by replacing the libk4a library file linked by the application executable with Orbbec SDK K4A Wrapper.

2.1 Windows Platform Application

2.1.1 Environment Configuration: Modify Registry

Due to the Windows system mechanism, for UVC protocol devices, if you need to get timestamp and other metadata information, you need to register in the registry first. You can execute the obsensor_metadata_win10.ps1 script to complete automatic registration according to the obsensor_metadata_win10.md document guide.

2.1.2 Replace Header Files

Orbbec SDK K4A Wrapper is developed based on Azure Kinect Sensor SDK V1.4.1 version. The header files directly use the original K4A library header files. If the user project originally used Azure Kinect Sensor SDK V1.4.1 version, this step can be ignored.

  • Find the header files in Orbbec SDK K4A Wrapper

  • Replace the corresponding header files in Azure Kinect Sensor SDK

2.1.3 Replace Library Files

Compile

  • First copy the k4a.lib and k4arecord.lib library files from the Orbbec SDK K4A Wrapper to the corresponding path of the Azure Kinect Sensor SDK.

  • Replace the corresponding k4a.lib and k4arecord.lib library files in the Azure Kinect Sensor SDK.

  • Then, copy the dll file from Orbbec SDK K4A Wrapper to the corresponding path in Azure Kinect Sensor SDK.

  • Replace the corresponding dll in Azure Kinect Sensor SDK.

Run

  • Find k4a.dll, k4arecord.dll, OrbbecSDK.dll, depthengine_2_0.dll library files in Orbbec SDK K4A Wrapper

Copy the above library files to the original executable application path of Azure Kinect Sensor SDK, and replace the original k4a.dll, k4arecord.dll, depthengine_2_0.dll library files to run the application program.

2.2 Linux Platform Application

2.2.1 Environment Configuration: Install udev rules Configuration

By default, Linux systems require root permissions for direct access to USB devices, which can be solved through the rules configuration file. Orbbec SDK K4A Wrapper provides a 99-obsensor-libusb.rules configuration file and install_udev_rules.sh installation script to complete the installation by executing the install_udev_rules.sh script.

Execution method:

sudo chmod +x ./install.sh # Make sure the installation script is executable
sudo ./install.sh # Execute the script with sudo

2.2.2 Replace Header Files

Orbbec SDK K4A Wrapper is developed based on Azure Kinect Sensor SDK V1.4.1 version. The header files directly use the original K4A library header files. If the user project originally used Azure Kinect Sensor SDK V1.4.1 version, this step can be ignored.

Find the original header files of Orbbec SDK K4A Wrapper and replace the corresponding header files under Azure Kinect Sensor SDK.

2.2.3 Replace Library Files

  • Find the library files in Orbbec SDK K4A Wrapper, including: libdepthengine, libk4a, libk4arecord, libOrbbecSDK.so.

  • Copy the above library files to the /usr/lib/x86_64-linux-gnu directory to replace the Azure Kinect Sensor SDK libraries to compile and use the application program with Orbbec SDK K4A Wrapper.

Note:

On Linux platforms, the depth engine library requires OpenGL. If the application layer also uses OpenGL and makes cross-thread calls to OpenGL instances, context conflicts may occur, causing problems during depth engine initialization and inability to start the depth data stream properly.

If the application layer uses OpenGL rendering, the application layer needs to switch the Context under OpenGL: (https://www.khronos.org/opengl/wiki/OpenGL_and_multithreading)

The following is our solution using the glfw library:

  1. Call glfwMakeContextCurrent(NULL) before starting the stream.
  2. Then start the stream.
  3. After the stream is finished, glfwMakeContextCurrent(currentContext).
// 参考github: file: tools/k4aviewer/k4adevicedockcontrol.cpp
GLFWwindow *currentContext = glfwGetCurrentContext(); // store the current context
glfwMakeContextCurrent(NULL); // make current context to NULL

StartCameras(); // will initialize the DepthEngine

glfwMakeContextCurrent(currentContext); // restore the current context

 

3.AKDK User Skeleton Algorithm Adaptation

The following uses the Azure Kinect Body Tracking SDK Sample as an example to describe how to obtain skeleton data by using Orbbec Femto Bolt camera data after replacing Azure Kinect Sensor SDK through Azure Kinect Body Tracking SDK (hereinafter referred to as K4ABT). (Users can switch by directly replacing the library files without recompiling)

3.1 Windows Platform

3.1.1 Skeleton Installation Package Download and Installation

Complete the installation according to Microsoft’s installation documentation: https://learn.microsoft.com/en-us/azure/kinect-dk/body-sdk-setup

Download Azure Kinect Body Tracking SDK 1.1.2.msi, then execute the file to complete the installation.

3.1.2 Demonstrate Skeleton Algorithm Effects

Complete the environment configuration (metadata registration) according to Chapter 2, then replace the following libraries of Azure kinect Sensor SDK in the installation directory with Orbbec SDK K4A Wrapper libraries (k4a.dll, OrbbecSDK.dll, k4arecord.dll, depthengine_2_0.dll), then run simple_3d_viewer.exe with administrator rights.

The effect of the skeleton algorithm is as follows:

 

3.1.3 Skeleton Algorithm Secondary Development (sample compilation)

1.Download Azure Kinect Samples

URL: https://github.com/microsoft/Azure-Kinect-Samples

Clone code:

git clone https://github.com/microsoft/Azure-Kinect-Samples.git

2. Open Microsoft skeleton sample with Visual Studio

Microsoft skeleton sample only supports opening with visual studio. Use VS2019 to open the following projects.

3.Use Orbbec SDK K4A Wrapper header files and library files to replace Azure Kinect Sensor SDK header files and library files.(dll, OrbbecSDK.dll, k4arecord.dll, depthengine_2_0.dll)。

  • Replace header files

  • Replace library files

4.Compile & Run

  • The simple_3d_viewer.exe example relies on the following 2 libraries, which are copied from the installation package to the bin directory where the compiled application is generated.

  • Copy the dependent libraries to the running directory:

 

  • Compilation running interface:

3.2 Running Kinect Skeleton Algorithm on Linux

Since Microsoft Azure Kinect Sensor SDK libraries only provide installation on Ubuntu 18.04, it is recommended to complete the following on Ubuntu 18.04 system.

3.2.1 Install Azure Kinect Sensor SDK

Refer to the Linux installation instructions section of Microsoft’s installation documentation to complete the installation. Document link:

Azure Kinect Sensor SDK download | Microsoft Learn

Installation instructions:

curl -sSL -O https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
rm packages-microsoft-prod.deb 
sudo apt-get update
sudo apt-get install libk4a1.4-dev
sudo apt-get install k4a-tools

3.2.2 Install Azure Kinect Body Tracking SDK

Refer to the Linux installation instructions section of Microsoft’s installation documentation to complete the installation. Document link:

Azure Kinect Body Tracking SDK download | Microsoft Learn

The following instructions were executed during installation of Azure Kinect Sensor SDK and do not need to be repeated:

curl -sSL -O https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
rm packages-microsoft-prod.deb
sudo apt-get update

Install libk4abt:

sudo apt install libk4abt1.1-dev

3.2.3 Use Orbbec SDK K4A Wrapper to Replace Azure Kinect Sensor SDK Library

Complete the environment configuration (udev rules script installation) according to Chapter 2, then replace the Azure Kinect Sensor SDK library files with the library files in the Orbbec SDK K4A Wrapper package (libk4a.so, libOrbbecSDK.so, depthengine2). After replacing and connecting the Orbbec camera, enter the simple_3d_viewer command in the terminal to see the following running effect:

 

4.Differences between Orbbec SDK K4A Wrapper and Azure Kinect Sensor SDK

4.1 Functional points with differences

No. Function Point Orbbec SDK K4A Wrapper Azure Kinect Sensor SDK Impact on Application
1 Recording c++ typedef struct _k4a_record_configuration_t { /** * The timestamp offset of the start of the recording. All recorded timestamps are offset by this value such that * the recording starts at timestamp 0. This value can be used to synchronize timestamps between 2 recording files. */ uint64_t start_timestamp_offset_usec; } k4a_record_configuration_t; c++ typedef struct _k4a_record_configuration_t { /** * The timestamp offset of the start of the recording. All recorded timestamps are offset by this value such that * the recording starts at timestamp 0. This value can be used to synchronize timestamps between 2 recording files. */ uint32_t start_timestamp_offset_usec; } k4a_record_configuration_t; Need to replace and recompile the header file of Orbbec SDK K4a Wrapper

4.2 Unimplemented interfaces in Orbbec SDK K4A Wrapper (return empty value or exception state)

No. Azure Kinect Sensor SDK Interface Meaning Impact of Differences
1 c++ k4a_result_t k4a_set_allocator(k4a_memory_allocate_cb_t allocate, k4a_memory_destroy_cb_t free)

Pass in external user-defined memory manager for SDK internal memory application

User cannot use their custom memory manager for SDK internal use. Basic functioning of SDK itself is barely affected.
2 c++ void k4a_capture_set_temperature_c(k4a_capture_t capture_handle, float temperature_c)

Set temperature information for capture

User cannot modify this value, impacting storage of custom values
3 c++ float k4a_capture_get_temperature_c(k4a_capture_t capture_handle)

Get temperature information for capture

User cannot get this value, impacting algorithms or applications that rely on this value
4 c++ void k4a_image_set_exposure_usec(k4a_image_t image_handle, uint64_t exposure_usec)

Set exposure value for image

User cannot modify this value, impacting storage of custom values
5 c++ void k4a_image_set_white_balance(k4a_image_t image_handle, uint32_t white_balance)

Set white balance value for image

6 c++ void k4a_image_set_iso_speed(k4a_image_t image_handle, uint32_t iso_speed)

Set ISO speed for image

7 c++ uint64_t k4a_image_get_exposure_usec(k4a_image_t image_handle)

Get exposure value for image

User cannot get this value, impacting algorithms or applications that rely on this value
8 c++ uint32_t k4a_image_get_white_balance(k4a_image_t image_handle)

Get white balance value for image

9 c++ uint32_t k4a_image_get_iso_speed(k4a_image_t image_handle)

Read ISO speed for image

10 c++ k4a_result_t k4a_device_get_sync_jack(k4a_device_t device_handle, bool *sync_in_jack_connected, bool *sync_out_jack_connected)

Get sync cable connection status for device

User application cannot rely on this interface to determine multi-camera sync cable linkage status

 

Stay updated

Be the first to learn about our new
products and updates.