OpenXR/OpenVR (virtual reality support), new Python workbench WiP

Have some feature requests, feedback, cool stuff to share, or want to know where FreeCAD is going? This is the place.
Forum rules
Be nice to others! Read the FreeCAD code of conduct!
Post Reply
User avatar
kwahoo
Posts: 680
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

OpenXR/OpenVR (virtual reality support), new Python workbench WiP

Post by kwahoo »

This thread continues discussion from a thread about Oculus Rift.
I would like discuss here about wider VR hardware support. A few weeks ago an open VR/AR standard has been was presented: OpenXR.

Waiting for OpenXR hardware adoption (manufacturers like Oculus or Valve announced future support) I've started experimenting with another API: OpenVR.
OpenVR is not quite open as name suggest - it still needs a proprietary runtime (SteamVR), but unlike Oculus SDK it is supported on different operating systems and supports hardware from other manufacturers.


UPDATE OpenXR
The OpenXR support is done. Should work on Windows: Index, Vive, WMR (via SteamVR) and Oculus Rift (not tested) and Linux: Index or Vive + AMD GPU. Update: there is a workaround for Nvidia/Linux now.

phpBB [video]

This is a release introducing OpenXR support. Open View->FreeCAD-XR to view scene in a virtual reality headset. Click View->FreeCAD-XR second time to close the viewer.

Requirements:

virtual reality headset
OpenXR runtime supporting OpenGL (XR_KHR_opengl_enable extension) eg. SteamVR. Windows Mixed Reality users have to use SteamVR, since the WMR runtime does not support OpenGL directly
(optionally) motion controllers

Demo video
Scene navigation using motion controllers:
Press the left trigger (first controller) to activate the menu (one of bars will become green), select navigtion scheme and press the trigger second time to apply changes. You cannot move when the menu is active.

"Free" scheme:

analog stick/trackpad of the first (usually left) controller moves viewer forward or backward along the controller axis
analog stick/trackpad of the second (usually right) controller rotates viewer around center of the controller

"Arch" scheme:

analog stick/trackpad of the first controller (usually left) moves viewer up/down and left/right
analog stick/trackpad of the second controller (usually right) rotates viewer around center of the HMD and moves forward/backward.

Teleport ray is now available. Point the place when you want to move and press the right (second controller) trigger. Release the trigger and your feet will go to the selected point. Ray stays after teleport to show where you was before teleportation.

Rotation and translation speed can be adjusted now. Press the left trigger (first controller) to activate the menu, adjust speed and press the trigger second time to apply changes.

How to start:

Enable/run your OpenXR runtime (eg. by starting SteamVR) - make sure that OpenXR loader path is set up (Windows registry key "ActiveRuntime" in HKEY_LOCAL_MACHINE\SOFTWARE\Khronos\OpenXR\1, Linux: /etc/xdg/openxr/1/active_runtime.json)
Turn on motion controllers if you want use them
Run FreeCAD, open a 3D model
Start OpenXR session clicking View->FreeCAD-XR menu

Known limitations and issues:

[Linux] Linux users have to use SteamVR 1.14 because of these issues ValveSoftware/SteamVR-for-Linux#421 ValveSoftware/SteamVR-for-Linux#422
[Linux] Nvidia is not supported because of lack of asynchronous reprojection support in SteamVR 1.14.
If you want to run on Nvidia/Linux you have to compile from git and use the latest SteamVR. The workaround is in this commit 5517192
Viewer may crash if reopened several times.
You can download Windows build.

The code can be found in this fork https://github.com/kwahoo2/FreeCAD

Notes about compilation:
0. Download and install OpenXR SDK from https://github.com/KhronosGroup/OpenXR-SDK (note: on Windows build the OpenXR Loader as a DLL)
1. Set BUILD_OPENXR to ON

Commands to build and run from git (recommended):

Code: Select all

git clone https://github.com/kwahoo2/FreeCAD.git
cd FreeCAD
mkdir -p build
cd build
cmake .. -DBUILD_OPENXR
make -j`nproc`
./bin/FreeCAD
For better performance in the most cases you can try the LinkDaily branch, based on the realthunder's LinkDaily branch:

Code: Select all

git clone https://github.com/kwahoo2/FreeCAD.git
cd FreeCAD
git checkout LinkDaily
mkdir -p build
cd build
cmake .. -DBUILD_OPENXR
make -j`nproc`
./bin/FreeCAD
Outdated: OpenVR support:

The Windows binary and Linux Appimage release can be downloaded here https://github.com/kwahoo2/FreeCAD/rele ... 19_preVR-1
Connect headset, run SteamVR and then start bin/FreeCAD.exe. Open a model and run View->FreeCAD OpenVR. A new, preview window should appear as soon as SteamVR detects active HMD.
Use analog stick or touchpad of first controller to translate world. Move stick forward - view will move where the controller is pointing. Use second controller to rotate world around two axes. Position of the controller sets center of rotation.

The implementation should work with all OpenVR compatible headsets, including Oculus Rift, HTC Vive, Valve Index and Windows Mixed Reality HMD's. Please report if this works (or not) for you.


I stopped working on the Python script and moved to C++. The code can be found in this fork https://github.com/kwahoo2/FreeCAD
Notes about compilation:
0. Download and install OpenVR SDK from https://github.com/ValveSoftware/openvr
1. Set BUILD_OPENVR to ON
2. Set BUILD_QT5 to ON
3. You may need to specify OpenVR_INCLUDE_DIR (/usr/local/include/openvr in my case)
4. You may need to specify OpenVR_LIBRARY (/usr/local/lib/libopenvr_api.a in my case)

Additional notes about compilation on Windows:
1. OpenVR_LIBRARY should point to lib/win64/openvr_api.lib distributed with the OpenVR SDK
2. openvr_api.dll can be found in openvr\bin\win64

Outdated: OpenVR python demo

Initially, I started tinkering with jriegel Rift's implementation, but later i found Python bindings for OpenVR pyopenvr. Basic idea is: write (and experiment with) an implementation in pyopenvr, and then port the code to C++.

Here is the repository: https://github.com/kwahoo2/freecad-pyopenvr
Steps to run the code:
1. Install pyopenvr, and SDL + OpenGL Python packages
2. Run SteamVR
3. Paste the code into FreeCAD

TODO list:
- disable touchpads on WMR - done
- adjustment options for movement, rotation speed - done
- save/load speed values from a config file - done
- remove/comment out experimental code - done
- second, arch-friendly movement scheme - done
- teleport movement - done
- a new testing Windows build - done 22.08.2021, next after collecting feedback
- fix performance issues - partially done, needs deeper investigation. There is a new build a top of the LinkDev branch.
- ability to reopen closed window - TODO crashes at 3rd attempt (SteamVR linux v 1.14) Update: this seems not happen with Monado?
- fix rotation center issue - done
- better navigation for small parts based on user feedback 1, 2 - TODO - the simplest idea is probably a movable center of rotation that can be placed anywhere in the scene
- rotate menu forward for better visibility - TODO
- improve lighting for better visibility of a model, some ambient light? - TODO
- autostart of Oculus software - TODO, not sure if can be done?
- fix issues on exit - TODO
- fix the mirror viewer on Windows (is this AMD-only issue?), it should not be semi-transparent TODO
Last edited by kwahoo on Sun Aug 06, 2023 2:45 pm, edited 26 times in total.
User avatar
Kunda1
Veteran
Posts: 13434
Joined: Thu Jan 05, 2017 9:03 pm

Re: OpenXR/OpenVR (virtual reality support)

Post by Kunda1 »

kwahoo wrote: Sat Sep 21, 2019 7:05 pm Steps to run the code:
1. Install pyopenvr, and sdl + opengl python packages
2. Run SteamVR
3. Paste the code into FreeCAD

Known issues:
1. The code renders two exactly same pictures - wrong buffer configuration?
2. Head tracking is not implemented yet
3. SoFrustumCamera needs further investigation. I'm trying to get necessary variables for the camera from IVRSystem::GetProjectionMatrix and IVRSystem::GetEyeToHeadTransform matrices, but maybe there is a better way?

Future work:
1. Fix issues
2. Remove SDL dependency, use QOpenGLWidget (?)
3. Port to C++
4. Add basic navigation
5. Add controllers support (for navigation)
6. Move to OpenXR
@kwahoo
Fantastic!
Lets put this on Github or Gitlab, so people can pull and test quickly and contribute code more easily. You can also create a Roadmap etc...
Alone you go faster. Together we go farther
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
User avatar
kwahoo
Posts: 680
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support)

Post by kwahoo »

Kunda1 wrote: Sat Sep 21, 2019 7:43 pm Lets put this on Github or Gitlab, so people can pull and test quickly and contribute code more easily.
Done. I updated the original post.

At this moment mostly(at least I hope so...) correct rendering with rotation and position tracking is possible.

I found the culprit of rendering the same image twice:
creating two scenes this way:

Code: Select all

rootScene = [SoSeparator()] * 2
...
for eye in range(2):
  m_sceneManager.setSceneGraph(rootScene[eye])
does not work. The second scene changes the first one.
schurik
Posts: 2
Joined: Wed Sep 11, 2019 10:35 pm

Re: OpenXR/OpenVR (virtual reality support)

Post by schurik »

This sounds great.
Is it possible to run my Rift S on CATIA with this?
User avatar
kwahoo
Posts: 680
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support)

Post by kwahoo »

schurik wrote: Mon Sep 23, 2019 10:09 pm Is it possible to run my Rift S on CATIA with this?
Rift S: Yes.
Catia: No, at least not with this script.

----
I have some issues to be ironed out, there is probably an error in cameras configuration (objects are too big, moving too fast). Update: this needs more testing, added a simple test file to the repository: a room with pillars and a table.

But meanwhile some performance teaser (click to see a bigger pic):
Image

To be honest, I was expecting worse fps;)
Last edited by kwahoo on Fri Sep 27, 2019 4:32 pm, edited 1 time in total.
User avatar
Kunda1
Veteran
Posts: 13434
Joined: Thu Jan 05, 2017 9:03 pm

Re: OpenXR/OpenVR (virtual reality support)

Post by Kunda1 »

Sent another PR to soup up the README.
Alone you go faster. Together we go farther
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
User avatar
kwahoo
Posts: 680
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support)

Post by kwahoo »

Thanks!

There is one thing, that worries me - SoSceneManager::render is pretty slow with huge models. In the worst scenario I got ~13 fps and ~30% GPU utilisation. Using SoRenderManager (since event handling is not very important for VR) instead does not change much. Low fps and big latency (especially latency) makes this a very uncomfortable experience.

Maybe I can use SoGLRenderAction, and instead using Open Inventor camera, use pure OpenGL projection and model matrices, similar as shown here in the part Applying a Render Action Inside a GLX Window And then instead of traversing full scenegraph I could just manipulate matrices.
User avatar
Kunda1
Veteran
Posts: 13434
Joined: Thu Jan 05, 2017 9:03 pm

Re: OpenXR/OpenVR (virtual reality support)

Post by Kunda1 »

kwahoo wrote: Sun Sep 29, 2019 5:07 pm There is one thing, that worries me - SoSceneManager::render is pretty slow with huge models. In the worst scenario I got ~13 fps and ~30% GPU utilisation. Using SoRenderManager (since event handling is not very important for VR) instead does not change much. Low fps and big latency (especially latency) makes this a very uncomfortable experience.

Maybe I can use SoGLRenderAction, and instead using Open Inventor camera, use pure OpenGL projection and model matrices, similar as shown here in the part Applying a Render Action Inside a GLX Window And then instead of traversing full scenegraph I could just manipulate matrices.
I would pose this as a question on the Open Issues subforum so it gets more exposure.
Alone you go faster. Together we go farther
Please mark thread [Solved]
Want to contribute back to FC? Checkout:
'good first issues' | Open TODOs and FIXMEs | How to Help FreeCAD | How to report Bugs
User avatar
kwahoo
Posts: 680
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support)

Post by kwahoo »

I ported a basic OpenVR support to C++. It has the same functionality as the Python script.

The implementation can be found in this repository https://github.com/kwahoo2/FreeCAD Commit introducing OpenVR https://github.com/kwahoo2/FreeCAD/comm ... b8ab0c1e28

Notes about compilation:
1. Set BUILD_OPENVR to ON
2. Set BUILD_QT5 to ON
3. You may need to specify OpenVR_INCLUDE_DIR (/usr/local/include/openvr in my case)
4. You may need to specify OpenVR_LIBRARY (/usr/local/lib/libopenvr_api.a in my case)

There is no OS-specific code in the implementation - it should run both on Linux and Windows. Mac has some beta SteamVR support, but I do not have any experience with it .

Running a scene:
1. Start SteamVR
2. Open a model in the FreeCAD
3. Run View->FreeCAD-OpenVR
User avatar
kwahoo
Posts: 680
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support)

Post by kwahoo »

Another update:
Added some navigation using controllers - see video in the first post:
stick/trackpad of the first controller controls world translation, the second one controls rotation.

Dear VR users, please let me know if you need precompiled packages for testing. If not, the first post contains some building from source tips.
Post Reply