OpenXR/OpenVR (virtual reality support), new Python workbench WiP

Have some feature requests, feedback, cool stuff to share, or want to know where FreeCAD is going? This is the place.
Forum rules
Be nice to others! Read the FreeCAD code of conduct!
catman
Posts: 412
Joined: Fri Jan 11, 2019 10:42 pm

Re: OpenXR/OpenVR (virtual reality support), arch-friendly update

Post by catman »

kwahoo wrote: Tue Aug 16, 2022 9:37 pm There are results, when I loaded a motorcycle engine model: about 42-43ms per frame for both pyopenvr and C++ OpenXR implementations.
Did not see the fps values in the images you posted but if both are the same fps thats a good sign to try pyopenxr, don't you think?
User avatar
kwahoo
Posts: 680
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support), arch-friendly update

Post by kwahoo »

catman wrote: Wed Aug 17, 2022 8:06 am Did not see the fps values in the images you posted but if both are the same fps thats a good sign to try pyopenxr, don't you think?
Because there is no fps values in the screenshot, just frame times. FPS = 1000 / (frame time in ms)
It's good sign indeed, but I cannot promise any work on it in the near future:(
User avatar
kwahoo
Posts: 680
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support), arch-friendly update

Post by kwahoo »

Hi, after a long break...

I've started experimenting with picking objects with controllers. So far on flat screen, so no HMD (or even working compositor) is needed.
This is the first attempt:
phpBB [video]


To do this I implemented a new getObjectInfoRay function, that does the same things as getObjectInfo, but uses a ray (described by 2 vectors, start point and direction) in 3D view as input instead of using 2D mouse cursor coordinates.
"Return a dictionary with the name of document, object and component. The"
"dictionary also contains the coordinates of the appropriate 3d point of"
"the underlying geometry in the scenegraph."
The script (and controller model) I used for creating the demo is here. I used pyopenvr, mainly because pyopenxr cannot run headless yet. The script is a bit buggy and unfinished but was good enough for simple demo.

You can test the function, after applying commit mentioned earlier, even if you do not have any VR controllers. For example, create a standard box in the Part Workbench and then paste this script:

Code: Select all

import FreeCADGui as Gui
import FreeCAD as App
import Part

def pick_ray(start_vec, dir_vec):
    Gui.Selection.clearSelection()
    if App.ActiveDocument.getObject('Line_dir'):
        App.ActiveDocument.removeObject('Line_dir')
    if App.ActiveDocument.getObject('Line_ray'):
        App.ActiveDocument.removeObject('Line_ray')
    if App.ActiveDocument.getObject('Sphere_sect'):
        App.ActiveDocument.removeObject('Sphere_sect')
    view = Gui.ActiveDocument.ActiveView
    info = view.getObjectInfoRay(start_vec, dir_vec)
    App.ActiveDocument.addObject("Part::Line","Line_dir") # line have to be created after picking to avoid intersection
    App.ActiveDocument.Line_dir.Label='Dir'
    App.ActiveDocument.Line_dir.X1=start_vec.x
    App.ActiveDocument.Line_dir.Y1=start_vec.y
    App.ActiveDocument.Line_dir.Z1=start_vec.z
    App.ActiveDocument.Line_dir.X2=start_vec.x + dir_vec.x
    App.ActiveDocument.Line_dir.Y2=start_vec.y + dir_vec.y
    App.ActiveDocument.Line_dir.Z2=start_vec.z + dir_vec.z
    if (info):
        sect_pt = App.Vector(info['x'], info['y'], info['z'])
        App.ActiveDocument.addObject("Part::Line","Line_ray") # line have to be created after picking to avoid self-intersection
        App.ActiveDocument.Line_ray.Label='Ray'
        Gui.ActiveDocument.Line_ray.LineColor = (1.00,0.67,0.00)
        App.ActiveDocument.Line_ray.X1=start_vec.x
        App.ActiveDocument.Line_ray.Y1=start_vec.y
        App.ActiveDocument.Line_ray.Z1=start_vec.z
        App.ActiveDocument.Line_ray.X2=sect_pt.x
        App.ActiveDocument.Line_ray.Y2=sect_pt.y
        App.ActiveDocument.Line_ray.Z2=sect_pt.z
        App.ActiveDocument.addObject("Part::Sphere","Sphere_sect")
        App.ActiveDocument.Sphere_sect.Label = "Intersection point"
        App.ActiveDocument.Sphere_sect.Radius = '0.5 mm'
        App.ActiveDocument.Sphere_sect.Placement = App.Placement(sect_pt,App.Rotation(App.Vector(0.00,0.00,1.00),0.00))
        Gui.ActiveDocument.Sphere_sect.ShapeColor = (1.00,0.33,0.00)
        Gui.Selection.addSelection(info['Document'], info['Object'], info['Component'], info['x'], info['y'], info['z'])
    App.ActiveDocument.recompute()

pick_ray(App.Vector(-3,-10, 1), App.Vector(0.2, 0.5, 0))
You should see something like this:
Image

Modify vectors inside pick_ray(App.Vector(-3,-10, 1), App.Vector(0.2, 0.5, 0)) to move the ray. They represent start point and direction.
---
There is another place, where headless mode may be useful. FreeCAD on Raspberry Pi. RPi is not powerful enough to run HMD and cannot run SteamVR at all[1], but why do not use a VR controller as input device? I had some success with libsurvive Python bindings + FreeCAD, including both tracking and buttons. Hope I can share more in future.

[1] excluding box86/box64 x86 on ARM emulation
User avatar
kwahoo
Posts: 680
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support), arch-friendly update

Post by kwahoo »

catman wrote: Wed Aug 17, 2022 8:06 am
kwahoo wrote: Tue Aug 16, 2022 9:37 pm There are results, when I loaded a motorcycle engine model: about 42-43ms per frame for both pyopenvr and C++ OpenXR implementations.
Did not see the fps values in the images you posted but if both are the same fps thats a good sign to try pyopenxr, don't you think?
Initial work for Python XR workbench https://github.com/kwahoo2/freecad-xr-workbench

Image

Implementation of controllers and artificial movement is planned soon.

06.08.2023 - controller representation and input logging added.
Edit

Code: Select all

self.xr_widget = XRwidget(log_level=logging.WARNING) 
to

Code: Select all

self.xr_widget = XRwidget(log_level=logging.DEBUG) 
in commonXR.py to see controller input info in the FreeCAD console.
catman
Posts: 412
Joined: Fri Jan 11, 2019 10:42 pm

Re: OpenXR/OpenVR (virtual reality support), arch-friendly update

Post by catman »

kwahoo wrote: Sat Jul 29, 2023 6:34 pm Initial work for Python XR workbench https://github.com/kwahoo2/freecad-xr-workbench
Great. Thats a wonderful idea.

It took me some time to reactivate the CV1, but I got it running in direct mode now. Trouble is that I did something wrong with the workbench installation. I used the official 21.0 version as AppImage in Linux. The OpenXR runtime is Monado.
There are not errors on startup. But I noted that the CV1 switches on when I start FreeCAD. So something is starting the OpenXR device. There is only a black image, though.
However, the workbench does not load. It prints errors and I do not get any toolbar buttons (I saw in the source there should be 4 buttons).

Below is the ReportView output. My guess would be that the python module installation is the issue. I installed pyOpenXR with pip and confirmed that in ".local/lib/site-packages/python3.10" there is the pyOpenXR and an pyOpenGL package. They should also have installed all their dependencies since I did not get any pip install errors. But shouldn't I have added the python modules to the AppImage? Also the libglfw.so is in the ".local/lib../glfw/X11/" folder, but it does not seem to find it.

Code: Select all

20:43:50  DEBUG:OpenGL.platform.ctypesloader:Loaded libGL.so => libGL.so.1 <CDLL 'libGL.so.1', handle 7f890bf89ac0 at 0x7f88fdd9c910>
20:43:50  INFO:OpenGL.acceleratesupport:No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
20:43:50  invalid syntax (<string>, line 1)
20:43:50  Traceback (most recent call last):
  File "<string>", line 60, in Initialize
  File "/tmp/.mount_FreeCAWOrqRS/usr/lib/python3.10/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 139, in _import
    return original_import(name, *args, **kwargs)
  File "/home/$User/.local/share/FreeCAD/Mod/xr-workbench/startXR.py", line 27, in <module>
    import commonXR as cxr
  File "/tmp/.mount_FreeCAWOrqRS/usr/lib/python3.10/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 139, in _import
    return original_import(name, *args, **kwargs)
  File "/home/$User/.local/share/FreeCAD/Mod/xr-workbench/commonXR.py", line 46, in <module>
    import xr
  File "/tmp/.mount_FreeCAWOrqRS/usr/lib/python3.10/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 139, in _import
    return original_import(name, *args, **kwargs)
  File "/home/$User/.local/lib/python3.10/site-packages/xr/__init__.py", line 6, in <module>
    from . import (
  File "/tmp/.mount_FreeCAWOrqRS/usr/lib/python3.10/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 139, in _import
    return original_import(name, *args, **kwargs)
  File "/home/$User/.local/lib/python3.10/site-packages/xr/classes.py", line 14, in <module>
    import glfw
  File "/tmp/.mount_FreeCAWOrqRS/usr/lib/python3.10/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 139, in _import
    return original_import(name, *args, **kwargs)
  File "/home/$User/.local/lib/python3.10/site-packages/glfw/__init__.py", line 40, in <module>
    from .library import glfw as _glfw
  File "/tmp/.mount_FreeCAWOrqRS/usr/lib/python3.10/site-packages/shiboken2/files.dir/shibokensupport/feature.py", line 139, in _import
    return original_import(name, *args, **kwargs)
  File "/home/$User/.local/lib/python3.10/site-packages/glfw/library.py", line 194, in <module>
    glfw = _load_library(['glfw', 'glfw3'], ['.so', '.dylib'],
  File "/home/$User/.local/lib/python3.10/site-packages/glfw/library.py", line 59, in _load_lib
Any ideas?
User avatar
kwahoo
Posts: 680
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support), arch-friendly update

Post by kwahoo »

catman wrote: Tue Aug 08, 2023 8:58 pm It took me some time to reactivate the CV1, but I got it running in direct mode now. Trouble is that I did something wrong with the workbench installation. I used the official 21.0 version as AppImage in Linux. The OpenXR runtime is Monado.
I use the same Appimage. Interesting, GLFW is used by pyopenxr itself, not by the workbench.

I have also package for GLFW that was installed with pyopenxr (as dependency):

Code: Select all

~/.local/lib/python3.10/site-packages/glfw/library.py
Can you check if this example does work for you?
catman
Posts: 412
Joined: Fri Jan 11, 2019 10:42 pm

Re: OpenXR/OpenVR (virtual reality support), arch-friendly update

Post by catman »

kwahoo wrote: Tue Aug 08, 2023 9:25 pm I use the same Appimage.
Thats good to know.
kwahoo wrote: Tue Aug 08, 2023 9:25 pm I have also package for GLFW that was installed with pyopenxr (as dependency):

Code: Select all

~/.local/lib/python3.10/site-packages/glfw/library.py
Yes I can confirm this is installed in the same location. also the GL example worked fine: the HMD fired up and showed some striped color pattern.

I run monado-service manually. I thought it might be missing the runtime location, so I tried to start it with the 'XR_RUNTIME_JSON" env var set, but I get the same errors.
I noted that there are two SO's. One in a folder "wayland" and one in "X11". The sources in the library loading code did not seem to contain X11 or wayload. Although the example worked, Maybe somewhere a selection need to be done to complete the path? (just a wild guess)
User avatar
kwahoo
Posts: 680
Joined: Fri Nov 29, 2013 3:09 pm
Contact:

Re: OpenXR/OpenVR (virtual reality support), arch-friendly update

Post by kwahoo »

Did you paste the gl_example.py directly into FreeCAD Python console? FreeCAD will freeze, but the HMD should display the example.
If it works that way, the issue is somewhere on the workbench side.
catman wrote: Wed Aug 09, 2023 7:55 am Maybe somewhere a selection need to be done to complete the path? (just a wild guess)
The only thing what I've seen in the pyGLFW docs is:
Although pyGLFW will try to detect whether the GLFW library for Wayland or X11 should be used, you can set the PYGLFW_LIBRARY_VARIANT variable to wayland or x11 to select either variant of the library.
(...)
pyGLFW will search for the library in a list of search paths (including those in LD_LIBRARY_PATH). If you want to use a specific library, you can set the PYGLFW_LIBRARY environment variable to its path.
catman
Posts: 412
Joined: Fri Jan 11, 2019 10:42 pm

Re: OpenXR/OpenVR (virtual reality support), arch-friendly update

Post by catman »

kwahoo wrote: Wed Aug 09, 2023 7:00 pm Did you paste the gl_example.py directly into FreeCAD Python console? FreeCAD will freeze, but the HMD should display the example. If it works that way, the issue is somewhere on the workbench side.
When pasting the full example the line import xr causes the same error.

Although pyGLFW will try to detect whether the GLFW library for Wayland or X11 should be used, you can set the PYGLFW_LIBRARY_VARIANT variable to wayland or x11 to select either variant of the library.
Also no difference.


I can cause the error on the FreeCAD console with

Code: Select all

 import glfw 
On a xfce console with Python3 the same works fine. Maybe its AppImage related - but strange that its working in your setup. My understanding of Python is quite limited, I'm afraid. So I do not know how /tmp/.mountFreeCADYXC/usr/lib/python3.1/site-packages/numpy is not clashing with my globally installed ~/local/share/lib/pythong3/site-packges/numpy. But the AppImage can not use a virtualenv since it would not find local packages like py_Slvs any more which it does.
catman
Posts: 412
Joined: Fri Jan 11, 2019 10:42 pm

Re: OpenXR/OpenVR (virtual reality support), new Python workbench WiP

Post by catman »

I saw that on the system /usr/lib/x86../libglfw.so was already installed with its normal symlinks to a 3.3 version file. From the binary info those differes to the pip installed local file. I switched all symlinks to point to the python installed file and ran ldconfig again, but same result. I also tried to put the lib in the FreeCAD AppImage folder according to the DevDocumentation it should check there for the library, again same result.

The loader command said ''glfw,glfw3''. The local file is named just "x11/libglfw.so" and there are no symlinks. That how pip installed it. Do you have symlinks to a "*.so.3" in that folder?
Post Reply