[Bug?] AppImage version of FreeCad can't use AppImage of OpenScad (and possible other externals)

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
gneiss
Posts: 75
Joined: Tue Mar 05, 2019 4:13 pm

[Bug?] AppImage version of FreeCad can't use AppImage of OpenScad (and possible other externals)

Post by gneiss »

In principle this is a "special" problem of the AppImage Version of Freecad.
I had already a discussion with the developer there: https://github.com/FreeCAD/FreeCAD-Bund ... 1201589944

There is a generell problem when calling an AppImage from another AppImage.
The first one adds some settings to the environment which might influence the second one.

Here especially the setting of LD_LIBRARY_PATH is problematic.
FreeCad-AppImage sets it so that the lib's from the mounted rootfs of the AppImage was used.
That's OK/needed/correct for executable's that reside "inside" the rootfs.
But it is not OK for executable's outside of rootfs.
In my case OpenSCAD could not find its lib's, because they are not inside the FreeCad-rootfs, they are inside the OpenSCAD-rootfs, but (i believe) AppRun of OpenSCAD did not set/change LD_LIBRARY_PATH because it's already set.
BTW: Using a wrapper around OpenSCAD-AppImage, that unsets LD_LIBRARY_PATH will fix this specail problem

More generally the best approach should be:
When calling an executable outside of the current rootfs, switch back to the environment as it was before AppRun of the AppImage was called.

AFAIK there is no such automatism build into the AppImage-World.

So an App that will be delivered as an AppImage, AppRun should do some extra work (saving the environment) and the Apps inside should call a Helper-function/script when they will call an external executable.
This Helper restores the environment, then calls the given executable with the given parameters.
All This might be done by two simple shell-scripts.

OK, long introduction, but I like to ensure that my intention and the use-case was clear.

The concrete case here was, that inside the WB OpenSCAD we have to supply a path to an external executable.
I believe that isn't the only situation inside FreeCad (there are too many WB's, that I don't know, each may need to call external executable's).

So even when not following my above sketched "full solution", FreeCad should "reset" (at least) LD_LIBRARY_PATH before calling an external executable.
OK, this might be some "overhead" for No-Appimage-FreeCad, but having a central Wrapper for calling external executable's is IMHO a good approach.

This will although support something that seams to be not implemented yet:
Using environment variables when specifying a path to an executable!
ex.: "$HOME/OpenSCADWrapper.sh"
Having one central wrapper only this must be adopted to support such extra functionality

what do you think ?
User avatar
adrianinsaval
Veteran
Posts: 5541
Joined: Thu Apr 05, 2018 5:15 pm

Re: [Bug?] AppImage version of FreeCad can't use AppImage of OpenScad (and possible other externals)

Post by adrianinsaval »

Note that while I have contributed to the FreeCAD-Bundle repo I'm not actually the dev behind it. Did you test changing the transfer mechanism as suggested in the report view message?

It appears there have been previous discussions about this:
https://forum.freecadweb.org/viewtopic.php?f=10&t=40848
https://github.com/FreeCAD/FreeCAD-Bundle/issues/107
https://github.com/FreeCAD/FreeCAD_Conda/issues/19
https://github.com/FreeCAD/FreeCAD-Bundle/issues/56
And there is a proposed fix: https://github.com/FreeCAD/FreeCAD/pull/7182 but doesn't look very well put together, if you could maybe review or provide a better PR it would be nice.
gneiss
Posts: 75
Joined: Tue Mar 05, 2019 4:13 pm

Re: [Bug?] AppImage version of FreeCad can't use AppImage of OpenScad (and possible other externals)

Post by gneiss »

@adrianinsaval

First:
Did you test changing the transfer mechanism as suggested in the report view message?
The problem has nothing to do with the 'transfer mechanism'.
OpenSCAD will simply not run, because of wrong/missing lib's (.so)

About Your Links:
1 https://forum.freecadweb.org/viewtopic.php?f=10&t=40848
2 https://github.com/FreeCAD/FreeCAD-Bundle/issues/107
3 https://github.com/FreeCAD/FreeCAD_Conda/issues/19
4 https://github.com/FreeCAD/FreeCAD-Bundle/issues/56
5 https://github.com/FreeCAD/FreeCAD/pull/7182

Links 1, 2 & 5: These Links describe the same problem, but the solution given will only address WB OpenSCAD (and are still not implemented). My approach is a more general that will fix all problems when calling external executable's.

Link 3: This was a (user-)problem not setting the correct path to the OpenSCAD executable...
Link 4: Is a "request" to include OpenSCAD in the FreeCad Appimage


My intention was to point out, that: YES this is an AppImage/Snap/Flatpack specific problem, but it could not be fixed without some "help" from the main App.
At least the operation "restore environment to the one it was before AppRun was called" must be implemented inside FreeCad(-core?) whenever doing a "subprocess.Popen(*args, **kwargs)" (meaning a call of an external executable).

A simple implementation (on linux, using python) will be:

1. Save environment (part of AppRun)
At Start of Apprun do:

Code: Select all

with open('/tmp/EnvAtFreeCadEntry', 'wb') as f:
        pickle.dump(os.environ, f)
This will save the current environment into a file.

2. Restore environment before executing external executable (Part of FreeCad whenever a "subprocess.Popen..." is used):
If the file EnvAtFreeCadEntry does not exist do as normaly.
Otherwise use the EnvAtFreeCadEntry to restore the environment before calling the subprocess.

So using the solutions given inside the above Links, we could use:

Code: Select all

# This should be a "global" or "core" function
def CallExternalExecutable(*args, **kwargs):
        kwargs.update({'stdout':subprocess.PIPE,'stderr':subprocess.PIPE})
        if os.path.isfile('/tmp/EnvAtFreeCadEntry') #!!! and it is (still) valid !!!
                with open('/tmp/EnvAtFreeCadEntry', 'rb') as f:
                        env = pickle.load(f)
                kwargs.update({'env':env})
        p = subprocess.Popen(*args, **kwargs)
        stdoutd, stderrd = p.communicate()
        return p, stdoutd, stderrd
Using this "central" function, the Implementation of a "interpretation" auf the args (e.g.: replace "$HOME" with the path to home) will be easy.

The function inside OpenSCADUtils.py can the be written as:

Code: Select all

def check_output2(*args, **kwargs):
        p, stdoutd, stderrd = CallExternalExecutable(*args, **kwargs)
        stdoutd = stdoutd.decode("utf8")
        stderrd = stderrd.decode("utf8")
        if p.returncode != 0:
            raise OpenSCADError('%s %s\n' % (stdoutd.strip(),stderrd.strip()))
            #raise Exception,'stdout %s\n stderr%s' %(stdoutd,stderrd)
        if stderrd.strip():
            FreeCAD.Console.PrintWarning(stderrd+u'\n')
        if stdoutd.strip():
            FreeCAD.Console.PrintMessage(stdoutd+u'\n')
            return stdoutd
There is one problem open:
How to ensure that the file /tmp/EnvAtFreeCadEntry is valid (the AppImage is still running).
May be we could use the existens of the mount-point of the AppImage-rootfs as an indication...
Last edited by gneiss on Thu Aug 04, 2022 7:11 pm, edited 1 time in total.
gneiss
Posts: 75
Joined: Tue Mar 05, 2019 4:13 pm

Re: [Bug?] AppImage version of FreeCad can't use AppImage of OpenScad (and possible other externals)

Post by gneiss »

I think I found a good solution to the open problem mentioned in my last answer:
If we modify the code like this

At Start of AppRun (assumig it's a python script) do:

Code: Select all

env = os.environ.copy
env.update({'FreeCadAppRunDir':os.path.dirname(__file__)})
with open('/tmp/EnvAtFreeCadEntry', 'wb') as f:
        pickle.dump(env, f)

Code: Select all

# This should be a "global" or "core" function
def CallExternalExecutable(*args, **kwargs):
        kwargs.update({'stdout':subprocess.PIPE,'stderr':subprocess.PIPE})
        FreeCadAppRunDir = os.environ.get('FreeCadAppRunDir') 
        if FreeCadAppRunDir is not None and os.path.isdir(mountPath) and os.path.isfile('/tmp/EnvAtFreeCadEntry'):
                with open('/tmp/EnvAtFreeCadEntry', 'rb') as f:
                        env = pickle.load(f)
                kwargs.update({'env':env})
        p = subprocess.Popen(*args, **kwargs)
        stdoutd, stderrd = p.communicate()
        return p, stdoutd, stderrd
User avatar
adrianinsaval
Veteran
Posts: 5541
Joined: Thu Apr 05, 2018 5:15 pm

Re: [Bug?] AppImage version of FreeCad can't use AppImage of OpenScad (and possible other externals)

Post by adrianinsaval »

The idea sounds good but using a fixed absolute path is a no go, it has to be associated to the session (you can have more than one instance of the appimage running at the same time under different envs) and use the appropriate tmp directory for the system
gneiss
Posts: 75
Joined: Tue Mar 05, 2019 4:13 pm

Re: [Bug?] AppImage version of FreeCad can't use AppImage of OpenScad (and possible other externals)

Post by gneiss »

@adrianinsaval
The idea sounds good but using a fixed absolute path is a no go
That's a thing I just realised myself :-)
Here is a approach that uses a unique file...

At Start of AppRun (assumig it's a python script) do:

Code: Select all

f=tempfile.NamedTemporaryFile()
os.environ['FreeCadEnvFilePath']=t.name
pickle.dump(os.environ, f)
f.close()

Code: Select all

# This should be a "global" or "core" function
def CallExternalExecutable(*args, **kwargs):
        kwargs.update({'stdout':subprocess.PIPE,'stderr':subprocess.PIPE})
        FreeCadEnvFilePath=os.environ.get('FreeCadEnvFilePath') 
        if FreeCadEnvFilePath is not None and os.path.isfile(FreeCadEnvFilePath):
                with open(FreeCadEnvFilePath, 'rb') as f:
                        env = pickle.load(f)
                kwargs.update({'env':env})
        p = subprocess.Popen(*args, **kwargs)
        stdoutd, stderrd = p.communicate()
        return p, stdoutd, stderrd
User avatar
chennes
Veteran
Posts: 3881
Joined: Fri Dec 23, 2016 3:38 pm
Location: Norman, OK, USA
Contact:

Re: [Bug?] AppImage version of FreeCad can't use AppImage of OpenScad (and possible other externals)

Post by chennes »

This seems useful enough that I wonder if there is already such a function and I (and other OpenSCAD WB devs) just didn't know it was in there. @wmayer does such a thing already exist?
Chris Hennes
Pioneer Library System
GitHub profile, LinkedIn profile, chrishennes.com
wmayer
Founder
Posts: 20243
Joined: Thu Feb 19, 2009 10:32 am
Contact:

Re: [Bug?] AppImage version of FreeCad can't use AppImage of OpenScad (and possible other externals)

Post by wmayer »

does such a thing already exist?
This is a problem we actually have never encountered before AppImages, Snap packages and co. So, there is no such mechanism implemented yet.
gneiss
Posts: 75
Joined: Tue Mar 05, 2019 4:13 pm

Re: [Bug?] AppImage version of FreeCad can't use AppImage of OpenScad (and possible other externals)

Post by gneiss »

@wmayer, chennes
I am willing to implement this and supply a PR, but I need some "hints".

As far as I know the correct way is:
- create a fork of https://github.com/FreeCAD/FreeCAD (done)
- clone the fork to a local disk (done)
- make the changes, compile & Test

So far all is clear, but now I wan't to make the AppImage.
I think I need the source from https://github.com/FreeCAD/FreeCAD-Bundle
But what must I do to create an AppImage.

Next problem will be, that this will create changes in two separate repositories...
User avatar
adrianinsaval
Veteran
Posts: 5541
Joined: Thu Apr 05, 2018 5:15 pm

Re: [Bug?] AppImage version of FreeCad can't use AppImage of OpenScad (and possible other externals)

Post by adrianinsaval »

@looo would be the person to ask on what to do to make an appimage, I think you need to compile freecad from within a conda environment and somehow package that.
Post Reply