Mac build and python C API

Having trouble installing or compiling FreeCAD? Get help here.
Forum rules
Be nice to others! Respect the FreeCAD code of conduct!
User avatar
ipatch
Posts: 112
Joined: Wed Apr 08, 2020 3:40 pm
Location: dfw
Contact:

Re: Mac build and python C API

Post by ipatch »

doia wrote: Tue Jul 19, 2022 9:18 am Has anyone success in compiling the latest FC source on arm64 architecture and could share a reproducible build setup? Preferably using Homebrew and Python 3.10. I'm stuck with the Py_complex and Py_UNICODE errors mentioned above. Thanks.

yeah i built a non app bundle yesterday on an m1. i'm trying to figure out how i can get

```
brew install freecad
```

to install everything and build an app bundle without having to manually link shiboken and pyside

i'm getting closer, but still haven't quite figured everything out.

as for your Py_complex issue i believe that may be to the cmake setup process finding a different version of pyside. homebrew-core provides a `pyside@2` it seems recently the pyside@2 formula file provided by homebrew-core switched to the limitied api as one of the configure options for pyside. and when i built and installed the homebrew-core version the cmake scripts found libraries related to abi3 whereas the pyside version provided by homebrew-freecad tap will use a cpython

also make sure to unlink python@3.9 and adjust your

Code: Select all

PATH
accordingly. also i had to setup and build boost-python3 using python@3.10 and currently the homebrew-core version is still using python@3.9 but just editing the formula file and replacing 3.9 with 3.10 got the job done for me.

my command was looking like the below as of yesterday (july 27 2022)

Code: Select all

cmake \
-DHOMEBREW_PREFIX=$bp \
-DPYTHON_EXECUTABLE=$bp/opt/python@3.10/bin/python3 \
-DPYTHON_INCLUDE_DIR=$bp/opt/python@3.10/Frameworks/Python.framework/Headers \
-DCMAKE_PREFIX_PATH="$pthmed;$pthboostpy3;$bp/Cellar/pyside2@5.15.5/5.15.5/lib/cmake/PySide2-5.15.5" \
-DCMAKE_BUILD_TYPE="Release" \
-DFREECAD_CREATE_MAC_APP=0 \
-DCMAKE_INSTALL_PREFIX=$code/freecad-git/installs/build.release.0.20.no.bundledir \
-DCMAKE_OSX_ARCHITECTURES="arm64" \
--debug-find-pkg=PySide2 \
--debug-find-var=PYSIDE_INCLUDE_DIR \
../../freecad-src
User avatar
doia
Posts: 251
Joined: Sat May 29, 2021 5:47 am
Location: Düsseldorf

Re: Mac build and python C API

Post by doia »

@ipatch, thank you. I‘m on vacation right now and will further test when I‘m back.
ipatch wrote: Thu Jul 28, 2022 4:11 pm as for your Py_complex issue i believe that may be to the cmake setup process finding a different version of pyside. homebrew-core provides a `pyside@2` it seems recently the pyside@2 formula file provided by homebrew-core switched to the limitied api as one of the configure options for pyside.
Up to now I used the homebrew-core provided pyside@2, which did work prior to the internal switch to use the Py_LIMITED_API.

Your solution suggests using the homebrew-freecad/pyside@2 formulae, which uses some workarounds to not use the Py_LIMITED_API. I think this is not future proof and should be avoided.

In my view the error lies deeper and should be fixed within the FreeCAD source. FC uses the PyCXX library v6.2.8 as part of its source, located in src/CXX. The current version of PyCXX is v.7.1.7 with added support for the Py_LIMITED_API from v7.1.0 on in mid 2018 (!). http://cxx.sourceforge.net

Py_complex and Py_UNICODE are not part of the Py_LIMITED_API. That's where the error occures. The current PyCXX library uses the correct Py_LIMITED_API endpoints with PyComplex_FromDoubles and so on.

So the main tasks are:
- update the underlying PyCXX library to v7.1.5+ ->@wmayer @yorik are there any intermediate patches which would prevent that?
- maybe remove the Python2 part other integrated PyCXX library, as these are not used anyways
User avatar
yorik
Founder
Posts: 13640
Joined: Tue Feb 17, 2009 9:16 pm
Location: Brussels
Contact:

Re: Mac build and python C API

Post by yorik »

doia wrote: Sun Jul 31, 2022 7:36 pm - update the underlying PyCXX library to v7.1.5+ ->@wmayer @yorik are there any intermediate patches which would prevent that?
I don't think so. We have upgraded PyCXX several times in the past, it's usually straightforward.
It's a pity pycxx is not more widespread through linux distros ;)
User avatar
ipatch
Posts: 112
Joined: Wed Apr 08, 2020 3:40 pm
Location: dfw
Contact:

Re: Mac build and python C API

Post by ipatch »

doia wrote: Sun Jul 31, 2022 7:36 pm @ipatch, thank you. I‘m on vacation right now and will further test when I‘m back.
ipatch wrote: Thu Jul 28, 2022 4:11 pm as for your Py_complex issue i believe that may be to the cmake setup process finding a different version of pyside. homebrew-core provides a `pyside@2` it seems recently the pyside@2 formula file provided by homebrew-core switched to the limitied api as one of the configure options for pyside.
Up to now I used the homebrew-core provided pyside@2, which did work prior to the internal switch to use the Py_LIMITED_API.

Your solution suggests using the homebrew-freecad/pyside@2 formulae, which uses some workarounds to not use the Py_LIMITED_API. I think this is not future proof and should be avoided.

In my view the error lies deeper and should be fixed within the FreeCAD source. FC uses the PyCXX library v6.2.8 as part of its source, located in src/CXX. The current version of PyCXX is v.7.1.7 with added support for the Py_LIMITED_API from v7.1.0 on in mid 2018 (!). http://cxx.sourceforge.net

Py_complex and Py_UNICODE are not part of the Py_LIMITED_API. That's where the error occures. The current PyCXX library uses the correct Py_LIMITED_API endpoints with PyComplex_FromDoubles and so on.

So the main tasks are:
- update the underlying PyCXX library to v7.1.5+ ->@wmayer @yorik are there any intermediate patches which would prevent that?
- maybe remove the Python2 part other integrated PyCXX library, as these are not used anyways
not further exasberate the issue, but i think the source handling the setup of shiboken & pyside could be updated to work with both the

Code: Select all

abi3
prefix / suffix along the with current

Code: Select all

cpython310
that is currently implemented.

below is a link to the cmake source where one could modify to look for either the

Code: Select all

cpython
or the

Code: Select all

abi3
https://github.com/FreeCAD/FreeCAD/blob ... side.cmake
User avatar
doia
Posts: 251
Joined: Sat May 29, 2021 5:47 am
Location: Düsseldorf

Re: Mac build and python C API

Post by doia »

@wmayer thanks for the update of PyCXX.

There before mentioned errors regarding Py_complex and Py_UNICODE are gone. But now new errors for PyUnicode_AsUTF8 and Py_buffer appear, see the build log. I will investigate.

Code: Select all

[ 47%] Building CXX object src/Gui/CMakeFiles/FreeCADGui.dir/BitmapFactory.cpp.o
In file included from /Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:42:
In file included from /Users/dirkolbrich/FreeCAD/freecad-source/src/Base/Interpreter.h:44:
In file included from /Users/dirkolbrich/FreeCAD/freecad-source/src/CXX/Extensions.hxx:42:
In file included from /Users/dirkolbrich/FreeCAD/freecad-source/src/CXX/Python3/Extensions.hxx:183:
/Users/dirkolbrich/FreeCAD/freecad-source/src/CXX/Python3/PythonType.hxx:206:9: warning: declaration does not declare anything [-Wmissing-declarations]
        std::unordered_map<int, void*>  slots;
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/Application.cpp:1587:30: error: use of undeclared identifier 'PyUnicode_AsUTF8'
        const char* wbName = PyUnicode_AsUTF8(key);
                             ^
[ 47%] Building CXX object src/Mod/Mesh/App/CMakeFiles/Mesh.dir/WildMagic4/Wm4ApprGaussPointsFit3.cpp.o
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1219:5: error: use of undeclared identifier 'Py_buffer'; did you mean 'setbuffer'?
    Py_buffer content;
    ^~~~~~~~~
    setbuffer
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/stdio.h:368:7: note: 'setbuffer' declared here
void     setbuffer(FILE *, char *, int);
         ^
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1219:14: error: expected ';' after expression
    Py_buffer content;
             ^
             ;
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1219:5: warning: expression result unused [-Wunused-value]
    Py_buffer content;
    ^~~~~~~~~
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1219:15: error: use of undeclared identifier 'content'; did you mean 'cgetent'?
    Py_buffer content;
              ^~~~~~~
              cgetent
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/stdlib.h:285:6: note: 'cgetent' declared here
int      cgetent(char **, char **, const char *);
         ^
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1219:15: warning: expression result unused [-Wunused-value]
    Py_buffer content;
              ^~~~~~~
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1221:54: error: use of undeclared identifier 'content'; did you mean 'cgetent'?
    if (!PyArg_ParseTuple(args, "ss*|s", &iconName, &content, &format))
                                                     ^~~~~~~
                                                     cgetent
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/include/stdlib.h:285:6: note: 'cgetent' declared here
int      cgetent(char **, char **, const char *);
         ^
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1227:27: error: use of undeclared identifier 'content'
        PyBuffer_Release(&content);
                          ^
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1231:55: error: use of undeclared identifier 'content'
    const char* contentStr = static_cast<const char*>(content.buf);
                                                      ^
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1232:32: error: use of undeclared identifier 'content'
    QByteArray ary(contentStr, content.len);
                               ^
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1236:54: error: use of undeclared identifier 'content'
        QString file = QString::fromUtf8(contentStr, content.len);
                                                     ^
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1240:23: error: use of undeclared identifier 'content'
    PyBuffer_Release(&content);
                      ^
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1303:28: error: use of undeclared identifier 'PyStructSequence_GET_ITEM'; did you mean 'PyStructSequence_GetItem'?
        PyObject *pyfile = PyStructSequence_GET_ITEM(*info,1);
                           ^~~~~~~~~~~~~~~~~~~~~~~~~
                           PyStructSequence_GetItem
/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Headers/structseq.h:44:23: note: 'PyStructSequence_GetItem' declared here
PyAPI_FUNC(PyObject*) PyStructSequence_GetItem(PyObject*, Py_ssize_t);
                      ^
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1411:12: error: use of undeclared identifier 'PyRun_String'
    return PyRun_String(sCmd, Py_file_input, dict, dict);
           ^
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1437:12: error: use of undeclared identifier 'PyRun_String'
    return PyRun_String(sCmd, Py_file_input, dict, dict);
           ^
[ 47%] Building CXX object src/Mod/Mesh/App/CMakeFiles/Mesh.dir/WildMagic4/Wm4ApprLineFit3.cpp.o
wmayer
Founder
Posts: 20243
Joined: Thu Feb 19, 2009 10:32 am
Contact:

Re: Mac build and python C API

Post by wmayer »

/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/Application.cpp:1587:30: error: use of undeclared identifier 'PyUnicode_AsUTF8'
That's strange because it's already used in Base and App and there the compiler didn't complain.

But what's also strange is this comment in unicodeobject.h (for Py 3.6):

/* Returns a pointer to the default encoding (UTF-8) of the
Unicode object unicode.

Like PyUnicode_AsUTF8AndSize(), this also caches the UTF-8 representation
in the unicodeobject.

_PyUnicode_AsString is a #define for PyUnicode_AsUTF8 to
support the previous internal function with the same behaviour.

Use of this API is DEPRECATED since no size information can be
extracted from the returned data.

*** This API is for interpreter INTERNAL USE ONLY and will likely
*** be removed or changed for Python 3.1.

*** If you need to access the Unicode object as UTF-8 bytes string,
*** please use PyUnicode_AsUTF8String() instead.

*/

#ifndef Py_LIMITED_API
PyAPI_FUNC(char *) PyUnicode_AsUTF8(PyObject *unicode);
#define _PyUnicode_AsString PyUnicode_AsUTF8
#endif
But at https://docs.python.org/3/c-api/unicode ... ode_AsUTF8 it says the function has been added with Py3.3 and no word about internal use only.
/Users/dirkolbrich/FreeCAD/freecad-source/src/Gui/ApplicationPy.cpp:1219:5: error: use of undeclared identifier 'Py_buffer'; did you mean 'setbuffer'?
Py_buffer is also part of the extended API.

Code: Select all

#ifndef Py_LIMITED_API
/* buffer interface */
typedef struct bufferinfo {
    void *buf;
    PyObject *obj;        /* owned reference */
    Py_ssize_t len;
    Py_ssize_t itemsize;  /* This is Py_ssize_t so it can be
                             pointed to by strides in simple case.*/
    int readonly;
    int ndim;
    char *format;
    Py_ssize_t *shape;
    Py_ssize_t *strides;
    Py_ssize_t *suboffsets;
    void *internal;
} Py_buffer;

User avatar
doia
Posts: 251
Joined: Sat May 29, 2021 5:47 am
Location: Düsseldorf

Re: Mac build and python C API

Post by doia »

So essentially as the homebrew pyside@2 formulae now enforces the use of the Py_LIMITED_API, it trips up on all those occasions where unsupported identifiers in the FC source are used. -> The supported identifiers are listed here https://docs.python.org/3/c-api/stable. ... imited-api

Py_buffer is another story as it is currently not supported by the Py_LIMITED_API and has no replacement, but will be with version 3.11 -> https://docs.python.org/3.11/c-api/buff ... .Py_buffer

This leaves the question to:
a) refactor the complete FC source to comply to the Python Staple API and use .abi3, which seems to be the intended goal of the Python maintainers to facilitate compatibility of 3rd party Python projects -> lots of work for FC but IMHO more future proof
b) use a custom pyside@2 formulae on Mac/homebrew based builds which does not enforce the Py_LIMITED_API - easy workaround (and maybe necessary for version up to Python 3.10) - but what if one of the other platforms Linux/Win decides to use the limited API one day? What if some other 3rd party libraries do?

What do y'all think?
User avatar
doia
Posts: 251
Joined: Sat May 29, 2021 5:47 am
Location: Düsseldorf

Re: Mac build and python C API

Post by doia »

OK, so I switched to using the homebrew-freecad formulaes for pyside and shiboken, as these don't enforce the Py_Limited_API. This builds FreeCAD with my configuration.

Here is the setup:

Code: Select all

# installing homebred-core dependencies
brew install boost-python3 coin3d opencascade pybind11 qt@5 swig vtk xerces-c

# install homebrew-freecad formulaes and linking these
brew install freecad/freecad/pyside2@5.15.5 freecad/freecad/shiboken2@5.15.5
brew link freecad/freecad/pyside2@5.15.5 freecad/freecad/shiboken2@5.15.5

# export path to installed dependencies
export PREFIX_PATH="/opt/homebrew/opt/qt@5"

# prepare cmake
cmake \
  -DCMAKE_BUILD_TYPE="Release" \
  -DBUILD_ENABLE_CXX_STD=C++17 \
  -Wno-dev \
  -DPYTHON_EXECUTABLE=/opt/homebrew/opt/python@3.10/bin \
  -DPYTHON_INCLUDE_DIR=/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Headers \
  -DPYTHON_LIBRARY=/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib/libpython3.10.dylib \
  -DBUILD_FEM=OFF \
  -DBUILD_WEB=OFF \
  -DCMAKE_PREFIX_PATH="$PREFIX_PATH" \
  -DCMAKE_INSTALL_PREFIX="./.." \
  ../freecad-source

# run build and measuring build time
time sh -c 'cmake --build . -j10'
So, now I have a question regarding the overall FC projects approach in using/supporting the Py_limited_API. Is there any intend/discussion/consideration within the project to use the Py_Limited_API in the near or longterm future? Is there any interest in starting this discussion?

There are a lot of pro and cons for supporting the Py_limited_API. One of them is that Pyside@2 (for QT5) as well as Pyside6 (for use with QT6) use the Py_limited_API by default. This is not very well documented on their site, but one can infer it from the setup.py file used in their source code, see https://github.com/pyside/pyside2-setup ... #L159-L160 and https://github.com/pyside/pyside-setup/ ... #L787-L788.

My view is, that this would be a good thing to do, as it would kind of future-proof the project to further Python versions. But would need a lot of work of porting the FC code base to support this.
berniev
Posts: 247
Joined: Wed Apr 13, 2022 10:45 pm
Location: Oz

Re: Mac build and python C API

Post by berniev »

It seems "No answer" was the stern reply. I suspect there is only one person in this ecosystem who can help. @wmayer of course.

I'm bumping into

Code: Select all

Use of undeclared identifier 'PyUnicode_AsUTF8'
right now in C++20/Python310/QT6 experimental build.

There are 90 occurences of PyUnicode_AsUTF8 in FC. I can get a build by

Code: Select all

PyBytes_AsString(PyUnicode_AsUTF8String(..))
But my knowledge is severely lacking.

https://docs.python.org/3/c-api/unicode.html
const char *PyUnicode_AsUTF8(PyObject *unicode)

As PyUnicode_AsUTF8AndSize(), but does not store the size.

New in version 3.3.

Changed in version 3.7: The return type is now const char * rather of char *.
Unlike many other functions it doesn't mention ABI at all.
Post Reply