Page 2 of 3

Re: Obsoleting material handling bash scripts with python

Posted: Sat Feb 23, 2019 1:28 pm
by PrzemoF
Looks like a file can be opened with encoding specified:

Code: Select all

with open(oldfile, 'r', encoding='utf-8') as infile:
I'm not sure yet how it will affect yaml

Re: Obsoleting material handling bash scripts with python

Posted: Mon Feb 25, 2019 7:11 pm
by PrzemoF
Edited!
I think we should aim for getting the material cards in yaml together with python3. Making it compatible with python2 is a pain and it would be used for a short time only. I think we have a problem with the existing content - I'm getting things like:

Code: Select all

- Meta:
    AuthorAndLicense: !!python/str '(c) 2019 Uwe Stöhr (CC-BY 3.0)'
    CardName: PA6-Generic
- General:
    Name: !!python/unicode 'Polyamide 6'
Random thoughts
- dump the comments in yaml file completely,
- use Meta section to store info like "created by",
- add API version for future use? API could be simply FreeCAD version

Re: Obsoleting material handling bash scripts with python

Posted: Mon Feb 25, 2019 8:50 pm
by PrzemoF
That's a good starting point for me. Then your commit. This is in the master branch:

Code: Select all

$ file *.FCMat
ABS.FCMat:                   ASCII text
ABS-Generic.FCMat:           ASCII text
Acrylic-Glass-Generic.FCMat: UTF-8 Unicode text
CalculiX-Steel.FCMat:        ASCII text
Concrete-Generic.FCMat:      ASCII text
Glass-Generic.FCMat:         UTF-8 Unicode text
None.FCMat:                  ASCII text
PA6-Generic.FCMat:           UTF-8 Unicode text
PET-Generic.FCMat:           UTF-8 Unicode text
PLA.FCMat:                   ASCII text
PLA-Generic.FCMat:           UTF-8 Unicode text
PP-Generic.FCMat:            UTF-8 Unicode text
PTFE-Generic.FCMat:          UTF-8 Unicode text
PVC-Generic.FCMat:           UTF-8 Unicode text
Steel-Generic.FCMat:         ASCII text
TEMPLATE.FCMat:              ASCII text
Wood-Generic.FCMat:          ASCII text
Edit: Looks like ASCII text is a UTF-8 subset, so it' not the source of the problem. I'm trying to figure out why some strings are encoded as python/str and some as python/unicode in the same file.

Re: Obsoleting material handling bash scripts with python

Posted: Fri Mar 01, 2019 7:47 pm
by bernd
strange because the cards are all written by your bash scripts on my linux debian machine. I wonder why they have different encoding ...

I am not sure about when Python 2 support will be dropped in FreeCAD I do not thing in 0.19 already. But I would like to switch in 0.19 to this yaml cards.

Re: Obsoleting material handling bash scripts with python

Posted: Fri Mar 01, 2019 9:15 pm
by Jee-Bee
If python 2 will be supported while python don't support it ( https://pythonclock.org) would be strange...

Re: Obsoleting material handling bash scripts with python

Posted: Fri Mar 01, 2019 9:56 pm
by PrzemoF
bernd wrote: Fri Mar 01, 2019 7:47 pm strange because the cards are all written by your bash scripts on my linux debian machine. I wonder why they have different encoding ...

I am not sure about when Python 2 support will be dropped in FreeCAD I do not thing in 0.19 already. But I would like to switch in 0.19 to this yaml cards.
The encoding is OK. I'm digging into ascii/utf-8/python2-3 and yaml. If file contains only ascii characters it's a valid utf-8 file as well. I tried quite a few combinations, yet python2/yaml exports strings like "szkło" (unicode/utf-8) as !!python/str and I have no idea why. But I learned a lot :D

Re: Obsoleting material handling bash scripts with python

Posted: Sat Mar 02, 2019 8:50 pm
by PrzemoF
The problem is with existing cards or the way the existing cards are read. Test from python 2.7.15:

Code: Select all

>>> y = yaml.load("""\n    - szkło\n    - test\n    """) ; print(yaml.dump(y))
["szk\u0142o", test]

>>> with open('y', 'w') as f:
...     yaml.dump(y, f, default_flow_style=False, allow_unicode=True)
and the result file looks perfect:

Code: Select all

$ cat y
- szkło
- test

Code: Select all

$ file y
y: UTF-8 Unicode text
Same code works with python 3.7.2 with identical (at least visually) results. Same result from FreeCAD python console. Also reading the file using yaml.load works fine on 2.7.15. I also did a test with glass card from the FreeCAD python console and it's all good.

We might need to edit existing cards and it might all work fine. Unless FreeCAD is making something strange "behind the scene". I'm talking about things like:

Code: Select all

pythonopen = open
or

Code: Select all

unicode = str
I should know that soon.

Re: Obsoleting material handling bash scripts with python

Posted: Sat Mar 02, 2019 10:00 pm
by PrzemoF
This code works fine from FreeCAD python console:

Code: Select all

>>> with open('x-fc.yml') as f:
...         x = yaml.load(f)
... 
>>> with open('x-fc.yml', 'w') as f:
...         yaml.dump(x, f, default_flow_style=False, allow_unicode=True)
x-fc.yml is utf-8 file without any !!python/srt or !!python/unicode tags.

Re: Obsoleting material handling bash scripts with python

Posted: Sat Mar 02, 2019 10:42 pm
by PrzemoF
Side note: we currently redefine python "open" and other build-in functions. Should we instead change it to fc_open across FreeCAD code? Renaming build-in functions is very confusing thing. I'm aware I'm asking for a huge thing, but unless there is a good reason to do it (C++/python hooks?) I'd avoid it even if it means changing it in the whole FreeCAD code.

Re: Obsoleting material handling bash scripts with python

Posted: Mon Mar 04, 2019 8:50 am
by bernd
PrzemoF wrote: Sat Mar 02, 2019 10:42 pm Side note: we currently redefine python "open" and other build-in functions. Should we instead change it to fc_open across FreeCAD code? Renaming build-in functions is very confusing thing. I'm aware I'm asking for a huge thing, but unless there is a good reason to do it (C++/python hooks?) I'd avoid it even if it means changing it in the whole FreeCAD code.
To start a disscusion, a topic in developer part of the forum makes sence IMHO.