brainstorm: storage-efficient enums

Here's the place for discussion related to coding in FreeCAD, C++ or Python. Design, interfaces and structures.
Post Reply
User avatar
yorik
Founder
Posts: 12866
Joined: Tue Feb 17, 2009 9:16 pm
Location: Brussels
Contact:

brainstorm: storage-efficient enums

Post by yorik »

Most objects created with the Arch WB have an IfcType App::PropertyEnum with more than 100 choices. When saving a FreeCAD file with many of these objects, the Document.xml becomes extremely large, mostly with repetitive enum stuff.

I'd like to implement a different kind of App::PropertyEnum that would avoid saving all these enums repeatedly...

I have two ideas:

1) The property doesn't store the enum values in the FCStd file. Instead, it has a py module + method stored, and, on restore, it would call that method to get the enum values. That would be kind of a module-dependent enum property. Problem: what if the module is not there?

2) The enum values are stored in a separate file in the .FCStd file. The enum property specifies a filename, creates the file on save if none is there, and all subsequent saves skip saving if the file exists. Problem: Not sure what happens if the file is already there from a previous file load...

Any other idea here? @wmayer ? ;)
User avatar
doia
Posts: 196
Joined: Sat May 29, 2021 5:47 am
Location: Düsseldorf

Re: brainstorm: storage-efficient enums

Post by doia »

Maybe there could be a new subcategory of <SharedData> under <Document> within the Document.xml file, where this property of type="App::PropertyEnumeration" is saved once and referenced by the single object which uses this shared property. Kind of like IFC does it.

Code: Select all

<?xml version='1.0' encoding='utf-8'?>
<!--
 FreeCAD Document, see https://www.freecadweb.org for more information...
-->
<Document SchemaVersion="4" ProgramVersion="0.20R29177 (Git)" FileVersion="1">
    <Properties Count="15" TransientCount="3">
    </Properties>
    <SharedData Count="1"></SharedData> // <-- new subcategory for shared data blocks
    <Objects Count="2" Dependencies="1">
    </Objects>
    <ObjectData Count="2">
    </ObjectData>
</Document>
spanner888
Posts: 202
Joined: Tue May 28, 2019 10:51 am

Re: brainstorm: storage-efficient enums

Post by spanner888 »

Based on my simplistic view that the complete content of each enum, is always available within FreeCAD code/module, why not only save current selected value of each enum in FCStd file?

Also is this only an IFC issue?
User avatar
doia
Posts: 196
Joined: Sat May 29, 2021 5:47 am
Location: Düsseldorf

Re: brainstorm: storage-efficient enums

Post by doia »

spanner888 wrote: Tue Jun 21, 2022 9:44 pm why not only save current selected value of each enum in FCStd file?
This is probably the right approach, as it addresses the cause of the excessive data duplication within a file, instead of just mitigating the symptoms. And it conforms to the single responsibility principle. An object should only know which IFCType it is derived from, not what other IFCTypes are available.

Another thing is, there is no used IFC schema specified within the FCStd file. There should be some property declaration stating if IFC2x3, IFC4 or (soon) IFC4x3 is used.

Edit/Add:
This holds for all other CustomEnumList as well, i.e. <Property name="PredefinedType" and <Property name="Align", which duplicate the corresponding enum lists for each object.
vm4dim
Posts: 83
Joined: Tue Nov 23, 2021 1:05 am

Re: brainstorm: storage-efficient enums

Post by vm4dim »

I think that need a pure enum property, which stores only the index
and uses a list of values from other properties.
wmayer
Founder
Posts: 18801
Joined: Thu Feb 19, 2009 10:32 am

Re: brainstorm: storage-efficient enums

Post by wmayer »

yorik wrote: Tue Jun 21, 2022 10:52 am Any other idea here? @wmayer ?
You can declare the enum as transient property to avoid to save it to the XML file. Now all what you need is to save the index for each object independently and there you can override the __getstate__ and __setstate__ methods of your Python proxy class.
1) The property doesn't store the enum values in the FCStd file. Instead, it has a py module + method stored, and, on restore, it would call that method to get the enum values. That would be kind of a module-dependent enum property. Problem: what if the module is not there?
How do you fill the enum at the moment? Isn't it already implemented in a Python module to add the values?
User avatar
yorik
Founder
Posts: 12866
Joined: Tue Feb 17, 2009 9:16 pm
Location: Brussels
Contact:

Re: brainstorm: storage-efficient enums

Post by yorik »

wmayer wrote: Fri Jun 24, 2022 9:27 am You can declare the enum as transient property to avoid to save it to the XML file. Now all what you need is to save the index for each object independently and there you can override the __getstate__ and __setstate__ methods of your Python proxy class.
That's a really simple solution, thanks!! Can you make a transient property form Python? I didn't know...
wmayer wrote: Fri Jun 24, 2022 9:27 am How do you fill the enum at the moment? Isn't it already implemented in a Python module to add the values?
At object init, yes. But you are right, of course that could also be ran at restore time! How silly I am :D
wmayer
Founder
Posts: 18801
Joined: Thu Feb 19, 2009 10:32 am

Re: brainstorm: storage-efficient enums

Post by wmayer »

Code: Select all

doc = App.newDocument()
obj = doc.addObject("Part::Feature", "Shape")
obj.addProperty("App::PropertyEnumeration", "Enumeration", "Base", "The enumeration property", App.PropertyType.Prop_Transient)
obj.Enumeration = ["Item1", "Item2", "Item3", "Item4", "Item5"]
When saving the document then only the meta-information about the property itself is written to the Document.xml but not its content. This means the entries Item1, ... are not saved.
User avatar
yorik
Founder
Posts: 12866
Joined: Tue Feb 17, 2009 9:16 pm
Location: Brussels
Contact:

Re: brainstorm: storage-efficient enums

Post by yorik »

Excellent, I'll use that.Thanks @wmayer !
Post Reply