About PDM for FreeCAD

Have some feature requests, feedback, cool stuff to share, or want to know where FreeCAD is going? This is the place.
Forum rules
Be nice to others! Read the FreeCAD code of conduct!
Jee-Bee
Veteran
Posts: 2566
Joined: Tue Jun 16, 2015 10:32 am
Location: Netherlands

Re: About PDM for FreeCAD

Post by Jee-Bee »

I don't think that adding custom attributes should be a user activity. Much more one that is handled on project level or admin level.
I can only think of a hand full of circumstances where i can think of that a attributes needs to be added that is never used before...

I think most difficulties starts when a company uses more than one CAD system... But for now if they are able to run multiple CAD software systems they end up with Windchill(from ptc) or Teamcenter(from Siemens) and not freePDM :P
heda
Veteran
Posts: 1348
Joined: Sat Dec 12, 2015 5:49 pm

Re: About PDM for FreeCAD

Post by heda »

well, i know of pdm's with hundreds of attrs, so took fifty to not come off as excagerating :-)
ah, well...
obviously that is not a handful of people pdm type of thing, regardless i do not think one needs a very large activity before a substantial amount of attrs becomes very useful in organizing things

anyways - the point is that all "regular" pdms generally do not allow for flexible attrs,
imho that sucks and it does not have to be that way.
maybe it is just me, but allowing users to do their own attributes on the fly is likely to turn out to be really handy for users - does not have to be heavy on the db either, every group/project can have their own table in the db with their own attributes - just need some joining code to get the full picture - come to think of it, that is probably why none of the regular pdm's are doing it, the have early on built a fixed architecture that is basically limited to one table and then it inherently is the situation you have all or nothing in terms of attributes... (and they have a driver to set it up this way, since anyone wanting smallest mod is likely to end up paying the sw-provider for it to be adjusted...)

about mainly using one field (description) for keeping and searching metadata.
hm, imho that is a too simplistic view of the expected capabilities of a pdm,
if it is only one field, in essence a text-file, it is more like any "search in files" functionality available in many text editors (also something that would be super easy to implement in python, i.e. search a folder of files for text content).

if that is the target maybe this thread is already there...
https://forum.freecadweb.org/viewtopic.php?f=22&t=63794
- in a file system the open files are locked to one user (i think)
- one could have a command in fc that saves the file, in a folder with a running name...
- etc

imagine that you would like to reuse standard fasteners that you have used earlier in your portfolio.
how would you enter that in a description field, and how would you find that back?
(as motivator for why one should not put artifical limits on what attributes and how many "should" be used)
or what happens when an organisation takes on a new product, and they will want to add new attributes to cover the needs of that new product.
the real benefit of attributes/pdm is that searches can be combined (in better ways than through a file system search), and different configurations can be found, using attributes to distinguish between products/configurations is really powerful - but that power largely lies in freely evolving the attributes over time (otherwise it tends to rot into uselessness over time)...

having fixed or flexible datastructures is largely how one chooses to think about the world,
practically the code is different, so it is an early choice to make.

let's make a practical example.

Code: Select all

from zipfile import ZipFile
import xml.etree.ElementTree as ET
import datetime as dt


with ZipFile('test.FCStd') as fczip:
    with fczip.open('Document.xml') as docxml:
        docx = docxml.read()

root = ET.fromstring(docx)


# fixed version where everyting has to be picked up explicitly
el = root.find(".//Property[@name='CreationDate']/String")
creationdate = el.attrib.get('value')
print(creationdate)

def dtparse(dtz):
    if dtz.endswith('Z'):
        return dt.datetime.fromisoformat(dtz[:-1])
    else:
        raise UserWarning('sorry, date format not known')

# flexible version, where one can for example maintain a skiplist...
# that skiplist could easily be a standalone config-file
skip = ('Material',)
converter = dict(CreationDate=dtparse)

docprops = dict()
properties, *_ = list(root)

for el in properties:
    name, ptype = el.attrib.values()
    if name in skip:
        continue
    child, = list(el)
    if 'Map' in ptype:
        # do something useful to parse a map-property
        pass
    else:
        if name in converter:
            docprops[name] = converter[name](child.attrib.get('value'))
        else:
            docprops[name] = child.attrib.get('value')

print(docprops)            

with the latter one, if fc adds additional properties, they will be parsed without any codechange...

same goes for how to handle any data structure, if the actual naming within the structure is abstracted away,
apart from some checks/conditionals here and there to choose right way to parse things,
the whole story is imho much more flexible and requires less to handle changes in content.

principally there is nothing stopping the fields in a database to be dynamic/flexible,
the fastest way there is to not store the attributes in a db at all, but rather store a python object that is flexible like the one above, one way to do that is simply to store a pickeled object in the db and a hook that unpickles (to memory) on db-read (one can pickle/unpickle any homegrown class that way), the downside of that would possibly be speed when searching larger db's (hence i favour actually making the number and content of database fields/tables flexible). obviously any presenting code for gui needs to be flexible in the same way.


get the feeling that you have "sketched" a full-blown pdm, with all that it entails,
at the same time as there should be well fenced functionality - that for me is a mismatch in intention vs outcome.

the version (outlined in linked thread) where one would store for example an xml directly in the fc file together with the capabilities of any filesystem will go a long way imho.
the xml (text file) in the fc file should be encoded, i.e. noone would touch it without going through dedicated gui. revisions could be handled in that scenario as well, gui could be made integrated or stand alone, when one makes a "release" or "revision" one would just alter filename acc to some predefined scheme, while at the same time updating a standalone db with that metadata. sure it would not be very user proof, but one does not always get everything one wants :-) and if something is mocked around with, it should be easy enough to patch things up.

that to me sounds like a very pragmatic and functional standalone poor mans pdm...
does not sound like much coding either.
as soon as you involve external revision handling, users, groups etc - it inherently will be much more involved and complex to make it work.

at last, have no real skin in the game (other than that i would like to see fc grow in functionality and userbase), so it does not matter much if it is one way or another - in the end the potential future users are the ones that will give thumbs up or not.
Jee-Bee
Veteran
Posts: 2566
Joined: Tue Jun 16, 2015 10:32 am
Location: Netherlands

Re: About PDM for FreeCAD

Post by Jee-Bee »

heda wrote: Tue Jun 07, 2022 10:59 pm well, i know of pdm's with hundreds of attrs, so took fifty to not come off as excagerating :-)
Don't be scared the amount of standard Attributes is just a number. ;)
heda wrote: Tue Jun 07, 2022 10:59 pm anyways - the point is that all "regular" pdms generally do not allow for flexible attrs,
imho that sucks and it does not have to be that way.
maybe it is just me, but allowing users to do their own attributes on the fly is likely to turn out to be really handy for users
I don't agree that it is easy because i think that if a user can add a attribute the user shall add attributes independent if it is needed or not.
I think this should bedone by a Admin or Super-User.
See https://github.com/grd/FreePDM/blob/mai ... seSetup.md for a setup of possible attribute tables. be aware that i cut tables for visual purpose...
I see that i forgot to add the description column(s). I even think there should be two of them because quite often the first description is most of the time the same as the name on drawing...
heda wrote: Tue Jun 07, 2022 10:59 pm about mainly using one field (description) for keeping and searching metadata.
hm, imho that is a too simplistic view of the expected capabilities of a pdm,
heda wrote: Tue Jun 07, 2022 10:59 pm imagine that you would like to reuse standard fasteners that you have used earlier in your portfolio.
how would you enter that in a description field, and how would you find that back?
(as motivator for why one should not put artifical limits on what attributes and how many "should" be used)
or what happens when an organisation takes on a new product, and they will want to add new attributes to cover the needs of that new product.
the real benefit of attributes/pdm is that searches can be combined (in better ways than through a file system search), and different configurations can be found, using attributes to distinguish between products/configurations is really powerful - but that power largely lies in freely evolving the attributes over time (otherwise it tends to rot into uselessness over time)...

having fixed or flexible datastructures is largely how one chooses to think about the world,
practically the code is different, so it is an early choice to make.
I think this is more a filter problem then a search problem.
Since most file systems can only search in what is available for them. my experience is that they don't find much information except when it is manually added to the file(what at least i never did...) and what also differ per operating system.
heda wrote: Tue Jun 07, 2022 10:59 pm
if that is the target maybe this thread is already there...
https://forum.freecadweb.org/viewtopic.php?f=22&t=63794
- in a file system the open files are locked to one user (i think)
- one could have a command in fc that saves the file, in a folder with a running name...
- etc
Blocking on the user side is much more difficult than blocking on the server-side. If on the server-side is said that only heda has write acces. that means that grd can't upload a file. If grd needs a different change he has to wait on you or communicate with you that you do also his change...
heda wrote: Tue Jun 07, 2022 10:59 pm same goes for how to handle any data structure, if the actual naming within the structure is abstracted away,
apart from some checks/conditionals here and there to choose right way to parse things,
the whole story is imho much more flexible and requires less to handle changes in content.
...
At this point @grd and i start to diverge. Where my focus is more or less on read/ write access of the user etc. is the focus of grd more reading out properties out of FC files.
I'm a fan of reading out data, up to some point. My current problem is easy FC it self. If i create a part (no it is a body... :? ... I don't care! why creating bodies with part design :x ) i am unable to add a material or a surface finish.
I prefer to create the versioning system on front and automatically filling attributes later on. My point is that a attributes is only changed a hand full of times over it's lifespan, but checking-in, checking-out happens hundred times easy...
grd
Posts: 328
Joined: Wed Apr 13, 2022 5:13 am
Location: Eindhoven, The Netherlands

Re: About PDM for FreeCAD

Post by grd »

heda wrote: Tue Jun 07, 2022 10:59 pm well, i know of pdm's with hundreds of attrs, so took fifty to not come off as excagerating :-)
Can you please name one?
heda wrote: Tue Jun 07, 2022 10:59 pm if that is the target maybe this thread is already there...
https://forum.freecadweb.org/viewtopic.php?f=22&t=63794
Thanks for the link. But they have different goals than ours.
heda wrote: Tue Jun 07, 2022 10:59 pm get the feeling that you have "sketched" a full-blown pdm, with all that it entails,
at the same time as there should be well fenced functionality - that for me is a mismatch in intention vs outcome.
Why do you believe that?
heda wrote: Tue Jun 07, 2022 10:59 pm that to me sounds like a very pragmatic and functional standalone poor mans pdm...
does not sound like much coding either.
as soon as you involve external revision handling, users, groups etc - it inherently will be much more involved and complex to make it work.
heda wrote: Tue Jun 07, 2022 10:59 pm at last, have no real skin in the game (other than that i would like to see fc grow in functionality and userbase), so it does not matter much if it is one way or another - in the end the potential future users are the ones that will give thumbs up or not.
The thing is that some times when you have a itch you just need to scratch it. Right now there is no PDM for FC, so we are gonna make one. There are gonna be lots of problems that we need to solve. That is the reality. Thanks to the specs that @Jee-Bee made we now have a way to solve it. But we are gonna make one. This is gonna be a long term project, keep that in mind.

Why I mentioned "A Poor man's PDM" is because it isn't finished yet and do you really believe that we are gonna be good enough to match the professional PDM's that have lots of guys working there on the code? Once we are there then I am gonna remove those lines of course ;)
About Nim. Latest Release 2.0.2. Here is Nim in 100 seconds and a Nim package. There are Qt and OCCT packages.
heda
Veteran
Posts: 1348
Joined: Sat Dec 12, 2015 5:49 pm

Re: About PDM for FreeCAD

Post by heda »

all of them (industrial grade)? it is not so much a matter of system, but rather organisation that is using them (and putting in their "requirements" of what is needed plus all the built-in, normally hidden attrs used internally by the db - and how much cash one is willing to throw at a pdm implementation for an organisation)
it all depends on how and for what one intends to use a pdm db (and surrounding it architecture)...

not at all scared of high numbers of attributes, the true power of a pdm lies in custom attributes, but it has to be used with a strategy and purpose in mind (that strategy and purpose will be different per user-base) the reason for me arguing for flexible attributes, even on user/project level is simple. if your admin/key-user is blocking a request, or it is perceived to hard to add one, what do people do? i would say that they resort to what is available to them, i.e. they create their own spreadsheet with all the manual labour and sync issues that arise from that... (completely agree thought that attributes has to be governed in one way or the other, but it does not have to be through admin..., could as well be by approval of 3 regular users, or something like that)

still do not completely grasp how you see this. it is rather black or white (for me), either there is a fixed number of attrs set by the "code-owners" or it is free to the deployer to add whatever a user/deployer want in terms of attrs - without changing code (the not changing code is the important part, and it completely depends on how the code is written, changing a config file is ok, but even better is doing it through a gui, and without the need for a db-migration).


an example, maybe more close to home.
freecad is extensible to whatever one wants, the only limitation is the occ-kernel (well actually this is not principally true - hypothetically one could use a different kernel, but practically it is true).
this freedom is created by having a sw architecture and code written with that in mind, users should always be allowed to add whatever they want through for example "feature python objects".
the file format allows for adding your own objects, without knowing how they will look like in advance - obviously there are some rules that must be followed for those objects.

just trying to argue for that extensibility is imho, a key feature of an endeavour like this, and if it is not explicitly part of the aim from the getgo - it is more than likely that it will not be extensible...

"it isn't finished yet"
fair enough

"do you really believe that we are gonna be good enough to match the professional PDM's that have lots of guys working there on the code?"

from an architectural and core functionality point of view - yes, with help from community, yes.
from a polished gui point of view, a matter of taste of course, but generally yes (but then again, i will be content with a quite stripped gui)
from a mime interaction point of view - not really, but little of this is truly needed imho
from a bloatware special use case point of view - na, not really, but do not see that as a negative :-)

anyhow, will stay silent on this (for now abstract) topic and let you get on with it.
Jee-Bee
Veteran
Posts: 2566
Joined: Tue Jun 16, 2015 10:32 am
Location: Netherlands

Re: About PDM for FreeCAD

Post by Jee-Bee »

heda wrote: Sun Jun 12, 2022 9:21 am ... it is not so much a matter of system, but rather organisation that is using them (and putting in their "requirements" of what is needed plus all the built-in, normally hidden attrs used internally by the db - ...)
heda wrote: Sun Jun 12, 2022 9:21 am still do not completely grasp how you see this. it is rather black or white (for me), either there is a fixed number of attrs set by the "code-owners" ...
Up til now we have only busy with attributes that ere there by default. We never spoke about a fixed amount or so. for me the number 50 was just baseline number. If it grows to 100, it's fine up to me. Since we havent code much i don't expect that we have hidden attributes yet, bu i'm sure we get them too. Of course there will be a time that custom properties are possible too.
I don't have in mind how and what methods are the best for creating new attributes. When creating a new database it is probably easy. but when a database is in use we have to think more how to change tinks, how to add data. for example we have a database and at some point in time the value added CE approved(easy Boolean: True / False) but where all systems before this one not CE approved? probably not. so do we accept empty fields...
heda wrote: Sun Jun 12, 2022 9:21 am not at all scared of high numbers of attributes, the true power of a pdm lies in custom attributes, but it has to be used with a strategy and purpose in mind (that strategy and purpose will be different per user-base) the reason for me arguing for flexible attributes, even on user/project level is simple. if your admin/key-user is blocking a request, or it is perceived to hard to add one...
completely agree thought that attributes has to be governed in one way or the other, but it does not have to be through admin..., could as well be by approval of 3 regular users, or something like that)
In my opinion it comes close to a how a company behave and what their internal regulations are. If a company decide that at least 3 users want a certain attribute that it should be added... then is that a company regulation and how to add such methods... are it 3 specialized users than a special role can be applied(at least is implemented) or has this done by a admin...
At the same time an Admin probably request methods to stop just adding attributes when unexpected growth of the database is seen(dimensions, probably not disc space).
At the same time not everybody knows the same. i can remember from my first company i thought a certain attribute was handy(i can't remember what or the situation) and i spoke a superuser / admin had some discussion about that what i needed and he agreed to discuss with the other superusers / admins. after let say a day he came back to me and he shows me how to achieve what i want. Basically it was already possible but not the way i expected it.
heda wrote: Sun Jun 12, 2022 9:21 am just trying to argue for that extensibility is imho, a key feature of an endeavour like this, and if it is not explicitly part of the aim from the getgo - it is more than likely that it will not be extensible...
I agree on extend ability. What the best way to extend end where is more difficult. for example initial there is only an addon planed for FC. But why not also an addon for KiCAD?
User avatar
Zolko
Veteran
Posts: 2213
Joined: Mon Dec 17, 2018 10:02 am

Re: About PDM for FreeCAD

Post by Zolko »

try the Assembly4 workbench for FreCAD — tutorials here and here
Jee-Bee
Veteran
Posts: 2566
Joined: Tue Jun 16, 2015 10:32 am
Location: Netherlands

Re: About PDM for FreeCAD

Post by Jee-Bee »

No i didn't see these two. At least not that i'm aware of.
dan-miel
Posts: 391
Joined: Thu Sep 13, 2018 12:29 am
Location: Spokane WA. USA

Re: About PDM for FreeCAD

Post by dan-miel »

Zolko wrote: Tue Jun 14, 2022 10:56 am Did you see these ?
I believe most PDM systems use the file extension to determine whether a file is a part, assembly or a drawing. None of them are setup to sort through a FreeCAD file to find what it is, so finding a PDM for FreeCAD would be hard.

The following is my opinion.

Overview of PDM
Three items would be needed to setup a PDM.
1 Programs to determine which files are assembly files, which are part files, which are drawing files and which are other files.
2 A data base to store the information.
3 A drive location to store the files.

Number three is the easiest since any hard drive will do.

Number one is a bit harder. For A2plus it is fairly easy because finding the string “a2plus” in the file’s Content indicates that it is an assembly file, then looping through the objects can give the parts in the assembly. I’m not sure how to separate A3, A4 or Arch files.
Inventor and SolidWorks make this easy because you only need to examine the file extension for asm, prt or drw.

Number 2 Database: I would save the file with a unique name that would never change, such as file1. For each new version of that file the file could be saved as file1v00, file1v01 or saved in a new directory that has the file name and version. The database would have tables that contain the name the user gave it and another table with the file's saved location. A user wanting to find a file would enter the file name he knows it by, the database would connect the table with the users file name with the unique name it was saved by, rename the file to the users file name and copy it to a directory that the user chooses. The user can then change his local files and put it back into the database as a new version if he wishes.
The reason the file is saved under a unique name and the users file name is saved in a table is so if the user changes the file name the data base updates the users name but can still find the file.

Database attributes: Part files normally have a material table and the assembly files has a BOM and/or parts table. Other tables could include Designer, Drafter, Finish, Description, checked out by and many others but a provision should be included for these to be added by the user.
Sorry I didn't notice this post earlier.
Dan
grd
Posts: 328
Joined: Wed Apr 13, 2022 5:13 am
Location: Eindhoven, The Netherlands

Re: About PDM for FreeCAD

Post by grd »

@dan-miel

About 1) That is why I wrote https://forum.freecad.org/viewtopic.php ... 2505151ed0

and also (because no one like it), that is why I added some objects into FreePDM, look at https://github.com/grd/FreePDM/blob/mai ... tamodel.py at the end. But it is not finished yet.

About 2) that is @Jee-Bee

About 3) That is obvious.

If you want to help, you are welcome.
About Nim. Latest Release 2.0.2. Here is Nim in 100 seconds and a Nim package. There are Qt and OCCT packages.
Post Reply