Benchmarking and Evaluating FreeCAD against other CAD/CAM/CAE/PLM software

Have some feature requests, feedback, cool stuff to share, or want to know where FreeCAD is going? This is the place.
Forum rules
Be nice to others! Read the FreeCAD code of conduct!
User avatar
Jzuken
Posts: 11
Joined: Sun May 17, 2020 10:17 pm

Benchmarking and Evaluating FreeCAD against other CAD/CAM/CAE/PLM software

Post by Jzuken »

Although as of right now I am working in Solidworks on my job, I see a great potential for FreeCAD to become one of the best, if not the best CAD software there is, like Blender once did, as it has some great community features, or features that could be added that wouldn't be available in 5-digit price commercial CAD software, and also some feature that are already more easy to use than with commercial CAD software (take navigation and hotkeys, for example).

But on the other hand there are significant milestones that prevent FreeCAD from reaching Blender levels of success, there are still some rough edges and Achilles heels FreeCAD has that prevent it's adoption in commercial sector. To identify those shortcoming I propose developing a series of benchmarks and evaluation exercises that can be used to compare FreeCAD with commercial software, eradicate those shortcoming, and ultimately expand it's userbase and funding.

So having worked in engineering, I decided to put a list of benchmarks I'd like to see FreeCAD pass that would allow it to get widespread commercial adoption. But I always welcome the advice and ideas from seasoned engineers and power users that can identify potential problem domains, suggest new solutions and advice how they can be improved.
User avatar
Jzuken
Posts: 11
Joined: Sun May 17, 2020 10:17 pm

Re: Benchmarking and Evaluating FreeCAD against other CAD/CAM/CAE/PLM software

Post by Jzuken »

Here is a list of benchmark exercises, mostly from my experience. Your contributions are welcome.

1. Modelmania exercise
First and the easiest to define, as such benchmark has already been done.
Goal: Model some of the specified models from Solidworks modelmania challenge, as was done here: https://forum.freecadweb.org/viewtopic. ... 2&p=388030 Models with most complex shapes and edges for filleting should be chosen, such as 2001 model, 2002 model, 2004 model, 2006 model, 2018 model, 2020 model, etc. Other models may be chosen - your suggestions welcome.
Passing result: All of the models can be modeled up to specification. Models can be achieved in different ways, using different techniques - including mirror modeling, shape extrusions, rotational symmetry, boolean operations, etc. Weight is consistent with other commercial software, feature count is not 10% more than that of minimally achievable in commercial software.
Optimal result: Passing result. Fillets on many edges can be easily achieved (no manual clicking 10's of edges). Weight can be easily calculated (in GUI, no console scripts), and consistent with material properties.
Priority: High

2. Will it fillet exercise
Goal: Achieving such complex fillets as shown in those video series: https://www.youtube.com/watch?v=lJ4aVAg ... _Q&index=4
Passing result: All fillets are able to be achieved using techniques similar to those on video
Optimal result: Passing result. Fillets can be achieved with simpler techniques and faster
Priority: Medium

3. Curvature continuous exercise
Goal: Design curvature continuous fan blade in FreeCAD: https://dpt3.dptcorporate.com/Content/H ... cation.htm
Passing result: Whole fan blade (except mating edges) is G3 continuous
Optimal result: Whole fan blade (except mating edges) is G4 continuous
Priority: Medium/Low

4. DFMA/DFQ/DFR design exercise
Goal: A basic design requirement is specified, and an assembly has to be produced according to DFMA/DFQ/DFR principles
Passing result: Achieving "compliance" up to DFMA/DFQ/DFR principles is not harder than in commercial software without addons
Optimal result: Achieving "compliance" up to DFMA/DFQ/DFR principles is not harder than in commercial software with addons or additional software
Priority: Medium

5. Complex mold design exercise
Goal: Complex mold is designed for different molding processes, including those that produce fasteners, snap fits, plastic screws, thin walls, etc.
Passing result: Such mold can be designed without complications, except design challenges
Optimal results: Passing result. Such features as plastic molding toolbox is present, which helps evaluate the plastic mold design for stresses, cavities, etc.
Priority: Medium

6. Top-down design exercise
Goal: Product is designed top-down within assembly context. It's parts are defined relative to the whole assembly and to other parts
Passing result: Product can be designed top-down with various parts. Changes can be made to parts and their location without breaking the assembly or with easy fixes.
Optimal result: Passing result. Circular or semi-circular references don't break the assembly or the parts. Parts don't "fly all over the place" when a mate is misplaced.
Priority: High

7. Multiuser exercise
Goal: Multiple users work on a design of a single product consisting of multiple parts, over the network, maybe across the world.
Passing result: Up to 10 users are able to work on a product together, either simultaneously or in different time zones without interfering with each others work. Python console is sparingly used, preferably not used.
Optimal result: Passing result. Assembly view changes in real time or close to it. One part (at least in different configurations) can be worked on by multiple users. Other suggestions welcome.
Priority: High

8. Significant assembly exercise
Goal: Product with over 200 separate parts is assembled and edited on a adequate modern PC: 8 cores or more, 32GB or RAM, GPU is RTX 1060 or better.
Passing result: Assembly with over 200 parts can be worked on. Parts can be frozen. Rebuild can be interrupted without breaking the models.
Optimal result: Passing result. Assembly with 200-2000 parts can be worked on.
Excellent result: Passing result. Assembly with 2000-20000 parts can be worked on.
Priority: High

9. Many parameters part exercise
Goal: A parametrized part is specified with many parameters and complex features, including cosmetic. An example would be a surface-defined solid grille with hexagonal pattern, hexagonal pattern is controlled by the size and curvature of the initial surface, all hexagon cutouts are filleted. Or some other model.
Passing result: Model can be parametrized and parameters edited to achieve a different model in no significant time. A table of parameters might be specified, and different models have to build up to these parameters. As long as parameters remain adequate, the model rebuilds without problem and with minimal user interaction.
Optimal result: Passing result. Later in the modeling stage a new parameter is introduced. Model does not need significant work to adjust for the new parameter.
Priority: High/Medium

10. Standard drawing exercise
Goal: Using TechDraw or other drawing workbench produce an engineering drawing that covers most of the specified standard, without using hacks. I'll take GOST as I'm more familiar with it.
Passing result: TechDraw covers 60% of the GOST standard, same is true for other standards.
Optimal result: TechDraw covers 80% of GOST and other standards.
Excellent result: Tech draw not only covers more than 90% of the standard, but also has additional features for managing drawings. LibreCAD integration?
Priority: Low/Medium

11. Standard company catalogue/part indexes
To be added later

12. PLM exercise
To be added later

13. Synchronous modelling exercise
Goal: Produce a model using a workflow similar to Solid Edge/Siemens NX synchronous editing.
Passing result: Model is produced used technique similar to Solid Edge/Siemens NX synchronous exercise. Model doesn't break when sparingly edited.
Optimal result: Passing result. Model doesn't break when aggressively edited.

Update 12.12.2021: Added 13. Synchronous modelling exercise
Last edited by Jzuken on Sun Dec 12, 2021 11:49 am, edited 1 time in total.
paullee
Veteran
Posts: 5097
Joined: Wed May 04, 2016 3:58 pm

Re: Benchmarking and Evaluating FreeCAD against other CAD/CAM/CAE/PLM software

Post by paullee »

Jzuken wrote: Fri Dec 10, 2021 12:17 am 4. DFMA/DFQ/DFR design exercise
Goal: A basic design requirement is specified, and an assembly has to be produced according to DFMA/DFQ/DFR principles
Passing result: Achieving "compliance" up to DFMA/DFQ/DFR principles is not harder than in commercial software without addons
Optimal result: Achieving "compliance" up to DFMA/DFQ/DFR principles is not harder than in commercial software with addons or additional software
Priority: Medium

Seems to be a very comprehensive and structured assessment framework!

The term DFMA catch my attention, in building construction industry, this would means Design for Manufacturing and Assembly. Same here? Any assessment criteria on CAD for construction industry?

Thanks!
drmacro
Veteran
Posts: 8864
Joined: Sun Mar 02, 2014 4:35 pm

Re: Benchmarking and Evaluating FreeCAD against other CAD/CAM/CAE/PLM software

Post by drmacro »

When do you plan to have the results of the benchmark ready for publication?

FreeCAD is an incredible tool as it sits. And the points you make might be valid if there was a goal to be any of the things you describe.

You mention Blender as an example. But, though there are definitely commonalities of origin between Blender and FreeCAD, at this point there is a chasm of differences.

Blender has implemented the foundation. It consists of a 30 member paid team. There is a management team doing outreach and defining roadmaps. With annual funding of over a million Euro. They have a headquarter office building.

Alas, there is nothing like that for FreeCAD.
Star Trek II: The Wrath of Khan: Spock: "...His pattern indicates two-dimensional thinking."
User avatar
Jzuken
Posts: 11
Joined: Sun May 17, 2020 10:17 pm

Re: Benchmarking and Evaluating FreeCAD against other CAD/CAM/CAE/PLM software

Post by Jzuken »

paullee wrote: Fri Dec 10, 2021 12:37 am Seems to be a very comprehensive and structured assessment framework!

The term DFMA catch my attention, in building construction industry, this would means Design for Manufacturing and Assembly. Same here? Any assessment criteria on CAD for construction industry?

Thanks!
Yes, DFMA is design for manufacturing and assembly. DFQ is design for reliability and DFR is design for reliability. I'm not sure how that benchmark exercise can be constructed yet, but it would certainly have things like manufacturing complex parts - e.g. plastic covers with catches, CAMing complex machined parts, investment casting, etc.

I'm not sure how FreeCAD can be assessed for building industry, as I am working in mechanical engineering. But for example something along the lines of "designing 5-storey building, with different variations and complete piping" could be used.
drmacro wrote: Fri Dec 10, 2021 1:29 am When do you plan to have the results of the benchmark ready for publication?

FreeCAD is an incredible tool as it sits. And the points you make might be valid if there was a goal to be any of the things you describe.

You mention Blender as an example. But, though there are definitely commonalities of origin between Blender and FreeCAD, at this point there is a chasm of differences.

Blender has implemented the foundation. It consists of a 30 member paid team. There is a management team doing outreach and defining roadmaps. With annual funding of over a million Euro. They have a headquarter office building.

Alas, there is nothing like that for FreeCAD.
As of now I do not plan on doing any benchmarking myself. In fact, I think people that specify benchmarks, people that test them and people that evaluate the results should be different people to exclude subjective bias. For example if someone is both specifying and testing a benchmark, they might undercut the rigor of the requirements and don't run into some problems that might arise in the real scenario. Someone that is both specifying the benchmark might be either too lenient or too strict.

FreeCAD is an incredible tool for hobby, as of now, but it's not suited for actual commercial production environment. FreeCAD lacks foundation and funding, but that is also a chicken and egg problem - you need funds to get the foundation going, you need the foundation to get the funds from the big guys, like auto industry, aerospace, hardware, etc.
Which is why FreeCAD needs those benchmarks/goals that I listed, so that it can become a choice not just for hobbyists and small open source projects, but also for smaller companies and universities that would be willing to donate enough money to establish a foundation.

But then if FreeCAD team resolves at least some of the goals I mentioned, including most of the high priority ones, they could really get the ball rolling and get the big players to jump in on funding. After all, FreeCAD has such features even for big players that no other CAD software can afford to offer:
1. It's code is open source, and it can run on Linux, so it can be used in higher security environment, such as military or aerospace where code can be checked for unwanted or malicious features.
2. Everything can be automated and parametrized much better than in other CAD software, that either has terrible in-house macro language or equation engine that lacks features.
3. It is much easier to write or find someone to write a custom workbench/integration, which are very useful for any size of business for managing catalogues of parts, project management, PLM/PDM, database integration, manufacturing and QC integration, etc.
4. FreeCAD has an open standard that can be implemented by anyone, even if some company still decided to stick with other software, it would still be able to import FreeCAD files, possibly even with keeping the feature trees.
5. Can be easily taught in university or self-taught, which greatly expands the number of potential employees.
6. Can at some point become de-facto standard for most companies, which means much easier interchangeable files and integrity.
7. Can have a great integration with other software such as Redmine/Taiga for project management, Inkscape and Blender for much finer artistic surfacing/production, LibreOffice for automated specification generation, Octave and python for automated calculations.
8. FreeCAD UI can be customized to mimick that of any CAD software, from CATIA and Siemens NX to Solidworks and Fusion 360, therefore minimizing retraining time.

But to get that going, FreeCAD needs to secure a stream of funding, at least from small companies, because there isn't much in terms of how much hobbyists can afford, and most people wouldn't pay more than 10$/mo out of their own pocket, but even small businesses could afford to pay in range of 100 to 1000$ a month to fund FreeCAD development.
But to get them using FreeCAD, those fundamental goals that I listed should be achieved, so that the productivity and quality at smaller companies does not suffer from FreeCAD bugs and shortcomings.
drmacro
Veteran
Posts: 8864
Joined: Sun Mar 02, 2014 4:35 pm

Re: Benchmarking and Evaluating FreeCAD against other CAD/CAM/CAE/PLM software

Post by drmacro »

Jzuken wrote: Fri Dec 10, 2021 10:04 am But to get them using FreeCAD, those fundamental goals that I listed should be achieved, so that the productivity and quality at smaller companies does not suffer from FreeCAD bugs and shortcomings.
And the goals and benchmarks you describe are all laudable. And I'd note your thread and those descriptions are not new and are recurring themes here on the forum.

But, you assume there is a goal or plan for some sort of industry adoption. I don't think this is the case.
Star Trek II: The Wrath of Khan: Spock: "...His pattern indicates two-dimensional thinking."
paullee
Veteran
Posts: 5097
Joined: Wed May 04, 2016 3:58 pm

Re: Benchmarking and Evaluating FreeCAD against other CAD/CAM/CAE/PLM software

Post by paullee »

Thanks @Jzuken for the explanation :)
User avatar
Jzuken
Posts: 11
Joined: Sun May 17, 2020 10:17 pm

Re: Benchmarking and Evaluating FreeCAD against other CAD/CAM/CAE/PLM software

Post by Jzuken »

drmacro wrote: Fri Dec 10, 2021 12:27 pm
Jzuken wrote: Fri Dec 10, 2021 10:04 am But to get them using FreeCAD, those fundamental goals that I listed should be achieved, so that the productivity and quality at smaller companies does not suffer from FreeCAD bugs and shortcomings.
And the goals and benchmarks you describe are all laudable. And I'd note your thread and those descriptions are not new and are recurring themes here on the forum.
I do agree that my descriptions might not be new, but in this thread I try to collect and systemize FreeCADs shortcomings that are important, and those that are less important. To achieve recognition between professionals and grow it's user base some of those kinks have to be ironed out, especially the ones that affect workflow greatly.
drmacro wrote: Fri Dec 10, 2021 12:27 pm But, you assume there is a goal or plan for some sort of industry adoption. I don't think this is the case.
And why is it not the case? Do you speak on behalf of developers?

Not only industry adoption would be the best thing that could happen to FreeCAD, but it also would vastly benefit a tremendous number of people, I think it's a noble goal to be pursued.
Just think of how many students, makers and smaller companies would greatly benefit from a CAD that not only they don't have to pay out of their nose for, but that is also so easy to teach and to modify to suit your needs. Even for larger companies FreeCAD would not only mean cost savings in terms of software licenses, but also a much greater workforce, greater interoperability, greater security and more fine-tuned workflows.
The good that FreeCAD can do for the people if it gets into industry is huge.
drmacro
Veteran
Posts: 8864
Joined: Sun Mar 02, 2014 4:35 pm

Re: Benchmarking and Evaluating FreeCAD against other CAD/CAM/CAE/PLM software

Post by drmacro »

Jzuken wrote: Sun Dec 12, 2021 11:45 am ...
I do agree that my descriptions might not be new, but in this thread I try to collect and systemize FreeCADs shortcomings that are important, and those that are less important.
And, as I've noted, you are far from the first first to note these things.

To achieve recognition between professionals and grow it's user base some of those kinks have to be ironed out, especially the ones that affect workflow greatly.

...
And why is it not the case? Do you speak on behalf of developers?
No I do not speak for the developers. But, I do note the things I've observed while watching FreeCAD for almost half of it's existence.

I know of no "mission statement". The concept of industry adoption and growth of the user base are a projections by you (and many others) yet there is no supporting evidence that it is being sought.

Quoting one of the founding members: "FreeCAD has a very small team of developers. We have not as much time as we would like to dedicate to FreeCAD, and things are not planned ahead, they are done when some developer sees it fit and when he finds time to do it. So we don't maintain a list of tasks to be done. It is up to you to find something you would like to do, for example a defect you would like to correct, or a small feature you think is missing."
Not only industry adoption would be the best thing that could happen to FreeCAD, but it also would vastly benefit a tremendous number of people, I think it's a noble goal to be pursued.
Just think of how many students, makers and smaller companies would greatly benefit from a CAD that not only they don't have to pay out of their nose for, but that is also so easy to teach and to modify to suit your needs. Even for larger companies FreeCAD would not only mean cost savings in terms of software licenses, but also a much greater workforce, greater interoperability, greater security and more fine-tuned workflows.
The good that FreeCAD can do for the people if it gets into industry is huge.
But...a noble goal for who? There is no board of directors (as in the Blender foundation), no steering committee to make roadmaps, no executive officer (as in the Blender foundation) to do outreach and sign up corporate sponsors, etc.

AFAIK there is no concerted effort to create those things.

Without these things, discussions like this are "pie in the sky" musings.

(unless it happens to, unlike so many similar discussions of the past, spur actions)
Star Trek II: The Wrath of Khan: Spock: "...His pattern indicates two-dimensional thinking."
keithsloan52
Veteran
Posts: 2756
Joined: Mon Feb 27, 2012 5:31 pm

Re: Benchmarking and Evaluating FreeCAD against other CAD/CAM/CAE/PLM software

Post by keithsloan52 »

drmacro wrote: Fri Dec 10, 2021 1:29 am When do you plan to have the results of the benchmark ready for publication?

FreeCAD is an incredible tool as it sits. And the points you make might be valid if there was a goal to be any of the things you describe.

You mention Blender as an example. But, though there are definitely commonalities of origin between Blender and FreeCAD, at this point there is a chasm of differences.

Blender has implemented the foundation. It consists of a 30 member paid team. There is a management team doing outreach and defining roadmaps. With annual funding of over a million Euro. They have a headquarter office building.

Alas, there is nothing like that for FreeCAD.
Not to forget that Blender has a great leader in Ton Roosendaal https://en.wikipedia.org/wiki/Ton_Roosendaal] who has elevated Blender into a different league.
Post Reply