BlenderBIM Experimental IFC Git add-on

brunopostle

As everyone knows by now, BlenderBIM is a Free and Open Source tool for working with Native IFC files. IFC is the open standard for BIM models, and Native IFC is a technique for working with IFC directly as first class data rather than as an exported dump of a proprietary database.

Native IFC applications keep data in place, only regenerating parts of the model that have been changed by the author. This has lots of advantages, among them are that it is easy to compare two versions of the same model and tell exactly what has changed between them, it also allows for very efficient storage and transmission of just the changes between versions.

Git is universal in the IT industry for distributed version control. Git is robust and secure, it can store large amounts of data, tagging and retrieving versions in an instant, branching and sharing data is reliable, every revision has a unique cryptographic hash allowing full verification and traceability.

Native IFC and Git are a perfect fit, the combination already works well, but existing GUI tools are designed for
source-code management not BIM workflows.

IFC Git is a new add-on that lets you manage your BIM data in a local Git repository without leaving BlenderBIM. Instead of just saving a file, with IFC Git you can create revisions with descriptions. These revisions are browsable and retrievable, changes between two revisions can be visualised in 3D. Any revision can be forked, creating a branch for developing options or playing with designs. IFC Git even has an experimental tool to merge branches using an IFC specific three-way merge for conflict resolution.

This short screencast shows all these IFC Git features in action:

Ace

Amazing amazing amazing going to try this out today!

Coen

This is absolutely amazing, this just proves open-source is the way forwards.

brunopostle

Note this isn't a single download install, yet. You will need git, GitPython and ifcmerge. I think something like this will work on Windows:

Install git

Using the cmd.exe command-line (hit the Windows key and type 'cmd'):

winget install --id Git.Git -e --source winget

This should install git, close cmd.exe and start it up again, now git should be in your %PATH%.

Install ifcmerge

Download and unzip the ifcmerge.exe file:

https://github.com/brunopostle/ifcmerge/releases/tag/2022-06-20

...and put it somewhere in your %PATH%, like this folder here just created by installing git:

C:\Users\John Doe\AppData\Local\Programs\Git\cmd

Install GitPython in blender

I have no idea if this is the recommended technique, in the blender python console:

import pip
pip.main(["install", "GitPython"])

Install IFC Git

Download the single ifc-git.py file and install using blender Edit > Preferences > Add-ons:

https://raw.githubusercontent.com/brunopostle/ifc-git/main/ifc-git.py

brunopostle

..also don't expect too much from the branch merging. BlenderBIM is not quite as Native IFC as it could be yet, and conflict resolution in a three-way merge is asymmetrical, so the order that you merge multiple branches is important.

bernd

how about usage from python only?

brunopostle

@bernd GitPython is a partial git implementation, mostly it interacts with the git database directly, but for some stuff it forks out the git binary.

The ifcmerge.exe windows executable is actually a compiled perl script that doesn't have any library dependecies. I'm sure it can be rewritten in python, contributions welcome, you need to be good with regular expressions ;)

theoryshaw

works!
A small addition...

...and put it somewhere in your `%PATH%`.  The path might look like (1) of the following. 

- `C:\Users\John Doe\AppData\Local\Programs\Git\cmd`
- `C:\Program Files\Git\cmd\`
- If you don't know the path, use the command `where git.exe` in the `cmd.exe`.  

Coen

Could this be used for collaborating on a remote git server as well?

brunopostle

@Coen yes! but at the moment there are no remote commands (fetch, remote, pull, push, clone etc..) in the GUI (contributions welcome).

This shouldn't stop you from using an external tool (a Git GUI or even the command-line Git) for this stuff, no need to restart blender. ie, if you clone a remote repository the 'origin' will be set in the repository, so you can push any locally committed work to the 'origin'. You can add a remote for somebody else's version of the same project, and fetching this will create a branch in your local repository that you can browse/diff/merge within the blender add-on just like any other local branch.

Moult

I've recently been using this website to help people deal with how %PATH% works, maybe someone might find it useful: https://superuser.com/questions/284342/what-are-path-and-other-environment-variables-and-how-can-i-set-or-use-them

Bimlooser

Really useful. I have a question: "how are data changes handled: they are only highlighted or I can see details somewhere?"

brunopostle

@Bimlooser at the moment all it does is highlight blender 3d objects where the corresponding IFC element has changed or is new.

It isn't looking any deeper, so in practice the object will only be highlighted if its name, location, or representation etc.. is different ie. it is literally only looking to see if the entity text has changed. As it happens, blenderbim will always create a new representation or location entity whenever there is a change, so this works, for now. It wouldn't be hard to look a bit deeper and check to see if entities that reference or are referenced by the 3d element have changed, and this would give a bit more coverage.

Regarding pure non-3d data, the add-on knows which entities are changed/added/removed, but there is no obvious way to show this in blenderbim because the data is presented in different parts of the gui (if at all). We could offer a list of entities and browse them by id in the IFC debug panel, but that would be no fun :(

Bimlooser

Thanks. No matter. The add-on rocks

brunopostle

A quick update: the add-on has had a number of fixes and enhancements in the last week. I've also added some items to the github tracker for things that need some more thought, basically I'm asking for input on: tagging, visual diffs, remote operations, and merge conflicts.

tlang

@brunopostle love this project. Some questions: Regarding merge assymetry as explained on github, would it be helpful to first assign GUID's instead of relying on instable STEP-ID's? And, regarding attribute change lists, changed attributes might be highlighted as in Tekla Structure change managment? ()?

brunopostle

@tlang Using GUIDs instead of step IDs ought to work, but it would need a lot of heuristics and assumptions to track down and compare non-rooted entities (which don't have GUIDs). BlenderBIM preserves step IDs for other good reasons, and this add-on is exploring the potential of this new Native IFC world where we don't have to deal with IFCs as data dumps from proprietary databases. For instance, the function that identifies all the added/modifed/deleted entities between two git versions of a Native IFC model is 25 lines of code and runs in milliseconds, this functionality would be a lot more complex and maybe even not fully resolvable if it had to work with GUIDs only.

Having said that, you are right, a tool that did this job using GUIDs would help with merging IFCs where step IDs had already been rewritten to accomodate a Native IFC three-way-merge conflict resolution.

Another way to avoid asymmetrical merge resolution (and I'll add this to the github issue) would be for authors to pre-agree working ranges of step IDs - currently BlenderBIM allocates new step IDs based on the highest existing number, but if my BlenderBIM was configured to only create entities in the range 10000001, 10000002, etc.. and your BlenderBIM was configured for 20000001, 20000002 etc.. then the asymmetrical nature of the merge conflict wouldn't apply, since there would be no need to rewrite local IDs to make space.

The Tekla demonstration is very impressive. We should have a list of added/modified/deleted entities, where selecting an element in the list would select the equivalent object in the 3d vew and would therefore display info for the element in Blender Object Properties. This wouldn't help for entities that don't have a 3d object representation, which is the point I was trying to make (and which Tekla have conveniently ignored) - What if the only thing that has changed between the two versions is a material pset? the material doesn't appear as a 3d object and the material psets are actually under Blender Material Properties, not Object Properties - Access to non-object entities such as materials is scattered across the BlenderBIM GUI for logical reasons.

This doesn't get prettier, if you are viewing a material pset changeset, you want to be able to show all the objects, layersets and profilesets that use this material.

cvillagrasa

This thing of the non-rooted entities in IFC has always bothered me. Wouldn't it be possible to bypass it somehow? May the IFC gods forgive me for these thoughts, but what about a certain IfcPropertySet per project, with a bunch of IfcPropertySingleValue, whose Name happens to be a GUID, and its Specification a Step ID. And then having one of such properties per non-rooted entity, thus linking all of them to a GUID?

Well, the filesize could become a problem I guess, as well as additions/deletions outside of software taking this into account (assuming that external software exports all entities which were imported, which perhaps is a lot to assume). Ahh, too dificult. When will IFC have all rooted entities? IFC5?

brunopostle

@cvillagrasa I don't know if proprietary BIM tools would be able to attach GUIDs to everything, in IFC even a single coordinate can be a separate entity. It would be as easy to ask them to preserve step IDs.

Maybe you could write a tool that traverses the graph of a traditionally exported IFC, and maps all the non-rooted entities to equivalent entities in a previously exported IFC, this would bring all the benefits of native IFC.

cvillagrasa

@brunopostle said:
@cvillagrasa I don't know if proprietary BIM tools would be able to attach GUIDs to everything, in IFC even a single coordinate can be a separate entity. It would be as easy to ask them to preserve step IDs.

Sure, proprietary software is not going to follow any possible community originated IFC unofficial approach. I was more thinking on a workflow between open source and proprietary software, where at each loading in the open source one, new non-rooted entities were tagged on that artificial pset, and properties of that pset corresponding to deleted non-rooted entities were deleted accordingly.

That would allow to keep track of every entity, but it very well could be that proprietary tools ingested that invented pset, but never exported it out again. Maybe that risk could be minimized by instancing a dummy IfcTypeObject and referencing the pset from it. But yes, the alternative of preserving Step IDs may be easier to achieve in practice.

I really don't particularly enjoy neither of them... the elephant in the room is that every entity should be rooted (in my view). Because an IfcDirection from an IfcAxis2Placement can refer to the global X or the global Z, and there's no need to have it repeated endlessly for every new IfcRepresentationMap. Or in similar ways, individual entities like IfcCartesianPoint can almost always have meaning by themselves and hence be subject to be referenced and tracked.

@brunopostle said:
Maybe you could write a tool that traverses the graph of a traditionally exported IFC, and maps all the non-rooted entities to equivalent entities in a previously exported IFC, this would bring all the benefits of native IFC.

I guess that rabbit hole leads to the Weisfeiler-Lehman isomorphism test and so on. I guess that the downside is that it could really last for a while in large files.

theoryshaw

If you'd like to help fund the development behind Bruno's GIT/IFC Interface in BlenderBIM.
https://opencollective.com/osarch/projects/git_ifc

brunopostle

@theoryshaw said:
If you'd like to help fund the development behind Bruno's GIT/IFC Interface in BlenderBIM.
https://opencollective.com/osarch/projects/git_ifc

Note also that this funding matched by osarch is available to anyone who wants to work on these IFC/Git tools, this could be porting the add-on to be part of BlenderBIM, any of the items on the ifc-git tracker, FreeCAD integration, a web app for interacting with IFC/Git repositories, etc.. etc..

Coen

Man, this is so cool.

@brunopostle

Install GitPython in blender
I have no idea if this is the recommended technique, in the blender python console:

import pip
pip.main(["install", "GitPython"])

Is there any reason this can't be bundled with the IFC git add-on? :-)

I might be using the tooling wrong, but I can't manage to create a new branch

System console gives no output
In Git fork I see

I made a branching in git fork it shows up in BlenderBIM, however I don't see any changes I am making

Futhermore questions

• Is the browse branch a git checkout?
• Where could I run git commands? Just in the Blender terminal?
• How would I collaborate with someone using pull requests? This is the Ifc Merge tool I guess?
• How would I set up a remote server connection? It's just storing the ifc file on a remote git server somewhere I think? Just as easy as that? :-)

If anyone is up for experimenting with this nice tool on Github please let me know :-)

This might be a useful resource for people who never have done anything with git and like to use this tool:
https://github.com/skills/introduction-to-github

brunopostle

@Coen said:

Install GitPython in blender

Is there any reason this can't be bundled with the IFC git add-on? :-)

Yes, GitPython is all python, so it can be stuffed into a zip installer and it should work anywhere. Though it uses the git executable for some operations and I don't see an easy way of bundling this, Git itself will still be a separate install.

I might be using the tooling wrong, but I can't manage to create a new branch

Basically a Git branch or tag name has restrictions, in particular it can't have any spaces, the operator won't be enabled unless you enter a valid name in the text box. I don't like this UX, but I don't know what the best way of doing this is, I could enable the button and put a message in the status bar when you try and use an invalid name, or even a pop-up error, maybe.

I made a branching in git fork it shows up in BlenderBIM, however I don't see any changes I am making

The add-on shows the Working branch, you should see all commits if you select the same branch in Browse branch

Futhermore questions

• Is the browse branch a git checkout?

No the browser is for navigating around the history, but nothing happens to the current project unless you click on the switch revision or merge branch buttons.

• Where could I run git commands? Just in the Blender terminal?

You can run git commands (creating branches, tagging etc..) in the system console (cmd.exe) or in a Git GUI, the add-on should pick them up when you click the refresh button.

• How would I collaborate with someone using pull requests? This is the Ifc Merge tool I guess?

The add-on doesn't do any remote commands (yet), but this shouldn't stop you from doing this stuff. Basically you are managing a local repository in the add-on, but this repository can have a remote origin that points to an online repository (or it can have multiple remotes if you really want to make your head hurt).

• How would I set up a remote server connection? It's just storing the ifc file on a remote git server somewhere I think? Just as easy as that? :-)

So assuming that you have a local IFC file with some history in a local repository, you can create an empty remote repository on github - which then gives you instructions for adding this as your origin, and then pushing your main (sometimes called master) branch to it - after this anyone can clone the repository from github, they can open your IFC on their own computer, make their own changes, browse the history etc..

The procedure for merging somebody else's remote repository is basically: create a local branch for them and pull their remote work into this local branch (for now these steps need to be done outside the add-on, github gives instructions for doing this whenever you have a pull request waiting). At this point you can browse their commits in the add-on just like any other branch, and even switch revision to view their model (which won't break any work you have committed to your own branches). But to merge their work, you need to be switched to the HEAD of the branch you want to 'merge-into', then find the branch you want to 'merge-from' in the browser and click the merge branch button.

If anyone is up for experimenting with this nice tool on Github please let me know :-)

This might be a useful resource for people who never have done anything with git and like to use this tool:
https://github.com/skills/introduction-to-github

I coming around to thinking that Git is a core technology of the 21st century (like the internal combustion engine in the 20th), but it is unintuitive and the documentation only makes sense when you have a solid overview. I think this add-on should centre the functionality for users to store, browse and retrieve revisions, as this is conceptually straightforward and immediately valuable. Branches, pushing, remotes, merging, pull-requests etc.. are powerful but they can easily overwhelm the GUI and cause much confusion, so they need to be introduced carefully.

theoryshaw

Hi @brunopostle
Is the intention to always use branches when pulling a change into a file?
Or can collaborators pull changes into files if they are both on the master branch?

brunopostle

@theoryshaw Git itself assumes that you only ever merge branches. So when you get a pull request you are expected to pull it into a local temporary branch, do the merge (then remove the temporary branch if you want).

ifcmerge doesn't care about this stuff, so if you are happy working on the command-line, you can merge a file that you received by email without worrying about branches.

theoryshaw

@brunopostle are you able to merge ifc-git-branch into master? I got a failed merge. It seems like it should work.
https://gitlab.com/openingdesign/clothing_boutique

brunopostle

@theoryshaw there is a merge conflict, ifcmerge reports the location, but ifc-git doesn't show it in the UI yet:

$ git merge remotes/origin/ifc-git-branch
Auto-merging testy.ifc
CONFLICT (content): Merge conflict in testy.ifc
Automatic merge failed; fix conflicts and then commit the result.

$ git mergetool --tool=ifcmerge
Merging:
testy.ifc

Normal merge conflict for 'testy.ifc':
  {local}: modified file
  {remote}: modified file
entity #1067 attribute [6] conflict!
merge of testy.ifc failed

$ git merge --abort

entity #1067 is an IfcWall, ifcmerge says that attribute [6] ObjectPlacement has been edited in both branches.

However #1067 is identical in both branches, so this looks like a bug, will investigate...

brunopostle

@theoryshaw ifcmerge is adamant that this ObjectPlacement started as #1080, was changed to #1177 in 'master' and to #1130 in 'ifc-git-branch', but I see #1130 in both branches. I've run out of time to investigate, will look again later.

theoryshaw

Np! I can run some more tests, to see.

brunopostle

@theoryshaw Basically you had a project with a single wall, then in master you made a copy of this wall, and in a branch of the original project you made a different copy of this wall. The merged result should have had all three walls.

The problem is that when you copy a wall, blenderbim creates a new wall, but also creates a new ObjectPlacement for the original wall, even though from a user perspective nothing about the original wall has actually changed.

Now ifcmerge is clever/stupid enough to realise that this ObjectPlacement is entirely new in both branches, even though due to the way you did it it happened to use the same set of step-ids in both branches (which was a boggle to say the least).

ifcmerge will refuse to merge an entity where an attribute has been modified in both branches - I'm not sure that randomly picking one to keep is a good idea because this will result in invalid IFC data such as orphan entities, but maybe we need to do this anyway with a 'prefer local/prefer remote' flag, since resolving this stuff manually is no fun at all.

But ultimately this is a BlenderBIM misfeature: copying a wall regenerates the ObjectPlacement of the original wall, even though as a Native IFC user you expect the original wall to be unchanged. None of this would have bothered anyone until now, so there are probably other similar glitches throughout BlenderBIM. I think a similar thing happens when you delete a window from a wall, the wall gets a new ObjectPlacement and/or Representation even though neither has changed. These need collecting and reporting in the tracker.