[IFC Merge] Proposal: IFC merge tool

edited June 2022 in General
This discussion was created from comments split from: Brainstorm on projects we'd like to be able to fund.

Comments

  • Proposal: IFC merge tool

    An IFC merge tool would enable fully collaborative BIM via git repositories.

    IFC STEP files are plain text and line-based, so in principle they should play nicely with tools like git/mercurial etc.. In practice, Blenderbim (I haven't tested Freecad) is already extremely git friendly, if you edit an IFC file and save it, the only things that change are: the FILE_NAME() header (where the time-stamp is updated); entities where parameters have changed are updated in-place, so this is a single line change; new entities are appended to the end of the file; and deleted entities are simply removed without renumbering any other entities. So currently if you are working on an IFC file in Blenderbim and comitting changes to git, this is extremely efficient - Updates are small and the git database grows very slowly - You should do this already for perfect change control, tracking and rollbacks.

    Collaborative working via git involves forking and merging (typically via pull requests), this doesn't work currently with IFC, at all. i.e. at the moment, if Amy and Bob want to work on the same IFC file, Amy has to commit her work and and push it to a repository, then Bob can check out this file, work on it and push it back - But Amy has to agree not to make any changes while Bob is working - otherwise there will be an unresolvable merge conflict. The main problem being that Blenderbim starts adding entities starting from the last id number already in the file: if the last entitiy in the file is #12345, both Amy and Bob will create #12346, and git won't be able to merge these new entities successfully.

    git has a mechanism for dealing with this, when attempting to merge branches or forks the user can specify a custom merge tool, this tool needs to understand the file format and be capable of rewriting a file syntactically-correctly in such a way that merging can proceed. There are three types of merge conflict that the tool needs to handle:

    1. Conflicting FILE_NAME() headers. This is easy, the header should be updated with a new timestamp corresponding to the time of the merge.
    2. Conflicting updates to a single entity. If different attributes have changed, then it may be safe enough to merge both changes (for some entities anyway). Sometimes a single attribute will need to be merged, such as with a list of aggregated entities. This doesn't need to be implemented initially in order to demonstrate the viability of the tool.
    3. Conflicting entity id numbers. The tool should determine which entities are new rather than just updated (by looking at the end of the file), and should renumber these in the local version of the file to allow the admission of entities from the remote version.

    If this can be made to work, then doing BIM with native IFC would be so obviously superior to any other workflow that everyone will do it :)

    CoenMassimotheoryshawNigeltlangJesusbillpeperiberavpajiccvillagrasa
  • @brunopostle this would be awesome!

  • Would love to see this as well.

    If this can be made to work, then doing BIM with native IFC would be so obviously superior to any other workflow that everyone will do it :)

    Agreed. :)
    @geometrygym played around with this concept awhile back.

  • @theoryshaw splitting the IFC into files and folders would also enable concurrent access and other cool stuff, i.e. multiple people editing the same model and seeing updates in real time. One file per entity is a valid approach, but I don't think it is necessary for a full git commit/pull/merge workflow - which I think is all we need.

    The difference is that native IFC modelling with blenderbim is now extremely git friendly compared to the old import/export workflow, it just needs a three-way merge tool.

  • I made a start on this. So far it integrates with git, but only merges branches with minor attribute differences: https://github.com/brunopostle/ifcmerge

    theoryshawJesusbilljchkoch
  • edited May 2022

    I've been playing around with IFC and git for some years now. It's not as easy as I initially hoped. In 2017 I wrote down my initial thoughts: https://bimagineering.blogspot.com/2017/09/thoughts-about-git-for-ifc.html

    In the end I implemented a system as described in my post. It did allow for most git tools, like version tracking and merging. But it did not scale to larger models. The thing with git is that when performing a merge, a full checkout needs to take place. If you store entities and links in separate files (as described in the blog), a lot of files need to be created and processed. And all these files require at least 4kb, even if the contents of the file is less. So large models result in super large checkouts.

    Your idea is to use the STEP file only. I have the feeling this won't hold either. First, you need to understand the way git versioning works: git looks at the hash of the complete file, not file parts. Line diffs are calculated only when you as a user want to see the file differences. Second, the STEP IFC file is just a way to write IFC data to disk. And the STEP definition does not state that line identifiers need to be fixed. So your idea would need an industry wide agreement on strict reading and writing of the STEP data. Industry wide standard...... Hmmm.....

    But what if we do find a way to write IFC to an ASCII file, or multiple files (which works better with git, since git versioning works with file hashes). Maybe there is a solution?

    JanFtheoryshawjchkochtim
  • @mtveerman this idea only works with a tool like blenderbim that does native IFC editing and that preserves the numeric ID of each step entity. So it is never going to work with tools that simply do an IFC export. This is fine for me because native IFC editing is a different paradigm that has other advantages.

    The tool I started (and haven't finished as I'm away from computers this week) doesn't use diffs, it is a custom three way merge tool for step files.

    I have also experimented with the 'lots of files' idea, this is potentially great, but these native IFC step files have some potential that should be explored. (my 'draft' proof of concept experiment: http://www.bruno.postle.net/older-stuff/2003/draft/ )

    JanFtheoryshaw
  • @brunopostle said:
    I have also experimented with the 'lots of files' idea, this is potentially great, but these native IFC step files have some potential that should be explored. (my 'draft' proof of concept experiment: http://www.bruno.postle.net/older-stuff/2003/draft/ )

    Cool, the way you write it I want to have it tomorrow! Is there any discussion on your concept somewhere? Would like to read some critique as well.

  • @JanF there was some discussion on the old freearchitecture mailing list, and the (also defunct and I can't remember the correct name exactly) free cad mailing list.

    The main problem with the 'lots of files' approach is that some filesystems are really bad at doing lots of files in one folder.

    JanF
  • I now have a 'working' ifcmerge tool: https://github.com/brunopostle/ifcmerge

    There is a test IFC file in the repository, you can help testing by forking the repository on Github, making some changes to the test IFC file, and doing a pull request, I'll see if I can merge the changes. Actually I need more than one pull request, each from a different fork.

    Note that this will only work if you use a native IFC editor (i.e. blenderbim). This merge tool will not support a traditional import/export workflow.

    tlangduncan
  • @brunopostle once ifcmerge is stable, do you think that something like ifcdiff can be done?

  • @agonzalezesteve do you mean visualising the difference between two versions of a file?

    It is easy to generate a list of entities that are modified, deleted or added between files (these changes do need to be done with a native ifc editor like blenderbim). Then these could be used to highlight elements in the blender GUI.

    One problem is that a lot of, or most, possible IFC changes are not visible as geometric changes to elements with representations. So if say a pset attached to a building changes, do you want the whole building marked as 'modified'? It isn't obvious what a useful GUI would look like.

    I think exported 2d drawings with automatically generated revision clouds that have git commit messages attached would be something. Or maybe you click on a revision note in an SVG drawing, and the associated changed elements get highlighted...

    jchkoch
  • @brunopostle I don't need to visualize them in a 3D model. For me it's enough having some text like git does.
    I think the output json file of ifcdiff it's what I'm looking for. https://github.com/IfcOpenShell/IfcOpenShell/blob/v0.7.0/src/ifcdiff/ifcdiff.py
    But it will be great having something similar to "git diff".

  • So, here's a demo, this is using working code.

    I have this simple model, it is in the ifcmerge repo, you can check it out:

    I created a branch add_stuff, created a water tank using blenderbim, added the tank to the storey 1, spatial zone, and committed this branch:

    Then I switched back to the main branch with the original building. In blenderbim I changed the colour of the masonry material to pink, saved and committed the IFC file:

    I now have two conflicting branches, git merge fails, so I need to use the ifcmerge tool:

    $ git merge add_stuff 
    Auto-merging test_model.ifc
    CONFLICT (content): Merge conflict in test_model.ifc
    Automatic merge failed; fix conflicts and then commit the result.
    $ git mergetool --tool=ifcmerge
    Merging:
    test_model.ifc
    Normal merge conflict for 'test_model.ifc':
      {local}: modified file
      {remote}: modified file
    

    ..and ¡tada! the model is merged:

    This is just adding stuff in one branch and modifying something else in the other. The tool can also handle deletions, modifications and additions in either or both versions of the file. It currently doesn't resolve conflicts between two versions of the same entity, but I'm working on this.

    VDobranovtheoryshawjchkoch
  • There is a test IFC file in the repository, you can help testing by forking the repository on Github, making some changes to the test IFC file, and doing a pull request, I'll see if I can merge the changes. Actually I need more than one pull request, each from a different fork.

    fun/fun...
    https://github.com/brunopostle/ifcmerge/pull/1

  • Merging attributes is now implemented. So if the same entity is edited in two different branches it can be merged if different attributes are edited:

    Original:

    #5348=IFCWALL('3yWUFAwebAXgkYb98hTepm',$,'Garden wall',$,$,#5393,#5363,$,.SOLIDWALL.);
    

    Local branch:

    #5348=IFCWALL('3yWUFAwebAXgkYb98hTepm',$,'Garden FOOLISH wall',$,$,#5393,#5363,$,.SOLIDWALL.);
    

    Remote branch:

    #5348=IFCWALL('3yWUFAwebAXgkYb98hTepm',$,'Garden wall',$,$,#5393,#5363,$,.USERDEFINED.);
    

    Merged:

    #5348=IFCWALL('3yWUFAwebAXgkYb98hTepm',$,'Garden FOOLISH wall',$,$,#5393,#5363,$,.USERDEFINED.);
    

    If the same attribute has been changed in both branches, merge fails with a useful error message:

    entity #5348 attribute [2] conflict!
    
    theoryshawJanFjchkoch
  • Merging lists within attributes is also implemented, so if both branches are messing around with a parent container, they can still be merged.

    For example, here is a list of objects in a spatial structure:

    #4586=IFCRELCONTAINEDINSPATIALSTRUCTURE('09hroIyDv55eMXcX5hlK4H',$,$,$,(#4584,#4598,#4610,#4622,#4634,#4646,#4658,#4670),#2382);
    

    In the local branch the user has deleted an element (#4598) and added a new one (#12345):

    #4586=IFCRELCONTAINEDINSPATIALSTRUCTURE('09hroIyDv55eMXcX5hlK4H',$,$,$,(#4584,#4610,#4622,#4634,#4646,#4658,#4670,#12345),#2382);
    

    ...but in the remote branch the user has also deleted an element (#4658):

    #4586=IFCRELCONTAINEDINSPATIALSTRUCTURE('09hroIyDv55eMXcX5hlK4H',$,$,$,(#4584,#4598,#4610,#4622,#4634,#4646,#4670),#2382); 
    

    The merged list contains both deletions and the addition:

    #4586=IFCRELCONTAINEDINSPATIALSTRUCTURE('09hroIyDv55eMXcX5hlK4H',$,$,$,(#12345,#4584,#4610,#4622,#4634,#4646,#4670),#2382);
    
  • edited June 2022

    I built a Windows version of ifcmerge.

    On the command-line it works the same as on Linux.

    If you use the Sourcetree GUI for managing your git repository (which seems pretty popular), you should (untested) be able to configure it here: Options -> Diff -> External Diff / Merge -> Merge Tool -> Custom

    Set Merge Command to C:\Program Files\ifcmerge\ifcmerge.exe (or wherever you have put it)

    ..and set Arguments to \"$BASE\" \"$LOCAL\" \"$REMOTE\" \"$MERGED\"

    theoryshawbruno_perdigaoAcejchkochMoult
  • I'm posting this since it may be relevant to the task:
    https://www.fast.ai/2022/08/25/jupyter-git/

    The fastai folks have adapted Jupyter Notebooks to Git. Maybe there are some insightful ideas on that git merge driver that could be applied to IFC (although that could potentially mean providing a deterministic entity sorting algorithm for non-native IFC Software, or sth else equivalent to their save hook).

  • @cvillagrasa the difflib and SequenceMatcher tools might be useful if anyone wants to rewrite ifcmerge in python (ifcmerge is basically feature complete, but I appreciate that perl isn't always easy to get running on all systems).

    Merging with non Native IFC tools is always going to be hard, but I can see that a preprocessor would help. Something that rewrote an exported IFC using an ancestor Native IFC as a reference. I.e. say there is a Native IFC file in git, you import this into a proprietary tool, modify and export, then you try and map the exported file to the STEP IDs in the original file.

    cvillagrasa
  • Could this be used to merge two ifc files into one?
    Let's say I have an IFC file "A" containing only walls, and an IFC Ffile "B" containing a window.
    Could I use file B to make modifications on the window, which I can then merge with B? Could this IFC file then be used as an instance? To create copies of the windows in file A?
    So I would have seprate IFC files functioning as Revit families?
    Or I completely misunderstanding the purposes of this technology?

  • @Coen in this case it would be better to make the window into an IFC Type, then it can be reused as many times as required.

    IFC doesn't have a way to represent repeats of larger aggregations, such as "these storeys are all identical", this is an unresolved problem.

    If you just want to join two unrelated IFC files, there is a patch recipe that will do it that ships with Blenderbim.

    ifcmerge is strictly for merging back two forked versions of the same IFC file.

    Coen
  • edited March 5

    @brunopostle said:
    ...
    If you just want to join two unrelated IFC files, there is a patch recipe that will do it that ships with Blenderbim.

    Where in BlenderBIM / IfcOpenShell source? ATM I do have exactly this problem. I would like to merge two IFC files which use the same spatial structure tree.

    cheers bernd

  • background ... In our company I have set up some automatic process which add properties to the IFC after exporting from CAD. For different object type there will be done totally different checking and added totally different properties. For some reason I need to combine such objects into one ifc. The simplest method would be to not change any checking or adding properties method, but just to merge the ifc after my smart process have finished.
    The opposite of ifcpatch which is a great tool :-)

    cheers bernd

  • Hi Bernd,
    what about this recipe?
    https://blenderbim.org/docs-python/autoapi/ifcpatch/recipes/MergeProject/index.html
    As of now, i only splitted Ifc files. So no personal expirience herewith.
    BR
    Marcus

Sign In or Register to comment.