Multiprocessing options in Blender

edited June 2020 in General

"For speed, can you retry the BlenderBIM Add-on with the multiprocessing option enabled,.. Speed is incredibly important."
I want to see how slow or fast Blender is at imprting IFC models.
Where can this option be found? @moult

Comments

  • @magicalcloud_75 it is worth noting that the BlenderBIM Add-on has not had much in terms of speed optimisations. All geometry is "meshed" by IfcOpenShell prior to import (even meshes are meshed again!) which means there is a ton of obvious inefficiency. A proper solution requires implementation of what I call "native IFC geometry" support, where Blender minimises geometry conversions and natively supports the raw IFC geometry tiself. The work required to implement this (and the ensuing speed improvements that will follow) is currently happening.

    Also, by default, the BlenderBIM Add-on will only use one CPU, which is, well, 4 times slower than if you had 4 CPUs in your computer :) To take advantage of your full computer, you need to enable the multiprocessing feature, which is disabled by default, as I am not fully convinced it is fully stable for production yet. Anyway, let's have some fun, enable it here prior to importing a file and you're good to go:

    By the way, the BlenderBIM Add-on also imports things which other BIM authoring packages do not necessarily import. For example, it will import the type geometry representation separately. I'll keep you posted whenever I offer new configuration options to customise it.

    Test away, @magicalcloud_75 ! Let's see if we can't fix things up!

  • edited June 2020

    Found it!
    Scene properties > Model View Definitions.
    There is a bunch of other stuff for tweaking as well.
    Testing Upfront.ifc ... 0%

  • edited June 2020

    I like the phase 'Automaticly Set Vendor Workaround
    sound like .. King of hill! :)

    Blender should be summitted for 'Best software development for IFC' soon ?
    https://www.buildingsmart.org/bsi-awards/

    Moult
  • edited June 2020

    UPFRONT.IFC
    details: https://3d.bk.tudelft.nl/projects/geobim-benchmark/uptown.html
    Timer is on : 14:51
    Blender crashed at 14:55, multiprocessor on, ifc2x3 schema, Revit workarounds not explicitly enabled
    Timer is on : 15:01
    Blender crashed at 15:06, multiprocessor on, ifc4 schema, Revit workarounds not explicitly enabled
    Timer is on : 15:11
    Blender crashed at 15:15, multiprocessor on, ifc2x3 schema, Revit workarounds explicitly enabled
    The four minutes seem to be a constant factor.
    BlenderBIM It does come come with crash warnings or -logs?

    MYRAN_FIXED.IFC
    details : https://3d.bk.tudelft.nl/projects/geobim-benchmark/ifcmyran.html
    Timer is on : 15:22
    Blender crashed at 15:23, multiprocessor on, ifc2x3 schema
    Timer is on : 15:24
    multiprocessor off (by default), import ok Timer : 15:28

    Conclusion in short: multiprocessing IFC is unstable

  • edited June 2020

    @magicalcloud_75 with the latest version of the BlenderBIM Add-on Myran_fixed.ifc (28MB) will succesfully import with the multiprocessing option enabled, in 17.49 seconds. See breakdown below:

    Starting import process :: 0.00
    Load existing rooted elements :: 0.00
    Load diff :: 0.00
    Caching file :: 0.08
    Loading file :: 1.49
    Setting file :: 0.00
    Set vendor worksarounds :: 0.00
    Calculate unit scale :: 0.00
    Set units :: 0.00
    Create contexts :: 0.00
    Create project :: 0.00
    Create classifications :: 0.00
    Create doc info :: 0.00
    Create doc refs :: 0.00
    Create spatial hierarchy :: 0.00
    Purge diffs :: 0.00
    Create type products :: 0.22
    Create aggregates :: 0.02
    Process element filter :: 0.00
    Filtering ifc :: 0.01
    Patching ifc :: 0.00
    Georeferencing ifc :: 0.00
    Creating groups :: 0.00
    Creating grids :: 0.00
    Creating native products :: 0.00
    250 elements processed in 0.75s ...
    500 elements processed in 0.95s ...
    750 elements processed in 1.84s ...
    1000 elements processed in 0.76s ...
    1250 elements processed in 0.41s ...
    1500 elements processed in 0.43s ...
    1750 elements processed in 0.44s ...
    2000 elements processed in 0.46s ...
    Done creating geometry
    Creating meshified products :: 13.39
    Relating openings :: 0.00
    Placing objects in spatial tree :: 0.53
    Info: Removed 7383 vertice(s)
    Mesh cleaning :: 1.75
    Import finished in 17.49 seconds
    

    I have also fixed some errors which result in crappy material assignment - see the screenshot below where the site topography has proper road/landscape rendering:

    Keep up the tests! I really appreciate it!

    Jesusbill
  • @magicalcloud_75 a few more points to make, before I give an update on UpTown.ifc:

    • Changing the IFC schema does nothing for an import. Don't bother :) Importing will auto-detect the schema of the receiving file.
    • Please continue to test for multiprocessing, and any crashes let me know so it can be fixed. It is slowly being stabilised.
    • I'd recommend leaving the "automatically set vendor workarounds" enabled, and not touching any other vendor workarounds, even if you know it comes from Revit. The way they are currently designed, it's best to just leave them alone unless you really know what you're doing.

    Process logs are available in the blenderbim/bim/data/process.log file in your Blender add-on's directory. However, it's best to just share the file as you have, as especially when it comes to multiprocessing and speed issues, they are a bit more intricate to solve.

  • Nice work. I am curious how that moster uptown will do now. So i just download and install the newest version, right? What version to look at before continuing the test?

  • @magicalcloud_75 these updates have not yet been released. You will need to wait until the next release.

    The UpTown.ifc is a bit of a strange one. I have imported 500MB of IFC data much faster than UpTown.ifc. There are a few odd ways in which that model is built that leads to inefficiencies which the BlenderBIM Add-on struggles to handle. I'll post again here when I've optimised the import process further with details.

  • Could you explain briefly how multiprocessing works @Moult ? Is it something specific to blenderbim, or in ifcopenshell core?

  • @yorik it's in IfcOpenShell core: there is ifcopenshell.geom.iterator which will iterate through all elements which have shape representations. The iteration and shape processing is done in IfcOpenShell, with multicore, so it is very fast.

    Here's some sample code:

    iterator = ifcopenshell.geom.iterator(
        self.settings, self.file, multiprocessing.cpu_count(),
        include=self.include_elements or None,
        exclude=self.exclude_elements or None
        )
    valid_file = iterator.initialize()
    if not valid_file:
        return False
    while True:
        shape = iterator.get()
        if shape:
            self.create_product(self.file.by_id(shape.guid), shape)
        if not iterator.next():
            break
    

    It returns a shape which includes faces, edges, and verts, as well as the object matrix. This makes it super useful in Blender. I think it is less useful in FreeCAD, since in FreeCAD a lot of the underlying shape representations are processed into native FreeCAD profiles and extrusions so the faces/edges/verts result isn't too useful - not like Blender where sometimes you don't need to retain the parametric shapes and you are just happy to view an IFC with meshes (although even processing the IfcFacetedBreps is faster due to multicore and C++).

    The iterator also allows you to call iterator.progress() to see how far you are in processing the file, and returns useful things like a shape ID to reuse mapped geometry, as well as material IDs for each polygon (so that you know for each polygon what material is assigned to it - super useful in setting Blender material slots).

    The include/exclude feature lets you whitelist or blacklist elements to be processed by the iterator. Not necessary when you are using create_shape directly, of course.

    Another difference is that create_shape can be used at the element level, or down to a single representation item, whereas the geometry iterator only processes the element level, and excludes type elements.

    carlopavJesusbill
  • Excellent, thanks for the explanation!! I'll play a bit with that. In any case indeed, just for building a mesh for viewing, it should already prove useful.

    Moultcarlopav
  • @yorik some good reading is @aothms 's original Blender import script which uses it and is quite a short read. By the way, there are some nice features like you can specify it to process curves / wireframes, or disable the boolean operations (if you have native booleans like Blender) ...

    carlopav
  • @magicalcloud_75 more optimisations, so that Myran_fixed.ifc will now import in 15.44 seconds, or a 10% speed increase.

    The UpTown.ifc (242MB), which previously was so slow that you would wait hours and it would not import, has been a great learning experience to optimise the import process. It now imports between 20-25 minutes, with multiprocessing enabled (and with no crashes!). It is worth noting that UpTown.ifc is a particularly inefficient IFC file. I have dealt with much larger files (>500MB) with similar import times of 20-25 minutes prior to all of these new optimisations. It's hard to give a percentage speed increase given that previously it would be so slow that you could wait hours and just give up, but I would now expect all files, not just UpTown, to have a nice speed boost! Again, thanks to your testing!

    It's interesting to compare the speed of a pure viewer, with a full BIM tool. For example, IfcOpenShell by itself parses all the geometry of UpTown.ifc in just over 2 minutes (similar to Solibri 1 min / BIMCollab Zoom 2 min), so I expect a pure IFC viewer (i.e. not a full BIM authoring tool like the BlenderBIM Add-on) would view it in about 3 minutes. The jump from the 3 minutes "best-case" to 20-25 minutes for a full BIM authoring tool comes from processing the data to be available in Blender for geometric editing.

    That said, the benchmark of ArchiCAD importing it at 5.5 minutes is very impressive. Admittedly, his computer has clearly got some beefy hardware (32GB ram, 8 core i9), but still cool. The BlenderBIM Add-on is written in Python, and a large portion of it is still single threaded, so in the future if portions are re-written in C++, I would highly expect that 20-25 minutes to drop down further. Don't expect it soon though - it's not a small task!

    Here's a snapshot of the imported UpTown happily in Blender :)

    carlopavJesusbilltlangmagicalcloud_75Cyril
  • edited June 2020

    I think especially Autodesk Revit is far too slow on development, lazy and uncontrolable on export and import. They could use a kick in the but to focus on better OpenBIM compliance. Surely they want keep up the advertising . Maybe someone should write the dutch guys from Revit Standards to raise some of the issues with IFC. Revit community and its RFO benchmark lacks IFC import. I think Revit development need focus.

    quote..
    "... The Dutch government has set itself the goal of designing and implementing as many construction projects as possible according to the BIM-process. For this purpose, the open standard IFC was included in the 'comply or explain' list of the Standardization Forum in 2011. With the NLRS, we ensure the correct implementation of IFC within the Netherlands. .... "

    Maybe I should write 'em to have some kind of RFO+ or NLRS benchmark that does do import and export test with IFC.
    But hey.. why not make a petition !?? :P Other angles?

    https://www.revitstandards.org/en/about/revit-standards/
    https://revitforum.org/showthread.php/35955-RFO-Benchmark-v3-X

  • I started playing with the iterator in FreeCAD... Multicore processing is highly unstable, but even using one core, first results are baffling. 47Mb schependomlaan imported in about 3 minutes. I'll have a try with the geobenchmark files above...

    carlopav
  • @yorik more recent builds of IfcOpenShell are more stable for multicore. I also make sure to report segfaults as soon as they come, to keep on pushing for increasing stability in multicore :)

    Great to hear it is creating some wonders in FreeCAD! Unfortunately, it meshifies everything, so you still need to go through manual processing if you want to retain native IFC geometry representations...

  • If you switch the BREP flag in the geom settings, it outputs brep data instead of mesh ;) You're right about decomposing shapes (extrusions, etc), but i'm more and more thinking in leaving that for later.. ie importing the file as fast as possible, then allowing to "decompose" objects on the fly, later on

    Cyrilcarlopav
  • @yorik I have the same idea! Throw everything in as a brep, then if you want to edit a particular object's geometry, let them choose it and 1) let then extract the original native geometry, or 2) let them agree to meshify it, especially if Blender/FreeCAD cannot support the particular native geometry, then it stores dirty geometry and only writes the dirty geometry back to the IFC file :)

    Of course, the user also has an option for "convert everything to native" on import if they want.

    baswein
  • edited June 2020

    Today this is a hot topic, but not just CPU, GPU too
    So the majority of movements are based on supporting heterogeneous computing based on CPU, GPU, TPU, ...

  • @yorik said:
    If you switch the BREP flag in the geom settings, it outputs brep data instead of mesh ;) You're right about decomposing shapes (extrusions, etc), but i'm more and more thinking in leaving that for later.. ie importing the file as fast as possible, then allowing to "decompose" objects on the fly, later on

    I also agree! perhaps we can add the shape as a wall subobject and try in different representation contexts if there is some footprint or wall axis and fill anyway the wall properties (hope it does make sense).

  • @yorik - I read in your latest blog that you saw that multiprocessing was not too stable ... are you running the latest IfcOpenBot builds? I've found that the latest builds are significantly more stable and I actually enable it for almost all of my imports now.

    If you've still got segfaults, can you perhaps share the file and/or report upstream to IfcOpenShell so that we can stabilise it as soon as possible? This is an option I'm very keen to start having enabled as default for users :) It's a big benefit!

    bitacovircarlopav
Sign In or Register to comment.