CSV Export from multiple models | future patch?

steverugi · April 9

introduction

there are a few models from which I need to extract properties and quantities, like when I link multiple ones in a federated arrangement
if it was for one model only I would use Spreadsheet Import/Export
combining multiple models requires some script
thankfully ChatGPT helped me out in putting this together, so if any programming bad style it's its fault..

modules/functions used:

import modules and

Load IFC models into a list of dictionaries

import ifcopenshell
from ifcopenshell.util.selector import filter_elements
from ifccsv import IfcCsv
import os

def load_model_list(model_paths: list[str]) -> list[dict]:
    return [
        {
            "model": ifcopenshell.open(path),
            "name": os.path.splitext(os.path.basename(path))[0],
            "path": path
        }
        for path in model_paths
    ]

Export filtered elements from a model to a CSV in the same folder

def export_model_to_csv(model, model_name: str, model_path: str, query: str, attributes: list[str]) -> None:
    elements = filter_elements(model, query)
    if not elements:
        print(f"⚠️ No matching elements in model '{model_name}'")
        return

    model_dir = os.path.dirname(model_path)
    output_file = os.path.join(model_dir, f"{model_name}_extract.csv")

    ifc_csv = IfcCsv()
    ifc_csv.export(
        model,
        elements,
        attributes,
        output=output_file,
        format="csv",
        delimiter=",",
        null="-"
    )

    print(f"✅ Exported model '{model_name}' to:\n   {output_file}")

Main function
change model_paths, query, and attributes to your need
in this example query and attributes select all IfcWall elements with type starting with "WAL-"
from selected elements extracts attributes like Name, type.Name, and NetSideArea
before running the script make sure that original models have quantity take-off already done

def main():
    model_paths = [
        r"path to ifc file\filename1.ifc",
        r"path to ifc file\filename2.ifc"
    ]

    query = 'IfcWall, type=/WAL-.*/' 
    attributes = ["Name", "type.Name", "Qto_WallBaseQuantities.NetSideArea"]

    models = load_model_list(model_paths)
    print("\n📂 Loaded Models:")
    for m in models:
        print(f"- {m['name']}: {m['path']}")

    for m in models:
        export_model_to_csv(
            model=m["model"],
            model_name=m["name"],
            model_path=m["path"],
            query=query,
            attributes=attributes
        )

Run the script

if __name__ == "__main__":
    main()

Code attached

I would hugely appreciate your feedback, thanks

falken10vdl · April 14

@steverugi Here is a possible PR Multiple ifc spreadsheet export #6548 to enhance the exporting capabilities of Quality & Coordination -> Spreadsheet Import/Export.
This is in line with Issue allow multiple .ifc file in Spreadsheet Import/Export, or linked ones #6252
It allows to select a number of files (no error checking yet) below the loaded ifc file directory structure. In the case of web output it concatenates the results. In the case of CSV it generates a file in the same directory adding the date-time suffix and the file format extension.

"

Thanks!

steverugi · April 15

@falken10vdl
wow! that's cool, thank you so much
only one thing if you like, the file selection I had in mind was similar to the one used in "Link IFC", at first impression looks more compact and allows individual selection of the files, also showing their path

but I'll be happy with any way the community wants to have it
really a useful feature, thanks again for your help

falken10vdl · April 15

@steverugi You are right. Probably best practice is as you say. Let me change it to follow your advice.
BTW, I was doing that tree structure because I was playing with the idea of some sort of "Project Management Panel/checklist". I have come to know that the IFC file is just a portion of a complete IFC based project. That is why I had the the idea of having a tool that will allow you to interact with all the files involved in the project so then it is simple to pack (zip) and provide to someone without losing information due to absolute paths or blender/bonsai "internal" directories. Also the links in relative paths so it can be loaded somewhere else.
Ex:

Example the "IFC Project Files Management" Provides details below the IFC main file. You can see that it shows the linked files and puts a Triangle icon for the case that is an absolute path (bad idea to transport), a hand if it is a relative path (good idea but the relative shoudl be below the main IFC file so a zip file package can easily be created) or a link if it is ok. So it is a sort of a checklist tool with good practice in order to make a whole package that you can deliver to someone else (probably info about blender/Bonsai/Sun/etc extensions should also appear)
What do you guys think about this "IFC Project Files Management" idea. Is this something you find interesting?
Thanks!

steverugi · April 15

@falken10vdl

BTW, I was doing that tree structure because I was playing with the idea of some sort of "Project Management Panel/checklist". I have come to know that the IFC file is just a portion of a complete IFC based project. That is why I had the the idea of having a tool that will allow you to interact with all the files involved in the project so then it is simple to pack (zip) and provide to someone without losing information due to absolute paths or blender/bonsai "internal" directories. Also the links in relative paths so it can be loaded somewhere else.

Example the "IFC Project Files Management" Provides details below the IFC main file. You can see that it shows the linked files and puts a Triangle icon for the case that is an absolute path (bad idea to transport), a hand if it is a relative path (good idea but the relative shoudl be below the main IFC file so a zip file package can easily be created) or a link if it is ok. So it is a sort of a checklist tool with good practice in order to make a whole package that you can deliver to someone else (probably info about blender/Bonsai/Sun/etc extensions should also appear)
What do you guys think about this "IFC Project Files Management" idea. Is this something you find interesting?

I like the idea of having a "Project Management Panel/ checklist", if possible it should be in a pop-up window, similar to what we use to import a file for instance. I am not sure how the same can be preserved in a full IFC environment though (I guess the same way we save Link paths..)

In general I tend to like consistent processes across the board, that's why my proposal to implement a "Link IFC" - like approach.

I am looking forward to seeing this feature implemented, hopefully more users will add more feedback to your video

Thanks again, cheers

brunopostle · April 15

@falken10vdl said:
What do you guys think about this "IFC Project Files Management" idea. Is this something you find interesting?

Something like this would be useful for managing the non-ifc files in a git repository

falken10vdl · April 15

Here a solution using UILists like in other parts of Bonsai
"
Do you know what the Generate SVG and Preserve Existing are supposed to do?

Probably another setting for "Performing Quantity Take off" would make sense? As another icon to check per file?
Thanks!

Massimo · April 16

@falken10vdl i think tree structure looks very nice but visually it needs more space than the UIList so i prefer the second one, also for consistency. It would be good also to have the possibility to select multiple files at the same time.
Btw, the tree structure is very interesting, i'm sure it is useful for some purpose...maybe for managing linked documents or attached images?

falken10vdl · April 16

@Massimo I agree. Let me add the multiple select file possibility. Since it is using the builtin file explorer I think that you can only select multiple files from the same directory right?
Thanks!

steverugi · April 16

Hi @falken10vdl
I don't want to spoil the game but can we just start with a simple panel like in "Link IFC"? There you have the path and option icons on the right. (Also useful for selection)
Additionally, if not too difficult, it would be handy to have a box to "select linked models" to operate in federated mode
Thanks a lot for your help

Massimo · April 16

@falken10vdl yep, i think so

steverugi · April 16

@falken10vdl said:

Do you know what the Generate SVG and Preserve Existing are supposed to do?

no idea, tried if it did something but couldn't find anything, maybe looking at the code it can be found out?
can you share the link to the code please?

Probably another setting for "Perform Quantity Take off" would make sense? As another icon to check per file?

mixed feelings about this, there are already two in the current UI

Object Metadata > Quantity Sets
Costing and Scheduling > Quantity Take-off

both of them with qto engine selection, wouldn't it be a bit confusing? sometimes you need to use either of them to capture specific quantities, or need manual quantification for particularly difficult ones or full manual entry.
I'd be careful before adding another one, my POV of course

falken10vdl · April 16

@steverugi A simple way to see the code is if you have enabled Edit->Preferences->Interface->Display->Developer Extras then you can right click in the pannel and select "Edit Source"

Then in Scripting you can see the relevant code:

It will actually mark the point in the code:

From there I find quite usefull to search in VSCode over the file directory:

The you click in it and there you have it:

Let me add this "tip" to the PR (https://github.com/IfcOpenShell/IfcOpenShell/pull/6494) I submitted in https://community.osarch.org/discussion/comment/24629#Comment_24629
Thanks!

falken10vdl · April 16

Ok. I have updated the panel to look like this:

There is a button "Add Linked Files" that will fetch them and add them to the list.
Each entry in the list has three options to select:
1. Tickmarck to select the file for exporting,
2. "eye" to open the file in a new blender instance (instead of doing Qty take off in settings, as per @steverugi suggestion, I have added this icon that could be usefull if one wants to check and save the file so it is ready for the csv/web export)
3. "cancel" to remove the entry

One can use both absolute and relative paths.
Please let me know if this is good enough for a PR or something else is missing

Thanks!

steverugi · April 16

@falken10vdl
good enough? it's perfect! :D

"eye" to open the file in a new blender instance (instead of doing Qty take off in settings, as per @steverugi suggestion, I have added this icon that could be useful if one wants to check and save the file so it is ready for the csv/web export)

for me to understand, does it open a new Bonsai session for potential editing in case of some change like QTO or else? would it be not simpler to adopt the "sync" icon Massimo activated no long ago in "Link IFC" ?

thank you so much for your help (and patience)

falken10vdl · April 16

@steverugi cool! Thanks for yor input. Yes it is as you say, it triggers a new process with an independent blender instance and automatically loads the ifc project so one can edit and make whatever adjustments.
@Massimo how does the "sync" icon work? If I take "load from.memory" for example, the export will dig into the items linked in the main file?
Thanks!

Massimo · April 16

@falken10vdl well, nothing special, it unlink and relink an ifc linked model so it can be updated "on the fly", look at here (https://github.com/IfcOpenShell/IfcOpenShell/blob/e8b74245ce8941f310c7c28c700129b79114dccc/src/bonsai/bonsai/bim/module/project/operator.py#L1427)
I think that "a new process" (with eye icon) and "sync" (refresh) have different purposes: the first one could be useful if i want to open the model in a new dedicated window and the second one could be useful if i want to stay in the same session and update the file without open it.
So, maybe, the "eye" icon is useful under the panel Project setup -> Links near the existing ones icons (Reload link, Unload link, etc...) and the "sync" icon is useful in the Spreadsheet import/export panel...no? What do you think?
Btw, the work you have done is great!

theoryshaw · April 16

backstory on 'preserve existing': https://github.com/IfcOpenShell/IfcOpenShell/issues/3609

steverugi · April 16

@Massimo & @falken10vdl

@falken10vdl well, nothing special, it unlink and relink an ifc linked model so it can be updated "on the fly", look at here (https://github.com/IfcOpenShell/IfcOpenShell/blob/e8b74245ce8941f310c7c28c700129b79114dccc/src/bonsai/bonsai/bim/module/project/operator.py#L1427)
I think that "a new process" (with eye icon) and "sync" (refresh) have different purposes: the first one could be useful if i want to open the model in a new dedicated window and the second one could be useful if i want to stay in the same session and update the file without open it.
So, maybe, the "eye" icon is useful under the panel Project setup -> Links near the existing ones icons (Reload link, Unload link, etc...) and the "sync" icon is useful in the Spreadsheet import/export panel...no? What do you think?

I was thinking:

do we really need a dedicated icon (eye) to open a project we need or want to edit in a different session? can't we just do it outside of that session (e.g. start new Blender/Bonsai and open file)
I find extremely useful to query multiple models, especially with the option where they are the current and linked ones, and generate a common schedule, but the "sync" (which I proposed earlier) can easily be the one from the "Link IFC" panel
to test the new feature I would just start with indispensable options, to avoid overpopulated UI that may disorient new users

Btw, the work you have done is great!

Super great!

steverugi · April 16

@theoryshaw

backstory on 'preserve existing': https://github.com/IfcOpenShell/IfcOpenShell/issues/3609

so it retains th[e]is column/row formatting of an existing .ods file, thanks for the explanation

we are now left with "Generate SVG" ;)

theoryshaw · April 16

a little fishing tour. :)
https://community.osarch.org/uploads/editor/y1/s0wzkvgco4cd.mp4

steverugi · April 16

@theoryshaw

a little fishing tour. :)

little help please, where is the 'svg' going to be saved? thanks

theoryshaw · April 16

in the same folder the .ods file is saved.

steverugi · April 16

thank you @theoryshaw
the final arrangement of the columns in the .svg file needs some help

I opened it in Inkscape and the text is there but overlapping,
Adjusting the column width in .ods > save it > run it again with "preserve existing" triggers an error
Better to export it from .ods maybe, unless I missed something
Cheers
PS why only .ods has the .svg option?

theoryshaw · April 16

¯\_(ツ)_/¯

falken10vdl · April 16

Please find attached the relevant documentation for the updated Export. Please take a look. Maybe we can add this to the PR as well.
IMHO I guess there are two types of users: the ones that like simplicity and clear UIs and the ones that like lots of features/options. I think I tend to be more in the second group :) Anyhow as you wish we can keep it or remove it. Maybe there should be a global switch in the addon settings to turn those ones on/off:

Thanks!

steverugi · April 16

hi @falken10vdl
impressive job, so much in so little time, truly amazing

one question: can it work with both activated? I mean from memory together from linked models or external files

the pdf looks very good, something to learn from!
very much appreciated

falken10vdl · April 16

@steverugi It is either one. However I have put in the following note:

So basically you can just add to the list the own file in memory. Just make sure to save before running the export to make sure that it gets all the latest done.
Thanks!

falken10vdl · April 16

Here is the PR Qty export multiple files #6563
Thanks!

falken10vdl · April 16

@Massimo said:
@falken10vdl well, nothing special, it unlink and relink an ifc linked model so it can be updated "on the fly", look at here (https://github.com/IfcOpenShell/IfcOpenShell/blob/e8b74245ce8941f310c7c28c700129b79114dccc/src/bonsai/bonsai/bim/module/project/operator.py#L1427)
I think that "a new process" (with eye icon) and "sync" (refresh) have different purposes: the first one could be useful if i want to open the model in a new dedicated window and the second one could be useful if i want to stay in the same session and update the file without open it.
So, maybe, the "eye" icon is useful under the panel Project setup -> Links near the existing ones icons (Reload link, Unload link, etc...) and the "sync" icon is useful in the Spreadsheet import/export panel...no? What do you think?
Btw, the work you have done is great!

Thanks @Massimo ! As I commented above I like to have lots of icons even if they are redundant :) but I agree that maybe that is not a popular preference... so I would say we put whatever you guys with AEC experience think is best. :)

CSV Export from multiple models | future patch?

introduction

modules/functions used:

import modules and

Comments