GSoC Project: Web-based UI integration with BlenderBIM

ZiadZiad
in General

Introduction

Hi everyone! My name is Ziad Ibrahim, and I'm a fourth-year CS student at Cairo University, Egypt. I'm happy to participate in GSoC as a contributor this year and work on the Web-based UI integration with Blender project.

Project goal

The goal of my project is to have a web interface running alongside Blender. This web interface ideally shows tabular data for objects that Blender knows about, has real-time updates, and executes operators on them. 

I'll use this thread to update you on my progress throughout GSoC. I am looking forward to engaging with the community.
Please feel free to ask any questions you might have, or if you have any additional ideas.

brunopostletheoryshawsteverugiNigelShegsJohnKoAracarlopavatomkarincaCSNand 8 others.

Comments

  • ZiadZiad

    Hello!

    Sorry for the absence, as I had my exams from May 25th to today (June 6th), so not much work was done during the previous period.

    Plan

    Blender UI

    So currently, the plan is to have a button somewhere on the UI, that once clicked would open the web browser and establish a WebSocket connection to allow for real-time data display.

    For now, the button will be in the Quality and Collaboration tab beside the header of the Spreadsheet import/export panel.

    (that's the best-looking "globe" icon I could find)

    I would love to hear your feedback on where the button should be.

    Implementation

    Once the operator is clicked from Blender, it will start a WebSocket server on another thread so it doesn't block the main Blender UI thread.

    NOTE: This can also be easily refactored to make the WebSocket server standalone and blender acts as a client connects to it instead of having serverside code.

    once the WebSocket server is opened, a web browser will automatically be opened using the webbrowser Python module. This allows the user not to worry about opening the web browser themselves.

    The browser instance will either serve an HTML file or open it on the local host URL and port specified.

    once the web UI is opened, it will connect to the WebSocket server using JavaScript. So we will be able to listen to any messages from the WebSocket server using on message listeners, and we will be able to send messages to the WebSocket server on user interaction with the web UI, such as when a user selects a row in the displayed tabular data, etc.

    And that is the general plan for the web UI. I would like to know if anyone has any feedback, suggestions, or questions about the process.

    Shegstheoryshawbdamay
  • JohnJohn

    Hi Ziad, I hope your exams went well. The technical scope and solution look good. For the button, I'd prefer a button with a descriptive text label, as remembering what another icon is for is a hassle for my old brain (as a very occasional user, Inkscape is a real challenge as it uses an icon based UI). I appreciate I'm probably in a tiny minority about this. Looking forward to trying your system.

    CSNZiad
  • CSNCSN
    edited June 2024

    Agreed @John - I think having it as one of those in the list there would would well for a start, ie:

    > BCF Project
> Spreadsheet Import/Export
> City JSON
> WebSocket (or similar)

    I'm excited for the potential of this work though - I can see the ability to have a browser window connection to Blender to be quite powerful - especially where we're limited by the customizability of the Blender UI.

    Ziad
  • ZiadZiad

    @John That's a valid concern, I'll keep that in mind.
    thanks for your feedback

  • MoultMoult

    Very cool! Can you use the "URL" icon? That would be consistent with all the other places we refer to a webbrowser clickable link in the UI.

  • ZiadZiad

    @Moult said:
    Very cool! Can you use the "URL" icon? That would be consistent with all the other places we refer to a webbrowser clickable link in the UI.

    Oh yes, that looks much better than the one I have!

  • vanuanvanuan
    edited June 2024

    Hi Ziad, great to see your progress and plans for the Web-based UI integration with Blender project! I've done a similar POC for FreeCAD and understand the challenge pretty well.

    I think you should focus primarily on developing the server solution, WebSocket API, and web client rather than modifying the Blender UI at this stage.

    For the Blender integration, I recommend using this addon as a starting point: https://github.com/KoltesDigital/websocket-server-for-blender. It automatically starts a WebSocket server and uses bpy.context to interface with Blender. The addon may need some updates as it's 9 years old, or just use it as inspiration.

    To create an inspiring and useful demo quickly, I suggest implementing the ability to:

    1. Upload an IFC file from the web client (browser)
    2. Display the uploaded IFC model in BlenderBIM
    3. Make changes in Blender
    4. Send the modified IFC back to the web client
    5. Display the updated model in the browser using a Web IFC library

    This flow would leverage the WebSocket technology nicely. Here are the key steps:

    1. Add an "Open File" button in the web UI that sends the IFC file contents via a WebSocket message
    2. Listen for this message in your Blender addon/server
    3. When received, save the IFC locally and open it in BlenderBIM (maybe using the bpy.context API)
    4. Listen for changes made by the user in Blender (or trigger on a timer), export the updated IFC and send the file contents back to the web client via WebSocket
    5. Use a Web IFC library like this one to display the received model: https://thatopen.github.io/engine_web-ifc/demo/

    For bonus points, you could build a simple backend server with authentication. It would maintain a single session connecting your web client and local Blender instance. This would allow you to stream your Blender edits in real-time and collaborate with others.

    Taking this further, there is exciting potential with features like:

    • Robust user authentication and authorization
    • Ability to create and manage multiple sessions
    • Automatic backup of IFC files and version history
    • Real-time collaboration allowing multiple users to view and edit models simultaneously using two BlenderBIM instances (needs a conflict resolution mechanism)

    Implementing these capabilities would elevate the project from a tech demo to a truly powerful tool enabling seamless import/export, live preview, backup, and multi-user editing of BIM models via a web interface and a desktop BlenderBIM.

    Let me know if you need any help. Excited to see what you will build.

    theoryshawZiadJohnbdamayJanF
  • vanuanvanuan
    edited June 2024

    I've just realised that the code should be a part of the BlenderBIM addon. Frankly, I don't think it is feasible that the pull request will be accepted. Much easier to make it a separate addon and merge it in once it matures. The challenge is the server lifecycle management which isn't properly described. For a POC, import/export of IFC files would be enough rather than replicating the BlenderBIM Gantt chart UI, IMHO.

    theoryshawZiad
  • MoultMoult

    A huge thanks for your feedback and interest!

    For loading / saving to the cloud, I would say that this is something more appropriate for OpenCDE - it has a workflow intended for loading and saving to collaboration servers.

    In contrast, the purpose here is to provide a supplementary interface to the Blender UI where Blender's UI has limitation. For example, unlike FreeCAD, Blender has its own strict UI toolkit rules. It cannot (easily) have grids of images, cannot show gantt charts, cannot show spreadsheet / tabular data easily, cannot show embedded drawing views, cannot show even things like textareas. So this would work alongside the Blender session on specific domains.

    And yes, @Ziad is using websockets - he can explain in more detail how the implementation works. The focus is on the server connection.

    I would say user auth is also the domain of OpenCDE. Similarly, you might be interested in IFC Git - which has Git integration which allows versioning, collaboration, and conflict resolution. This works on any native IFC platform, so you can even use it with FreeCAD's new Native IFC workbench.

    I'm hoping to have an unstable release in about a week with the code merged in so people can see an initial test. After that, the first goal is integration with the IfcCSV tool to interactively work with spreadsheets alongside the model.

    theoryshawZiadJohn
  • ZiadZiad

    @vanuan said:
    Hi Ziad, great to see your progress and plans for the Web-based UI integration with Blender project! I've done a similar POC for FreeCAD and understand the challenge pretty well.

    Hello, @vanuan! First of all, thanks very much for the great message.

    I think you should focus primarily on developing the server solution, WebSocket API, and web client rather than modifying the Blender UI at this stage.

    For the Blender integration, I recommend using this addon as a starting point: https://github.com/KoltesDigital/websocket-server-for-blender. It automatically starts a WebSocket server and uses bpy.context to interface with Blender. The addon may need some updates as it's 9 years old, or just use it as inspiration.

    Yes, that addon was helpful as a starting point. it helped me learn how to start a WebSocket server in Blender using threads.

    To create an inspiring and useful demo quickly, I suggest implementing the ability to:

    1. Upload an IFC file from the web client (browser)
    2. Display the uploaded IFC model in BlenderBIM
    3. Make changes in Blender
    4. Send the modified IFC back to the web client
    5. Display the updated model in the browser using a Web IFC library

    This flow would leverage the WebSocket technology nicely. Here are the key steps:

    1. Add an "Open File" button in the web UI that sends the IFC file contents via a WebSocket message
    2. Listen for this message in your Blender addon/server
    3. When received, save the IFC locally and open it in BlenderBIM (maybe using the bpy.context API)
    4. Listen for changes made by the user in Blender (or trigger on a timer), export the updated IFC and send the file contents back to the web client via WebSocket
    5. Use a Web IFC library like this one to display the received model: https://thatopen.github.io/engine_web-ifc/demo/

    Great suggestion. However, the purpose is to display the IFC file objects in a tabular format on the web UI (think of Excel after exporting a CSV in BBIM), and later on, be able to run operations on them.

    The steps are quite similar to those you suggested:

    1. let us assume the IFC file is already opened in Blender.
    2. The user uses the blender interface to start a websocket server on a specified port as a subprocess
    3. blender connects to it on a thread (so that it doesn't block the UI). (more on points 2 and 3 below)
    4. The user opens the web UI in the browser and connects to the WebSocket server.
    5. Once the user makes changes to the IFC file. They are detected by using the current BBIM architecture.
    6. The IFC objects are then encoded (JSON, CSV, etc...) and sent to the web client using the websocket server.
    7. The web client then renders the data received as a message in a tabular format.

    Later on, we can support operations from the web UI by doing something similar:

    1. Once an operation is executed on the web UI.
    2. using the same websocket server to send the operation to Blender.
    3. the operation data is saved in a Queue to ensure thread safety, as the thread listening to the websocket server is not the main thread.
    4. periodically, the operations are taken from the queue and applied in Blender.

    A big challenge in the described approach is the server management (starting, closing, connecting, listening, and sending data to the server) using Blender.

    The way it is currently done is by:

    1. Starting the websocket server using a new process from blender operators, Then keep a reference to the process which allows us to terminate the process using another operator hence closing the server.
    2. Using a thread running an asyncio's event loop (a Python module that allows us to run asynchronous tasks). Then, keep adding functions to the loop using blender operators, such as connecting, sending, and disconnecting tasks.

    I hope my explanation made sense!

    I've just realised that the code should be a part of the BlenderBIM addon. Frankly, I don't think it is feasible that the pull request will be accepted.

    I think it isn't tough to integrate with BlenderBIM due to the way BlenderBIM is organized (module/core/tool) which makes it easy to extend and use the current functionality.

    The challenge is the server lifecycle management which isn't properly described.

    Not really, as the server will be local, so there won't be server lifecycle management concerns beyond basic setup.

    Let me know if you need any help. Excited to see what you will build.

    Sure, will do.. Thanks for the feedback!

    John
  • vanuanvanuan

    Thanks for pointing out the limitations of Blender's UI toolkit. I hadn't considered the constraints that made opening drawings in the browser a necessity rather than just a design choice.

    While a web-based UI can certainly complement Blender for specific BIM tasks, I wonder if a separate browser window might feel a bit disconnected from desktop app. Have you considered an Electron-based approach to create a more native-feeling experience? Though packaging another platform-dependent app inside a blender extension could be technically much more challenging than just a python server.

    Still, focusing on a specific use case like layouting drawings (essentially displaying grids of images) rather than just a technical capability could be more impressive. Interactive spreadsheet editing with the IfcCSV integration and reimplementing gantt charts seem interesting as well, but a bit more complex.

    Looking forward to testing the initial version. Keep up the great work!

  • ZiadZiad

    Hello everyone!

    It's been a while since our last update. The web UI is now ready for testing, and we would love your feedback!

    In this comment, I'll explain how to use the web UI. In the next comment, I'll go into the technical details with diagrams for those interested.

    Starting and Connecting to the Web UI

    The web UI can be found under the Quality and Coordination tab.

    To start the web UI, enter a port number in the WebSocket Server Port field, then click the Connect/Start Websocket Server button. You can also keep the default value 0, and Blender will automatically assign a free port number.

    Multiple BlenderBIM instances can connect to the same web UI by specifying the same port number in all of them.

    Once started, you can:

    • Kill the web UI WebSocket server (only available to the BlenderBIM instance that started the web UI).
    • Disconnect this BlenderBIM instance from the web UI.

    Web UI Components

    After starting the web UI, a browser tab will open.

    The web UI includes:

    • Navigation Bar: Switch between different pages.
    • Connected Blenders Counter: Shows the number of connected BlenderBIM instances with details. (Currently partially working, full functionality expected next week.)
    • Main Container: Displays data for the current page.

    There are three pages in the web UI:

    • IFC Data Page: Displays tabular data for exported IFC data.
    • Gantt Chart Page: Shows Gantt charts for work schedules.
    • Drawings Page: Displays drawings/sheets saved in the IFC file.

    Note: Page names might be updated to reflect their functionalities better.

    IFC Data Page

    The IFC data page shows tabular data for the exported IFC data. This IFC data is exported from the Spreadsheet Import/Export Panel in BlenderBIM

    To export data to the web UI:
    1. Go to the Spreadsheet Import/Export Panel.
    2. Select the web UI output format.
    3. Specify the search group and CSV attribute.
    4. Click on Export IFC to WEB.

    Note: If the BlenderBIM instance is not connected to a web UI, pressing Export IFC to WEB will start the web UI and export the data.

    If multiple instances of the BlenderBIM export data then there will be multiple tables in the web UI.

    The table in the web UI allows you to:

    • Sort data by clicking on a header.
    • Show/hide specific headers by clicking the dots next to the header names.
    • Highlight objects in Blender by clicking on a row.

    Note: If the global ID is not included while exporting data, highlighting will not be available.

    If data changes in BlenderBIM, re-export it to update the web UI.

    Gantt Chart Page

    The Gantt chart page shows the work schedules Gantt charts. Once you have set up a work schedule gantt chart and connected to a web UI then press Generate Gantt to display it in the web UI.

    To display a Gantt chart:
    1. Set up a work schedule in BlenderBIM.
    2. Connect to the web UI.
    3. Click Generate Gantt.

    Once the Gantt chart is generated, you will see the Gantt chart in the web UI.

    You can edit task names and start/end dates directly in the web UI. Updates will be reflected in both the web UI and BlenderBIM.

    Note: If tasks are updated in BlenderBIM, re-generate the Gantt chart to refresh the Gantt chart on the web UI.

    If multiple instances of the BB generate Gantt charts then there will be multiple Gantt charts in the web UI.

    Drawings Page

    The drawings page shows the drawings and sheets that you have created. Once the page is opened, it will show a dropdown menu of the IFC files, from which you can select a specific IFC file to view the drawings and sheets.

    Once a file is selected, the side panels will show the drawings and sheets from which you can select one to view it in the web UI.

    Note: To refresh the drawings/sheets shown, refresh the page. This functionality is currently being improved.

    That covers the current functionalities of the web UI. I'd like for everyone to help in testing it and give any feedback!

    In the next comment, I'll explain the technical aspects with diagrams.

    GerardTbrunopostleJohntheoryshawMoultAceKoAraNigelatomkarincatlang
  • ZiadZiad

    For everyone interested in how the web UI works, here are several diagrams that should explain the general concepts. If anyone has any questions or feedback, please do not hesitate to let me know!

    First Diagram

    This diagram shows how the web UI WebSocket server starts/connects and the relationship between a BlenderBIM instance, a WebSocket server, and the web UI itself.

    Once a user clicks on the Connect/Start Websocket Server button, the operator connect_websocket_server is executed. This operator passes the specified port and the Web tool (tool/web.py) to the Web core (core/web.py) connect_websocket_server function.

    The Web core does the following:

    • Firstly, it checks if the port is available. If it is, then there is no server running on that port, so it calls the start_websocket_server function from the Web tool. If the port is not available, then there is a server running on that port, and it skips starting the server.

    The start_websocket_server function starts the server (sioserver.py) as a subprocess on the specified port.

    • Secondly, it calls the connect_websocket_server function from the Web tool to connect to the web UI WebSocket server.

    There can be multiple BlenderBIM instances connected to the same web UI WebSocket server. However, only the instance that started the server can kill it.

    There can also be multiple servers, provided they are started from BlenderBIM instances that are not connected to any server and the port is not used by any other server.

    Second Diagram

    This diagram shows the data flow between a BlenderBIM instance and the web UI.

    Both the IFC Data page and the Gantt Chart page have similar data flows. However, the Drawings page has a different data flow.

    IFC Data and Gantt Chart Pages Data Flow

    Once the user clicks Export IFC to WEB or Generate Gantt Chart in BlenderBIM, the data is fetched from their respective places (IFCCSV for the CSV data, and Sequence Tool for the Gantt data).

    The function send_webui_data from the Web Tool is called with the following arguments:

    • data: The data to be sent to the web UI.
    • data key: The key under which the data will be placed when sent to the WebSocket server.
    • event name: The name of the event emitted to the WebSocket server.

    The send_webui_data function loads the WebData if it is not already loaded. The WebData contains the IFC file name and a "is dirty" flag indicating whether there have been changes in BlenderBIM that are not reflected in the web UI.

    Then, the data is emitted to the server with the specified event name. The server forwards the data to the web UI along with the Socket ID for that specific BlenderBIM instance.

    The Socket ID is a unique ID given to each client connected to the WebSocket server. This ID is used for updating the BlenderBIM instance data on the web UI.

    Finally, once the data arrives at the web UI, it is rendered and displayed.

    If any operations are performed on the web UI, such as:

    • Highlighting a row in the IFC Data page tables.
    • Editing task details on the Gantt Chart page.

    The web UI emits a web_operator event with the details of the operator, such as Blender socket ID, page, operator type, etc. This data is received by the server and forwarded to the BlenderBIM web tool, specifically the sio_listen_web_operator function, to be passed to each page's operator handlers, which will map web operations into IFC/Blender operators.

    Drawings Page Data Flow

    The Drawings page has a different flow. It starts when the Drawings page is opened, emitting a web_operator event with the operator type being getDrawings.

    This operator is handled by the handle_drawings_operator function in the Web Tool. It retrieves the drawings data from the Drawing Tool (tool/drawing.py) and sends it to the server with the event drawings_data, which sends it back to the web UI to be displayed.

    GerardTbrunopostleJohntheoryshawMoultAceKoAraemiliotassoduarteframosShegsand 7 others.
  • theoryshawtheoryshaw
    edited August 2024

    Nicely done! This is awesome, of course! :)
    ...
    This is probably in the roadmap anyway, but for sheets, it would be awesome (and it seems a low hanging fruit) to use the webUI to move drawings around on the canvas.
    Would replace having to rely on Inkscape to compose sheets.

    AceBedsontlangShegsZiad
  • theoryshawtheoryshaw
    edited August 2024

    Also would be awesome to integrate a real-time whiteboarding tool like tldraw or FirstDraft to be able to markup sheets that could be shared with collaborators.

    fyi, tldraw used to be open source, by they changed their license to a more proprietary license back in December. Could be still a worthy fork, however, at that point.

    AceShegsZiad
Sign In or Register to comment.