A Real-time Ideation App for Client and Spatial Designer during Online Meeting
Utilizing natural language processing (NLP) to and Graph Neural Network(GNN) to To improve the efficiency of 3D scene design iteration and discussion between designers and non-professional customers by Integration Large Language Model and graph theory
From Whom?
Designer:
- Use visible information as a guide: Due to the ambiguity of the connection between language description and spatial design, they often need to guide users to express their needs through visual information or do the guesswork.
- Duplicate work for multiple solution: The workflow often requires listing a large number of solutions to guess the user's preferences and letting the customer choose from them, which is a tedious and time-consuming process.
Client:
- Abstractness of language description: As non-professionals, they often need to use language to describe the ideal space design, but this process is often full of ambiguity and abstraction.
My Contribution
End-to-end Design Spring
Throughout the agile sprint, I’ve brought up critical design decisions through user research, led the iteration on low-fi and hi-fi prototypes, and validated through usability studies
Software Development and System Design
Determine the scope and requirement engineering of software development. Responsible for the design and implementation of the front-end and back-end software frameworks and APIs, and help build AWS DevOps pipelines to ensure the scalability of the code.
Machine Learning Model Training
Collaborate with team members to train, optimize and evaluate donditional 3D Furnished House Layout Generation with Heterogeneous Graph Variational Autoencoders
Key Solution.
The Nestwork, a 3D web app that Utilizing natural language processing (NLP) to and Graph Neural Network(GNN) to To improve the efficiency of 3D scene design iteration and discussion between designers and non-professional customers by Integration Large Language Model and graph theory
One.
Natural language to spatial relationship Bubble diagrams Generation using LLM
Client’s needs are often descriptive, and it is difficult for them to convert such needs into relevance to the room layout. The basic knowledge of the big prediction model can initially achieve this and convert it into a bubble diagram, which is conducive to the initial consensus between customers and designers.
click to check diagram
Two.
Bubble diagrams to 3D Scene Generation
After the client and the designer reach an agreement, they can use bubble diagrams to generate 3D unit scenes. A single generation only requires a few saplings, which saves a lot of designer time and communication back and forth. And if you are not satisfied with the design, you can regenerate and adjust it. In contrast, it often takes several days for the typical feedback to the designer's initial solution.
click to check diagram
Three.
User Profile to spatial relationship Bubble diagrams
Client A thinks he has more specific requirements for the number and details of different rooms. He defines his user profile by setting their key requirements, thus generating bubble diagrams more accurately.
click to check diagram
Four.
Historical Generation Tracing
Customer D thinks that the result of this generation is not as satisfactory as the last one, and hopes to go back to the last generation result to continue the discussion with the designer.
Five.
Furniture Modifications(WIP)
After customer D confirmed the satisfactory space layout plan, he had some ideas about the furniture positions automatically generated by the algorithm, so he manually adjusted the position and type of furniture.
Research proccess
The research process focuses on understanding user needs for transforming text into 3D room models. It aims to enhance user control and satisfaction by making the design process intuitive. Ultimately, the goal is to create a seamless and engaging user experience.
Iteration Highlight.
One.
Optimize Interaction Layers
A streamlined interface that unifies bubble diagram generation, editing, and user profile settings into a single, intuitive control panel—minimizing navigation and simplifying the overall design experience.
Two.
Prioritize information and reduce operation
Reordered visual hierarchies to emphasize the 3D model to enhanceuser recall. By spotlighting essential information, it reduces cognitive load, and delivers a more intuitive, memory-oriented interface that aligns with users’ immediate demands.
AI Approach Details
The Nestwork, a 3D web app that Utilizing natural language processing (NLP) to and Graph Neural Network(GNN) to To improve the efficiency of 3D scene design iteration and discussion between designers and non-professional customers by Integration Large Language Model and graph theory
One.
Overall Machine Learning Pipeline
Afterconducting a round of user interviews, our pipeline allows user input eitherfree-form text description or structured user profile which can be convertedinto adjacency graph, just like a bubble diagram. Users can then interactivelyadjust this graph to refine the spatial relationships as well as specifying thegeneration boundary . The finalized graph together with boundary constraintwere fed into our trained Graph based generative model to predict finally 3Dmodel
Two.
Large Language Model
The LLMs pipe-line accept two types of user input: either a paragraph describing the desired layout or a structured user profile, a persona representing their lifestyle and family info. We then use chain-of-thought and in-context learning prompting techniques to parse these inputs into a graph format that represents the rooms and their connections, eventually visualized in the front-end
Three.
Graph Neural Network
Our project leverages graph representations that accurately model the compositional nature of architectural 3D layouts. This approach surpasses traditional image-centric inference methods by providing a structured and semantically rich depiction of spatial relationships and room connectivity.
Overall Pipeline
Graph Neural Network Pipeline
Existing Dataset from 3D Front and Self-collected Revit Model dataset
Our dataset includes two primary sources. First, 3D FRONT dataset available online, which consists of 2,000 professionally designed one-floor apartment units. Secondly, we supplement this with custom-collected data, specifically multi-level house models from across the U.S., sourced from Revit (A 3D Modeling Software). Each of these models is paired with its corresponding graph, which represents the spatial and functional relationships within the house.
Custom Script for Extracting Adjacency Information
Graph Representation: Extracted Semantic Graph
In our proposed graph representation, each node represents a room, with edges indicating adjacency. On the left is the full graph used during training, capturing room labels, bounding boxes, orientation, and furniture 3d information. The simplified semantic graph on the right is used during inference, focusing on room labels without any edge embedding.
Graph-based Generative Model (Conditional Graph VAE)
The LLM parsed graph is then fed into our graph-based generative model to predict3d layouts. In our current pipeline, we adapt a conditional graph VAE. We trainthis model on the largest 3D furnished layout dataset, 3D-Front. Duringtraining, the model learns to encode a 3D layout into latent variables, whichare then combined with the semantic graph constraints to predict new 3Dlayouts. During inference, we only use the semantic graph and sampled latentvariables to generate layout designs.
Illustration of Decoding 3D Shapes from Latent Space
Latent Space
One way to visualize the machine's "brain" is by plotting the latent space of the entire dataset to see if it learned any patterns. By combining the latent vectors from each graph node, we create an overall representation for each design. In the plot, these points are color-coded by bedroom count, showing clusters where layouts with similar bedroom numbers group together. This suggests the machine can categorize designs based on similar numerical features. From left to right, you can observe a transition from simpler 1-bedroom units to more complex layouts.
Graph Neural Network Latent Space
Sample Plots of latent space
By adjusting each node’s latent representation, we can create diverse layouts that follow the same adjacency graph. The animation shows a continuous shift in the layout. For example, while the bedrooms, bathroom, and living room maintain fixed relationships, the kitchen, represented by the dark blue box, only knows it should be close to the living room, causing it to float within the unit.
Software Development
Thoroughly document all aspects of the software development process, from designing the application's architecture and defining APIs to implementing reusable components and developing the DevOps pipeline. This includes detailing key decisions, best practices, and the rationale behind design choices to ensure a well-structured, maintainable, and scalable solution that aligns with development and operational needs.
One.
Application Architecture Design
After confirming the basic framework of the user interface, we conducted a preliminary software architecture design. We defined the front end as the entry point for displaying and collecting user input data. The back end mainly processes the user input data and calls the machine learning model to generate results and return them to the front end.
Two.
Front-End Object Oriented Programming and Design
Considering that there is no user login and registration system designed at present, we currently do not plan to include the database in the design. Therefore, all data will be managed and called by different managers, and different levels of object-oriented programming will be used to meet the business logic. Considering the development speed and small project scale, some data will be provided globally by the Context Provider.
Three.
Restful API Design
Considering that the backend will run our own Graph Neural Network model trained with PyTorch, we plan to deploy the backend as a Flask App and integrate the Open AI function of adding system prompts to meet business needs. Since the backend mainly processes the data input by the front-end user and then calls the AI model to generate and return the results, the Restful API uses POST
Four.
Reusable UI Building Strategy
Nestwork's basic UI is extended from the native NextUI library. Different basic UIs are combined to form UI Groups to meet design requirements, which is scalable and reusable.
Five.
DevOps Implementation
Utilize GitHub Action as to build DevOps pipline, containerize the front-end and back-end and push them to Docker Hub, AWS BeanStalk pulls images from Docker Hub for automatic deployment, and sets up Auto Scaling
Six.
Graph-D3.js, 3D Scene-ReactFiber Implementation
For presenting the bubble diagram (Graph data) with features like adding and deleting nodes, we selected D3.js, a widely-used SVG library for data visualization. D3.js natively supports nodes, links, and force simulations, making it ideal for dynamic graph interactions. For 3D model visualization, we chose ReactFiber as it offers a robust API, providing flexibility and scalability for future development needs.
Achievement
We are actively participating in Computer-aided Design and computing conferences—both presenting and publishing our research—to solicit valuable feedback and gather constructive suggestions from the community. Our innovation was accepted by the ACADIA (Association for Computer Aided Design in Architecture) Conference Ideation Lab, with an acceptance rate of less than 30%.
Two.
OpenAI’s Researcher Access Program
We wrote a research paper on 3D generation pipeline using Open AI API and applied for OpenAI’s Researcher Access Program. We were approved and received $2,000 in API usage credits.
Three.
SIGRRAPH 2025 Paper Under Review
The final paper, Nestwork: Conditional 3D Furnished House Layout Generation with Heterogeneous Graph Variational Autoencoders, is currently under review at the SIGRAPH 2025 Conference - the premier conference on computer graphics and interactive techniques