Searching 3D CAD assembly models using 2D images and prior knowledge

Background

In 2021, I applied to Vertex Systems for a summer trainee position and eventually secured it. I had previous experience in mechanical design from my previous job. It was the primary reason I was put to work as a developer on the G4 mechanical design software. Previous work experience confirmed that this job was not for me, so I wanted to change my career plan upon starting my master’s degree. During my master’s degree, I delved into robotics and programming because I had always been fascinated by them, even though I had no direct experience with them. As a trainee, I was given tasks that the end user would use in the next version of the software which made the work feel really mattering.

The thesis voyage began during the fall of 2023. Because the work had to be related on some level to my major, robotics, it was not a straightforward task to come up with a topic. My major interest in robotics has been machine learning thus the topic was being searched from that point of view. Originally, the topic of the study was related to the retrieval of 3D models. However, based on previous studies solutions that were considered state-of-the-art in the classification and retrieval of part models. For this reason, the topic eventually formed into 3D CAD assembly retrieval because it is an area of research that has not yet been fully explored.

3D CAD assembly model retrieval

Assembly models are the core of mechanical design composed of sub-assemblies and parts that are used to form a complex structure. The design process is an intensive and time-consuming stage in the product lifecycle. Traditional methods for finding similar assembly models rely primarily on the model properties, which are insufficient in terms of efficiency and performance. My thesis addresses the problem of how to enable users to reuse and find existing assembly designs more efficiently by implementing automatic methods. The high-level process pipeline of finding similar CAD assembly models is illustrated in the figure below.

The initial problem was how to represent the 3D CAD models so that they could effectively be searched. Based on a comprehensive literature review, view-based methods appeared to be both more efficient and more accurate in contrast to other methods used. Also, as they do not rely on the explicit CAD model’s data, they were considered appealing. View-based methods are used to extract features for example from models’ visual representations such as 2D images. For this thesis, the data used was collected from Vertex G4 3D CAD models. The models were exported into Graphics Language Transmission Format which was used to render them into images. Additional post-processing was applied to capture the models’ contour lines as shown in the figure above. Images were captured from different viewpoints around the model to capture their overall appearance.

Multiple convolutional neural network-based autoencoders (CNN-AE) were trained and carefully analyzed. The primary goal of an autoencoder is to learn a compact representation of input data, referred to as the feature vector. Different feature vector sizes were studied and based on the results feature vectors of size 512 yielded the best results combining both compact form and accuracy. A single main-level assembly model is represented using six feature vectors that describe the model’s overall characteristics. A novel method combining the assembly and its component’s feature vectors was found. Child models’ feature vectors are weighted by their relative volume and given less importance in the resulting vectors in contrast to the main assembly. This reduces the smaller component significance in the set of combined feature vectors.

Different metrics were studied to find similar assemblies based on the input model. Clustering was leveraged to reduce the number of comparisons between the input and models in the mockup database created for this proof-of-concept work. The models were grouped into different clusters based on their individual feature vectors. An average of feature vectors was used as an identifying key for each cluster. Eventually, models that are found to be the most similar in the clusters are given as a result for the user. The figure below illustrates the steps required both in the offline and online stages of assembly model search.

Summary

Gathering the data at the beginning of the master’s thesis project proved to be a cumbersome process since there was none available that could be used directly. All the data was collected and crafted for the purpose of this thesis. The second concern was how to effectively form a shape descriptor that captures the complexity of an assembly model while retaining a somewhat compact form to not induce too much overhead. Lastly, the validation of the proposed method was difficult because the search results are somewhat abstract in relation to what can be said to be the correct result in relation to the given input. During the implementation of proof-of-concept work some assumptions were made that would require fixing before it could be used in a real-world application.

The project had its ups and downs but in general, the process taught me a lot of new things. The most challenging part was writing the thesis itself – where staring at a blank page seemed like an impossible task to get ideas in. By the time I got into a course in writing, doing it was not nearly as challenging as it had been at the beginning. Gladly, Tampere University allows students to utilize GPT language models, making the writing process feel less like a solitary endeavor. This really helped to improve the quality of the text.