Journal Article

From text to DSM: evaluating the impact of writing style and entity naming on LLM-based retrieval of asymmetrical indirect design dependencies

Koh, Edwin C. Y.

2026

Overview

The design structure matrix (DSM) is an established method for modelling design dependencies but manually putting one together can be resource intensive. The Auto-DSM workflow integrates a large language model (LLM) with retrieval-augmented generation (RAG) to extract system dependencies from input data, which are then used to automatically generate a corresponding DSM. This paper reports on an evaluation study that uses the Auto-DSM workflow as a basis to evaluate the retrieval of asymmetrical direct and indirect system dependencies from text data. Five LLMs, namely GPT-4o, GPT-4, Llama 3, DeepSeek-R1, and TinyLlama were used in this work. Auto-DSM with GPT-4 produced a complete DSM with an accuracy of 0.981 ( SD  = 0.025, N  = 600) when plain dependency descriptions were used and reached a full accuracy of 1.000 ( SD  = 0.000, N  = 5) when the same dependencies were presented in the form of patent claims. It was revealed that the way system entities are named in input data can affect accuracy and the reporting of path distance between entities is influenced by the writing style and format of the data. The findings of this work can be used to support the development of automated DSM generation, enabling more advanced DSM techniques to be built on.

• Publisher: Springer London,Springer Nature B.V
• Subject: Accuracy / Asymmetry / Automation / CAE) and Design / Computer-Aided Engineering (CAD / Diesel engines / Engineering / Engineering Design / Large language models / Original Article / Product development / Retrieval augmented generation / Workflow