Fine-tuned multimodal large language model for autonomous state cognition system of shape-recognition 6-bar tensegrity integrated with flexible sensors Authors: Zebing Mao, Jianhui Wang, Jianhua Zhang, Junji Ohgi, Yanqiu Zheng, Yanhong Peng, Luyang Zhao, Qi Su, Weidi Huang, Bing Xu Published: 2026 ABSTRACT AbstractConducting dynamic exploration in complex and unpredictable environments, particularly in space exploration, reveals the great potential of systems based on tensegrity structures. However, the implementation of such systems faces a series of intelligent challenges, including the reliability of wireless monitoring, the efficiency of human-computer interaction, and the optimization of intelligent analysis and recommendation capabilities. In this study, we introduce a 6-bar tensegrity system equipped with 24 flexible sensors, leveraging fine-tuned multimodal large language model to enable autonomous state cognition system including self-shape recognition, alarm system, as well as fault diagnosis. Supported by long and short-term memory models, the tensegrity is able to reconstruct its own shape via conductive flexible tendons without relying on external sensors. By combining the flask server with the fine-tuned large language model, the tensegrity automatically transmits data to the iPhone for wireless monitoring. Finally, we developed the fine-tuned LLM and employed it to facilitate fault diagnosis and human interaction, enabling users to effectively obtain the requisite information through natural language processing techniques. This autonomous state cognition system relying on tensioning bars shows great potential for future exploration and becomes a powerful tool for multifunctional applications in the real world. Subject terms: Electrical and electronic engineering, Sensors Introduction Tensegrity structures consist of rigid struts and flexible cables that form mechanically stable and mass-efficient networks1–4. Their structural efficiency has enabled applications ranging from architectural design to deployable lightweight systems capable of sustaining compressive and bending loads with minimal mass5–7. In robotics, tensegrity principles have been widely adopted to realize diverse locomotion behaviors, including rolling8, swimming9,10, climbing11, flying12, bending13, and crawling14. The achieved locomotion mode strongly depends on actuator selection and structural topology15,16. For example, rolling configurations typically employ symmetrically arranged structures with motor-driven cable actuation, whereas climbing systems are adapted to pillar-like terrains. Modular tensegrity robots capable of terrain adaptation and self-assembly have also been demonstrated17. Despite these advances in mechanical design and locomotion strategies, most tensegrity robots operate under open-loop control without integrated sensing18,19. While state estimation and contact sensing have been introduced to enhance closed-loop control performance20,21, existing efforts remain largely confined to improving actuation accuracy or gait regulation. For instance, tension-profile sensing with fiber Bragg gratings has enabled closed-loop control in cable-driven continuum robots22, and shape-changing tensegrity wheels have been used to achieve adaptive locomotion over uneven terrain23. However, these approaches primarily address local sensing and control, rather than system-level structural cognition, such as planetary exploration illustrated in Fig. 1—tensegrity robots must operate in uncertain and uneven environments where deformation, entrapment, or structural malfunction may occur. Under such conditions, locomotion stability alone is insufficient. The system must autonomously recognize its global configuration, interpret structural states, diagnose abnormalities, and provide interpretable feedback for human operators. Although numerous soft and flexible sensors have been developed24,25, few studies have established an integrated framework that unifies physics-based structural reconstruction with higher-level cognitive reasoning. Existing solutions often decouple state estimation from semantic interpretation, resulting in fragmented monitoring architectures lacking adaptive reasoning capability. LLM-integrated tensegrity robot for lunar applications. 6-bar tensegrity integrated with flexible sensors enabling self-shape recognition, warning system from astronauts and interaction with the lunar landing base by means of fine-tuned multimodal large language model Therefore, the absence of a unified cross-layer framework that links geometric–mechanical reconstruction of tensegrity configurations with multimodal AI-driven diagnostic cognition. To address this problem, we propose an autonomous state cognition architecture that integrates interpretable geometric–mechanical reconstruction, temporal dynamic modeling, and ontology-guided large language model (LLM) reasoning. A 6-bar tensegrity structure—whose dormant configuration forms an irregular icosahedron composed of eight equilateral and twelve triangular surfaces—is designed and fabricated as the experimental platform. The system incorporates distributed sensing for shape recognition, WiFi-based wireless monitoring, and a Python-based Flask server for real-time data transmission. On top of the physical reconstruction layer, a fine-tuned multimodal LLM is introduced to perform semantic-level reasoning, enabling condition interpretation, human–machine interaction, and fault diagnosis26–31,32. Unlike conventional monitoring approaches that focus solely on state estimation or fault classification, the proposed framework establishes a closed-loop cognitive architecture linking physical configuration reconstruction with semantic diagnostic reasoning. The framework is modular and transferable: while demonstrated on a 6-bar tensegrity structure, the reconstruction module can be replaced by alternative state estimators, and the reasoning layer operates on structured descriptors independent of specific topology. Consequently, the methodological contribution extends beyond the presented tensegrity application and provides a generalizable paradigm for physics-informed multimodal monitoring and autonomous structural cognition in engineered systems. Problem This section outlines the problem description and methodological analysis. It primarily discusses the challenges associated with visualization and sensing in tensegrity structures, which are difficult to address using conventional approaches. In terms of methodology, we provide a general description of the structural characteristics of tensegrity, employing machine learning for self-recognition and fine-tuning large language models (LLMs) for fault diagnosis. Problem description The tensegrity structure in this study consists of six wood-made struts and twenty-four rubber tendons, forming the shape of an icosahedron. The model comprises multiple independent rigid rods and multiple independent flexible lines, where each rigid body has 6 degrees of freedom, denoted as qi=[Tx,Ty,Tz,θx,θy,θz]T. According to Lagrange’s equations, the dynamic model of the entire system (comprising six rods) is given by: where q=[q1T,q2T,…,q6T] represents a total of 36 degrees of freedom, M is the inertia matrix, h means the Coriolis force and gravitational torque, and is a 36 × 1 matrix. The forces on the right-term of the equation include Fsupport: The support force from the ground, Finternal: the elastic force coupling the rods; Fexternal: external forces acting on the system, each being 36 × 1 in dimension. Assuming there are N grounding points, the constraint at each grounding point ensures sliding motion in the plane while remaining stationary in the z-direction: The total number of constraints is 3N, expressed as C = 0, where C has dimensions of 3N×1. The calculation of vx and vy involves a complex coupled relationship based on the friction model. By applying the principle of virtual work and combining C = 0 with Eq. (1), Fsupport can be determined as follows: This expression allows for the computation of the ground support force based on constraint conditions, external forces, and internal coupling forces. Given that the number of grounding points varies with the motion state, Fsupport is discretely dependent on motion. Equation (3) can then be substituted back into Eq. (1) to eliminate and derive the relationship concerning Fexternal and Finternal, defined as: Note that this represents a discrete nonlinear relationship dependent on the motion states (q˙,q,¨q). Fexternal can be expressed as Fexternal=JTF, representing the collection of each independent coupling force (line tension), where JT is a Jacobian matrix collection defined by geometric relations and motion states, representing a nonlinear relationship. F is a collection of each coupling force (line tension) defined as a function of each line length l, such that F=g(l). Thus, Eq. (4) can be further expressed as: where shows that f2 is a highly nonlinear discrete mapping relationship. Methodology System setup and structural sensing framework The proposed monitoring framework is designed to reconstruct structural integrity states through deformation-based sensing and hierarchical cognitive reasoning. Flexible strain sensors are integrated along selected tensile members to directly capture inter-nodal distance variations. By measuring deformation-induced resistance changes rather than relying on force estimation, the system reduces dependence on force modeling assumptions and enables geometric state reconstruction under tension-dominated configurations. To establish the mechanical–electrical mapping relationship, controlled loading experiments were conducted using a tensile testing machine. External force–displacement relationships, rotation angle–strain curves, and load–strain mappings were obtained to quantify the physical correspondence between structural deformation and sensor response, thereby forming the foundation for subsequent data-driven modeling. Sensor characterization and reliability evaluation Prior to algorithmic modeling, the sensing units were systematically characterized to ensure measurement reliability under practical operating conditions. The relative resistance response was examined under mechanical deformation scenarios, including bending, stretching, compression, and cyclic tensile loading, as well as environmental variations such as temperature and humidity changes. These experiments enabled the assessment of dynamic response stability, hysteresis behavior, repeatability, and environmental robustness. Through this evaluation, the operational validity range of the sensing layer was established, ensuring that downstream modeling operates on physically consistent and stable measurements. Data processing and state estimation modeling Following sensor validation, a data-driven mapping was constructed to estimate structural state variables from measured resistance variations. Multiple estimation strategies were comparatively investigated to determine the most suitable modeling approach for nonlinear resistance–deformation relationships. Polynomial regression, residual neural networks, and sparse identification of nonlinear dynamics were evaluated in terms of fitting accuracy, computational efficiency, and interpretability. Based on this comparative analysis, the optimal model was selected and integrated into the structural reconstruction pipeline. This stage converts raw resistance measurements into structured state descriptors that serve as intermediate representations for higher-level diagnostic reasoning. Multimodal feature fusion architecture To enhance diagnostic discrimination capability, multimodal information was incorporated into the framework. The multimodal diagnostic framework adopts a hierarchical feature-level fusion architecture. Strain-derived temporal signals are encoded via a two-layer LSTM network to obtain dynamic structural state representations, while visual inputs are processed through a convolutional feature extractor to obtain spatial degradation descriptors. The modality-specific feature vectors are projected into a shared embedding space and concatenated to form a unified latent representation. A fully connected fusion layer maps this representation to a structured diagnostic descriptor, which is subsequently interpreted by the ontology-guided reasoning module. This feature-level fusion strategy preserves modality-specific interpretability while enabling cross-modal correlation modeling prior to semantic reasoning. For specific LSTM training parameters, please refer to Table SI in the Supplementary Materials. Ontology-guided cognitive diagnosis and LLM fine-tuning For interpretable fault reasoning, the structured diagnostic embeddings were mapped onto an ontology-based knowledge graph encoding fault categories and causal relationships. A large language model was fine-tuned to generate structured diagnostic explanations conditioned on reconstructed state descriptors and ontological mappings. Model performance was evaluated by analyzing the influence of training epochs, prompt length, and input modality on perplexity, BLEU score, logical coherence, alignment with expert knowledge, and classification accuracy. Confusion matrices were used to assess diagnostic performance under sensor-only, image-only, and multimodal input configurations, thereby quantifying the contribution of multimodal fusion to fault discrimination. System integration and experimental demonstration The sensing layer, modeling module, multimodal fusion mechanism, and ontology-guided reasoning component were integrated into a unified monitoring framework. An alarm mechanism was implemented to detect abnormal sensor behavior, including disconnection events, and system-level response characteristics were analyzed to characterize transmission and computational delays. A comprehensive demonstration experiment was conducted to validate the autonomous integrity cognition capability of the framework, illustrating its ability to generate expert-level diagnostic explanations, identify structural faults, and detect sensor breakage conditions within an integrated deployment environment. Results The sections encompass the structure’s performance, sensor’s performance, machine learning methods, alarm system, fine-tuned LLM, and fault diagnosis, collectively addressing the integration of structural analysis, sensing capabilities, and AI-driven diagnostic methods for tensegrity systems. Structure’s performance To evaluate the functional performance of the tensegrity structure, we constructed a model of a six-strut tensegrity system and developed an experimental platform comprising six bars, 24 sensors, and two Arduino Mega microcontrollers (Fig. 2a, b). The data acquisition system employs an Arduino Mega 2560 microcontroller for synchronized signal collection. Strain sensors are distributed symmetrically across representative tension members according to a structural observability principle, ensuring sufficient geometric reconstruction constraints while minimizing redundancy. The sampling frequency is set to 10 Hz, which is sufficient to capture the dynamic structural responses under driving forces and external disturbances. We systematically conducted the experiments, including deformation displacements, rotational angles, strain variations, and torsional angles during the dynamic testing of the 6-bar tensegrity structure under distinct kinematic loading modes. More specifically, the platform’s single-cycle compression response exhibited characteristic nonlinear behavior in the external force-displacement relationship (Fig. 2c). The structure exhibited negligible deformation under increasing external force until a critical threshold was reached during loading process, at which point a distinct transition to a stiffness-stiffening response occurred due to geometric realignment and internal force redistribution. Pronounced hysteresis was observed during the unloading process: the structure required only 13 N, compared to 32 N during the loading process at a displacement of 10 mm. This quantitatively illustrates energy dissipation induced by internal friction and geometric nonlinearity. Moreover, the relationship between stress and rotation angle exhibited a quasi-linear correlation within the strain magnitude range of 0–0.5 (Fig. 2d). Beyond this threshold, the compressive members reached their elastic buckling limit, transitioning the system into a tension-dominant deformation regime in which the rotational angles asymptotically approached 80° with progressive stabilization. This nonlinear phase arises from prestress redistribution in tension cables, forming a spatial truss configuration that enforces kinematic constraints through three-dimensional self-locking mechanisms. As the applied load increases, the corresponding strain intensifies throughout the period of tensile deformation. Specifically, when the load reaches 25 N, the strain attains a magnitude of 0.3. However, as the load continues to increase, the rate of strain accumulation gradually decelerates, likely due to the intrinsic tensile characteristics (Fig. 2e). This trend is further reflected in the overall shear angle of the tensegrity structure, as illustrated in Fig. 2f. The shear angle exhibits a linear increase with escalating load, reaching a maximum value of 22° at an applied load of 47 N. Performance validation of 6-bar tensegrity structure. a 3D model of the tensegrity structure; b Optical image of the 6-bar tensegrity structure; c External force-displacement curve illustrating the platform’s single-cycle compression response; d Rotation angle-strain curves of the tensegrity structure under compressive loading; e Load-strain curves of the tensegrity structure under tensile loading; f Shear angle-load curves of the tensegrity structure under torsional loading Sensor’s performance In this study, we investigated the performance and characteristics of the strain sensor under dynamic conditions. Figure 3a illustrates the fitting strategy for the relationship between relative resistance variation and strain in the flexible sensor fabricated from diene rubber under normal, bending, and stretching conditions. When the sensor is in its normal state, the relative resistance can be expressed as follows: Performance and characteristics of strain sensor under dynamic conditions. a Fitting strategies for the relative resistance variation of the sensor under bending and stretching deformations. b Relative resistance variation of the sensor under compressive strain. c Resistance variation of the sensor at different temperatures. d Relative resistance variation of the sensor under varying relative humidity levels. e Changes in the relative resistance of the sensor over 540 cycles of tensile strain. Insets present the detailed relative resistance values at the beginning, middle, and end of the tensile test (from left to right) Both the strain characteristics and the relative resistance variation change accordingly when the sensor undergoes deformation. In the bending condition, a polynomial function fitting approach is employed to describe the relative resistance variation. In the stretching condition, a machine learning-based estimation method is utilized to construct a predictive model of the resistance change. Figure 3b demonstrates the relative resistance characteristics of the sensor under a compressive strain ranging from −0.9 to 0 mm, at a bending rate of 50 mm/min. The sensor demonstrates consistent responsiveness across varying levels of compressive strain; specifically, as the strain increases, the relative resistance variation also increases. A polynomial fitting method was applied to the experimental data, yielding a coefficient of determination (R²) of 0.99. This high R² value indicates that the fitted regression function accurately represents the observed data, thereby validating the applicability of the numerical model. We investigate the influence of environmental conditions on the performance of the sensor. Figure 3c shows the effect of temperature on the sensor’s resistance. Within the temperature range of 20–80 °C, there exists a significant linear relationship between temperature and sensor resistance. Similarly, Fig. 3d shows that the relative resistance variation exhibits a consistent increasing trend with rising humidity from 20 to 90%. The sensor also demonstrates stable performance over more than 540 cycles of tensile strain. During the initial cycles (0–30) and the final cycles (510–540), no significant decrease is observed in the fluctuation range of the relative resistance variation, indicating excellent long-term durability (Fig. 3e). Notably, the fabricated device exhibits highly consistent conductive periodicity even under large deformations. Machine learning methods Accurate model fitting serves as a fundamental prerequisite for establishing robust and reliable simulation platforms. Here, we systematically introduced, scrutinized, and comparatively evaluated four representative models: polynomial regression, recurrent neural networks (ResNets), sparse identification of nonlinear dynamics (SINDy), and long short-term memory (LSTM) in terms of their fitting principles and predictive capabilities, using mean squared error (MSE) and computational time as key evaluation metrics. Polynomial regression captures nonlinear data relationships by constructing higher-order polynomial terms of input features, thereby embedding nonlinear patterns within a linear regression framework. Its fundamental principle involves expanding the feature space by incorporating polynomial terms, effectively transforming inherently nonlinear problems into linearly separable problems within a higher-dimensional space (Fig. 4a). The model fails to effectively capture complex data patterns, resulting in a relatively modest fitting score of 0.933 and a mean squared error (MSE) of 14.28 × 10−3, indicating relatively high errors (Fig. 4b). As the polynomial degree n increases, the model’s fitting performance improves, with the score rising from 0.933 to 0.966 and the MSE decreasing by 49.36%. Performance analysis of machine learning estimation methods for fitting sensor relative resistance change models. a Fitting principle of polynomial regression. b Score and mean squared error (MSE) of polynomial regression for sensor model fitting. c Fitting principle of residual neural networks (ResNets). d MSE of ResNets for model fitting. e Fitting principle of sparse identification of nonlinear dynamics (SINDy). f MSE of SINDy for model fitting. g Comparison of MSE and computational efficiency across the four estimation methods ResNets address the vanishing/exploding gradient problem in deep network training by introducing skip connections to construct residual blocks. This approach enables layer-wise capture of high-order nonlinear features while maintaining stable gradient flow, thereby granting the model robust nonlinear fitting capabilities (Fig. 4c). The relationship between training epochs and mean squared error (MSE) reveals two distinct phases (Fig. 4d). In the under-trained phase (fewer epochs), the model exhibits suboptimal alignment between training and test data, with an MSE of 63.24 × 10−3, indicating substantial errors. As training progresses, the MSE declines by 96.83%, and predictions on both training and test datasets converge to nearly identical patterns, signifying effective generalization and the attenuation of overfitting. Also, SINDy seeks to systematically identify sparse governing equations of a system from data, employing the minimal set of nonlinear terms necessary to accurately characterize its evolutionary dynamics (Fig. 4e). SINDy demonstrates pronounced limitations when applied to fitting sensor relative resistance change models (Fig. 4f). The fitted model’s MSE consistently exceeds 0.4 across varying rates, a level deemed unacceptably high for practical applications. This limitation originates from the derivative-dependent nature of SINDy: numerical differentiation is inherently sensitive to noise, and the presence of input noise in sensor signals likely amplifies errors during dynamics reconstruction, thereby compromising the model’s fitting accuracy. We compared the MSE and computational efficiency of four machine learning estimation methods for model fitting (Fig. 4g). Polynomial regression achieves rapid computation (∼2 s) but with a relatively high MSE of 0.00771, underscoring its limitations in nonlinear time series fitting. In contrast, ResNets balance accuracy and efficiency, attaining an MSE of 0.002 with moderate computation time (∼6 s). However, its susceptibility to the vanishing gradient problem impairs predictive accuracy for long-term dependencies, resulting in inferior performance to LSTM. Notably, SINDy performs anomalously, exhibiting an MSE of 0.414676: nearly 400 times that of LSTM, and the longest computation time (∼11 s). This outcome suggests that the complex nonlinear interactions inherent in real-world systems often surpass the representational capacity of predefined basis function libraries, hindering the effective capture of system dynamics via sparse differential equations. The LSTM model achieves the lowest MSE of 0.0011, markedly outperforming other methods. While its computational time (∼7 s) slightly exceeds that of ResNets, this discrepancy is increasingly acceptable in our applications. Alarm system We developed an alert system to improve the discernibility of the tensegrity robot’s operational states. Real-time resistance signals are obtained by directly measuring the inter-nodal distances. These resistance variation signals are processed using polynomial fitting and LSTM models to reconstruct bending and stretching states with high fidelity. We achieved synchronized state reconstruction of the tensegrity robot in both virtual and physical environments through the deployment of Arduino and Python12. Subsequently, data from fault detection are transmitted via two Arduino units to an LED control board to simulate emergency alerts, such as activating LEDs to notify users when sensor disconnections occur. Finally, we relay the monitoring data to mobile devices via Wi-Fi using a Python Flask server (Fig. 5a). Alarm system. a Simulated emergency alert: LED activation indicates sensor disconnection, notifying users. b Differentiated configuration strategy: verifying alert system performance through selective LED activation. c Data analysis post-disconnection: assessment of transmission and computational delays following sensor disconnection. d Implementation process: overview of the alarm system deployment, including hardware and software integration We devised a differentiation strategy to assess the efficacy of the alarm system. Specifically, the system was configured to activate a single LED upon disconnection of the A12B21 link, whereas the disconnection of the B21C12 link triggered the activation of two LEDs (Fig. 5b). Figure 5c presents the quantified transmission and computational delays observed during sensor failure scenarios. In the experiments, manually disconnecting two tendons from the structure resulted in prolonged computational time for LED activation. We systematically measured time intervals across three critical stages: (1) disconnection detection by the Arduino Mega, (2) data reception by the Python processing module, and (3) final LED activation. Each of the 24 conductive tendons undergoes parallel computational workflows for state monitoring. However, the cumulative processing load increases linearly with the number of tendons, thereby substantially extending the overall computation time. The Arduino Mega’s serial interface introduces delays due to its limited bandwidth, which necessitates the sequential processing of both incoming sensor data and outgoing computational outputs. To enhance remote monitoring capabilities, we integrated a router with a Python Flask server and developed a mobile-accessible web interface (Fig. 5d). The mobile application was implemented as a simple, responsive web page. The testing workflow is as follows: First, sensor data is collected by the Arduino Mega and transmitted to the Python Flask server. Subsequently, a pre-configured web interface on mobile devices retrieves the resistance data via Wi-Fi connectivity. Finally, the resistance values on the web interface are automatically refreshed upon sensor state changes (Supplementary Video S1). Fine-tuned LLM The system architecture for shape-recognition and fault diagnosis of our tensegrity structure based on a fine-tuned LLM is shown in Fig. 6. The architecture primarily consists: Implementation architecture of the 6-bar tensegrity based on fine-tuned LLM. The diagram illustrates the pipeline of sensor signal processing (concatenation, averaging, interleaving), multimodal input embedding (time-series, images, text prompts), LLM fine-tuning with LoRA, and downstream task solvers for anomaly detection and self-shape recognition Specifically, the time-series resistance signals from the 24 flexible sensors are first segmented into fixed-length temporal windows. Each window is treated as a one-dimensional temporal patch. A linear projection layer then maps each patch into a fixed-dimensional embedding vector. These embedding vectors are concatenated sequentially to form time-series tokens, which are subsequently fed into the multimodal fusion module. Positional encoding is applied along the temporal dimension to preserve chronological dependencies before fusion with textual and image embeddings. For each RGB structural image, we adopt a patch-based embedding strategy similar to Vision Transformer architectures. The image is divided into fixed-size spatial patches, and each patch is flattened and linearly projected into an embedding vector. Spatial positional encodings are added to preserve geometric layout information of the tensegrity structure. These image tokens are then aligned dimensionally with the time-series tokens before multimodal fusion. Then, employing a lightweight modal alignment adapter, the “Patch Re-programmer”33, converts heterogeneous embeddings (time-series and image tokens) into the semantic token space of pre-trained large language models. This adapter ensures compatibility between physical sensor embeddings and text embeddings through dimensionality projection and feature normalization. After tokenization and embedding, text prompt tokens, time-series tokens, and image tokens are concatenated into a unified token sequence. The pretrained LLM backbone processes this sequence via self-attention, enabling cross-modal interaction between structural geometry, temporal dynamics, and linguistic instructions. To specialize the model for domain-specific diagnostic tasks, we adopt Low-Rank Adaptation (LoRA). During fine-tuning, only low-rank adapter matrices are updated, while the original backbone weights remain frozen. This parameter-efficient strategy allows the model to learn tensegrity-specific structural reasoning without overfitting or incurring high computational cost. In the anomaly detection stage, an autoencoder is trained on normal operational data to minimize the mean squared reconstruction error. During inference, reconstruction errors are used as anomaly scores. Inter-feature normalization is applied to remove scale discrepancies across sensor channels, and statistical thresholds combined with domain-specific frequency criteria are used to categorize anomalies into sensor interference, sensor breakage, rod deformation, or normal state. Fault diagnosis We systematically compared the pretrained LLM and LLM tuned by LoRA models across three dimensions: training cycles, prompt length, and text generation quality. Figure 7a demonstrates the relationship between training cycles and perplexity. Both models exhibit significant reductions in perplexity, ultimately converging to comparable levels, indicating analogous model capacities. Notably, the LLM tuned by LoRA model shows a sharper perplexity decline early in training, suggesting that its low-rank adapters expedite convergence through parameter-efficient fine-tuning that reduces computational redundancy. Subsequently, we present comparative BLEU scores under varying prompt lengths (Fig. 7b). Both models exhibit statistically consistent trends in BLEU score progression as context length increases, indicating that the parameter-efficient fine-tuning approach employed by the LoRA method effectively preserves the model’s adaptability to long-text generation. The marginally higher BLEU scores achieved by the tuned LLM on extended sentences, in comparison to the pretrained LLM, suggest that LoRA may enhance the coherence of long texts through targeted fine-tuning strategies. The final evaluation compares text quality across fluency, alignment with human expression, and logicality (Fig. 7c, Supplementary Materials Fig. S1). In terms of fluency, the slightly higher scores of LLM tuned by LoRA suggest that its adapter-tuning may optimize vocabulary distributions, thereby reducing grammatical errors. Regarding alignment with human expression, LoRA’s minor improvement may result from leveraging human feedback data to enhance naturalness. Most notably, LLM tuned by LoRA exhibits substantial gains in logicality, indicating that low-rank adapters effectively reinforce contextual reasoning capabilities, thereby substantiating the efficacy of task-specific fine-tuning. Performance evaluation. a Perplexity decreases as epochs increase. b BLEU scores vary with prompt length. c Improvement in fluency, alignment, and logicality after applying LoRA. d Accuracy increases with sensor data over epochs. e The accuracy fluctuates slightly with image data over epochs. f Multimodal data input achieves higher accuracy over epochs. g Confusion matrix showing classification performance with sensor data input. h Confusion matrix illustrating classification performance with image data input. i Confusion matrix demonstrating classification performance with multimodal data input Additionally, we conducted an analysis of the proposed monitoring system’s overall accuracy under varying input modalities. The results indicate that all three input modalities exhibit progressive accuracy improvements as training epochs increase (Fig. 7d–f, Supplementary Materials Fig. S2). During the initial training phase (0–10 epochs), all input modalities demonstrate a rapid increase in accuracy. However, in the mid-phase training (10–40 epochs), heightened accuracy volatility emerges in unimodal systems, particularly evident in image-based models. This finding suggests that the learning processes of unimodal systems may be more susceptible to interference. In the final training phase (≥40 epochs), all models demonstrate convergence. Sensor data input achieves an accuracy of 0.88–0.90 (Fig. 7d), image data input stabilizes between 0.80 and 0.85 (Fig. 7e), while multimodal integration input attains approximately 0.95 accuracy (Fig. 7f). We classify fault states into four categories: sensor interference, sensor breakage, rod deformation, and normal. Confusion matrices are provided for monitoring systems trained on three distinct input data types, illustrating their classification performance across these states (Fig. 7g–i). Figure 7g presents the confusion matrix for the diagnostic strategy using only sensor data input. The matrix reveals classification accuracy rates of 91% for Class I (sensor interference), 89% for Class II (sensor breakage), 75% for Class III (rod deformation), and 82% for Class IV (normal). In comparison, Fig. 7h demonstrates that the image-based monitoring strategy achieves significantly lower classification confidence for Class I and II states (91% → 71 and 89% → 71% respectively) than the sensor-based approach. However, it shows improved confidence levels for Class III (75% → 84%). This discrepancy may be attributed to the inherent limitations of image recognition techniques. Classes I and II primarily involve internal structural fault that are inherently difficult to clearly capture in single-frame images, whereas sensor data can effectively reflect these internal state variations. In contrast, Class III state variations exhibit more visually discernible characteristics, rendering image-based classification more effective for this particular condition. Figure 7i introduce multimodal inputs to enhance the diagnostic strategy’s training model. This modification yields significant improvements in classification accuracy across all categories while effectively addressing the limitations inherent to both sensor-based and image-based single-modality approaches. The diagnostic module performs four-state classification, as quantitatively validated in Fig. 7. The fine-tuned LLM operates downstream, receiving structured diagnostic outputs and generating expert-level reasoning reports. Representative prompt–response examples corresponding to each classified state are provided in Fig. S1, demonstrating the integration of structural mechanics knowledge and probabilistic fault interpretation. In this study, we present a test case to assess the performance of the Autonomous Integrity Cognition System following the fine-tuning of the LLM (Fig. 8). Users induce structural deformation by compressing the tensegrity system, simulating ground collision scenarios (Fig. 8a). Concurrently, multimodal sensors capture real-time physical deformation data, which is transmitted to the fine-tuned LLM system (Supplementary Video S2). The system relies on fine-tuning LLM to integrate the machine body information and perception data, and finally realizes intelligent interaction through the natural language interface. To demonstrate the interaction mechanisms, we present typical dialog scenarios including structural status queries, sensor value readings, and proprioceptive parameter verification. Figure 8b demonstrates the logical architecture for the system’s fault diagnosis. The workflow initiates with physical fault triggers, employs multi-sensor fusion and synchronized structural image capture to transmit multimodal inputs to the fine-tuned large language model for cross-modal analysis. The monitoring system then precisely identifies and classifies fault characteristics, ultimately generating interpretable technical diagnostic reports. Demonstration of the autonomous integrity cognition system via fine-tuned LLM. a Expert-level response capability in human-machine interaction within the system is facilitated by the fine-tuned LLM. b Fault identification using fine-tuned LLM response mechanisms. c Faults diagnosis of sensor breakage via the fine-tune LLM Finally, we simulated a sensor fracture failure scenario (Fig. 8c, Supplementary Materials Fig. S2, Supplementary Video S3). During the experimental execution phase: the operator applied axial stress to the sensing unit until fracture occurred, triggering a redistribution of structural states. The mechanical response data obtained during the image generation phase reveals that, during the failure progression, sensing channels A12B21 and B21C12 exhibited abrupt decreases in length values at 24 s and 20 s, respectively, prompting virtual model reconstruction based on these state changes. The system achieved 94% confidence in Type I fault identification, significantly outperforming other fault categories (Type II-IV combined <6%), demonstrating exceptional autonomous integrity cognition capabilities during the diagnostic phase. The system ultimately generates both fault classification results and interpretable technical diagnostic reports through the LLM system. Conclusion Our study presents a comprehensive investigation of a six-bar tensegrity structure, sensor performance, machine learning estimation methods, an alarm system, language model fine-tuning, and multimodal fault diagnosis. Firstly, the tensegrity structure exhibited pronounced nonlinear deformation and self-locking behavior, dissipating energy as the force dropped from 32 N to 13 N at a 10 mm displacement, and achieving a maximum shear angle of 22° under a 47 N load. Secondly, the sensor demonstrated a reliable linear resistance-temperature response between 20 and 80 °C, showed humidity-dependent resistance increases from 20 to 90%, and exhibited excellent durability with stable conductivity over 540 tensile cycles, thereby confirming its robustness under significant deformation. Regarding machine learning models, LSTM achieved superior estimation accuracy with the lowest mean squared error (0.0011) and moderate computation time (~7 s), outperforming ResNets and significantly surpassing SINDy, which showed limited capability in modeling complex nonlinear dynamics. Furthermore, the proposed alarm system effectively distinguishes tendon disconnections through a visual LED signaling mechanism; its computational delays scale linearly with tendon count and it supports real-time resistance monitoring via a mobile-accessible web interface powered by a Python Flask server. In the domain of natural language processing tasks, LoRA-based fine-tuning of large language models (LLMs) yielded faster perplexity reduction, higher BLEU scores for long-text generation, and enhanced text quality, particularly in logical coherence. This underscores the efficiency and efficacy of parameter-efficient tuning strategies. Finally, the incorporation of multimodal inputs significantly improved diagnostic accuracy (∼0.95) compared to unimodal sensor (0.88–0.90) or image (0.80–0.85) approaches. This improvement effectively mitigates the limitations inherent to single-modality methods and advances fault classification performance. The present study primarily aims to validate the feasibility of integrating structural reconstruction with ontology-guided cognitive reasoning, rather than to provide a fully optimized engineering deployment. Consequently, several limitations remain. First, the current prototype is not optimized for strict real-time applications. The observed processing delay arises mainly from software-level data handling in the experimental implementation. Second, the experimental validation focuses on representative single-fault states to ensure clear causal separability, and systematic compound-fault datasets have not yet been established. Although the proposed architecture is theoretically extensible to multi-fault inference and latency-optimized deployment, comprehensive system-level optimization and large-scale benchmarking under complex operational conditions remain important directions for future work. Supplementary information SUPPLEMENTARY MATERIALS Figures S1 to S5, Table SI, Special Topic Discussion Supplementary Video S1 Supplementary Video S2 Supplementary Video S3 Acknowledgements This work was supported by the National Natural Science Foundation of China (No. 52505072 to Z.M.) and by the Independent Research Project of the Open Foundation of the State Key Laboratory of Fluid Power and Mechatronic Systems (Grant SKLoFP_ZZ_2513 to Z.M.), and by the National Key Laboratory of Metal Forming Technology and Heavy Equipment (SKLFM-2025-017 to Z.M.). Author contributions Zebing Mao: Conceptualization, funding acquisition, methodology, writing - review and editing, investigation. Jianhui Wang: Data curation, formal analysis, investigation, software, writing—original draft. Jianhua Zhang: Supervision, software, writing—review and editing. Junji Ohgi: Conceptualization, methodology, project administration, supervision, validation. Yanqiu Zheng: Data curation, software, writing—original draft. Yanhong Peng: Supervision, writing—review and editing. Luyang Zhao: Methodology, software, writing—review and editing. Qi Su: Writing—review and editing. Weidi Huang: Writing—review and editing. Bing Xu: Writing—review and editing. Conflict of interest The authors declare no competing interests. Contributor Information Jianhua Zhang, Email: jianhua.zhang@zju.edu.cn. Qi Su, Email: suqi@zju.edu.cn. Weidi Huang, Email: wdhuang@zju.edu.cn. Supplementary information The online version contains supplementary material available at 10.1038/s41378-026-01322-0. References Associated Data Supplementary Materials SUPPLEMENTARY MATERIALS Figures S1 to S5, Table SI, Special Topic Discussion Supplementary Video S1 Supplementary Video S2 Supplementary Video S3