Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spaces Pratham Yashwante, Rose Yu Abstract The Platonic Representation Hypothesis posits that learned representations from models trained on different modalities converge to a shared latent structure of the world. However, this hypothesis has largely been examined in vision and language, and it remains unclear whether time series participate in such convergence. We first examine this in a trimodal setting and find that independently pretrained time series, vision, and language encoders exhibit near-orthogonal geometry in the absence of explicit coupling. We then apply post-hoc alignment by training projection heads over frozen encoders using contrastive learning, and analyze the resulting representations with respect to geometry, scaling behavior, and dependence on information density and input modality characteristics. Our investigation reveals that overall alignment in contrastive representation spaces improves with model size, but this alignment is asymmetric: time series align more strongly with visual representations than with text, and images can act as effective intermediaries between time series and language. We further see that richer textual descriptions improve alignment only up to a threshold; training on denser captions does not lead to further improvement. Analogous effects are observed for visual representations. Our findings shed light on considerations for building multimodal systems involving non-conventional data modalities beyond vision and language. 1 Introduction Neural network representations are increasingly converging. Across architectures and training objectives, deep models tend to organize inputs according to similar notions of similarity, an observation formalized as the Platonic Representation Hypothesis (PRH) ( 18 ) . The hypothesis posits that representation learning models converge toward a shared statistical model of reality, shaped by the structure of the underlying data. Empirical evidence for this convergence is strongest for vision and language: as models scale, their similarity kernels become more aligned. Joint vision–language (VL) models such as CLIP ( 35 ) further demonstrate that contrastive learning (CL) can produce coherent shared representations across perceptual and linguistic domains. Figure 1 : Trimodal projections of a shared temporal process. A latent process Z gives rise to a numeric time series, a visual line plot, and a textual description, each representing the same signal in values, geometry, and language. Modality-specific encoders f_{\text{ts}} , f_{\text{img}} , and f_{\text{txt}} map inputs into representation spaces. Time series pose a distinct challenge for multimodal alignment because their semantic structure is not directly observable from raw values. Images encode structure explicitly through spatial geometry, and text encodes semantics explicitly through symbolic tokens. In contrast, numeric time series express meaning only implicitly through temporal variation: properties such as trends, periodicity, or anomalies are not discrete tokens or visual features, but latent properties that must be recovered from the signal. For example, while an image of a cat and the word “cat” explicitly denote the same concept, a temporal trend is neither an object nor a symbol, but a property that emerges only through computation over a numeric series. This mismatch raises a central question: can time series achieve the same degree of representational alignment with vision and language? Figure 1 illustrates the trimodal formulation studied in this work. Assuming that there exists a shared reality of a latent temporal process, we aim to understand the convergence behavior of learned representations from vision, language and time series. Figure 2 : Mean angular deviation between pretrained cross-modal representations on CaTS shows little inherent alignment. As a motivating observation, we first examine the representational geometry in the absence of external coupling on the CaTS dataset ( 59 ) , which consists of triplets of numeric time series, corresponding visual plots, and textual captions. Independently pretrained cross-modal encoders exhibit near-orthogonal structure across modalities (Figure 2 ). When external coupling via CL is applied, alignment behavior differs substantially across datasets: while image–text structure becomes coherent on Flickr ( 34 ) , the same objective gives more fragmented overlap on CaTS as shown in Figure 3 , highlighting the additional challenges posed by time series. Additional analyses of uncoupled representational geometry across multimodal time series datasets are provided in Appendix F . Figure 3 : UMAP visualizations of representations after contrastive training on Flickr (top) and CaTS (bottom). These observations suggest that alignment may not emerge uniformly across modalities and datasets, even under identical contrastive objectives, thereby raising the question of how time series alignment differs from standard VL settings. In this work, we conduct a systematic empirical study of trimodal alignment across time series, visual plots, and language, using CL as a controlled mechanism to probe representational compatibility across modality pairs. Our setup pairs pretrained encoders from each modality and trains projection heads to map representations into a shared space, following the CLIP paradigm ( 35 ) . Our experiments span four datasets varying in domain and annotation richness, 34 encoder combinations across multiple model families and scales, and controlled ablations isolating the effects of model capacity, optimization, and input characteristics. Our contributions and empirical findings highlight several aspects of trimodal alignment involving time series: 1. Trimodal Analysis: We present the first systematic empirical study of trimodal alignment involving time series, images, and language, and discuss the roles of information density and semantic explicitness in governing cross-modal alignment. 2. Asymmetric Convergence: Trimodal alignment is asymmetric, with time series aligning more strongly with visual plots than with language. This gap persists even with detailed textual descriptions, providing evidence that multimodal convergence is uneven. 3. Information Density Saturation: Increasing information density improves alignment only up to a threshold, indicating that denser text by itself is insufficient to induce further cross-modal convergence. 4. Grounding and Explicitness: Alignment is shaped not only by model scale, but also by how explicitly semantics are grounded across modalities and how well representational formats are matched; modality pairs with more explicitly observable correspondences consistently align more strongly than those with implicit semantic links. Figure 4 : Trimodal contrastive alignment framework. Frozen pretrained encoders independently map time series, visual plots, and text into their respective unimodal representation spaces. Trainable projection heads transform these representations into a shared embedding space. Alignment is learned via a symmetric contrastive objective applied jointly across all modality pairs (TS–IMG, TS–TXT, IMG–TXT). 2 Related Work Contrastive learning. CLIP ( 35 ) demonstrated that CL on large-scale image–text pairs produces representations with strong zero-shot transfer capabilities. The approach trains separate encoders for each modality and aligns their outputs using the InfoNCE loss ( 45 ) , which maximizes agreement between matched pairs while minimizing agreement with mismatched pairs within a batch. More work has extended this paradigm to additional modalities including audio ( 14 ) , video ( 53 ) , and 3D clouds ( 56 ) . Time series models. Recent work has introduced pretrained time series models for forecasting or masked prediction ( 52 ; 12 ; 3 ; 8 ) . While these models demonstrate strong downstream performance, their representational alignment with other modalities remains underexplored. Prior work has also explored pairing time series with other modalities for specific applications. In healthcare, clinical notes have been aligned with physiological time series for patient outcome prediction ( 4 ; 16 ) . Contrastive learning has also been applied to time series–text settings, including MedCLIP ( 51 ) and the recent TimesCLIP ( 7 ) . Our work instead treats representational alignment as the main focus, studying when and under what conditions it emerges, with potential implications for downstream use. Semantic content and density in language. The uniform information density hypothesis ( 25 ) proposes that speakers distribute information evenly across utterances to optimize communication efficiency. Subsequent work has operationalized this through surprisal-based metrics measuring how predictably information is distributed across tokens ( 30 ) . We adapt these ideas to study how the amount and distribution of semantic content in text affects alignment quality. Further discussion on alignment is provided in Appendix B , along with related work on ECG representations in Appendix N . 3 Trimodal Alignment Framework Design Rationale. As shown in Figure 4 , our design adopts the core contrastive alignment mechanism similar to that used in CLIP, repurposing it as a controlled framework for analyzing trimodal representational compatibility. We evaluate 34 trimodal configurations spanning 26 unique pretrained encoders, including 9 text, 9 vision, and 8 time series models across different scales. All configurations use identical hyperparameters for fair comparison. Encoder combinations appear in Appendix D , with experimental details and ablations in Appendix G . Each modality encoder is used as a frozen feature extractor, and its output is mapped into a shared embedding space via projection heads. All modalities share the same projection (Appendix G.6 ). Symmetric Contrastive Loss. We align representations using a symmetric contrastive objective applied jointly across all modality pairs: time series–image (TS–IMG), time series–text (TS–TXT), and image–text (IMG–TXT). Given \ell_{2} -normalized embeddings z_{\text{ts}},z_{\text{img}},z_{\text{txt}}\in\mathbb{R}^{d} , we apply a bidirectional InfoNCE loss to each modality pair. For a modality pair (x,y) , the forward loss is defined as \mathcal{L}_{x\rightarrow y}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp\!\left(z_{x}^{(i)\top}z_{y}^{(i)}/\tau\right)}{\sum_{j=1}^{N}\exp\!\left(z_{x}^{(i)\top}z_{y}^{(j)}/\tau\right)}, (1) where \tau is a temperature parameter. For example, the TS–IMG loss is defined as the average of the forward and reverse directions, \mathcal{L}_{\text{ts-img}}=\tfrac{1}{2}\left(\mathcal{L}_{\text{ts}\rightarrow\text{img}}+\mathcal{L}_{\text{img}\rightarrow\text{ts}}\right). (2) Analogous losses are defined for TS–TXT and IMG–TXT (see Appendix G.3 ). The final training objective is the equally weighted sum of all three modality-pair losses: \mathcal{L}_{\text{total}}=\mathcal{L}_{\text{ts-img}}+\mathcal{L}_{\text{ts-txt}}+\mathcal{L}_{\text{img-txt}}. (3) We also evaluate ablated variants that restrict the set of modality pairs in the loss, including bimodal TS–IMG and TS–TXT settings, as well as joint VL with time series (VL–TS; full details in Appendix I ). Evaluation Metrics. We evaluate alignment using metrics that capture global similarity, retrieval, and geometric consistency of cross-modal representations. All metrics are computed on held-out test sets using \ell_{2} -normalized embeddings. Implementation details and definitions are provided in Appendix E . For each modality pair, we calculate: 1. Cosine similarity margin defined as the difference between matched and mismatched pairs. 2. Bidirectional cross-modal retrieval using Recall@ k (R@1, R@5, R@10). 3. Procrustes disparity measures how well one modality can be aligned to another via an optimal orthogonal transformation. 4. Centered Kernel Alignment (CKA) measures non-linear representational similarity using RBF kernels. 5. Mutual k NN overlap measures agreement in local neighborhood structure. We quantify the semantic richness of textual descriptions using information density (ID), defined as the total surprisal of a text under a pretrained language model. Higher ID corresponds to captions that are longer and semantically more informative and unique. Dataset-level ID is computed as the mean across captions; see Appendix H for full details. Datasets. We evaluate trimodal alignment using datasets that allow us to probe semantic explicitness, information density, and visual mediation. Because fully aligned trimodal time series datasets are rare, we combine human and synthetic data to construct modality-complete variants wherever necessary. Full details are provided in Appendix C . We use CaTS-Bench ( 59 ) as our primary dataset, as it natively provides aligned triplets of time series, visual plots, and captions. The captions serve as direct textual representations, describing observable temporal structure such as trends, phases, events, and summary statistics. To study ID, we construct multiple caption variants derived from the same underlying time series, including progressively condensed test-set captions and an expanded high-ID variant. Condensed captions reduce semantic explicitness by limiting descriptions to a fixed number of salient phrases, while the high-ID variant exceeds the original captions by more than a factor of two in measured ID. We also evaluate on TRUCE ( 19 ) , which contains short time series paired with concise, direct textual descriptions. Its short length enables explicit plot annotation of the time series. Owing to its small size, it is used only for evaluation, with three visual variants per signal (generic, styled, and annotated) to assess visual robustness. To study alignment under indirect textual supervision, we use MIMIC ( 13 ) and PTB-XL ( 46 ) , where text consists of diagnostic reports that do not explicitly describe waveform structure. MIMIC reports are in English, while PTB-XL reports are in German, allowing us to examine multilingual effects. For both datasets, we generate visual plots of ECG waveforms and construct train/test splits matched in size to CaTS. These datasets use longer series (5000 timesteps) than CaTS, enabling analysis of how temporal resolution affects multimodal alignment. 4 Results Figure 5 : Scaling behavior of alignment across 34 trimodal configurations on CaTS as a function of total model size (in millions of parameters). Each subplot reports alignment quality for three modality pairs (TS–IMG, TS–TXT, IMG–TXT). Each point corresponds to a distinct encoder configuration. Dashed lines indicate linear trends with Pearson correlation coefficients reported in the legend. RQ1. Do contrastive representation spaces converge uniformly as models scale? Increasing model capacity is often associated with stronger representations ( 20 ; 35 ; 17 ) , but whether scale alone leads to convergence across time series, vision, and language remains an open question. We therefore use our trimodal framework to analyze alignment on CaTS across encoder scales, while holding the contrastive objective and parameters fixed. Figure 5 reports alignment quality across modality pairs as a function of total model size. Overall alignment improves with scale, but gains are highly uneven across modality pairs. TS–TXT alignment remains the weakest in absolute terms across all model sizes, reflecting the difficulty of directly aligning numeric signals with language, yet it also exhibits the strongest positive relationship with scale. In contrast, TS–IMG alignment achieves substantially higher absolute performance even at smaller scales, but shows weaker scaling trends, suggesting earlier saturation driven by stronger shared visual grounding with temporal data. Notably, global alignment across modalities is consistently strong, as reflected by cosine similarity and geometric metrics, while local neighborhood structure remains weak. Across all configurations, mutual k NN overlap stays low, indicating limited fine-grained correspondence. This highlights a clear dissociation between global and local alignment under CL: strong global geometric similarity does not imply robust neighborhood-level semantic correspondence. Further, we observe a consistent asymmetry, with time series aligning more strongly with visual plots than with language. This pattern reflects mismatched inductive biases and differences in semantic explicitness across modalities: visual plots externalize latent temporal structure into geometric form, whereas time series encode it implicitly and text abstracts over it symbolically. As a result, alignment is strongest between modalities that expose structure in comparable representational formats, while implicit-to-abstract alignment remains challenging even at larger scales. Further detailed discussion and ablations are in Appendix M . We also find that trimodal alignment is sensitive to both optimization quality and encoder capacity. Larger batch sizes and stronger projection heads consistently improve alignment which confirms the importance of effective contrastive optimization. Also, scaling the time series encoder provides substantial gains for both TS–IMG and TS–TXT alignment. This identifies temporal representations as a key factor in trimodal alignment, with improvements to the time series encoder being particularly important for strengthening otherwise weak modality pairs such as TS–TXT. Additional analyses and ablations are provided in Appendix G.5 (optimization), K (time series encoder capacity), and G.6 (projection head design). Figure 6 : Effect of text ID on alignment across increasing levels of text density. Markers indicate distinct caption regimes with increasing semantic richness. Scores are averaged across 9 representative model configurations (see Appendix D.2 ). RQ2. Do pretrained VL models change alignment? Figure 7 : IMG–TXT alignment across model scales for VL–TS on CaTS. A–I correspond to select configurations (Appendix D.2 ). Jointly pretrained VL models exhibit strong intrinsic alignment between images and text ( 35 ; 27 ; 55 ) , raising the question of whether such coupling affects trimodal alignment. In particular, it is unclear whether strong IMG–TXT alignment in trimodal systems must be learned jointly, or can instead be inherited from existing VL pretraining. To isolate this effect, we conduct this comparison on CaTS, contrasting the full trimodal setting with configurations where we pair a pretrained VL model with a time series encoder. Both settings are trained using the same contrastive objective, allowing differences in alignment to be attributed to representational priors. As shown in Figure 7 , we see that VL–TS configurations achieve strong IMG–TXT alignment even at relatively small scales. This reflects the tightly coupled image–text geometry inherited from VL pretraining, which enables robust alignment without relying on increased model capacity. In contrast, trimodal encoders must jointly learn structure across all three modalities, and therefore depend more heavily on scale to compensate for weaker intrinsic coupling. RQ3. How does information density in text affect cross-modal alignment? Because textual descriptions vary widely in how explicitly they encode semantic structure about an underlying time series, we study how the information density of text affects the strength of cross-modal alignment. We vary textual semantic richness by constructing multiple caption variants for each sample with different ID. Specifically, we use LLMs (GPT-4o-mini and LLaMA-3.2-90B) to compress the original captions to a fixed number of salient semantic phrases, while keeping the underlying data, model architectures, and contrastive setup fixed. We observe that alignment quality improves consistently with increasing ID across all metrics, as shown in Figure 6 . Using the original CaTS captions with three progressively condensed variants, we find that as captions become denser and more information-rich, Procrustes disparity decreases, indicating tighter shared geometry, while k NN overlap increases, reflecting stronger neighborhood-level semantic consistency. Low-information captions provide insufficient semantic signal, leading to weakly structured or partially collapsed embeddings and high variability across model configurations. At very low ID, embeddings cluster tightly with limited neighborhood differentiation, indicating that sparse semantic content constrains the formation of meaningful relational structure. In contrast, higher-information captions encode richer attributes and contextual relationships, enabling more coherent cross-modal geometry. Larger models benefit most in this regime, but also degrade more sharply when captions are short and under-specified, reflecting a stronger dependence on explicit semantic content. These results show that the amount of explicitly encoded semantic information plays a key role in multimodal alignment when text is involved. Since alignment improves as captions become denser, we test whether further increasing caption ID during training provides additional gains. Specifically, we test whether training on captions with more than double the original CaTS ID improves alignment. We find that despite this substantial increase in semantic content, alignment metrics change only marginally across all modality pairs as shown in Table 1 . This indicates that once text reach sufficient semantic richness, further increases in training-time ID do not meaningfully improve cross-modal alignment. Table 1 : Effect of doubling text ID on alignment. We compare original captions (train ID = 417.2) and high-information captions (train ID = 870.2). \Delta denotes High ID - OG ID. Both models use same samples with their respective ID-specific caption sets. Scores are averaged across configurations listed in Appendix D.2 . Cosine Margin ↑ Procrustes ↓ Mutual kNN ↑ Pair OG High \Delta OG High \Delta OG High \Delta TS–IMG 0.72 0.72 0.00 0.47 0.47 0.00 0.13 0.13 0.00 TS–TXT 0.57 0.56 -0.01 0.74 0.74 0.00 0.08 0.07 -0.01 IMG–TXT 0.64 0.64 0.00 0.60 0.60 0.00 0.11 0.10 -0.01 A similar effect appears when evaluating CaTS-trained models on TRUCE for TS–TXT alignment. TRUCE captions are short, direct textual descriptions with ID comparable to our low-ID (2 phrase) caption variants (ID \approx 74.5 vs. 69.6 ) and similar length (12.5 vs. 10.2 words on average), and give comparable levels of TS–TXT alignment. This mirrors the behavior observed for CaTS captions in the low-ID regime and provides additional evidence that alignment is strongly governed by the amount and structural organization of semantic content in the text modality. RQ4. What happens when text does not directly describe the time series signal? In many real-world settings, text associated with time series provides high-level interpretations rather than direct descriptions of signal structure. We therefore ask whether alignment persists when language is only indirectly related to the underlying signal. To answer this, we evaluate trimodal alignment using the same contrastive framework under indirect textual supervision using ECG datasets. Specifically, we compare CaTS with MIMIC, where clinical reports only implicitly relate to waveform dynamics. We observe distinct alignment scaling behavior between CaTS and MIMIC across different trimodal configurations (Figure 8 ). Although increasing model capacity improves alignment across all modality pairs in both datasets, systematic differences persist depending on the nature of textual supervision. Note that CaTS captions exhibit substantially higher textual density (test ID = 416.81), whereas MIMIC reports have much lower ID (test ID = 149.48), and this disparity is directly reflected in cross-modal alignment strength. Figure 8 : Scaling behavior of alignment on CaTS and MIMIC across 5 representative trimodal configurations (see Appendix D.2 ). We report Procrustes disparity as a function of model size. Alignment involving text is consistently weaker on MIMIC, particularly for TS–TXT and IMG–TXT pairs. Even at comparable model scales, MIMIC exhibits lower cosine similarity and higher Procrustes disparity, indicating poorer global and geometric alignment when language provides only indirect semantic grounding. This gap is most pronounced for TS–TXT which shows that clinical language provides substantially less explicit signal for aligning numeric time series than direct descriptive captions. Scaling partially mitigates this effect: larger models improve TS–TXT alignment on both datasets, but gains are substantially weaker under indirect supervision, with alignment remaining consistently worse on MIMIC across scales, indicating a limit imposed by semantic explicitness. In contrast, TS–IMG alignment is often stronger on MIMIC despite weaker text alignment. This reflects the longer and more structured ECG signals in MIMIC, which provide richer numeric structure for alignment with visual plots. We also see that stronger TS–IMG alignment also yields large gains in cross-modal retrieval (Table 2 ), confirming that improved geometric alignment translates into retrieval performance even when textual grounding is weak. Table 2 : Cross-modal retrieval performance on MIMIC and CaTS for TS \leftrightarrow IMG retrieval. Results are macro-averaged over TS \rightarrow IMG and IMG \rightarrow TS and reported as percentages. Model acronyms: Dv2/Dv3 = DINOv2/DINOv3, SL = SigLIP2, Q = Qwen, Mo = MOMENT, C = Chronos; B/L denote Base/Large. Model Configuration Retrieval Performance (%) R@1 R@5 R@10 CaTS MIMIC CaTS MIMIC CaTS MIMIC Dv2-B + Q-0.6B + Mo-B 1.61 31.31 6.13 61.84 10.75 73.62 Dv2-L + Q-4B + Mo-L 2.25 21.36 8.65 49.48 14.71 62.21 Dv3-7B + Q-8B + Mo-L 6.19 23.70 18.35 51.80 27.12 65.13 SL2-B + Q-0.6B + C-B 3.96 20.33 12.64 45.79 19.85 58.69 SL-L + Q-4B + C-L 6.95 29.78 19.84 58.45 28.65 69.90 RQ5. How sensitive is alignment to language shifts? Thus far, we have examined how alignment depends on explicitness and density within a single language. In practice, however, textual supervision can also vary linguistically. To isolate the effect of language shift, we compare alignment between MIMIC and PTB-XL, which both pair ECG time series with clinical reports from the same domain but differ in report language (English vs. German). Figure 9 shows that alignment is consistently weaker on PTB-XL across all modality pairs. This degradation is pronounced for pairs involving text, indicating that linguistic shift further weakens semantic coupling even when the underlying time series distribution is comparable. All trends observed in earlier experiments persist: TS–TXT remains the most challenging pair, increased scale provides partial compensation, and TS–IMG alignment remains relatively robust for longer more structured time series. These results suggest that cross-modal alignment is sensitive not only to semantic explicitness and text ID, but also to how well the language of textual supervision aligns with the inductive biases of pretrained language encoders. Figure 9 : Comparison of multimodal alignment between MIMIC (English reports) and PTB-XL (German reports). English reports consistently achieve higher alignment across all modality pairs. Figure 10 : Role of trimodality as a semantic bridge. We compare bimodal and trimodal CL for TS–IMG (top row) and TS–TXT (bottom row). Introducing the image modality consistently improves TS–TXT alignment. RQ6. Do richer visual inputs help alignment? While earlier results show that textual semantic explicitness and information density strongly affect alignment, it remains unclear whether similar effects arise in the visual modality. To test whether increasing visual semantic content improves alignment, we compare TS–IMG alignment across plot variants from the TRUCE dataset with progressively richer visual annotations (Appendix C.2 ), holding the underlying time series and training setup fixed. Figure 11 shows that increasing visual semantic richness leads to consistent improvements in TS–IMG alignment across configurations. Annotated plots achieve higher alignment than generic or stylistically varied variants, indicating that richer and denser visual inputs provide stronger semantic anchors for aligning time series with images. Consistent with earlier trends, increasing model capacity (compare D and E) further improves TS–IMG alignment which indicates that visual expressiveness and scale act as complementary factors. Figure 11 : TS–IMG alignment across TRUCE plot variants with increasing visual input richness. Annotated achieves highest scores. RQ7. Does trimodality help weakly aligned pairs? Given the persistent asymmetry between TS–IMG and TS–TXT alignment, we ask whether introducing a third modality can facilitate alignment for otherwise weak modality pairs. To answer this, we measure alignment changes when one modality (IMG or TXT) is removed, comparing bimodal and trimodal contrastive learning on CaTS (Figure 10 ). For the weakest modality pair, TS–TXT, introducing the image modality produces a consistent upward shift across alignment metrics which indicates that visual representations provide intermediate semantic structure that neither time series nor text capture in isolation. This improvement reflects not only better global geometry but also more coherent neighborhood structure which suggests that images help organize weakly coupled representations. For a strong pair (TS–IMG), introducing a third modality often degrades performance, suggesting that it adds optimization complexity without providing new semantic signal. This shows that trimodality is most effective when it introduces missing semantic structure rather than redundant supervision. 5 Discussion We first examine whether the datasets studied here exhibit any inherent cross-modal structure prior to alignment and find that independently pretrained time series, vision, and language encoders produce near-orthogonal representations. This establishes a clear baseline: multimodal convergence does not arise without explicit coupling. When contrastive alignment is introduced, convergence remains highly non-uniform and strongly modality-dependent. While contrastive learning readily aligns vision and language, time series participate only partially, with the weakest alignment observed for TS–TXT. This asymmetry persists even at larger scales and is consistent with how semantics are expressed: time series encode structure implicitly, text abstracts symbolically, and visual plots externalize latent temporal structure. As a result, images can act as intermediaries, reducing abstraction gaps and helping stronger alignment between time series and text. We further show that semantic explicitness constrains how much shared structure can be recovered across modalities. Increasing information density improves alignment at low to moderate levels, but saturates beyond a latent threshold, indicating that alignment limits arise from representational mismatch rather than insufficient supervision or scale alone. Although scaling and improved optimization selectively help weaker modality pairs, architectural coupling and pretraining objectives also play a stronger role, as evidenced by the effectiveness of jointly pretrained VL models. All these findings refine how multimodal convergence should be understood for numeric modalities and suggest that representational format and explicitness are key factors shaping alignment behavior, beyond model size alone. Impact Statement This paper aims to advance the empirical understanding of multimodal representation learning involving time series, visual, and language modalities. The work focuses on analyzing alignment behavior under controlled experimental settings, with the goal of clarifying how different modalities interact in contrastive representation spaces. While the study is not intended as a deployment-facing system and does not introduce models trained for direct decision-making in real-world applications, the questions it addresses are relevant to domains such as scientific analysis and healthcare, where multimodal data are common. We do not foresee immediate negative societal impacts arising from this work. By improving understanding of the conditions under which multimodal representations align or fail to align, this work aims to inform more robust evaluation practices and support the responsible development of future multimodal systems. Appendix A Limitations and Future Directions Several questions remain open for future research. Our main goal was to provide a first systematic study of trimodal alignment in contrastive spaces involving time series, vision and language, rather than a definitive characterization of all possible settings. Our analysis focuses on univariate time series, and it remains an open question whether the alignment behaviors observed here extend to multivariate signals. CaTS captions are synthetic, which may not fully capture the diversity of human writing styles. We partially address this by evaluating on CaTS human-written, directly descriptive captions but is limited in scale. While we also test on real clinical text from MIMIC and PTB-XL, these datasets are domain-specific and rely on indirect language, which constrains their generalization for studying direct textual grounding. Also to ensure controlled and fair comparison across a large number of encoder combinations, we adopt a fixed contrastive training protocol with frozen encoders and train projection heads. This design isolates intrinsic representational compatibility across modalities, but precludes studying how end-to-end fine-tuning or modality-specific adaptation might further shape alignment. Exploring such training regimes would be a natural extension, but doing so exhaustively across all configurations considered in this work would be computationally prohibitive. An important direction for future work is to investigate how the alignment phenomena that we identify correlate with downstream task performance in multimodal time series applications. However, large-scale datasets that simultaneously include aligned time series, visual representations, and natural language supervision remain scarce, which limits systematic evaluation of task-level transfer. We hope that our findings highlight the need for richer multimodal time series datasets and stimulate further development, helping in more studies of how representational alignment in this trimodal setting translates into practical performance gains. Appendix B Continued Related Work A long-standing theme across machine learning and computational neuroscience is that learned representations can be meaningfully compared across models, layers, training runs, and even across fundamentally different modalities. Early work in neuroscience formalized this idea through representational similarity analysis (RSA), which compares the geometry of activation patterns via representational (dis)similarity matrices rather than attempting neuron-to-neuron correspondence ( 23 ) . This perspective has strongly influenced modern deep learning research, where alignment is typically operationalized as the extent to which two embedding spaces share structure up to simple transformations. Though a central challenge in measuring representational similarity is that different models may implement equivalent functions using different bases, permutations, or scalings which makes direct coordinate-wise comparison ill-defined. Canonical correlation analysis (CCA) and its variants address this by comparing subspaces rather than individual units, motivating widely used tools such as SVCCA ( 37 ) and projection-weighted CCA (PWCCA) ( 32 ) . CKA further refines this approach by providing a similarity index that is stable across random initializations and closely connected to CCA while importantly remaining computationally tractable ( 22 ) . Another common metric of alignment is Procrustes analysis, which measures how well two representation sets can be matched via an optimal orthogonal transformation, with classical roots in the orthogonal Procrustes problem ( 38 ) . More recent work has begun to systematically benchmark and compare these similarity measures across architectures, datasets, and modalities, demonstrating that different metrics emphasize different invariances ( 21 ; 42 ) . Beyond measurement, lots of works explicitly seeks to induce alignment through training objectives. Early multiview learning methods such as deep canonical correlation analysis (DCCA) learn nonlinear transformations of two views whose outputs are maximally correlated ( 2 ) , with extensions to more than two views enabling multiway alignment through DGCCA ( 5 ) . In modern models, cross modal alignment is most commonly induced via contrastive learning objectives. CLIP ( 35 ) and ALIGN ( 20 ) demonstrated that large-scale image–text contrastive training can produce shared embedding spaces with strong zero-shot transfer properties, while distillation-based methods such as ALBEF improve robustness and alignment quality under noisy supervision ( 28 ) . Theoretical and empirical analyses of contrastive learning characterize these objectives as balancing alignment between positive pairs and uniformity of the representation distribution ( 50 ) . While large-scale contrastive training can induce shared representation spaces across modalities, less is known about the geometry, asymmetry, and limits of such alignment particularly when modalities have fundamentally different inductive biases such as time series. Contrastive alignment has been extended beyond image–text to audio–text ( 10 ) , image–text–audio ( 14 ) , video–text ( 53 ) , time series-text ( 6 ) , and 3D–text ( 15 ) . More unified frameworks align multiple modalities within a shared embedding space, including VATT, which jointly learns from raw video, audio, and text ( 1 ) , and ImageBind ( 11 ) , extends a VL backbone to modalities via image-paired training. Appendix C Datasets We evaluate alignment using four datasets which we use to isolate the roles of semantic explicitness, visual mediation, and temporal complexity. All datasets are processed into aligned triplets of time series , visual plots , and text , with consistent train–test splits and identical preprocessing across modalities. Because naturally occurring trimodal datasets are extremely scarce for the three modalities that we study, we complement available data with carefully generated synthetic and human-authored modalities to complete missing components wherever required. C.1 CaTS CaTS is a multimodal dataset and benchmark which we use to study how numeric time series align with vision and language under controlled direct semantic conditions. Each sample consists of an univariate time series, a rendered plot of that series, and a natural language caption that describes observable temporal structure. The dataset spans 11 domains, including agriculture, air quality, border crossings, COVID statistics, crime, demography, diet, online retail, road injuries, CO 2 , and Walmart sales. Figure 12 shows a representative CaTS triplet. We use the original CaTS training split (16k) for all training experiments. Evaluation is performed on held-out test sets for each caption variant, with a fixed validation set of 1k samples. All variants share identical time series and plots, differing only in textual descriptions. Time Series 83.65, 81.28, 86.98, 93.55, 96.89, 100.21, 101.88, 100.0, 103.96, 109.67 Plot Text India’s Total Factor Productivity index demonstrates consistent upward momentum from 2008 to 2017. Starting at 83.65 in 2008, the index experiences an initial dip to 81.28 in 2009 before steadily climbing to a peak of 109.67 in 2017, a 31% increase from the initial value …The overall pattern reveals sustained growth with minimal volatility following the initial 2009 recovery. Figure 12 : Example CaTS triplet consisting of a numeric time series, its visual plot, and a reference caption. Figure 13 : KDEs over projected embedding values and caption examples across CaTS variants with increasing ID. To study the effect of semantic expressiveness, we generate multiple caption ID variants derived from the same underlying time series, plot, and metadata. Figure 13 illustrates distributional shifts in caption embeddings, visualized as kernel density estimates (KDEs) over projected embedding values. Starting from the original captions, we create progressively condensed versions containing approximately one phrase, two phrases, or four phrases. The condensed caption variants are generated only for evaluation on test samples. All variants are generated using a fixed protocol that preserves numeric fidelity while controlling verbosity. Caption extraction/summarization are performed using two LLMs, GPT-4o-mini and LLaMA-3.2-90B, with identical instructions applied across all samples. For these condensed variants, we instruct the language models to compress the original caption to a target number of semantic phrases, prioritizing the most salient temporal patterns, trends, and numeric attributes while discarding secondary details. This phrase-level constraint enables controlled reduction of semantic explicitness without altering the underlying grounding or introducing new information. All synthetically generated caption variants are validated following the same verification protocol used in the original CaTS construction. Next to study whether substantially increasing semantic richness during training can further improve multimodal alignment, we construct a High-ID variant of CaTS for both training and test splits. This dataset preserves the original time series and visual plots and we only modify the textual modality. High-ID captions are designed to be significantly more detailed than the original captions while maintaining strict numeric fidelity. Captions are generated independently for each sample using large language models (GPT-4o-mini and LLaMA-3.2-90B). Each model receives only the sample-specific metadata, and raw time series values. All captions are generated with low temperature to ensure determinism and consistency and all captions are generated using the following fixed system prompt, which explicitly enforces exhaustive numerical reporting, pattern and phase-wise analysis, and strict grounding in the provided data. The resulting High-ID captions are substantially longer and more semantically dense than the original captions, exceeding them by more than a factor of two in measured ID. ⬇ Write an EXTREMELY detailed and comprehensive analysis of this time series data . Time Series : < ts > Metadata : < metadata > Your caption should include MOST of the following in 5-7 sentences : 1. CONTEXT : Full description of what this data represents , including location , measurement type , and exact time period covered 2. STATISTICS : Exact numerical values for mean , minimum , maximum , and standard deviation , with explicit comparisons to historical norms 3. OVERALL TREND : Primary direction and magnitude of change across the entire series 4. PHASE ANALYSIS : Break the series into distinct phases / segments , describe each with specific values and date ranges 5. PATTERNS : Any cyclical , seasonal , or periodic behavior with specific period lengths 6. ANOMALIES : Any unusual spikes , drops , or deviations with exact positions in the timeline and their values 7. COMPARISONS : Explicit numerical comparisons of this series to historical / global averages 8. TEMPORAL DETAILS : Specific dates , years , months , or time ranges for all key events and transitions CRITICAL RULES : - Be exhaustive and thorough - Include specific numbers for EVERY claim - Do not use speculative language - only describe what is evident in the data - Do not summarize briefly - provide full detail - Minimum 150 words - REMEMBER NOT ALL INFORMATION IS NEEDED TO BE PRESENT IF IT IS NOT EVIDENT IN THE DATA . C.2 TRUCE Figure 14 : Representative TRUCE visual variants: generic, styled, and annotated plots. TRUCE is a controlled dataset which we use to isolate the effect of visual mediation on time series alignment. It consists of short, fixed-length univariate time series paired with concise direct textual descriptions. All time series have a length of 12 timesteps which enables exact visual annotation of every point. The training split consists 1,968 samples and the test split consists 492 samples. Representative TRUCE captions include “Increases linearly throughout the entirety of the graph,” “Steady increase throughout,” and “Troughs near the beginning.” Captions are short (approximately 85 characters on average) and directly describe local temporal structure, patterns, and trends. For each time series, we generate multiple visual variations as illustrated in Figure 14 : (a) a generic plot showing only the raw time-series curve, with no axes, annotations, or semantic markers, (b) a styled plot with randomized visual properties (line styles, colors, markers), and (c) an annotated plot in which every time step is labeled with its numeric value and has styling. C.3 MIMIC-IV ECG The MIMIC-IV ECG dataset provides real-world clinical time series paired with unstructured diagnostic reports. Each sample consists of a full-length ECG waveform and an associated clinical note that does not explicitly describe waveform structure. We use 21,000 samples with non-empty reports and valid waveform files. Time series is of 5,000 steps per record. Waveforms are rendered as line plots with axes and units. Figure 15 illustrates a triplet from MIMIC. We randomly shuffle the dataset and split it into 16,000 training samples and 4,000 test samples, with a fixed 1k validation set. All splits are disjoint at the subject–study level. Clinical reports are concatenated across available fields and used verbatim, without summarization or restructuring. As a result, the text modality provides indirect semantic supervision, focusing on diagnoses and observations rather than explicit temporal descriptions. MIMIC also helps in analysis of alignment when time series are substantially longer than those in CaTS or TRUCE. Time Series 0.02, 0.02, 0.015, 0.015, 0.015, 0.02, 0.02, 0.015, 0.01, 0.01, 0.015, 0.02, 0.01, 0.0, 0.01, 0.015,  … 0.015, 0.01, -0.01, -0.01, -0.01, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -0.01, -0.015, -0.015, -0.015 Plot Text Atrial fibrillation with rapid ventricular response Inferior/lateral ST-T changes are nonspecific Low QRS voltages in limb leads Abnormal ECG Figure 15 : Example MIMIC triplet consisting of an ECG time series, its waveform visualization, and a report. C.4 PTB-XL PTB-XL is a another ECG dataset which we use with standardized waveform recordings and diagnostic reports written primarily in German. Each recording is a fixed-length 10-second ECG sampled at 500 Hz, giving time series of length 5,000. We select samples with non-empty reports and valid waveform files, resulting in 21,000 aligned triplets. As with MIMIC, we render the full waveform with axes and units. Reports are used verbatim and remain in German, introducing a multilingual setting. The text modality again provides indirect semantic supervision, referring to clinical conditions rather than explicitly describing waveform structure. PTB-XL allows us to disentangle the effects of indirect grounding from language mismatch, which complements MIMIC by holding the signal structure constant while varying linguistic properties. For PTB-XL, we evaluate a subset of configurations that use Qwen embedding models for the text modality. Qwen embeddings provide multilingual support, which is appropriate for the multilingual clinical text found in PTB-XL. Time Series -0.035, -0.035, -0.035, -0.035, -0.035, -0.035, -0.035, -0.035, -0.035, -0.035, -0.035, -0.031, -0.03, -0.03, -0.03, -0.03, -0.03, -0.027, -0.024, … 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12 Plot Text sinusrhythmus ueberdrehter linkstyp rechtsschenkelblock Figure 16 : Example PTB-XL triplet consisting of an ECG time series, its waveform visualization, and a report. Appendix D Encoders and Trimodal Combinations Table 3 : Triplet model combinations used in our experiments (Grouped by Size Bounds) # Vision Encoder Text Encoder Time Series Encoder Lower Bound Models 1 DINOv2-Base (86.6M) T5-Small (60.5M) Moirai (91M) 2 ViT-Base (86.4M) T5-Small (60.5M) MOMENT (125M) 3 DINOv2-Base (86.6M) T5-Small (60.5M) Chronos (201M) 4 DINOv2-Base (86.6M) E5-Base-v2 (109M) TimesFM (200M) 5 ViT-Base (86.4M) E5-Base-v2 (109M) Chronos (201M) 6 SigLIP2-Base (375M) E5-Base-v2 (109M) Moirai (91M) 7 SigLIP2-Base (375M) E5-Base-v2 (109M) MOMENT (125M) 8 SigLIP2-Base (375M) T5-Small (60.5M) TimesFM (200M) 9 ViT-Base (86.4M) Qwen3-Embedding (596M) Moirai (91M) 10 DINOv2-Base (86.6M) Qwen3-Embedding (596M) MOMENT (125M) 11 ViT-Base (86.4M) Qwen3-Embedding (596M) TimesFM (200M) 12 SigLIP2-Base (375M) Qwen3-Embedding (596M) Chronos (201M) Middle Bound Models 13 ViT-Huge (632M) T5 (3B) MOMENT (385M) 14 SigLIP (878M) T5 (3B) TimesFM (500M) 15 DINOv2-Giant (1.14B) T5 (3B) Moirai (311M) 16 DINOv2-Giant (1.14B) T5 (3B) Chronos (709M) 17 ViT-Huge (632M) Qwen3-Embedding (4B) Moirai (311M) 18 ViT-Huge (632M) Qwen3-Embedding (4B) TimesFM (500M) 19 DINOv2-Giant (1.14B) Qwen3-Embedding (4B) MOMENT (385M) 20 SigLIP (878M) Qwen3-Embedding (4B) Chronos (709M) 21 SigLIP (878M) E5-Mistral (7B) Moirai (311M) 22 SigLIP (878M) E5-Mistral (7B) MOMENT (385M) 23 ViT-Huge (632M) E5-Mistral (7B) Chronos (709M) 24 DINOv2-Giant (1.14B) E5-Mistral (7B) TimesFM (500M) Upper Bound Models 25 DINOv3 ViT (7B) Qwen3-Embedding (8B) MOMENT (385M) 26 EVA-CLIP (8B) Qwen3-Embedding (8B) Chronos (709M) 27 DINOv3 ViT (7B) Mistral-Nemo-Base (12B) Chronos (709M) 28 EVA-CLIP (8B) Mistral-Nemo-Base (12B) Moirai (311M) 29 EVA-CLIP (8B) Mistral-Nemo-Base (12B) TimesFM (500M) 30 EVA-CLIP (18B) Qwen3-Embedding (8B) Moirai (311M) 31 EVA-CLIP (18B) Mistral-Nemo-Base (12B) MOMENT (385M) 32 DINOv3 ViT (7B) Gemma-3 (27B) Moirai (311M) 33 EVA-CLIP (8B) Gemma-3 (27B) MOMENT (385M) 34 EVA-CLIP (18B) Gemma-3 (27B) Chronos (709M) Models used. We evaluate a broad and diverse set of pretrained models across vision, language, and time series modalities, covering multiple architectural paradigms, training objectives, and model scales. 1. Vision encoders. We use nine vision encoders spanning model sizes from 86M to 18B parameters. Self-supervised models include DINOv2 ( 33 ) (Base, 86.6M; Giant, 1.14B) and DINOv3 ViT (7B) ( 39 ) . Supervised models include ViT ( 9 ) (Base, 86.4M; Huge, 632M). Vision–language pretrained models include SigLIP ( 55 ) (Large, 878M), SigLIP2 ( 43 ) (Base, 375M), and EVA-CLIP ( 40 ) (8B, 18B). 2. Text encoders. We evaluate nine text encoders with scales ranging from 60M to 27B parameters. These include T5 ( 36 ) (Small, 60.5M; 3B), E5-Base-v2 (109M) and E5-Mistral (7B) ( 48 ; 49 ) , Qwen3-Embedding ( 54 ; 57 ) (596M, 4B, 8B), Mistral-Nemo-Base (12B) ( 31 ) , and Gemma-3 (27B) ( 41 ) . 3. Time series encoders. We use eight pretrained time series encoders with model sizes ranging from under 100M to over 700M parameters. These include TimesFM ( 8 ) (200M, 500M), Chronos ( 3 ) (201M, 709M), MOMENT ( 12 ) (125M, 385M), and Moirai ( 52 ) (91M, 311M). D.1 Representation Extraction. We adopt a consistent and controlled representation extraction protocol across text, vision, and time series models to ensure fair comparisons across modalities and families. All backbone models are kept frozen and used strictly as feature extractors. 1. Text representations. For text models (Qwen, Gemma, and Mistral), we extract token-level hidden states from the final transformer layer and apply mean pooling over valid tokens using the attention mask, yielding a single fixed-dimensional sentence embedding per sample. For embedding-style models (e.g., Qwen Embedding), we use the last-token or sequence-end representation following the model’s recommended pooling strategy. 2. Vision representations. Vision representations are obtained by encoding images through pretrained vision backbones (e.g., EVA-CLIP, DINOv3), using either the model’s native image embedding head or the [CLS] token from the final layer, depending on the architecture. 3. Time series representations. For time series encoders, we extract sequence-level embeddings by mean pooling over temporal hidden states returned by the encoder. In all cases, the resulting modality-specific embeddings are projected into a shared latent space of fixed dimension using trainable projection heads, while the backbone encoders remain frozen. D.2 Representative Configuration Subsets 1. Information density analysis (Figure 6 ). Results are averaged over 9 representative configurations corresponding to rows {31, 29, 32, 34, 14, 22, 3, 9, 2} in Table 3 . 2. VL–TS scaling analysis (Figure 7 ). Configurations labeled A–I in Figure 7 correspond to rows {A: 13, B: 18, C: 23, D: 14, E: 20, F: 22, G: 16, H: 19, I: 24} in Table 3 . 3. Scaling under indirect textual supervision (Figure 8 ). Results are reported over 5 representative configurations {10, 19, 25, 12, 20} in Table 3 . 4. High-ID saturation analysis (Table 1 ). Results are averaged over configurations {10, 20, 12, 5} from Table 3 . Appendix E Metrics We quantify cross-modal alignment using complementary metrics that capture global geometric correspondence, non-linear representational similarity, and local neighborhood consistency. All metrics are computed independently for each modality pair (TS–IMG, TS–TXT, IMG–TXT) using normalized embeddings. To ensure computational efficiency and stability, all geometric metrics are computed on a randomly sampled subset of at most 2000 paired samples when datasets exceed this size, using a fixed seed (42). Cosine is done on full test set. Let \mathbf{X}=\{x_{i}\}_{i=1}^{N} and \mathbf{Y}=\{y_{i}\}_{i=1}^{N} denote paired embeddings from two modalities, where each embedding is \ell_{2} -normalized. E.1 Cosine Similarity Cosine similarity serves as a global semantic alignment metric. For paired samples, we compute the full similarity matrix S=\mathbf{X}\mathbf{Y}^{\top}, (4) where S_{ij}=\langle x_{i},y_{j}\rangle . We report three statistics: Matched \displaystyle=\frac{1}{N}\sum_{i=1}^{N}S_{ii},\quad\text{Mismatched}=\frac{1}{N(N-1)}\sum_{i\neq j}S_{ij},\quad\text{Margin}=\frac{1}{N}\sum_{i=1}^{N}\left(S_{ii}-\frac{1}{N-1}\sum_{j\neq i}S_{ij}\right). (5) The margin reflects how well matched cross-modal pairs are separated from mismatched pairs. Higher values indicate stronger alignment. E.2 Procrustes Disparity (Global Geometry) We compute the normalized Procrustes disparity between embedding spaces. After centering both modalities, \displaystyle\tilde{\mathbf{X}} \displaystyle=\mathbf{X}-\frac{1}{N}\sum_{i}x_{i}, (6) \displaystyle\tilde{\mathbf{Y}} \displaystyle=\mathbf{Y}-\frac{1}{N}\sum_{i}y_{i}, (7) we solve the orthogonal Procrustes problem \min_{\mathbf{R}\in\mathbb{R}^{d\times d},\;\mathbf{R}^{\top}\mathbf{R}=\mathbf{I}}\|\tilde{\mathbf{X}}\mathbf{R}-\tilde{\mathbf{Y}}\|_{F}^{2}. (8) The optimal rotation \mathbf{R} is obtained via the Kabsch solution using singular value decomposition with explicit reflection correction. We report the normalized disparity \text{Disparity}=\frac{\|\tilde{\mathbf{X}}\mathbf{R}-\tilde{\mathbf{Y}}\|_{F}^{2}}{\|\tilde{\mathbf{Y}}\|_{F}^{2}}. (9) Lower values indicate stronger global geometric alignment. Figure 17 : Illustration of Procrustes alignment. Embeddings from one modality are rotated to best match the other under an orthogonal transformation. Normalized residual error measures global geometric mismatch. E.3 CKA (Non-linear Similarity) To capture non-linear representational similarity, we compute CKA using an RBF kernel. Given embeddings \mathbf{X} and \mathbf{Y} , we compute \displaystyle K_{ij} \displaystyle=\exp\left(-\gamma\|x_{i}-x_{j}\|^{2}\right), (10) \displaystyle L_{ij} \displaystyle=\exp\left(-\gamma\|y_{i}-y_{j}\|^{2}\right). (11) The bandwidth \gamma is selected using the median heuristic over pairwise distances from both modalities. Kernels are centered using \tilde{K}=HKH,\quad\tilde{L}=HLH, (12) where H=I-\frac{1}{N}\mathbf{1}\mathbf{1}^{\top} and where \mathbf{1}\in\mathbb{R}^{N} denotes the all-ones vector. CKA is then computed as \mathrm{CKA}(K,L)=\frac{\langle\tilde{K},\tilde{L}\rangle_{F}}{\|\tilde{K}\|_{F}\|\tilde{L}\|_{F}}. (13) Values closer to 1 indicate highly similar non-linear representational structure. E.4 Mutual k -Nearest Neighbors (Local Structure) To evaluate local neighborhood consistency, we compute mutual k -nearest neighbor overlap. For each embedding x_{i} , let \mathcal{N}_{k}^{X}(i) denote its k nearest neighbors in \mathbf{X} under cosine distance, excluding itself. Similarly define \mathcal{N}_{k}^{Y}(i) for \mathbf{Y} . The mutual k NN score is \text{Mutual-$k$NN}=\frac{1}{Nk}\sum_{i=1}^{N}\left|\mathcal{N}_{k}^{X}(i)\cap\mathcal{N}_{k}^{Y}(i)\right|. (14) This metric captures local neighborhood agreement across modalities. Higher values indicate stronger preservation of neighborhood structure. We use k=5 in all computations as want to preserve the results on local structure and increasing it more would turn it into a global geometry metric. Figure 18 : Illustration of Mutual k NN alignment. Mutual k NN measures local neighborhood agreement across modalities. Points that share neighbors in both spaces contribute to higher scores. E.5 Cross-modal retrieval. For each modality pair (TS–IMG, TS–TXT, IMG–TXT), we evaluate cross-modal retrieval by treating embeddings from one modality as queries and ranking all embeddings from the paired modality over the full evaluation set using cosine similarity in the shared embedding space. For a dataset of N paired samples, the correct match for query i is defined as the paired sample at index i . Recall@ k (R@1, R@5, R@10) is computed as the fraction of queries for which the correct match appears within the top- k ranked candidates. Retrieval is evaluated in both directions for each modality pair (e.g., TS \rightarrow IMG and IMG \rightarrow TS), with rankings computed independently per direction. No thresholds or learned classifiers are used; the metrics depend solely on the rank order induced by cosine similarity, which is invariant to any positive temperature scaling. Appendix F Learned Convergence Without External Coupling Before doing post-hoc alignment, we first understand whether representations learned independently within each modality already exhibit convergent structure, as suggested by PRH for vision and language models, as an initial motivation. We extend this diagnostic to a trimodal setting involving time series, vision, and language, and evaluate representational geometry in the absence of any external coupling. For each dataset for their test sets, we extract embeddings from independently pretrained encoders for time series, images, and text. No projection heads are trained and no contrastive or alignment objectives are applied. We then compare geometry directly using mean angular deviation (MAD) and cosine similarity. Figure 19 : Mean angular deviation between modality pairs across datasets and model configurations. Values remain near 90^{\circ} for all pairs, indicating near-orthogonal geometry and a lack of inherent cross-modal convergence in independently pretrained encoders. Figure 19 reports mean angular deviation between modality pairs across 9 representative trimodal configurations which we used in VL-TS experiment (A–I) (see Appendix D.2 ) and four datasets. Across all datasets and modality pairs, MAD values remain close to 90^{\circ} , indicating near-orthogonal geometry. This behavior is consistent across domains, including cases where representations are deterministic renderings of the underlying signal (CaTS, TRUCE) and cases where text is indirectly related to the signal (PTB-XL, MIMIC). These results indicate that independently pretrained encoders do not exhibit inherent geometric convergence across these datasets used. Figure 20 visualizes cross-modal cosine similarity matrices computed over matched samples. Off-diagonal similarities remain uniformly close to zero across all datasets, with no discernible separation between matched and mismatched pairs. This also confirms that the lack of alignment is not merely a matter of scale but reflects the absence of latent cross-modal correspondence in the embedding spaces. Notably, TS–IMG similarity also remains close to zero in this uncoupled setting, although it is consistently higher than other pairs. Figure 20 : Cross-modal cosine similarity matrices for matched samples across datasets. Off-diagonal similarities remain near zero with no separation between matched and mismatched pairs, confirming the absence of latent alignment without explicit coupling. These diagnostics establish a clear baseline: independently pretrained encoders for time series, vision, and language do not spontaneously converge to a shared representational geometry for different modality projections of time series data on the datasets studied here. This motivates the contrastive alignment experiments in the paper, where we introduce trainable projection heads to study how post-hoc alignment is induced. Appendix G Experimental Details, Ablations, Loss Functions In this section, we describe the experimental protocol used to train and evaluate our models, including optimization settings, batching and precision choices, data handling, and the contrastive objective. We additionally present targeted ablations to assess the sensitivity of alignment geometry to optimization hyperparameters, training duration, and projection head design. G.1 Optimization, Batching and Precision Models are trained for 50 epochs using AdamW with learning rate 1\times 10^{-4} and weight decay 1\times 10^{-5} . A cosine decay schedule with linear warmup (2% of total steps) is applied. A fixed temperature of 0.2 is used throughout. Gradient clipping with maximum norm 1.0 is used to stabilize training. Early stopping with patience 5 is applied uniformly. All experiments use an effective batch size of 256, obtained via gradient accumulation when necessary. Mixed-precision (bfloat16) is used for very large encoders. All hyperparameters are held constant across datasets and model combinations. G.2 Data Handling Time series keep their original length and no truncation is applied. Images are resized to 384 \times 384 pixels; Text inputs use a maximum length of 512 tokens; all samples fall within this limit, so full text is retained across all variants. G.3 Loss Let a minibatch consist of N aligned triplets \{(z_{\text{ts}}^{(i)},z_{\text{img}}^{(i)},z_{\text{txt}}^{(i)})\}_{i=1}^{N} , where all embeddings are \ell_{2} -normalized and lie in \mathbb{R}^{d} . Similarity is measured using the dot product s(a,b)=a^{\top}b and scaled by a temperature parameter \tau>0 . For a generic modality pair (x,y) , the forward contrastive loss x\rightarrow y is defined as \mathcal{L}_{x\rightarrow y}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp\!\left(s(z_{x}^{(i)},z_{y}^{(i)})/\tau\right)}{\sum_{j=1}^{N}\exp\!\left(s(z_{x}^{(i)},z_{y}^{(j)})/\tau\right)} (15) The reverse direction y\rightarrow x is defined analogously: \mathcal{L}_{y\rightarrow x}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp\!\left(s(z_{y}^{(i)},z_{x}^{(i)})/\tau\right)}{\sum_{j=1}^{N}\exp\!\left(s(z_{y}^{(i)},z_{x}^{(j)})/\tau\right)}. (16) The symmetric bidirectional loss for modality pair (x,y) is then \mathcal{L}_{x\text{-}y}=\frac{1}{2}\left(\mathcal{L}_{x\rightarrow y}+\mathcal{L}_{y\rightarrow x}\right). (17) In our trimodal setting, we compute losses for all three modality pairs: time series–image (TS–IMG), time series–text (TS–TXT), and image–text (IMG–TXT). The final training objective is the equally weighted sum: \mathcal{L}_{\text{total}}=\mathcal{L}_{\text{ts-img}}+\mathcal{L}_{\text{ts-txt}}+\mathcal{L}_{\text{img-txt}}. (18) In all cases, negative examples are implicitly provided by other samples within the same minibatch, i.e., each non-matching pair (z_{x}^{(i)},z_{y}^{(j)}) for j\neq i is treated as a negative. No external negative mining or memory bank is used. Equal Weighting of Modality Pairs in the Loss. We apply equal weights to all modality pairs in the trimodal contrastive objective. This choice reflects our goal of studying intrinsic representational compatibility rather than optimizing for a downstream task. Reweighting individual losses would introduce assumptions about the relative importance of different modality relationships, which would directly shape the geometry learned during training. In our setting, we treat each modality as a peer projection of a shared latent process, and no modality pair is favored a priori . There is therefore no principled basis for assigning greater weight to one interaction over another. For example, increasing the weight of TS–TXT would implicitly assume that linguistic alignment is more semantically important, while downweighting TS–IMG could assume that visual renderings are easier or partially redundant. Such choices would be arbitrary and closely tied to the hypothesis under study. Using equal weights provides the most neutral formulation, allowing cross-modal structure to emerge from the representations themselves rather than being imposed through optimization choices. Notably, the observed asymmetry between TS–IMG and TS–TXT alignment persists across model scales, encoder families, and text richness despite symmetric treatment in the loss. This suggests that the asymmetry reflects underlying representational compatibility. Exploring adaptive or modality-specific weighting is a natural direction for task-oriented settings for multimodal time series tasks, but is outside the scope of our goal. G.4 Compute Training cost varies substantially with model scale and the number of trainable parameters. For lower-bound configurations, each triplet combination required approximately 6–10 hours of training, while middle- and upper-bound configurations required between 1 and 3 days per run. All experiments were executed using 8 NVIDIA A100 GPUs on a shared high-performance computing node with sufficient CPU and memory resources to support large-scale multimodal training. These runtimes reflect the cost of training projection heads with frozen encoders. G.5 Setting Ablations We assess the sensitivity of trimodal alignment to contrastive optimization by comparing two configurations: b64 (batch size 64, temperature 0.7, 512-dimensional projection) and b256 (batch size 256, temperature 0.2, 1024-dimensional projection). We find that the b256 configuration consistently gives stronger alignment across modality pairs. These gains reflect the combined effects of larger negative sets, sharper contrastive separation, and higher-capacity projection heads. Differences are most pronounced in global geometric metrics: Procrustes disparity remains low for b256 (approximately 0.45) but degrades substantially under b64 (above 0.70). Epoch Ablations. We study the effect of training duration by performing an epoch ablation on a representative configuration, training the projection head for up to 100 epochs while keeping all encoders frozen. As summarized in Table 4 , metrics improve rapidly during early training and largely stabilize by 40–50 epochs. Extending training beyond this point yields no consistent improvements and in some cases leads to slight degradation in alignment which shows diminishing returns once cross-modal correspondences are established. This suggests that prolonged optimization of the projection head can introduce geometric drift without improving semantic alignment. Based on these observations, we adopt a fixed training budget of 50 epochs for all experiments which helps us to balance computational efficiency with stable and robust alignment. Table 4 : Epoch ablation on vit_h14 + t5 + moment-l . Metrics are averaged across all modality pairs. Epochs Cosine Margin \uparrow Procrustes \downarrow CKA \uparrow Mutual KNN \uparrow 50 0.676 0.565 0.658 0.103 100 0.671 0.577 0.637 0.104 G.6 Projection Head Ablations We study the effect of projection head design on alignment by comparing a simple baseline projection head with a more expressive variant, which we use as our default throughout the paper. The baseline head consists of a shallow two-layer MLP with a ReLU nonlinearity, similar to very early contrastive learning setups in the field. Given an encoder output x , the improved projection head is defined as: \displaystyle h(x) \displaystyle=\mathrm{LayerNorm}(W_{1}x+b_{1}), (19) \displaystyle u(x) \displaystyle=\mathrm{Dropout}(\mathrm{GELU}(h(x))), \displaystyle\mathrm{Proj}(x) \displaystyle=W_{2}u(x)+b_{2}. where W_{1}\in\mathbb{R}^{m\times d_{\text{in}}} , W_{2}\in\mathbb{R}^{d\times m} , m=\max(3d,768) , and d=1024 is the shared embedding dimension. The improved projection head increases capacity by introducing an expanded hidden layer, layer normalization, GELU activation, and dropout which provides more flexible feature re-mapping while keeping all encoders frozen. As shown in Figure 21 , the improved projection head gives substantial gains in cosine similarity margins across all modality pairs. At the same time, Procrustes disparity is uniformly reduced, indicating that the learned embedding spaces are not only closer in a pointwise sense but also better aligned geometrically. These improvements hold across different families and combinations. Importantly, while the improved head strengthens alignment, the relative ordering between modality pairs remains stable, indicating that projection head design improves alignment quality without altering the underlying cross-modal asymmetries. Figure 21 : Effect of projection head design on trimodal alignment. Comparison of baseline and improved projection heads across six model configurations. The top row shows cosine similarity margins (higher is better) for TS–IMG, TS–TXT, and IMG–TXT alignment, while the bottom row reports Procrustes disparity (lower is better). We additionally experimented with deeper projection head variants that further increase depth and nonlinearity beyond the improved design (e.g., additional hidden layers and normalization), but did not observe consistent gains over the reported configuration. In most cases, alignment metrics saturated or showed marginal improvements at the cost of increased instability and sensitivity to hyperparameters. As a result, we adopt the improved projection head as a principled trade-off between capacity and robustness. We leave a more thorough exploration of end-to-end encoder fine-tuning or parameter-efficient adaptation to future work. While such approaches may further improve cross-modal alignment for our trimodal setting, they substantially increase computational cost and complicate fair comparison across heterogeneous encoder families, which is a central goal of this study. Appendix H Information Density We define information density as the total surprisal of a caption under a fixed pretrained language model. Given a caption x=(w_{1},\dots,w_{T}) , we compute the per-token surprisal \overline{\ell}(x)=\frac{1}{T}\sum_{t=1}^{T}-\log p(w_{t}\mid w_{