Do VLMs Truly "Read" Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting CCS: Applied computing Forecasting
Kaiqi Hu, Linda Xiao, Shiyue Xu, Ziyi Tang, Mingwen Liu · 2026
Read Do VLMs Truly "Read" Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting CCS: Applied computing Forecasting by Kaiqi Hu, Linda Xiao, Shiyue Xu, Ziyi Tang, Mingwen Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12659.txt.
Introduction
Kaiqi Hu and four colleagues benchmark vision-language models on candlestick charts. Prior tests fail to isolate whether visual understanding truly aids price forecasts. Most setups use single-period or tabular inputs. Analysts read multi-scale charts for trend and turning points. The team builds a multi-scale dataset with a standard evaluation framework. Scoring mixes confusion matrices with information coefficient series. Most models shine only in persistent uptrends or downtrends. They show weak prediction in common scenarios, with bias and poor horizon sensitivity.
Catalog record
| CREATOR | Kaiqi Hu, Linda Xiao, Shiyue Xu, Ziyi Tang, Mingwen Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43007 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.12659
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Do VLMs Truly "Read" Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting CCS: Applied computing Forecasting Kaiqi Hu, Linda Xiao, Shiyue Xu, Ziyi Tang, Mingwen Liu Abstract . Vision–language models (VLMs) are increasingly applied to visual stock price forecasting, yet existing benchmarks inadequately evaluate their understanding of stock price in candlestick charts. First, prior studies fail to isolate VLMs’ comprehension of visual stock price inputs, and it remains unclear whether visual inputs genuinely improve predictive performance and whether VLMs truly comprehend candlestick patterns. Further, most existing datasets and evaluation setups are designed around single-period or tabular inputs. However, human analysts strongly rely on multi-scale candlestick charts, where longer-term horizons capture trend direction and shorter-term horizons provide cues for inflection points, making it difficult to systematically assess VLMs’ ability to integrate short-term and long-term visual market dynamics. To bridge this gap, we construct a multi-scale candlestick chart dataset and a standardized evaluation framework to assess VLMs’ ability to utilize multi-scale visual market signals. Evaluation combines confusion-matrix-based diagnostics with information coefficient (IC) time series metrics and includes XGBoost as a feature-based temporal baseline. Using this dataset, we benchmark representative VLMs and analyze their ability to leverage multi-scale stock price data. Experimental results show that most VLMs perform well only under persistent uptrend or downtrend conditions, while exhibiting weak predictive capability in more common market scenarios. We also identify significant prediction biases and limited sensitivity to explicitly specified forecast horizons in prompts, indicating inherent limitations in precise temporal reasoning. Figure 1. Pipeline for constructing visual inputs and evaluation. 1. Introduction With the rapid advancement of artificial intelligence, Vision-Language Models (VLMs) have demonstrated significant potential in financial applications, particularly in stock price forecasting ( 10 ) . Candlestick charts, as a core analysis tool, visually encapsulate rich market dynamics and price patterns ( 2 ) ( 12 ) ( 13 ) ( 16 ) , making chart-based prediction a key VLMs’ application in finance. However, existing evaluation benchmarks exhibit critical limitations in assessing VLMs’ genuine comprehension of stock price information. Current research faces a fundamental methodological challenge: dataset complexity hinders capability attribution analysis. Mainstream studies employ highly heterogeneous multimodal datasets combining visual and textual content ( 8 ) ( 23 ) . While mimicking real trading environments, this introduces critical flaws: unclear capability attribution when achieving strong performance, we cannot determine whether models truly “understand” chart patterns or primarily rely on textual signals ( 10 ) . Most studies lack rigorous ablation experiments to isolate modality contributions, preventing quantification of visual inputs’ marginal value. Furthermore, existing studies neglect the multi-time-scale analysis methodology central to technical analysis ( 10 ) , thereby limiting the comprehensive assessment of VLMs’ capabilities. Professional traders systematically integrate multiple timeframes ( 11 ) : long-term frames reveal macro trends forming strategic backdrops ( 6 ) ( 11 ) , while short-term frames provide precise trading signals, breakouts, momentum shifts, and inflection points ( 18 ) ( 25 ) . This hierarchical validation represents a critical cognitive advantage validated through decades of market practice. To address these limitations, this study proposes a systematic solution minimizing information interference while preserving essential analytical elements. Specifically, our contributions are: • We construct dataset samples that incorporate both daily and weekly candlestick charts with forecast targets, forcing VLMs to extract information purely from visual patterns and eliminating multimodal confounding bias. • We employ a multidimensional evaluation framework that goes beyond traditional information coefficient (IC) metrics by introducing dual-dimensional confusion matrix analysis, which quantifies VLMs’ behavioral characteristics across different market states and stock attributes. • We conduct fine-grained analyses examining predictive biases under varying conditions, including false positive and false negative distributions as well as trend capture abilities. This framework provides methodological support for understanding multimodal prediction mechanisms and offers valuable insights for trading strategy optimization. 2. Related Work 2.1. Traditional Stock Forecasting Methods Stock price forecasting has long been central to financial research. Traditional statistical models, such as ARIMA, demonstrate potential for short-term predictions ( 9 ) but struggle with complex nonlinear relationships. Machine learning approaches enhance performance through feature engineering: Wu et al. showed XGBoost achieves high accuracy ( 22 ) , while Sheng et al.’s multimodal LightGBM reached 75.85% balanced accuracy ( 19 ) , though susceptible to overfitting. Deep learning opened new possibilities, with Yu et al. demonstrating ARIMA’s trend forecasting utility ( 24 ) and Nichani et al. validating hybrid ARIMA-LSTM superiority ( 15 ) . Image-based approaches treat candlestick charts as visual inputs: Sim et al. found CNNs enhance learning through convolution and pooling ( 21 ) , Chootong et al. integrated neural networks with technical indicators and chart patterns for trading signals ( 2 ) , and Kusuma et al. demonstrated CNNs uncover hidden patterns within candlestick images, achieving peak performance across multiple metrics ( 12 ) .However, these methods require data or structured image processing and cannot extract effective information directly from candlestick charts for prediction like human traders do. 2.2. Image Applications and Challenges In recent years, financial images have been widely applied to forecasting tasks. However, existing research predominantly employs highly complex multimodal datasets, hindering capability attribution analysis. Typical studies combine images with multiple information sources: Karadaş et al. incorporated tweet metrics (likes, retweets, comments, follower counts) into feature extraction ( 8 ) ; Xu et al. provided models with pure time series, news sentiment, image trends, and fundamental tables ( 23 ) ; Huang et al. fed FinLLaMA with financial domain knowledge (papers, reports) alongside images, achieving 55.73% cumulative return and 2.45 Sharpe ratio on TSLA ( 5 ) ; Prado et al. visually encoded candlestick charts and identified 16 classic patterns through mathematical rules, finding statistical significance in some patterns ( 17 ) . While achieving high accuracy, these studies suffer from a fundamental flaw: the inability to determine whether performance stems from visual understanding or textual information. Critically, most lack ablation experiments isolating modality contributions, leaving visual comprehension capabilities a “black box”. 2.3. Multi-Time-Scale Analysis Methods Multi-time-scale analysis constitutes technical analysis’s core methodology. Classical Dow Theory distinguishes primary trends, secondary trends, and short-term fluctuations ( 4 ) ; Elliott Wave Theory depicts nested market structures ( 1 ) . Modern systems adopt multi-period confirmation mechanisms like the “three-screen trading system” requiring validation across timeframes ( 7 ) . In deep learning, multi-scale modeling has emerged: Xu et al. provided minute, daily, and quarterly resolutions to capture short-term fluctuations, medium-term trends, and long-term fundamentals ( 23 ) , though primarily targeting tabular data rather than visual candlestick charts. Existing research lacks a systematic evaluation of VLMs’ visual integration of multi-period charts. Shu et al. constructed time-series datasets by stitching ten consecutive years of financial charts ( 20 ) , yet these experiments failed to analyze whether models truly learned hierarchical reasoning, grasping long-term trends before capturing short-term opportunities, or merely identified stronger statistical correlations in certain periods. 3. Dataset This section describes our process for constructing a multi-scale candlestick chart dataset. The raw market data are obtained from two publicly accessible financial data sources. Historical data for the Chinese market are collected from TuShare, which provides structured daily OHLCV records for A-share stocks in China. Using this interface, we collect data for constituent