Deep Learning with Pretrained ’Internal World’ Layers: A Gemma 3-Based Modular Architecture for Wildfire Prediction
AYOUB JADOULI 1, CHAKER EL AMRANI 1 · 2025
Read Deep Learning with Pretrained ’Internal World’ Layers: A Gemma 3-Based Modular Architecture for Wildfire Prediction by AYOUB JADOULI 1, CHAKER EL AMRANI 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18562.txt.
Introduction
Ayoub Jadouli and Chaker El Amrani reuse the inner layers of a large AI model to predict wildfires. Their twenty twenty-five paper uses Gemma 3, a multimodal model from Google. Its middle layers hold rich learned knowledge, which the team calls an internal world.
Instead of training a fresh model, they freeze those middle layers and train only small input and output parts. A custom module turns table-style fire data, like weather and terrain, into the form the layers expect. On a Moroccan fire data set, the design beat standard networks on accuracy and held up better. Tests that remove parts confirm the frozen layers do the heavy lifting.
Catalog record
| CREATOR | AYOUB JADOULI 1, CHAKER EL AMRANI 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-18562
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deep Learning with Pretrained ’Internal World’ Layers: A Gemma 3-Based Modular Architecture for Wildfire Prediction AYOUB JADOULI 1, CHAKER EL AMRANI 1 Abstract Deep learning models, especially large Transformers, carry substantial "memory" in their intermediate layers—an internal world that encodes a wealth of relational and contextual knowledge. This work harnesses that internal world for wildfire occurrence prediction by introducing a modular architecture built upon Gemma 3, a state-of-the-art multimodal model. Rather than relying on Gemma 3’s original embedding and positional encoding stacks, we develop a custom feed-forward module that transforms tabular wildfire features into the hidden dimension required by Gemma 3’s mid-layer Transformer blocks. We freeze these Gemma 3 sub-layers—thus preserving their pretrained representation power—while training only the smaller input and output networks. This approach minimizes the number of trainable parameters and reduces the risk of overfitting on limited wildfire data, yet retains the benefits of Gemma 3’s broad knowledge. Evaluations on a Moroccan wildfire dataset demonstrate improved predictive accuracy and robustness compared to standard feed-forward and convolutional baselines. Ablation studies confirm that the frozen Transformer layers consistently contribute to better representations, underscoring the feasibility of reusing large-model mid-layers as a learned internal world. Our findings suggest that strategic modular reuse of pretrained Transformers can enable more data-efficient and interpretable solutions for critical environmental applications such as wildfire risk management. † † history: Date of publication April 15, 2025, date of current version April 15, 2025. This research was financed entirely with the authors’ own funds. † † doi: 10.1109/ACCESS.2025.XXXXXXX † † address: Computer Science and Smart Systems, Faculty of Sciences and Technology, Abdelmalek Essaâdi University, 90000 Tangier, Morocco (e-mail: [email protected]) † † corresponding: Corresponding author: Ayoub Jadouli (e-mail: [email protected]). I Introduction Wildfires are among the most devastating natural disasters, causing severe damage to ecosystems, economies, and human lives worldwide [ 29 , 30 ] . As climate variability increases and human influences grow, accurately forecasting wildfire occurrences has become an urgent global challenge. Traditional fire-risk rating systems, while useful, often fail to capture the intricate interactions between meteorological, ecological, and anthropogenic factors [ 3 , 4 ] . Recent advances in deep neural networks (DNNs) have opened new avenues for modeling these complex interdependencies. In particular, large language models (LLMs) and large vision models (LVMs) have emerged as powerful tools that internalize vast amounts of knowledge—spanning human logic, mathematical principles, and scientific theory—through pretraining on diverse datasets [ 31 , 32 ] . These models develop what can be considered an internal world: a rich repository of latent knowledge and reasoning capabilities embedded within their middle layers. This phenomenon raises an intriguing question: Could we leverage these pretrained "internal worlds" for specialized environmental prediction tasks like wildfire forecasting? Our research has progressively built towards answering this question through a series of studies addressing wildfire prediction in Morocco. Jadouli and El Amrani [ 10 ] first developed an advanced dataset specifically curated from multisource observations in Morocco, addressing local geographical and climatic characteristics. Building on this foundation, they proposed a Parallel Multi-path Feed Forward Neural Network (PMFFNN) [ 11 ] that effectively reduces complexity in long columnar datasets, offering an efficient approach to processing heterogeneous environmental data. This initial work revealed a critical insight: the temporal memory of past environmental states is crucial for accurate wildfire prediction. Subsequent research by Jadouli and El Amrani incorporated physical principles into the deep learning pipeline through a physics-embedded model [ 12 ] that integrates statistical mechanics into neural networks, thereby enhancing the interpretability of wildfire risk assessments. Further refinements demonstrated that integrating multisource spatio-temporal data via ensemble models and transfer learning significantly improves forecast accuracy [ 13 ] , and that bridging physical entropy theory with deep learning provides a hybrid approach that leverages satellite data for more interpretable environmental modeling [ 14 ] . This progressive research path has culminated in a key insight: pretrained models with their rich internal representations could potentially serve as sophisticated "memory modules" for wildfire prediction. While traditional approaches train models from scratch on domain-specific data, they miss the opportunity to leverage the knowledge and reasoning capabilities already embedded in large pretrained models. Our work addresses this gap by introducing a novel architecture that injects the middle layers of the state-of-the-art Gemma 3 model as an "internal world" module within a wildfire prediction network. By carefully designing input projection layers to map tabular wildfire features into Gemma’s hidden representation space, and bypassing its original embedding and positional encoding components, we access the rich knowledge captured in Gemma’s pretrained transformer layers. These frozen layers—which encapsulate extensive knowledge from diverse domains—function as a sophisticated feature processor that enhances wildfire prediction without requiring extensive retraining. This approach not only reduces the number of trainable parameters and mitigates overfitting but also potentially improves model interpretability by grounding predictions in the established knowledge embedded within Gemma’s weights. The primary contributions of this paper are threefold: Architectural Innovation: We propose a novel hybrid architecture that incorporates the middle layers of a large pretrained model (Gemma 3) as a modular "internal world" within a wildfire prediction network, demonstrating a new paradigm for leveraging pretrained knowledge in environmental modeling. Memory-Enhanced Prediction: By exploiting the rich representational capacity of pretrained transformer layers, our model captures complex temporal dependencies and environmental interactions critical for wildfire prediction, addressing the limitations of traditional approaches that lack sophisticated memory mechanisms. Efficient Transfer Learning: We demonstrate that by treating pretrained model layers as fixed knowledge modules, we can achieve superior performance with fewer trainable parameters, providing an efficient alternative to full fine-tuning or training specialized models from scratch. The remainder of this paper is organized as follows. Section II reviews related work in wildfire prediction and the application of transfer learning in environmental modeling. Section III details our methodology, including our approach to integrating Gemma’s "internal world" into the wildfire prediction architecture. Section IV presents experimental results and comparative analyses, and Section V concludes with implications and future research directions. II Related Work This section reviews three key areas relevant to our proposed approach: (1) wildfire prediction methodologies, (2) representational power of pretrained model middle layers, and (3) modular reuse for transfer learning. II-A Wildfire Prediction Methodologies Wildfire prediction methods have evolved from traditional weather index systems to advanced data-driven models. Early operational models like the Canadian Fire Weather Index [ 1 ] and the National Fire Danger Rating System by the U.S. Forest Service [ 2 ] effectively utilized meteorological data but were limited by regional specificity and insufficient ecological interaction modeling [ 3 ] . Recent machine learning approaches integrate diverse datasets to achieve superior performance. Cortez and Morais [ 4 ] first utilized data mining techniques on meteorological variables, while comprehensive reviews like Jain et al. [ 5 ] summarized various machine learning applications. Deep learning, especially Convolutional Neural Networks (CNNs), has significantly advanced wildfire prediction by leveraging spatial information from satellite imagery [ 6 ] . Notably, FireCast [ 7 ] , a CNN-based multimodal system, showcased the predictive power of deep learning. More recently, transformer-based models demonstrated exceptional capabilities by effectively capturing temporal dependencies. Lahrichi et al. [ 8 ] introduced a Transformer-based Swin-Unet model that substantially outperformed conventional methods by utilizing multi-day historical inputs. Zhu