Dynamic Multimodal Data Fusion via Reinforcement Learning for Robust Urban Climate Analysis in GRASS GIS

[yyyi1024]

Dear GRASS GIS Community and Potential Mentors,

I am a first-year graduate student at Tsinghua University (Dept. of Electronic Engineering), and I am highly interested in contributing to GRASS GIS for GSoC 2026.

My research focuses on Urban Computing and Multimodal Learning. I recently developed a framework called AESPA (Physics-Aware Multimodal Urban Heat Mapping), which has been accepted to Web4Good/WWW 2026. This framework utilizes Satellite imagery, Street-view panoramas, and Human mobility data to estimate fine-grained Land Surface Temperature (LST).

My GSoC Vision:
I noticed that the r.learn.ml2 module in GRASS is evolving towards deep learning. I would like to propose an Adaptive Multimodal Reinforcement Learning (RL) approach to upgrade the current urban mapping capabilities:

Adaptive Fusion: Use an RL Agent to dynamically adjust weights between different data modalities (Satellite, OSM, Street-view) based on data quality and availability.

Physics-Aware Regularization: Incorporate physical proxies (Albedo, Canopy, etc.) as constraints within the RL reward function to ensure scientific consistency in geospatial outputs.

Open-Source Integration: Port the AESPA logic into a GRASS-native module (e.g., r.urban.heat.rl) using pygrass and PyTorch.

Why GRASS GIS?
I believe my work on "Physics-aware" AI aligns perfectly with GRASS GIS's tradition of rigorous spatial analysis. I have experience optimizing complex Python/Gurobi tasks (reducing runtime from 47s to 0.6s) and would love to bring this performance-oriented AI approach to the community.

I would appreciate your feedback on:

Whether this "Adaptive Multimodal" direction fits the current roadmap of r.learn.ml2.

Any suggestions on handling multi-source data I/O efficiently within the GRASS environment.

I’ve attached a brief summary of my AESPA framework and would be happy to share more technical details.

Looking forward to hearing from you!

Best regards,

Yuanyi You
GitHub: https://github.com/tsinghua-fib-lab/AESPA
Tsinghua University

[omni-front]

Could you provide more details about how you plan to handle the integration of multisource data within GRASS GIS, particularly regarding the I/O operations? Understanding your approach to managing data from various sources like Satellite, OSM, and Street-view would be helpful. Additionally, are there specific challenges you anticipate with incorporating the RL framework into the existing r.learn.ml2 module? Knowing these details will assist in providing more targeted feedback.

[metzm]

I like the topic, Urban Heat Mapping, and the deep-learning approach using amongst others a vision transformer. According to the github repo, the workflow consists of 3 steps

Data Preparation
This is where GRASS could come in for the integration of multisource data and the creation of harmonized training and inference data. No PyTorch needed here it seems

Teacher Model Training (with Mobility)
PyTorch is needed here, but not GRASS. Training might be run on a separate GPU server, if GRASS is not needed, setup of the server could be a bit easier. Setting up PyTorch correctly on a given server can be difficult enough.

Student Model Distillation (Imagery Only)
As for Teacher Model Training, PyTorch is needed here, but not GRASS.

An additional step could be added:

Data postprocessing
Results could be postprocessed using GRASS, e.g. stitching together the tiles created by inference, removing noise and artifacts or any optional resampling or reprojection.

The easiest solution would be to write as far as possible GRASS modules as wrappers for the existing code base, with the exception of data preparation. This step could be mostly done in GRASS which is not only more memory-efficient than the current numpy approach, but also takes care of spatial referencing.

I do not see how r.learn.ml2 would be of help here. Can you provide some details on this?

[omni-front]

Ah good catch, you're right. It seems GRASS GIS is primarily needed for the data preparation step. Thanks for clarifying!