Foundation Models in language, vision, and audio have been a focal point of research in Machine Learning in 2024, with Graph Foundation Models lagging behind in development. However, this post argues that the era of Graph Foundation Models has already begun and provides examples of their potential applications. This post was authored by Michael Galkin and Michael Bronstein with contributions from Jianan Zhao, Haitao Mao, and Zhaocheng Zhu. The emergence of foundation models in graph- and geometric deep learning is explored in the timeline outlined in the post.
A Graph Foundation Model is defined as a single neural model that learns transferable graph representations capable of generalizing to any new, previously unseen graph. The challenges inherent in designing Graph Foundation Models revolve around the diverse nature of graphs in terms of connectivity and feature structure. Existing techniques like Graph Neural Networks (GNNs) struggle with varying feature dimensions and types across different graphs, making the creation of universal representations a complex task. The post discusses the importance of transferable graph representations and the difficulties in achieving this goal.
Several open research questions are highlighted in the realm of graph learning, including generalizing across graphs with heterogeneous features, generalizing across prediction tasks, and determining the optimal model expressivity for different types of graph tasks. The post emphasizes the need for a balance between expressivity, generalization, and optimization in the design of Graph Foundation Models to ensure their effectiveness across a wide range of graph-related tasks.
Examples of existing Graph Foundation Models are discussed, including GraphAny for node classification, UniLP for link prediction, ULTRA for knowledge graph reasoning, and Universal Algorithmic Reasoning models for solving various algorithmic tasks. These models showcase the potential of Graph Foundation Models in different application domains, such as molecular modeling, protein language modeling, and algorithmic reasoning.
The post also delves into the challenges of scaling Graph Foundation Models, especially in the context of data availability and model complexity. It explores the importance of data diversity and sample efficiency in training Graph Foundation Models and discusses the potential use of synthetic and black-box data to enhance model performance. The post concludes by emphasizing the need for further research in scaling laws, data scaling, and model expressivity to advance the field of Graph Foundation Models.