News

Press Releases

Theoretical Analysis of How Learning Processes Are Compressed into a Small Number of Synthetic Data Points
－The extraction of generalizable information and its efficient replay－

A joint research team led by Yuri Kinoshita, a trainee at the Laboratory for Neural Computation and Adaptation, RIKEN Center for Brain Science, and a doctoral student in the Department of Mathematical Informatics, Graduate School of Information Science and Technology, The University of Tokyo; Taro Toyoizumi, Team Director at RIKEN and Project Professor in the Department of Mathematical Informatics, Graduate School of Information Science and Technology, The University of Tokyo; and Naoki Nishikawa, a doctoral student in the Department of Mathematical Informatics, Graduate School of Information Science and Technology, The University of Tokyo, has theoretically analyzed a mechanism by which information acquired during the learning process can be efficiently compressed into a small number of synthetic data points and replayed as a form of learned memory.

These findings are expected to contribute to the development of methods for reducing the costs of training artificial intelligence (AI), as well as the costs of storing and transmitting training data that are typically large-scale. They may also help advance our understanding of efficient ways to compress knowledge acquired during learning into reusable memory.

In machine learning, dataset distillation has recently attracted increasing attention as a learning-aware data compression technique that compresses large training datasets into a small set of synthetic data points. In this study, the research team theoretically analyzed how dataset distillation extracts the low-dimensional structure that represents the essence of a learning task.

Focusing on mathematical models in which the input data are high-dimensional but the task-relevant information for prediction depends on only a small number of variables, the team proved that essential information acquired through gradient-based learning can be efficiently encoded into distilled synthetic data. They also showed that a mathematical model retrained using these distilled data can reproduce performance comparable to that of a model trained on the original large-scale dataset.

This research was accepted by the International Conference on Machine Learning 2026 (ICML 2026), a leading international conference in the field of machine learning, and was presented at the conference held in Seoul, South Korea, on July 7, 2026.

Mechanism of dataset distillation and extraction of low-dimensional structure

Conference Information

Conference: International Conference on Machine Learning 2026
Title：Dataset Distillation Efficiently Encodes Low-Dimensional Representations from Gradient-Based Learning of Non-Linear Tasks
Authors： Yuri Kinoshita, Naoki Nishikawa, Taro Toyoizumi

(This English article was translated with the assistance of AI.)