NTT Research and NTT Scientists Engage at NeurIPS 2024

NTT Research Co-Authored Paper Offering Breakthrough in Learning Dynamics Earns Spotlight Status,
NTT R&D Labs Deliver Five Presentations on Training, Parameters and more

News Highlights:

• Spotlight paper introduces framework to analyze generative models’ learning dynamics
• The findings answer the fundamental question of what determines the concepts a model learns, the order in which it learns them and its ability to manipulate those concepts, revealing that modern models may have deeper capabilities than previously understood
• Five NTT R&D papers address issues involving imbalanced classification, processing time, expressivity of kernel-based functions, data-driven physics simulation and stability of gradient methods.

SUNNYVALE, Calif. & CAMBRIDGE, Mass.--(BUSINESS WIRE)--NTT Research, Inc., a division of NTT (TYO:9432) and NTT R&D, today announced that scientists from their respective organizations are delivering six presentations at this year’s conference on Neural Information Processing Systems (NeurIPS). Conference organizers designated one of those six as a Spotlight presentation, a status awarded to the top 2-3 percent of submitted papers. Researchers from the NTT R&D Computer and Data Science (CD), Communication Science (CS), Social Informatics (SO), and Human Informatics (HI) Labs are delivering five other presentations on papers involving data classification, latent functions, partial differential equations and other topics. A top-tier conference in the field of artificial intelligence, NeurIPS 2024 is being held December 10-15 in Vancouver, British Columbia.

The paper given Spotlight status, titled “Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space,” answers the question of whether generative AI models have reached certain limits by arguing that they are instead concealing far greater capabilities than scientists can currently gauge with standard protocols. The concept space is their key to unlocking these capabilities. This paper was co-authored by Harvard Ph.D. candidate Core Francisco Park*, NTT Research Scientist Maya Okawa*, University of Michigan Post-doctoral Fellow Andrew Lee, Harvard Post-doctoral Fellow Ekdeep Singh Lubana† (who was at the University of Michigan when the paper was drafted) and NTT Research Scientist Hidenori Tanaka†. Park, Okawa, Lubana and Tanaka are also affiliated with the Harvard Center for Brain Science-NTT Program on the Physics of Intelligence, where Tanaka serves as Group Leader.

“This research seeks to address fundamental questions, such as what determines the concepts a model learns, the order in which it learns them, and its ability to manipulate those concepts,” said Okawa. “By observing moments of sudden turns in the direction of the models’ learning dynamics, we find that these points correlate to the appearance of hidden capabilities in the model’s capacity to manipulate a concept, transforming our understanding of generative model learning dynamics.”

How precisely to elicit these capabilities – naïve input prompting does not yet suffice – remains a matter of ongoing investigation, but the team is encouraged by results to date. “These findings highlight the value of NTT Research’s collaborative efforts to explore the emerging field of the Physics of Intelligence,” Tanaka said. “We are honored that our joint progress in unearthing a deeper understanding of modern models’ capabilities has been recognized by NeurIPS, and we look forward to building upon our achievements in this area.”

The five NTT R&D presentations delivered at NeurIPS 2024 address a range of other vital topics in the field of neural networking, as indicated by the titles and areas covered by their related papers:

• “AUC Maximization under Positive Distribution Shift;” Atsutoshi Kumagai, Tomoharu Iwata, Hiroshi Takahashi, Taishi Nishiyama, and Yasuhiro Fujiwara (CD, SO, SI, CS Labs). An approach to improving training and classification in imbalanced scenarios where one category is much more common than another.
• “Fast Iterative Hard Thresholding Methods with Pruning Gradient Computations;” Yasutoshi Ida, Sekitoshi Kanai, Atsutoshi Kumagai, Tomoharu Iwata, and Yasuhiro Fujiwara (CD, CS Labs). This team has found a way to accelerate a technique commonly used in problems involving sparse solutions by as much as 73 times over the status quo, with no loss of accuracy.
• “Inverse M-Kernels for Linear Universal Approximators of Non-Negative Functions;” Hideaki Kim (HI, CS Labs). The kernel method is often used to estimate latent functions, while adjusting flexibly to data. This paper elucidates, arguably for the first time, the existence of linear universal approximators of non-negative functions.
• “Understanding the Expressivity and Trainability of Fourier Neural Operator: A Mean-Field Perspective,” Takeshi Koshizuka, Masahiro Fujisawa, Yusuke Tanaka, Issei Sato (University of Tokyo, RIKEN, CS Lab). This finding provides a practical prerequisite for the stable training of the Fourier Neural Operator (FNO), which enables a solution of partial differential equations.
• “Polyak meets Parameter-free Clipped Gradient Descent;” Yuki Takezawa, Han Bao, Ryoma Sato, Kenta Niwa, and Makoto Yamada (CS Lab, Kyoto University, Okinawa Institute of Science and Technology). Polyak momentum accelerates gradient descent, while clipped gradient descent stabilizes training and prevents issues caused by large updates to model parameters.

NeurIPS 2024 marks the 37th year of this annual event, a prestigious and competitive international conference in machine learning and computational neuroscience. The multi-track interdisciplinary annual meeting includes invited talks, demonstrations, symposia, and oral and poster presentations of refereed papers. Along with the conference is a professional exposition focusing on machine learning in practice, tutorials, and topical workshops that provide a less formal setting for the exchange of ideas.

About NTT Research

NTT Research opened its offices in July 2019 as a new Silicon Valley startup to conduct basic research and advance technologies that promote positive change for humankind. Currently, three labs are housed at NTT Research facilities in Sunnyvale: the Physics and Informatics (PHI) Lab, the Cryptography and Information Security (CIS) Lab, and the Medical and Health Informatics (MEI) Lab. The organization aims to upgrade reality in three areas: 1) quantum information, neuroscience and photonics; 2) cryptographic and information security; and 3) medical and health informatics. NTT Research is part of NTT, a global technology and business solutions provider with an annual R&D budget of $3.6 billion.

NTT and the NTT logo are registered trademarks or trademarks of NIPPON TELEGRAPH AND TELEPHONE CORPORATION and/or its affiliates. All other referenced product names are trademarks of their respective owners. © 2024 NIPPON TELEGRAPH AND TELEPHONE CORPORATION