AI Can Play Video Games Now: Introducing Google DeepMind's SIMA
- Name
- Jennie Rose
Published on
In the world of artificial intelligence (AI), video games have proven to be an ideal testing ground for AI systems, offering a rich learning environment with real-time settings and dynamic goals. Building upon their expertise in AI and games, Google DeepMind has unveiled their latest milestone: a versatile AI agent called SIMA (Scalable Instructable Multiworld Agent). SIMA is capable of understanding natural-language instructions and performing tasks in a variety of 3D virtual environments, making it a significant breakthrough in the field of AI.
SIMA: Generalist AI Agent for 3D Virtual Environments
To develop SIMA, Google DeepMind partnered with eight game studios and trained the agent on nine different video games, including popular titles such as No Man's Sky and Teardown. Each game in SIMA's portfolio presents a unique virtual world with its own set of challenges and skills to master. From simple navigation and menu use to complex tasks like mining resources or flying a spaceship, SIMA was immersed in diverse gaming environments to learn and adapt.
As part of the training process, Google DeepMind recorded pairs of human players interacting with the games, with one player instructing and the other executing the instructions. This data helped train SIMA to understand natural-language instructions and correlate them with appropriate in-game actions. The agent consists of pre-trained vision models and a main model with memory, enabling it to generate keyboard and mouse actions to execute the instructions.
A Versatile AI Agent
SIMA's design prioritizes versatility and accessibility. The agent doesn't require access to a game's source code or specialized APIs. It only needs two inputs: the on-screen visuals and the user's simple, natural-language instructions. This interface mimics how humans interact with virtual environments, enabling SIMA to potentially engage with any virtual setting. Currently, SIMA's capabilities are evaluated across 600 basic skills, ranging from navigation and object interaction to menu use. The agent performs simple tasks within approximately 10 seconds.
https://t.co/qz3IxzUpto (opens in a new tab) pic.twitter.com/02Q6AkW4uq
— Google DeepMind (@GoogleDeepMind) March 13, 2024
Generalizing Across Games
Through extensive evaluations, Google DeepMind found that SIMA agents trained on multiple games outperformed agents trained on individual games. This highlights the agent's ability to generalize learnings across different gaming worlds. Even when faced with an unseen game, SIMA trained in multiple environments performed remarkably well, nearly matching the performance of an agent specifically trained on that particular game. The ability to adapt to new environments demonstrates SIMA's potential to operate beyond its training and navigate unfamiliar challenges.
Advancing AI Agent Research
Google DeepMind's development of SIMA marks an important step towards the creation of generalist, language-driven AI agents. With further training in a wider range of environments and the integration of more advanced models, SIMA is expected to become increasingly versatile and adaptable. The goal is to develop AI systems and agents capable of understanding and executing high-level language instructions, paving the way for AI that can effectively assist humans in various tasks.
Google DeepMind's commitment to advancing AI agent research is driven by the desire to create AI systems that are helpful and safe for people online and in real-world scenarios. As the development of SIMA progresses, the potential for a new wave of generalist AI agents becomes more promising.
Conclusion
The introduction of SIMA by Google DeepMind signals a significant advancement in the field of AI and gaming. By training the agent on a diverse range of virtual environments and enabling it to understand natural-language instructions, SIMA demonstrates the potential for AI systems to become more versatile and helpful across various domains. With further research, the aim is to develop AI agents capable of executing complex tasks and strategic planning, ultimately enhancing human-AI interaction and enabling AI to become an invaluable asset in everyday life.
