Large language models (LLMs) are traditionally perceived as text generators. However, with a bit of ingenuity, they can be made to perform much more complex tasks. For instance, developer Andrea Ricci has found a way to teach the Claude Sonnet model to play the classic shooter DOOM. To achieve this, the game was adapted for the SCINTIX P4 platform – a single-board computer based on Raspberry Pi CM4/CM5 with ESP32-P4 and ESP32-C6 processors. The game runs on the P4 and is displayed on a MIPI DSI screen with a resolution of 1024×600. However, instead of a continuous stream of frames, the game transmits them to the Claude Sonnet model via WebSocket only step-by-step, simulating human perception. The LLM receives information about the game world and independently generates movement and shooting commands. The system demonstrates an impressive capability: Claude generally accurately describes the environment, navigates through levels, opens doors, and shoots at enemies. Special methods are used to help the model orient itself: fan-shaped depth mapping to identify walls and prevent collisions, as well as an ASCII auto-map to track traversed areas. Importantly, the LLM plays without any additional hints beyond those available to a regular player. This project shows how text models can adapt to visual perception and interaction with graphical environments, opening new horizons for the application of artificial intelligence.
Games
Claude Taught to Play DOOM
Large language models (LLMs) are traditionally perceived as text generators.
Author: Хана Окада · Aug 262 min

Category: Games
Sources and verification
- Source: Hackaday
Original publication ↗
Related reading
- NOPIA Harmony Synthesizer: Chords You Can Feel
Music theory often seems abstract, requiring memorization and painstaking study.
- Lamps Without Switches: Design That Senses You
Forget fumbling for a switch in the dark. The new Call It a Day lighting collection from Tongqi Design offers a different kind of interaction: simply touch, pull, or tap the soft silicone shade, and the light turns on. This isn't magic, but an engineering approach where the entire lamp surface becomes a sensor.
- AI Assistant for Medics: AI-GUIDE Receives Award for Saving Lives
The portable medical device AI-GUIDE, developed at MIT Lincoln Laboratory in collaboration with Massachusetts General Hospital, has received the prestigious Excellence in Technology Transfer Award.
