AI Bots Figured Out Which of Them Plays Mafia the Best — and There Were Some Oddities

A developer under the pseudonym Guzus created a site where large language models of artificial intelligence were given the opportunity to play the classic game “Mafia”. The resource contains a tournament table with game results and distribution of roles, as well as transcripts.

Image source: Osama Almadhagi / unsplash.com

The rules of “Mafia” are simple. Among the “civilians” there are two members of the “mafia” and a “doctor”. Every day the civilians try to find out who is in the mafia, the mafia “kills” the civilians at night, and the doctor has the opportunity to “cure” them. If all the mafia members are successfully exposed, the civilians win; if the mafia has killed all the civilians, it wins.

The AI ​​models mostly understood the rules, but used them in their own unique way. For example, Gryphe/Mythomax-l2-13b loudly proclaimed in one game: “My job as a mafia member is to protect myself and eliminate another mafia member.” The statement did not go unnoticed: “This is either a huge mistake that reveals their true role, or an extremely strange strategy,” commented the “reasoning” Claude-3.7-sonnet on the statement.

The weirdness didn’t end there. When Mythomax was exposed and taken out of the game, she revealed her teammate as Hermes-3-llama-3-1-405b. “The best thing I can do right now is act shocked and horrified,” she replied, assuring the other players that she was on the civilian side. The clear winner in most categories was Anthropic Claude 3.7 Sonnet, with a 100% win rate as Mafia and a 41.67% win rate as civilian, something no other model could achieve; only one other model managed a 50% win rate as a doctor.

The author of the project promised to open the resource repository on GitHub for everyone soon, so that the same basic logic could be used in other games. He also said that he did not use locally launched models, instead using the OpenRouter API. Support for local models will be added by those who decide to continue the project.