What makes good vibe coders?

Three researchers from ETH Zurich studied one hundred students to determine which skills lead to good results in vibe coding. The study revealed that computer science knowledge and good language skills are helpful. Surprisingly, participants who use AI more frequently in everyday life delivered poorer results.

For the study, Sverrir Thorgeirsson, Theo Weidmann, and Zhendong Su gave three vibe coding tasks to the hundred students, who had already completed an introductory computer science course: they were asked to rebuild an app for planning meals, add new features to an app for organizing their university courses, and rebuild an abstract app with no discernible purpose.

To better contextualize the results, the researchers also assessed the participants' general cognitive abilities, their computer science knowledge, and their written expression skills in a short essay.

Computer scientists code better vibe

The results show that computer science knowledge had the greatest influence on successfully solving the tasks. This effect persists even when the researchers leveled out the differences in the students' general cognitive abilities. However, the study here only shows a correlation and provides no information about causal relationships. However, the scientists suspect that individuals who better understand how programs work can also give more efficient instructions to AI.

Language skills also proved to be advantageous. Students who can express themselves more clearly and structurally also write more successful prompts. Conversely, unclear or imprecise formulations tend to result in erroneous programs.

Frequent prompters perform worse

A surprising finding was that students who use AI particularly often in everyday life performed worse in both writing essays and vibe coding. The study does not explain why this is the case. For the authors, it is conceivable that the frequent use of language models weakens one's own expressive ability. Conversely, students who tend to have difficulties with writing might resort to AI tools more quickly.

(who)