AI as a programming aid: more efficient code, but more errors
A data scientist tests what AI can do as a programming assistant. His result: if the human checks again and threatens a fine, the code becomes more efficient.
(Image: everything possible/Shutterstock.com)
In order for generative chatbots to output efficient program code, developers need to guide the AI with precise instructions. This is indicated by an experiment conducted by Max Woolf, Senior Data Scientist at Buzzfeed. He comes to the conclusion that prompt engineering can improve the speed of the generated code. However, the highly optimized program code contains more errors because language models are not designed for this, Woolf wrote.
AI generates faster code after being threatened with a fine
For his experiment, Woolf had Anthropic Claude generate a Python code. From a list of one million random integers between 1 and 100,000, he was asked to calculate the difference between the smallest and largest number whose sum of the digits was 30. The AI provided Woolf with a working code block. He then asked the chatbot four times to improve the code. After the first iteration, the code was almost three times faster. The second optimization suggestion was five times faster with multithreading, but returned bugs. With the last suggestion, the AI accelerated the code by a factor of 100.
In a second run, Woolf instructed the chatbot that the code needed to be fully optimized. In addition, he gave examples of the improvements in the prompt, such as the use of parallelization and vectorization and the reuse of code. Woolf threatened the AI with a fine if the code was not fully optimized. The result was code that was nine times faster. He repeated the optimization three times and in the last two iterations found code execution to be almost a hundred times faster.
Videos by heise
However, the strong optimizations with prompt engineering in Woolf's experiment led to a higher frequency of errors. In all outputs of the second run, he found errors in the code where he had to rework. Nevertheless, Woolf praised the language models for their inventiveness and suggestions. However, he does not expect artificial intelligence to be able to replace workers. It needs specialist knowledge to be able to recognize really good ideas and resolve problems, Woolf wrote in his blog.
(sfe)
