GPT Engineer - The Threat of AI

I recently watched a YouTube video about a new technology called GPT Engineer (https://youtu.be/FPZONhA0C60?si=pSzZEXcJdCkzt8h_). It is based on the same generative pre-trained transformer (GPT) model that ChatGPT is based on. (For more information on what ChatGPT is, check out my previous article: https://www.codersjournal.com/post/what-is-chatgpt) It is a fascinating method to create AI that replicates our method of communicating via the written word. GPT Engineer, instead of responding to requests for information, was designed to generate code. For example, a user might tell GPT Engineer to “create a website that I can use to sell my collection of video games using Python, as well as a way to modify the website with React code,” and GPT Engineer would create the files on your computer automatically.

This is a fascinating piece of technology, but one that has widespread implications for not only the future careers of computer scientists but for society as a whole. Let me start with the implications for computer scientists. I would like to think this technology was created to make the lives of a programmer easier, as an aid to developers. Developers often use Google or websites like Stack Overflow to help develop and troubleshoot their code. But now, instead of searching for a solution to their problem on the internet, a developer could just use GPT Engineer to troubleshoot their issues which would save them time and energy. The problem is that GPT Engineer can be used to replace the work of junior engineers. Its ability to quickly develop code, even if it isn’t perfect, is a threat to the future careers of computer scientists. A company could choose to, instead of hiring a team of junior developers, demand senior developers use GPT Engineer to create a rough draft of the desired code which they would subsequently polish up for distribution. This saves the company time and money.

If this happens junior developers will begin to struggle to find employment. Without the on-the-job experience, a junior developer would not be able to gain the experience necessary to become a senior developer. If this prediction rings true, as the current senior developers retire, the technology companies will struggle to find trained staff and the economy may take a hit. But this is only one way society may feel the impact of this AI model.

There is a second, more studied, negative consequence of the widespread use of large language models (LLMs) like ChatGPT and GPT Engineer. These LLMs are commonly trained on human-generated content found on the internet. But, because of the widespread use of these LLMs, the internet is now filling up with LLM-generated content, most likely ChatGPT. There is nothing wrong with generating content using these LLMs, but problems may arise when future generations of the LLMs are created. As I stated earlier, the models are trained off of content found on the internet, and as the models are increasingly used, output from the previous generation of LLMs will make their way to becoming training material for the next generations of the LLMs. This is where the problems will begin.

As can be seen by these articles, The Curse of Recursion: Training on Generated Data Makes Models Forget (https://arxiv.org/pdf/2305.17493v2.pdf), and Self-Consuming Generative Models Go MAD (https://arxiv.org/pdf/2307.01850.pdf), if an artificial intelligence (AI) model is trained of its own output artifacts may emerge, as well as a decrease of original output. The second article shows artifacts being generated as early as the third generation of an AI mode. The problem with models getting trained off their own output is that the next generation’s output has less variation. As a model is trained off of its own output, its bell curve regresses toward the mean, and the original content seen during the first generation will disappear. This regression toward the mean will mean that future generations of ChatGPT may output the same response to a similar prompt, or GPT Engineer may create the same code for different companies trying to compete with similar products. These problems may stay hidden until it is too late, and the companies that rely on AI to create their products may begin to suffer.

The future economy is dependent on us solving these problems before it is too late. I am not against AI, on the contrary, I would like to create better, smarter AI, but I do see the problems that may occur if we stay on this path. If we are to have a society where AI can be used to help cure diseases thought incurable, or solve problems thought unsolvable, we will need to solve the issues brought up in this article. Might a solution be more regulation by the government on the use of AI by corporations, or might it be that the output generated by AI is marked in a way that it cannot be used as training material during the next generation? I don’t know what the best solution to these problems is but I know that the first step in solving them is to identify that there is a problem. I wrote this article to present these issues to my readers in hopes they might help find solutions to these problems, as well as start discussions with their peers about these issues.

If you liked this article please share it with a friend. Also if you would like to learn how to code your own AI application, or want to learn be basics of Python, please leave a comment below or follow this link for more information about my classes https://www.codersjournal.com/general-clean.

Thank you :)