I'm Sorry Dave, ChatGPT Can't Do That

As a computer science major, one of my biggest pet peeves is the way people talk about and use artificial intelligence (AI).

Here’s the deal: a lot of the messaging around AI is all hype. Major tech companies are exalting the immense power of AI as a tool because they want more people to buy into it, and therefore make their billions of dollars of investments into the technology worthwhile.

This technology is not as powerful as you have been led to believe.

First of all, what we call “artificial intelligence” lacks the ability to think. What we call AI today is nowhere near the realm of something like HAL from Space Odyssey: 2001. A more accurate term might be machine learning—statistical algorithms that are trained on large sets of data.

Unfortunately, this means a lot of the AI tools that are developed are used for jobs they have no place being. This is because they are described as an all-purpose technology rather than specific tools for specific tasks.

AI already exists in a lot of places. Amazon recommendations are powered by machine learning algorithms. Travel apps like Google Maps use AI to estimate destination time and best route. Even the spell check on your phone uses natural language processing machine learning tools. The technology is already heavily used in day-to-day life.

Generative AI is a new step forward in AI innovation, but it is incredibly limited and not a replacement for most anything. ChatGPT, a large-language model and chatbot, is a very good example of the limitations of AI.

How does it work?

Generative AI, like most AI, is an input-output machine that is trying to predict something. Travel apps predict the time it takes to get to a destination, ChatGPT tries to predict what response you want.

At the simplest level, models like ChatGPT are trained by trying to predict the next word in a sentence. Think like the autocomplete in email scaled to a larger degree. Instead of trying to guess the next word in a sentence, generative AI models are trying to guess what a plausible response to your input would be.

As a model trained on an absurdly large set of data—basically the entire internet—ChatGPT can be pretty good at generating responses that look good and seem human. However, this appearance of good responses means people may not realize a lot of the issues with a lot of the technology.

You don’t know how ChatGPT gets its information

The way AI generates information is called a “black box” because you don’t know exactly how it comes up with a result. If you ask a human to explain their thought process or how they did something, they can usually walk you through the individual steps. AI cannot do this, because it does not think and is created via layers and layers of statistical algorithms.

You may be able to ask ChatGPT how it came up with an answer, and it will give you an answer because that is what it is trained to do. That answer though is not going to be how it actually came to any conclusion.

Again, it is trying to predict the response you want because it is just a more complex version of your email autocorrect.

Since the entire internet is its dataset, the information ChatGPT is drawing from for any particular question could range from a published legal case, to a 15 year old’s blog, to someone spreading misinformation on social media.

It is very easy to see how quickly ChatGPT falls apart when you press it on information. Asking it for sources does not actually provide you the sources where ChatGPT got its information, because it does not know where it got its information, but rather potential sources for further research.

Frequently, these sources straight up do not exist.

ChatGPT lies to you, all the time

The AI hallucination problem is something researchers in the area are desperately trying to solve, but may never be able to.

As stated previously, AI is just trying to develop a response based upon what it predicts a good response to your input would be. The response it develops itself is then a mishmash of hundreds of thousands of other responses online.

Sometimes, this means the information it presents as truth is entirely false.

In a particularly notable example of this from 2023, a lawyer named Steven Schwartz used ChatGPT to write a legal case for him. When presented in court, the opposing lawyers were unable to find the court decisions cited within the case.

It was quickly realized ChatGPT completely fabricated a lot of the evidence cited, with the judge presiding over the case calling portions of it “legal gibberish.”

This isn’t unique to large scale legal briefs - I tested this just a moment ago by asking the model to provide sources for “why wearing shoes are bad.” As a response, it provided titles like "The Science of Running: How to Run Faster, Farther, and Injury-Free for Life" by Steve Magness, which does not exist.

If you can’t know where something gets its information, and it is notorious for lying, then you should not trust it for help on anything worthwhile.

So what does this mean?

If you need to create something that requires accurate, specific information and sources you should not use ChatGPT or any other generative AI model. They can’t think. They can’t provide you with accurate information, and they fail to even work as a starting point for research.

These models can be fun to mess with and can help in specific use cases, but if you need something done right, do it yourself or get an expert in the field to do it for you.

Generative AI is not an expert in anything except grifting tech companies out of billions of dollars.