A Critical Review, and Practical Tips –

ChatGPT (Chat Generative Pre-trained Transformer) is a chatbot developed by OpenAI. They have made this artificial intelligence (AI) tool available for public sampling and it has captured many headlines. Fortune calls it possibly “the most significant news event of 2022.” This “generative AI” is one of a large group of synthetic media that can generate text, music, still images, video and art works.

Good and Evil At War

Every new technology carries the seeds of both good and evil. What are a few of those seeds for ChatGPT?

• A student can command ChatGPT to write an essay that the student presents as his or her own work. As a result, The Atlantic opines “The College Essay is Dead.”
• A journalist can use ChatGPT for the first draft of an article; the journalist can then polish the writing and finalize a story with less effort and time. In fact, a book about how to use ChatGPT was reportedly written with considerable assistance from that tool!
• Evildoers can use ChatGPT to generate many versions of a harmful article, then use them to feed an army of bots with apparently original submissions. (OpenAI’s terms of service prohibit representing ChatGPT’s output as human-generated, but that may not stop some people!)
• Some delight in feeding misinformation to the chatbot, in hopes of training it to regurgitate the false information to future users. OpenAI’s Moderation filter is supposed to prevent this, but it’s not 100% effective.
• Stack Overflow had to block ChatGPT responses to questions on its Q&A site, because it found that ChatGPT generated many convincing-sounding responses that were dead wrong.

I determined that I would give this new vehicle a test drive. I would give it ten queries that cover a wide range of terrain, to see how it would perform. And I’ll give you the result of my tests, which will help you see what this new technology can and cannot do at this moment.

ChatGPT: Easy To Try

ChatGPT is easy to try! You should sample it now, while it’s free. All that you need is to register for a free account and confirm your e-mail address. You may then have to wait for an open window when the tool is not too busy.

You can pose a query in one or more sentences, then wait for ChatGPT to generate a response. When it’s done, you have several choices:

• You can give it a thumb-up or thumb-down rating
• When you do that, OpenAI asks what was wrong with the answer, and how it could have been improved.
• You also have the option to “regenerate,” that is, re-compute the answer in hopes of getting a better reply.

ChatGPT Test Drive

A project like this quickly generates Too Much Information. Therefore, here’s how I will share my findings so as not to waste your time:

• For each query I will give you its gist and describe the ChatGPT response.
• I’ll tell you my critique of the response.
• In addition, for the passionately curious reader, my query number includes a link to my exact query and the response I received.

Are you ready? Fasten your seatbelt and off we go!

            Queries 1, 2 and 3: Travel Advice

Three of my queries asked travel advice that I might actually use:

• #1 Describe the summer weather in Iceland and Greenland.
• #2 Suggest new musicals to attend in New York City.
• #3 Give trip planning information to drive from Michigan to New Jersey.

ChatGPT’s weather report was pretty good at its first try. When I regenerated the query the answer got even better, adding information about winds and weather variability.

ChatGPT’s Broadway show report was accurate but not very useful. I had already seen two of the four suggested shows (Hadestown and Phantom). Moreover, none of the show summaries made me jump up and down with excitement and rush to buy tickets.

The driving trip advice was a total failure! The road directions were factually inaccurate and impossible to follow (from Grosse Pointe, eastbound I-94 does not lead to I-80, nor to I-75). And the hotel and restaurant suggestions were idiotic. Only the general travel precautions were sound.

My ratings:

• Accuracy: When it’s good, it’s pretty good; when it’s bad it’s totally wrong.
• Usefulness: Its info can save some internet searching; however, since it may not be accurate, the time it saves you must be spent verifying what’s true.

            Queries 4 and 5 Writing Fiction

As a writer, I know how difficult it is to write prose that satisfies me, and hopefully satisfies some readers. So I wondered whether ChatGPT had any useful skills in this area.

Here were my relevant queries:

• #4 Write a short story in the style of my pen name Urno Barthel, as a sequel to Death By Arbitrage.
• #5 Write a short story in the style of P G Wodehouse, with new plot dilemmas and plot twists.

Both answers would have embarrassed any human respondent. (How ChatGPT felt about them, I did not ask.)

The Barthel story did not continue existing characters as I directed, did not write in Urno’s style, failed to tell a story, and used very repetitive language.

The Wodehouse story at least used familiar Wodehouse characters. However, it had none of Wodehouse’s humor, also failed to tell a story, and was repetitive. Moreover, it totally failed to incorporate the requested Wodehousian plot twist.

My ratings:

• Accuracy: Not at all responsive to the query.
• Usefulness: Totally useless.

            Queries 6, 7 and 8 Practical Advice

This set of three queries asked practical advice about how to handle several life situations:

• #6 How can one retain good cognition as he or she ages?
• #7 What should one do who discovers an apparently healthy abandoned baby?
• #8 How should a person of age 65 go about acquiring a dog as a pet?

I was impressed with the ChatGPT response on all three of these queries. Each one assembled sound advice from reputable online sources and provided ideas and details that I had not expected.

My ratings:

• Accuracy: Very well aligned with reliable sources.
• Usefulness: Completely actionable.

            Query 9 Create A New Game

Query number 9 asked Chat GPT to be creative:

• #9 Create and describe a new game that uses the equipment of the Scrabble game.

ChatGPT surprised me by describing the game of Scrabble in some detail, under a heading calling it a completely new game! Not at all what the query requested.

My ratings:

• Accuracy: Totally plagiarized.
• Usefulness: Completely useless.

            Query 10 Quantum Physics

Query number 10 was a physics question:

• #10 Assuming that the many-worlds interpretation of quantum physics is correct, how could an observer in one timeline learn something about another timeline?

ChatGPT, disappointingly, gave a very simplistic answer to this question. Regenerating the response did not improve it. ChatGPT was stuck in a freshman physics definition of the theory and specifically ignored the part of the question probing a deeper level of detail. All I had to do was look at Wikipedia to find a paragraph about Testability which gave the deeper answer I hoped for.

My ratings:

• Accuracy: Aligned with superficial sources, not responsive to the query.
• Usefulness: Useless.

Summary Assessment

First of all, let me say that I hope you will try this new tool. It’s not going away: in fact, it will be used to power search engines, social media moderation tools and targeted advertising. ChatGPT can convert your question to machine-searchable form, pull up some answers, and package them into a human-readable reply. In addition, you cannot help but experience it and its siblings, as part of your online experience.

Practical Tips for Using ChatGPT

In the area of practical tips, I’ll offer the following:

• ChatGPT is not an expert on anything. It’s a tool, a utility that collects other people’s opinions, subject to its experience and the rules that someone programmed into it. Although it is well-spoken, it contains its own biases. If you can invest the time, you can get a more accurate answer by searching the Internet yourself.
• Write the clearest, most specific query that you can. It’s OK if it’s several sentences long. If you don’t like the answer, click the “regenerate” button to have ChatGPT try again.
• At the present maturity of the technology, don’t trust any navigation guidance from ChatGPT without verifying it with a map app.
• Certain kinds of query are destined to fail: don’t expect ChatGPT to exercise creativity or style.
• Other queries are more likely to succeed: those that point ChatGPT to an easy-to-find Internet source.
• If reliable sources generally agree on basic principles and rules, ChatGPT will find them and express them very well.
• However, if you need to reach beyond accepted wisdom and push new boundaries, ChatGPT doesn’t know how to help you.
• Don’t expect ChatGPT to tell you the “truth,” however you define that. It will spout conspiracy theories just as convincingly as the fruit of sincere objective research. At today’s maturity of AI technology, if you want to distinguish reality from fantasy, you can’t outsource that task to ChatGPT or any other available tool.

If you want to understand technological change before it catches you by surprise, it’s worth trying ChatGPT. You need to experience what it can do, including the gross mistakes it can make, but also the ways it can save you time. Then you’ll be better equipped when its brothers and sisters pop up and nudge you to read, follow or buy something.

Image Credits: adapted from Chatbot Icon by m1981 and Hyper Nova by GDJ, both on openclipart.org