My issue with ChatGPT is when you have to double check the AI’s work to make sure everything is accurate. At that point, why waste time using the generative AI in the first place? I’m glad that we have real human reporters such as yourself, Stephen! Unless you’re a robot in disguise. Surely you’d be upfront about that, of course.
I am not pro-AI in any way (I think it’s awful) but your methodology is slightly flawed. If you’re going to use a chatbot to do this, you need to instruct it to do something with function calling. Something like “use beautifulsoup to build an array of all the titles inside the doc with this ID” and then “use python to sort the array by string length.” Chatbots can’t count, but they can use tools that can.
I asked Gemini for a game that has four words in the title, each beginning with the letter "K" (for a 4K joke that's not important now). It told me there weren't any released ones that it knew about, but there was King K. Rool's Krazy Kartoon Kapers, a "well-known" canceled SNES game. I didn't believe that was a thing, so I asked for sources mentioning it. It said it was "well documented," and gave me three real-looking links. Every one of them was broken.
Not only did it hallucinate the game, it hallucinated entire sources on the spot. When I called it out on that, it apologized and said it failed.
In my mind, the danger isn't the lack of accuracy. It's the confidence in the wrong answers, and the fact that it won't say "I don't know" or even "maybe it's this, but I'm not sure" unless you prove that you know it's wrong.
Trust but verify. I was doing a similar thing with the number of Metroids in Metroid II. It had been a while since I played the game and I couldn't remember how many Metroids Samus had to defeated. AI said 47, but something seemed wrong about the number. I checked a playthrough video and the answer is actually 39. So close AI but no cigar
I'd ask that you don't start using ChatGPT. Its information as shown here is wrong, and it's fueled by plagiarism, environment destruction and many many other problems. In fact, if you are using it going forward, please tell me so that I can cancel my subscription because I avoid supporting generative AI work - which involves avoiding a number of games as well as different outlets and work.
The tone throughout the article was very tongue-in-cheek, and I didn't get the impression that he was happy with the results. More of a "oh, alright, since everyone's bugging me about it, I suppose I'll see if it's as useful as they say." And it decidedly wasn't.
I find that you can't ask LLMs about specific web pages unless you've printed that page to PDF and sent it as part of the query. Otherwise it will either not be able to access the page at all or will possibly provide you dated info or info it is "inferring" from old knowledge.
"In fact it would likely land near the very top (possibly top 3)"
This is the thing I can't stand about these chatbots. Do you not know this? Is it in the top 3 or not? If so, just say that! The conversational chatty tone does a lot of legwork to cover up the fact that it's not actually smart, or as magical as OpenAI wants you to believe.
My issue with ChatGPT is when you have to double check the AI’s work to make sure everything is accurate. At that point, why waste time using the generative AI in the first place? I’m glad that we have real human reporters such as yourself, Stephen! Unless you’re a robot in disguise. Surely you’d be upfront about that, of course.
That said, ChatGPT correctly pointed out that Fandom pages are bad ;)
I am not pro-AI in any way (I think it’s awful) but your methodology is slightly flawed. If you’re going to use a chatbot to do this, you need to instruct it to do something with function calling. Something like “use beautifulsoup to build an array of all the titles inside the doc with this ID” and then “use python to sort the array by string length.” Chatbots can’t count, but they can use tools that can.
I asked Gemini for a game that has four words in the title, each beginning with the letter "K" (for a 4K joke that's not important now). It told me there weren't any released ones that it knew about, but there was King K. Rool's Krazy Kartoon Kapers, a "well-known" canceled SNES game. I didn't believe that was a thing, so I asked for sources mentioning it. It said it was "well documented," and gave me three real-looking links. Every one of them was broken.
Not only did it hallucinate the game, it hallucinated entire sources on the spot. When I called it out on that, it apologized and said it failed.
In my mind, the danger isn't the lack of accuracy. It's the confidence in the wrong answers, and the fact that it won't say "I don't know" or even "maybe it's this, but I'm not sure" unless you prove that you know it's wrong.
Trust but verify. I was doing a similar thing with the number of Metroids in Metroid II. It had been a while since I played the game and I couldn't remember how many Metroids Samus had to defeated. AI said 47, but something seemed wrong about the number. I checked a playthrough video and the answer is actually 39. So close AI but no cigar
I'd ask that you don't start using ChatGPT. Its information as shown here is wrong, and it's fueled by plagiarism, environment destruction and many many other problems. In fact, if you are using it going forward, please tell me so that I can cancel my subscription because I avoid supporting generative AI work - which involves avoiding a number of games as well as different outlets and work.
The tone throughout the article was very tongue-in-cheek, and I didn't get the impression that he was happy with the results. More of a "oh, alright, since everyone's bugging me about it, I suppose I'll see if it's as useful as they say." And it decidedly wasn't.
Ya, I know but I'm hoping this isn't a case of him starting to experiment with it everywhere to see if its useful somewhere.
I have to spend enough time watching and writing about the BS stuff that I don't want to see it here.
I find that you can't ask LLMs about specific web pages unless you've printed that page to PDF and sent it as part of the query. Otherwise it will either not be able to access the page at all or will possibly provide you dated info or info it is "inferring" from old knowledge.
"In fact it would likely land near the very top (possibly top 3)"
This is the thing I can't stand about these chatbots. Do you not know this? Is it in the top 3 or not? If so, just say that! The conversational chatty tone does a lot of legwork to cover up the fact that it's not actually smart, or as magical as OpenAI wants you to believe.
A few more trillion will correct this mishap