Skip to content
PSHIR.COM
/Articles

Distrusting the AI Bot Research

Author
Pinchas Shir
Share

    Large language models like ChatGPT, Grok, and Gemini are so popular these days. They can do so much and so fast, compared to old-fashioned search engine queries. Millions of people use them because they are packaged with core services offered by Google, Microsoft, Meta, and many other software tools. I use them, too. Some people build entire livelihoods on AI bots because they not only locate the needed data but can also synthesize it. These are powerful tools, and their reasoning and thinking functions are new and very useful. Today, these bots can be used for research in any area, including biblical studies, Jewish studies, and ancient languages too. Should we embrace the progress and let AI do the research?

    I grew up watching DOS turn into Windows, and Macintosh computers replace typewriters in libraries, so I am a big fan of using tech. I firmly believe that we should use the best tools we have. But is AI good at this? Generally speaking, yes. In my experience, however, most AI bots still look an inch deep to come up with a very quick answer, and then, satisfied, move on to the next task. What I call "lazy research." Yes, some specialized reasoning tools are much better than others (they are quite niche), but I am referring to popular language models used by most people today. Allow me to illustrate.

    I often use AI bots to gather lists of textual references and help me find general information about the topics I study. Doing this, I noticed that many bots lack common sense and nuance, often believing everything they read way too quickly. I am not satisfied with this. I end up arguing with AI pretty much every week, just as I would with a colleague. To illustrate, I would like to share an excerpt from my recent conversation with ChatGPT. I was researching Josephus' references about first-century religious teachers. So, I asked AI to pull up a list of Josephus' texts where he talks about "rabbis," which it happily did with lightning speed. Then, looking at the list of passages, I retorted skeptically, "No, Josephus does not really speak of rabbis, if ever..." and ChatGPT quickly pushed back.

    💡
    CHAT: yes, some translations are anachronistic—but not because Josephus never uses “rabbi.” He does. The problem is that translators sometimes use rabbi as a convenient English label for any Jewish religious authority, whereas the ancient Greek vocabulary is considerably more differentiated.

    So, I asked the bot to prove it and to show me some of those references to "rabbis" in Josephus that it just listed for me a minute ago, in Greek, of course. The entire collection of Josephus' works is available online in ancient Greek, so this is a tedious data-sifting task, perfect for AI super-brains. And Chat admitted that it was wrong!

    💡
    CHAT: I need to correct something from my previous answer: I was wrong when I said that Josephus explicitly uses the Greek term ῥαββί for Sameas in Antiquities 17.2.4 (§39). I cannot substantiate that claim, and the Greek text of Josephus does not contain ῥαββί there. I conflated Josephus's description of Jewish teachers with the later/other evidence for the title. Thank you for asking me to cite it. In fact, Josephus does not appear to use the noun ῥαββί in his extant Greek works in the way I previously claimed. The Greek term is well attested in the New Testament, but that is different evidence.

    Now I could have easily accepted the preliminary answer ChatGPT gave me that "here is a list of passages from Josephus that mention the rabbis" and simply repeated this to others as if it were a fact. Only I saved myself some embarrassment because I pulled up the Greek reference to verify this and quickly saw the problem. No superintelligence is necessary to check the original source text, but apparently ChatGPT decided to give me a list based on its research in English. Translations, however, can be all over the place.

    The point of my story here is very simple. I love the new tech, but I don't blindly rely on AI tools to do careful research for me. Before giving in to widespread panic that AI will take over everyone's job (or end us all), consider that a bot can be wrong even on the most basic things as often as any human. I happily use AI data processing capabilities, but I still opt to check the results manually and draw my own conclusions. And so should you, especially after the example I just shared with you. If you are an enthusiastic AI user, go ahead, distrust its research on occasion, dig deeper, use skepticism, argue with the bot, and you might be surprised what will come up next.

    Bless PSHIR.COM with a Gift of Support?
    Many people benefit from PSHIR.COM publications, but not everyone wishes to be a subscriber. That’s fine. Now you can support independent publishing by giving a simple gift (in any amount and anytime) as a thank you without subscribing. SUPPORT PSHIR.COM The gift works the same way as
    PSHIR.COMPinchas Shir