What can and can't language models do? Lessons learned from BIGBench

So what exactly can and can’t language models do? What's the least impressive thing GPT-4 won't be able to do? What will GPT-4 be incapable of? BIGBench is kind of a way to figure this out. BigBench, aka “The Beyond the Imitation Game” Benchmark, is an attempt to explore the capabilities of large language models over a wide variety of tasks. All the tasks are enumerated here. I looked through every BIGBench task and took the ones that compared both GPT3 and PaLM against humans. * Spreadsheet

Using cognitive psychology to understand GPT-3

Language Models Perform Reasoning via Chain of Thought – Google

Hidden abilities of large language models: Is emergence the norm?

GitHub - uncbiag/Awesome-Foundation-Models: A curated list of

A Survey of Large Language Models

444 Authors From 132 Institutions Release BIG-bench: A 204-Task

Google PaLM: Scaling Language Modeling with Pathways

Language Models Perform Reasoning via Chain of Thought – Google

Emergent Abilities in AI: Are We Chasing a Myth?

Generative AI and large language models: background and contexts