GPT-3 is the future. But what can NLP do in the present?

GPT-3 is the future. But what can NLP do in the present?


This post was originally published by Alex C-G at Towards Data Science

In short, neural search is a new approach to retrieving information. In an ideal world, there would be lots of new data to search, but most of it is already stored in a relational database. That’s why most of the time it’s best not to store any information in the database at all. And in the future, what we want is that the information is simply searchable. But even if the information is readily searchable, there is still the problem of how to combine multiple search terms in the database. One way to combine search terms is to have search engines, but when we search for something on Google or any other search engine, they can often be quite good, but sometimes they’ll make mistakes, and they’ll only show some of the results, and sometimes they won’t show all of the results or they’ll say something very specific but you can’t look it up in that particular search. One of the most important goals of neural search is to let them search even more easily so they can make better decisions. But there’s still one issue: some types of problems don’t lend themselves to it and this can lead to a lot of lost information if the search engine doesn’t have the right tool in the right place. Nurture Now, I think we can take some lessons from neural search to solve this problem…

Looks somewhat coherent to me, though it doesn’t keep a clear narrative going, and Dear Lord, some of those run-on sentences!


distilGPT2 is a more efficient version of GPT-2 that had some excess neurons pruned away. Judging by its outputs, maybe a few too many neurons! It took about a minute to generate each result:

In short, neural search is a new approach to retrieving information. But in recent years, the field has gained little traction, and now there is evidence that it is more accurate in detecting and analyzing neural networks on human skin. Here is the gist of this story. The researchers of the Chinese National Scientific Research Foundation (CSRF) have teamed up with researchers from the University of California, San Diego, who are working on a neuroimaging and learning network that will map up and reconstruct neural networks on human skin. This brain project aims to look into the brain’s visual cortex, so that it can learn from individuals without losing information. The CSRF will also use a neural network to map the brain’s sensory cortex and the cerebral cortex, as well as the anterior and caudate cortex, which is part of the brain’s central cortex. But this may not be the first step in the quest to decipher neural networks, a method used to search for patterns of information on a wide variety of people, including those who are not on the receiving end of a drug. Researchers at University of California, San Diego (USDA) in collaboration with scientists at the Chinese National Scientific Research Foundation (CSRF) in collaboration with CSRF, have developed a technique that allows the scientists to search for patterns of information on skin. The researchers analyzed the brain’s sensory…

What if we throw that seed text at something meant to generate something different, like movie scripts?


gpt2_film_scripts was made by fine-tuning GPT-2 on 1,300 screenplays, complete with the weird spacing and margins they have compared to regular text. This really shows when you generate examples. Like GPT-2 (which this model is based on), examples took about 10 minutes each:

In short, neural search is a new approach to retrieving information. There have been many, many studies demonstrating that what we find, we search for. This is a kind of computer-aided search.

I don’t like that. I feel like an a**h*le.

She is still getting out of bed, having gotten the morning’s lesson. She sits at the desk, studying the computer screen. It is a huge pile of documents, and she quickly glances through them, and quickly notices the picture of a cat.
She stares at the picture, and takes out a tiny file.

Via gfycat

As we can see, the language model has picked up how to swear and isn’t afraid to do it! (I had to manually censor its potty mouth). We can also see it understands character lines and stage directions. And just like every netizen out there, it has a thing for cats. (If it seems a little short compared to other samples, it’s because it adds a lot of spacing (for magins) that Medium strips out.


One of my dreams is to create Star Trek episodes fully generated by an AI. Lieutenant Commander Data Science, if you will. So I trained gpt2_film_scripts on every Star Trek script I could find.

And…after making that tragic Star Trek pun, my easy_text_generator crashed on me. I have a feeling these AI’s may be smarter (and more emotional) than we think.

Via gfycat

After restarting:

In short, neural search is a new approach to retrieving information. A new way forward.

FARRIS: The key is the neural patterns themselves, not the computer.

SISKO: But a computer can do this.

FARRIS: It’s a breakthrough in computational neural net theory. It is clear they need to be designed to learn, replicate, and then share their knowledge for practical applications.

SISKO: Then you’ve got a problem. How do we know it’ll work?

FARRIS: We’ll probably never see it on paper, but the neural pattern generator is the single most promising piece of hardware I’ve ever seen written about, all things considered.

SISKO: We’ll get there eventually. When we do, can we trust this?

[Farris’ office]

FARRIS: A neural pattern generator will be the single most promising piece of hardware I’ve ever seen written about all things considered.

[Captain’s office]

(Morn is giving a speech at Starfleet Headquarters, reading.)

FARRIS: My colleagues and I have spent a year developing the prototype, which will be ready for launch by the end of the year.

SISKO: How will that affect Starfleet operations?

Not bad at all. For anyone who knows Star Trek, that sounds mostly convincing (though Morn giving a speech is plainly hilarious.)

The not-so-talkative Morn. Via gfycat

As we can see above, GPT-2 is pretty powerful. both for generating general-purpose text and for more specific use cases (like Star Trek scripts).

For these specific use cases though, GPT-2 needs a lot of data thrown at it, and a lot of epochs going over all that data. GPT-3 promises “few-shot learning”, so we could throw just a few examples at it and it would quickly pick up the style. So if you fancy a new book in the Lord of the Rings series, or a press release about <insert product here>, GPT-3 might be the language model for you!

Spread the word

This post was originally published by Alex C-G at Towards Data Science

Related posts