ChatGPT has swept the world off its feet in the past few months since it was released to the public. In fact, has it only been a few months? The impact it has made makes it feel like it has been around for years.
But for all its virtues, it had one tiny flaw that wasn’t so tiny; it did not have access to recent information. The only information it had was up to mid-2021. But OpenAI is finally starting to change that. No, it hasn’t been trained on newer data. However, OpenAI is finally implementing early support for plugins in ChatGPT that allow it to connect to the internet as well as some third-party services!
What are ChatGPT Plugins?
Plugins are tools that have been designed for language models specifically. They enhance the chatbot’s capabilities and let it undertake previously impossible tasks. For example, it can now retrieve real-time information, like sports scores, stock prices, perform actions on behalf of a user, like booking flights, and retrieve knowledge base information, like company documents, etc.
Ever since ChatGPT’s launch, plugins were the most demanded item from users, and OpenAI has finally delivered. But their release is going to be a gradual, iterative rollout. Initially, OpenAI has only released a few third-party plugins and a couple of plugins of their own.
The access for users, as well as developers who want to build plugins, is currently limited as well. Moreover, as an end user, access is only being granted to ChatGPT Plus users at the moment, but ChatGPT says that they do plan to expand the rollout in the future.
Access can only be requested after joining their waitlist, which entails filling out a small questionnaire as to why you would like access and whether you would be willing to provide feedback.
The list of third-party plugins includes:
- Expedia – This plugin will allow you to plan your next trip with ChatGPT, with complete information about the availability and price of hotels, flights, etc.
- FiscalNote – This ChatGPT plugin would allow you access to real-time legal, political, and regulatory information and data.
- Instacart – Use ChatGPT to order groceries from nearby grocery stores and supermarkets.
- KAYAK – Use KAYAK within ChatGPT to find cars, hotels, rentals, etc. within your defined budget.
- Klarna Shopping – Search and compare prices from various online shops within your ChatGPT conversation.
- Milo Family AI – The plugin for parents to supercharge parenting.
- OpenTable – Get restaurant recommendations and reservation links in chats.
- Shop from Shopify – Search for products from different brands.
- Slack – Use ChatGPT with Slack to streamline communication
- Speak – Get an AI-powered language tutor
- Wolfram – Get ChatGPT to access calculations, math knowledge, etc.
- Zapier – Use this plugin to interact with more than 5000 apps within ChatGPT.
There are also two plugins from OpenAI itself: Browsing (a web browser) and a Code interpreter, and an open-source plugin, called Retriever. All the plugins are currently in the Alpha testing phase.
How Do These Plugins Work?
According to OpenAI, plugins are “eyes and ears” for language models like ChatGPT. The fact about language models is that they can only learn from their training data and that learning can be limited. ChatGPT can only provide text-based instructions on its own. These plugins can make it follow these instructions as well as make available information that is too recent, too personal, or too specific and could not be included in the training data.
But there’s a safety concern that arises when you think about an AI model following instructions and performing actions on your behalf. That’s the reason why plugins are being rolled out slowly. OpenAI is building them with safety as their core principle and will be monitoring their real-world usage.
Let’s take a look at how some of these plugin models work.
Browsing
This is the plugin that connects ChatGPT to the internet and is one of the two plugins from OpenAI itself. The plugin uses Bing search API from Microsoft in a not-so-curious turn of events; the two companies have had deals going back years. Apart from initial investments, Microsoft now also uses OpenAI technology to power their new Bing AI.
The model not only knows how to browse the internet but it also knows when to browse the internet and when not to. For example, when a user asks ChatGPT for information about Oscars 2023, it accurately searches the internet for the query. But when asked about the first Oscars ever held, it does not search the internet, because that information is part of its training data.
Note: Based on initial appearances, it seems like ‘Browsing’ and ‘Code Interpreter’ are separate models from ‘Plugins’. The way it seems to work is that you would select the model you currently want to use from the drop-down menu, which consists of three options: Browsing, Code Interpreter, and Plugins.
So, to make ChatGPT browse the internet, you’ll have to select the ‘Browsing’ model/ plugin.
Now, when a user enters a prompt that requires it to browse the internet, it will do so; the process will take some time as the bot will need time to get itself acquainted with the information. You can see ChatGPT’s flow of action until it generates a response by expanding the ‘Browsing the web..’ tile.
There, you can see what queries it searched for, what links it clicked, and when it’s reading in real-time. It uses a text-based web browser so it can go beyond the search results and actually read and navigate the websites.
Once ChatGPT is caught up with the events, it will give you an answer in its natural language model, as before. But the answer will include citations when it is generated with the Browsing model. Clicking the citation will take you to the website. This particular aspect is the same as Bing Chat AI.
From a safety point of view, the text-based browser can only make GET requests, which reduces certain risks. For example, the model can only retrieve information from the internet but it cannot perform “transactional” operations, like submitting a form.
Code Interpreter
The Code Interpreter model, the second plugin from OpenAI, provides ChatGPT with a Python interpreter. It also provides it with some short-lived disk space.
The session is alive during a single chat, so the next call can build on top of a previous call but there is an upper-bound time limit. Additionally, the Code Interpreter also supports uploading files and downloading files with the results.
To keep it safe, OpenAI is keeping it in a sandboxed, firewalled execution environment. Internet access is also disabled for Code Interpreter. According to OpenAI, even though this move limits the model’s functionality, they feel it is the right move initially.
The ‘Code Interpreter’ model must be selected before beginning the conversation.
Once the user enters the prompts that requires the use of the Code Interpreter plugin, ChatGPT would start using it to perform the required calculations. As with Browsing, the user can see ChatGPT’s flow of calculations by clicking ‘Show Work’ and all the steps in the calculation will be visible.
In initial tests, OpenAI has discovered this plugin to be useful in certain scenarios. These include:
- Solving mathematical problems, both quantitative and qualitative, which quite frankly, users found ChatGPT was rubbish at before.
- Doing data analysis and visualization, which has a lot of users excited.
- Converting files between formats
OpenAI expects users to discover more useful tasks the Code Interpreter can perform as they try it out.
Third-Party Plugins
The rest of the plugins fall under the Plugins model. This also includes the open-source Retriever plugin from OpenAI that developers will be able to use and 12 third-party plugins.
Once the Plugins model is selected from the drop-down, the user can install desired plugins from the store.
Here’s how plugins will work in a nutshell:
Once the users activate a plugin by installing it (they won’t be automatically activated) and begin the conversation, OpenAI will inject a compact description of the plugin to ChatGPT in a message. This message is not visible to end users but it will include the plugin’s description, endpoints, and examples. So, until you choose to use a plugin in a conversation, ChatGPT won’t have any knowledge about it. You must activate the plugins you want to use in every conversation.
Now, you can put your query to ChatGPT. If the bot finds it relevant to invoke the plugin, it will do so using an API call. In other words, it can decide itself whether a plugin needs to be invoked.
It will then include the results it gets from the plugin in the response it generates for you.
Here’s an example of how ChatGPT uses plugins from OpenTable, Wolfram, and Instacart. A user asks ChatGPT to recommend a vegan restaurant for Saturday and a vegan recipe for Sunday. They also ask it to calculate the calories for the recipe it recommends using Wolfram as well as to order the ingredients for the recipe from Instacart. The AI bot does exactly that.
First, it uses OpenTable to recommend a restaurant and a link to make a reservation.
It recommends a vegan recipe (which it could previously do) and then calculates the calories for the recipe using Wolfram.
Finally, it adds all the required ingredients to the cart on Instacart and presents the user with the link that the user can simply click to complete the order!
Plugins are completely going to change the way ChatGPT will work. The speed with which AI is progressing in the past few months is making it an equally scary and great time to be alive, isn’t it?
Member discussion