There are currently two ways to use ChatGPT API. One through the API provided by OpenAI and another by using Azure OpenAI service provided by Microsoft.

In this blog, I will compare the differences between the two to help you decide which best fits your situation.

Latency

Latency is one of the biggest struggles when building an LLM-integrated application. They don't have the computing power to keep up with the ever-growing demand. Responses can be delayed or it is not rare for them to be never returned at all.

I ran a test using both APIs to compare the response time to process a request for ChatCompletion. The code below is for OpenAI.

import os
import openai
import time
from tqdm import tqdm

from dotenv import load_dotenv
load_dotenv()

openai.api_key = os.getenv("OPEN_AI_ACCESS_KEY")

def measure_response_time():
    start_time = time.time()
    _ = openai.ChatCompletion.create(
        model="gpt-3.5-turbo-0301",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Who is the founder of Apple?"},
        ]
    )
    end_time = time.time()
    latency = end_time - start_time
    return latency

latencies = []
for i in tqdm(range(20)):
    latencies.append(measure_response_time())

print(f"Average Latency: {sum(latencies) / len(latencies)} seconds")
print(f"Maximum Latency: {max(latencies)} seconds")
print(f"Minimum Latency: {min(latencies)} seconds")

Here are the results.

Azure OpenAI outperformed OpenAI API in all criteria. For products, setting the timeout to 5 seconds will suffice for Azure whereas for OpenAI it has to be much longer.

One thing to note is that I observed a rate limit of 24 requests per minute for Azure OpenAI when running this test. The limit is too small so maybe I need to contact Azure to increase the quota (needs more investigation).

Winner: Azure OpenAI API

Availability

At the time of this blog, OpenAI's entire API has 99.82% of uptime. Azure OpenAI Service has a Service Level Agreement which guarantees 99.9%. As OpenAI does not provide SLA, Azure OpenAI API is favoured for production usage.

Winner: Azure OpenAI API

Network Security

Azure OpenAI API is part of Azure cloud service so you can use their IAM and Azure Active Directory to manage multi-cloud identities and secure access to resources.

It's also good to mention that Azure OpenAI service allows you to deploy ChatGPT API as a Private Endpoint inside your VNet. This means that you will be able to call ChatGPT without your request and API KEY being passed through the public Internet.

OpenAI's API does not have any privacy control at the moment so again Azure is the winner if you require security.

Winner: Azure OpenAI API

Prompt Injection

Prompt injection is the elephant in the room when it comes to building LLM-powered applications. It is causing many headaches for companies as nobody has found a great solution to this yet. Still, there are strategies you can perform to decrease the risk.

Microsoft offers a basic "Content filtering" that works alongside the deployed model to filter inappropriate content.

An excerpt from their documentation explains this process:

"This system works by running both the input prompt and generated content through an ensemble of classification models aimed at detecting misuse. If the system identifies harmful content, you'll receive either an error on the API call if the prompt was deemed inappropriate or the finish_reason on the response will be content_filter to signify that some of the generation was filtered."

Although you still need an application-specific mitigation strategy in place having these security available by default is helpful.

OpenAI API does not offer any support. You would have to build one from scratch.

Winner: Azure OpenAI API

Privacy policy

As long as ChatGPT is invoked via API, all the data sent will not be used for training, for both APIS.

However, OpenAI is a California-based company so its privacy policy adheres to Californian law. This might not be ideal if the data you process needs to reside in Europe, as per GDPR.

On the other hand, Azure's OpenAI API can be deployed per region so they are more favourable for companies outside of the US.

Winner: Azure OpenAI API

New features

New features will roll out earlier with OpenAI. At this point, Azure OpenAI API does not have the plugin features. In fact, even GPT4 is not available on Azure.

To use their shiny new features, OpenAI is the way to go.

Winner: OpenAI API

Development

For development, I find OpenAI API to be easier and better documented. There are a bunch of examples online as most tutorials and blogs use OpenAI API.

Winner: OpenAI API

Conclusion

To create proofs-of-concept, experiment with new features, and replicate a blog or paper, I find the OpenAI API to be particularly user-friendly and well-suited.

On the other hand, when it comes to product development, my preference leans towards the Azure OpenAI Service due to its notable attributes such as reliability, availability, and robust security measures.

It's important to acknowledge that software development inherently involves making trade-offs. There isn't a one-size-fits-all solution, and developers must carefully evaluate their specific requirements before selecting an API that best aligns with their needs. I hope that this blog helps make the decision!

P.S.

At Seeai, we help with LLM-powered software development. For any questions contact us here!