Monitor Microsoft Azure OpenAI

This tutorial shows you how to use the Elastic Azure OpenAI integration, the Azure portal, and Elastic Agent to collect and monitor Azure OpenAI logs and metrics with Elastic Observability.

You’ll learn how to:

The Elastic Azure OpenAI integration captures audit logs and request and response logs.

For more on the fields ingested from audit and request and response logs, refer to the Azure OpenAI integration documentation.

Before Elastic Agent can collect your logs and send them to Kibana, complete the following steps in the Azure portal:

Azure Event Hubs is a data streaming platform and event ingestion service that you use to store in-flight Azure logs before sending them to Elasticsearch. For this tutorial, you create a single event hub because you are collecting logs from one service.

To create an Azure event hub:

Every Azure service that creates logs has diagnostic settings that allow you to export logs and metrics to an external destination. In this step, you’ll configure the Azure OpenAI service to export audit and request and response logs to the event hub you created in the previous step.

To configure diagnostic settings to export logs:

The Elastic Agent stores the consumer group information (state, position, or offset) in a storage account container. Making this information available to Elastic Agents allows them to share the logs processing and resume from the last processed logs after a restart. The Elastic Agent can use one storage account container for all integrations. The agent uses the integration name and the event hub name to identify the blob to store the consumer group information uniquely.

To create the storage account:

The Azure OpenAI integration metric data stream collects the cognitive service metrics specific to the Azure OpenAI service. Before Elastic Agent can collect your metrics and send them to Kibana, it needs an app registration to access Azure on your behalf and collect data using the Azure APIs.

Complete the following steps in your Azure instance to register a new Azure app:

To register your app:

Credentials allow your app to access Azure APIs and authenticate itself, so you won’t need to do anything at runtime. The Elastic Azure OpenAI integration uses client secrets to authenticate.

To create and add client secrets:

To add a role assignment to your app:

To collect Azure OpenAI logs, specify values for the following required fields:

To collect Azure OpenAI metrics, specify values for the following required fields:

After you’ve finished configuring your integration, click Save and continue. You’ll get a notification that your integration was added. Select Add Elastic Agent to your hosts.

You can install Elastic Agent on any host that can access the Azure account and forward events to Elasticsearch.

It takes a few minutes for Elastic Agent to enroll in Fleet, download the configuration specified in the policy, and start collecting data. You can wait to confirm incoming data, or close the window.

Now that your log and metric data is streaming to Elasticsearch, you can view them in Kibana. You have the following options for viewing your data:

The Elastic Azure OpenAI integration comes with a built-in overview dashboard to visualize your log and metric data. To view the integration dashboards:

From here, you’ll find visualizations of important metrics for your Azure OpenAI service, like the request rate, error rate, token usage, and chat completion latency. To zoom in on your data, click and drag across the bars in a visualization.

For more on dashboards and visualization, refer to the Dashboards and visualizations documentation.

Go to Discover from the Kibana menu under Analytics. From the data view drop-down, select either logs-* or metrics-* to view specific data. You can also create data views if, for example, you wanted to view both logs-* and metrics-* simultaneously.

From here, filter your data and dive deeper into individual logs to find information and troubleshoot issues. For a list of Azure OpenAI fields you may want to filter by, refer to the Azure OpenAI integration docs.

For more on using Discover and creating data views, refer to the Discover documentation.

To view Azure OpenAI logs, open Kibana and go to Observability → Logs Explorer. With Logs Explorer, you can quickly search and filter your log data, get information about the structure of log fields, and display your findings in a visualization.

From Logs Explorer, you can select the Azure OpenAI integration from the data selector to view your Kubernetes data.

From here, filter your log data and dive deeper into individual logs to find information and troubleshoot issues. For a list of Azure OpenAI fields you may want to filter by, refer to the Azure OpenAI integration documentation.

For more on Logs Explorer, refer to:

The Azure OpenAI API provides useful data to help monitor and understand your code. Using OpenTelemetry, you can ingest this data into Elastic Observability. From there, you can view and analyze your data to monitor the cost and performance of your applications.

For this tutorial, we’ll be using an example Python application and the Python OpenTelemetry libraries to instrument the application and send data to Observability.

To start collecting APM data for your Azure OpenAI applications, gather the OpenTelemetry OTLP exporter endpoint and authentication header from your Elastic Cloud instance:

With the configuration values from the APM integration and your Azure OpenAI API key and endpoint, set the following environment variables using the export command on the command line:

export AZURE_OPENAI_API_KEY="your-Azure-OpenAI-API-key"
export AZURE_OPENAI_ENDPOINT="your-Azure-OpenAI-endpoint"
export OPENAI_API_VERSION="your_api_version"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20<your-otel-exporter-auth-header>"
export OTEL_EXPORTER_OTLP_ENDPOINT="your-otel-exporter-endpoint"
export OTEL_RESOURCE_ATTRIBUTES=service.name=your-service-name

Install the necessary Python libraries using this command:

pip3 install openai flask opentelemetry-distro[otlp] opentelemetry-instrumentation

The following code is from the example application. In a real use case, you would add the import statements to your code.

The app we’re using in this tutorial is a simple example that calls Azure OpenAI APIs with the following message: “How do I send my APM data to Elastic Observability?”:

import os

from flask import Flask
from openai import AzureOpenAI
from opentelemetry import trace

from monitor import count_completion_requests_and_tokens


# Initialize Flask app
app = Flask(__name__)

# Set OpenAI API key
client = AzureOpenAI(
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),
    api_version=os.getenv("OPENAI_API_VERSION"),
    azure_endpoint=os.getenv("AZURE_OPENAI_ENDPOINT"),
)

# Monkey-patch the openai.Completion.create function
client.chat.completions.create = count_completion_requests_and_tokens(
    client.chat.completions.create
)

tracer = trace.get_tracer("counter")


@app.route("/completion")
@tracer.start_as_current_span("completion")
def completion():
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[
            {
                "role": "user",
                "content": "How do I send my APM data to Elastic Observability?",
            }
        ],
        max_tokens=20,
        temperature=0,
    )
    return response.choices[0].message.content.strip()

The code uses monkey patching, a technique in Python that dynamically modifies the behavior of a class or module at runtime by modifying its attributes or methods, to modify the behavior of the chat.completions call so we can add the response metrics to the OpenTelemetry spans.

The monitor.py file in the example application instruments the application and can be used to instrument your own applications.

def count_completion_requests_and_tokens(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        counters["completion_count"] += 1
        response = func(*args, **kwargs)

        token_count = response.usage.total_tokens
        prompt_tokens = response.usage.prompt_tokens
        completion_tokens = response.usage.completion_tokens
        cost = calculate_cost(response)
        strResponse = json.dumps(response, default=str)

        # Set OpenTelemetry attributes
        span = trace.get_current_span()
        if span:
            span.set_attribute("completion_count", counters["completion_count"])
            span.set_attribute("token_count", token_count)
            span.set_attribute("prompt_tokens", prompt_tokens)
            span.set_attribute("completion_tokens", completion_tokens)
            span.set_attribute("model", response.model)
            span.set_attribute("cost", cost)
            span.set_attribute("response", strResponse)
        return response

    return wrapper

Adding this data to our span lets us send it to our OTLP endpoint, so you can search for the data in Observability and build dashboards and visualizations.

Implementing the following function allows you to calculate the cost of a single request to the OpenAI APIs.

def calculate_cost(response):
    if response.model in ["gpt-4", "gpt-4-0314"]:
        cost = (
            response.usage.prompt_tokens * 0.03
            + response.usage.completion_tokens * 0.06
        ) / 1000
    elif response.model in ["gpt-4-32k", "gpt-4-32k-0314"]:
        cost = (
            response.usage.prompt_tokens * 0.06
            + response.usage.completion_tokens * 0.12
        ) / 1000
    elif "gpt-3.5-turbo" in response.model:
        cost = response.usage.total_tokens * 0.002 / 1000
    elif "davinci" in response.model:
        cost = response.usage.total_tokens * 0.02 / 1000
    elif "curie" in response.model:
        cost = response.usage.total_tokens * 0.002 / 1000
    elif "babbage" in response.model:
        cost = response.usage.total_tokens * 0.0005 / 1000
    elif "ada" in response.model:
        cost = response.usage.total_tokens * 0.0004 / 1000
    else:
        cost = 0
    return cost

To download the example application and try it for yourself, go to the GitHub repo.

After ingesting your data, you can filter and explore it using Discover in Kibana. Go to Discover from the Kibana menu under Analytics. You can then filter by the fields sent to Observability by OpenTelemetry, including:

Then, use these fields to create visualizations and build dashboards. Refer to the Dashboard and visualizations documentation for more information.

Now that you know how to find and visualize your Azure OpenAI logs and metrics, you’ll want to make sure you’re getting the most out of your data. Elastic has some useful tools to help you do that: