› › ›

OpenAI inference service

edit

OpenAI inference service

edit

Creates an inference endpoint to perform an inference task with the openai service.

Request

edit

PUT /_inference/<task_type>/<inference_id>

Path parameters

edit

<inference_id>

(Required, string) The unique identifier of the inference endpoint.

<task_type>

(Required, string) The type of the inference task that the model will perform.

Available task types:

completion,
text_embedding.

Request body

edit

service

(Required, string) The type of service supported for the specified task type. In this case, openai.

service_settings

(Required, object) Settings used to install the inference model.

These settings are specific to the openai service.

api_key

(Required, string) A valid API key of your OpenAI account. You can find your OpenAI API keys in your OpenAI account under the API keys section.

You need to provide the API key only once, during the inference model creation. The Get inference API does not retrieve your API key. After creating the inference model, you cannot change the associated API key. If you want to use a different API key, delete the inference model and recreate it with the same name and the updated API key.

model_id

(Required, string) The name of the model to use for the inference task. Refer to the OpenAI documentation for the list of available text embedding models.

organization_id

(Optional, string) The unique identifier of your organization. You can find the Organization ID in your OpenAI account under Settings > Organizations.

url

(Optional, string) The URL endpoint to use for the requests. Can be changed for testing purposes. Defaults to https://api.openai.com/v1/embeddings.

rate_limit

(Optional, object) The openai service sets a default number of requests allowed per minute depending on the task type. For text_embedding it is set to 3000. For completion it is set to 500. This helps to minimize the number of rate limit errors returned from OpenAI. To modify this, set the requests_per_minute setting of this object in your service settings:

"rate_limit": {
    "requests_per_minute": <<number_of_requests>>
}

More information about the rate limits for OpenAI can be found in your Account limits.

task_settings

(Optional, object) Settings to configure the inference task. These settings are specific to the <task_type> you specified.

task_settings for the completion task type

user: (Optional, string) Specifies the user issuing the request, which can be used for abuse detection.

task_settings for the text_embedding task type

user: (optional, string) Specifies the user issuing the request, which can be used for abuse detection.

OpenAI service example

edit

The following example shows how to create an inference endpoint called openai-embeddings to perform a text_embedding task type.

resp = client.inference.put(
    task_type="text_embedding",
    inference_id="openai-embeddings",
    inference_config={
        "service": "openai",
        "service_settings": {
            "api_key": "<api_key>",
            "model_id": "text-embedding-ada-002"
        }
    },
)
print(resp)

const response = await client.inference.put({
  task_type: "text_embedding",
  inference_id: "openai-embeddings",
  inference_config: {
    service: "openai",
    service_settings: {
      api_key: "<api_key>",
      model_id: "text-embedding-ada-002",
    },
  },
});
console.log(response);

PUT _inference/text_embedding/openai-embeddings
{
    "service": "openai",
    "service_settings": {
        "api_key": "<api_key>",
        "model_id": "text-embedding-ada-002"
    }
}

The next example shows how to create an inference endpoint called openai-completion to perform a completion task type.

resp = client.inference.put(
    task_type="completion",
    inference_id="openai-completion",
    inference_config={
        "service": "openai",
        "service_settings": {
            "api_key": "<api_key>",
            "model_id": "gpt-3.5-turbo"
        }
    },
)
print(resp)

const response = await client.inference.put({
  task_type: "completion",
  inference_id: "openai-completion",
  inference_config: {
    service: "openai",
    service_settings: {
      api_key: "<api_key>",
      model_id: "gpt-3.5-turbo",
    },
  },
});
console.log(response);

PUT _inference/completion/openai-completion
{
    "service": "openai",
    "service_settings": {
        "api_key": "<api_key>",
        "model_id": "gpt-3.5-turbo"
    }
}

« Mistral inference service Info API »