This documentation contains work-in-progress information for future Elastic Stack and Cloud releases. Use the version selector to view supported release docs. It also contains some Elastic Cloud serverless information. Check out our serverless docs for more details.

« Mistral inference service Watsonx inference service »

› › ›

OpenAI inference service

edit

OpenAI inference service

edit

Creates an inference endpoint to perform an inference task with the openai service.

Request

edit

PUT /_inference/<task_type>/<inference_id>

Path parameters

edit

<inference_id>

(Required, string) The unique identifier of the inference endpoint.

<task_type>

(Required, string) The type of the inference task that the model will perform.

Available task types:

completion,
text_embedding.

Request body

edit

chunking_settings

(Optional, object) Chunking configuration object. Refer to Configuring chunking to learn more about chunking.

max_chunking_size: (Optional, integer) Specifies the maximum size of a chunk in words. Defaults to 250. This value cannot be higher than 300 or lower than 20 (for sentence strategy) or 10 (for word strategy).
overlap: (Optional, integer) Only for word chunking strategy. Specifies the number of overlapping words for chunks. Defaults to 100. This value cannot be higher than the half of max_chunking_size.
sentence_overlap: (Optional, integer) Only for sentence chunking strategy. Specifies the numnber of overlapping sentences for chunks. It can be either 1 or 0. Defaults to 1.
strategy: (Optional, string) Specifies the chunking strategy. It could be either sentence or word.

service

(Required, string) The type of service supported for the specified task type. In this case, openai.

service_settings

(Required, object) Settings used to install the inference model.

These settings are specific to the openai service.

api_key

(Required, string) A valid API key of your OpenAI account. You can find your OpenAI API keys in your OpenAI account under the API keys section.

You need to provide the API key only once, during the inference model creation. The Get inference API does not retrieve your API key. After creating the inference model, you cannot change the associated API key. If you want to use a different API key, delete the inference model and recreate it with the same name and the updated API key.

dimensions

(Optional, integer) The number of dimensions the resulting output embeddings should have. Only supported in text-embedding-3 and later models. If not set the OpenAI defined default for the model is used.

model_id

(Required, string) The name of the model to use for the inference task. Refer to the OpenAI documentation for the list of available text embedding models.

organization_id

(Optional, string) The unique identifier of your organization. You can find the Organization ID in your OpenAI account under Settings > Organizations.

url

(Optional, string) The URL endpoint to use for the requests. Can be changed for testing purposes. Defaults to https://api.openai.com/v1/embeddings.

rate_limit

(Optional, object) The openai service sets a default number of requests allowed per minute depending on the task type. For text_embedding it is set to 3000. For completion it is set to 500. This helps to minimize the number of rate limit errors returned from OpenAI. To modify this, set the requests_per_minute setting of this object in your service settings:

"rate_limit": {
    "requests_per_minute": <<number_of_requests>>
}

More information about the rate limits for OpenAI can be found in your Account limits.

task_settings

(Optional, object) Settings to configure the inference task. These settings are specific to the <task_type> you specified.

task_settings for the completion task type

user: (Optional, string) Specifies the user issuing the request, which can be used for abuse detection.

task_settings for the text_embedding task type

user: (optional, string) Specifies the user issuing the request, which can be used for abuse detection.

OpenAI service example

edit

The following example shows how to create an inference endpoint called openai-embeddings to perform a text_embedding task type. The embeddings created by requests to this endpoint will have 128 dimensions.

resp = client.inference.put(
    task_type="text_embedding",
    inference_id="openai-embeddings",
    inference_config={
        "service": "openai",
        "service_settings": {
            "api_key": "<api_key>",
            "model_id": "text-embedding-3-small",
            "dimensions": 128
        }
    },
)
print(resp)

PUT _inference/text_embedding/openai-embeddings
{
    "service": "openai",
    "service_settings": {
        "api_key": "<api_key>",
        "model_id": "text-embedding-3-small",
        "dimensions": 128
    }
}

The next example shows how to create an inference endpoint called openai-completion to perform a completion task type.

resp = client.inference.put(
    task_type="completion",
    inference_id="openai-completion",
    inference_config={
        "service": "openai",
        "service_settings": {
            "api_key": "<api_key>",
            "model_id": "gpt-3.5-turbo"
        }
    },
)
print(resp)

const response = await client.inference.put({
  task_type: "completion",
  inference_id: "openai-completion",
  inference_config: {
    service: "openai",
    service_settings: {
      api_key: "<api_key>",
      model_id: "gpt-3.5-turbo",
    },
  },
});
console.log(response);

PUT _inference/completion/openai-completion
{
    "service": "openai",
    "service_settings": {
        "api_key": "<api_key>",
        "model_id": "gpt-3.5-turbo"
    }
}

« Mistral inference service Watsonx inference service »