IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Amazon Bedrock inference service Azure AI studio inference service »

› › ›

Anthropic inference service

edit

Anthropic inference service

edit

Creates an inference endpoint to perform an inference task with the anthropic service.

Request

edit

PUT /_inference/<task_type>/<inference_id>

Path parameters

edit

<inference_id>

(Required, string) The unique identifier of the inference endpoint.

<task_type>

(Required, string) The type of the inference task that the model will perform.

Available task types:

completion

Request body

edit

service

(Required, string) The type of service supported for the specified task type. In this case, anthropic.

service_settings

(Required, object) Settings used to install the inference model.

These settings are specific to the anthropic service.

api_key

(Required, string) A valid API key for the Anthropic API.

model_id

(Required, string) The name of the model to use for the inference task. You can find the supported models at Anthropic models.

rate_limit

(Optional, object) By default, the anthropic service sets the number of requests allowed per minute to 50. This helps to minimize the number of rate limit errors returned from Anthropic. To modify this, set the requests_per_minute setting of this object in your service settings:

"rate_limit": {
    "requests_per_minute": <<number_of_requests>>
}

task_settings

(Required, object) Settings to configure the inference task. These settings are specific to the <task_type> you specified.

task_settings for the completion task type

max_tokens

(Required, integer) The maximum number of tokens to generate before stopping.

temperature

(Optional, float) The amount of randomness injected into the response.

For more details about the supported range, see the Anthropic messages API.

top_k

(Optional, integer) Specifies to only sample from the top K options for each subsequent token.

Recommended for advanced use cases only. You usually only need to use temperature.

For more details, see the Anthropic messages API.

top_p

(Optional, float) Specifies to use Anthropic’s nucleus sampling.

In nucleus sampling, Anthropic computes the cumulative distribution over all the options for each subsequent token in decreasing probability order and cut it off once it reaches a particular probability specified by top_p. You should either alter temperature or top_p, but not both.

Recommended for advanced use cases only. You usually only need to use temperature.

For more details, see the Anthropic messages API.

Anthropic service example

edit

The following example shows how to create an inference endpoint called anthropic_completion to perform a completion task type.

resp = client.inference.put(
    task_type="completion",
    inference_id="anthropic_completion",
    inference_config={
        "service": "anthropic",
        "service_settings": {
            "api_key": "<api_key>",
            "model_id": "<model_id>"
        },
        "task_settings": {
            "max_tokens": 1024
        }
    },
)
print(resp)

const response = await client.inference.put({
  task_type: "completion",
  inference_id: "anthropic_completion",
  inference_config: {
    service: "anthropic",
    service_settings: {
      api_key: "<api_key>",
      model_id: "<model_id>",
    },
    task_settings: {
      max_tokens: 1024,
    },
  },
});
console.log(response);

PUT _inference/completion/anthropic_completion
{
    "service": "anthropic",
    "service_settings": {
        "api_key": "<api_key>",
        "model_id": "<model_id>"
    },
    "task_settings": {
        "max_tokens": 1024
    }
}

« Amazon Bedrock inference service Azure AI studio inference service »