Anthropic inference service
editAnthropic inference service
editCreates an inference endpoint to perform an inference task with the anthropic
service.
Request
editPUT /_inference/<task_type>/<inference_id>
Path parameters
edit-
<inference_id>
- (Required, string) The unique identifier of the inference endpoint.
-
<task_type>
-
(Required, string) The type of the inference task that the model will perform.
Available task types:
-
completion
-
Request body
edit-
chunking_settings
-
(Optional, object) Chunking configuration object. Refer to Configuring chunking to learn more about chunking.
-
max_chunking_size
-
(Optional, integer)
Specifies the maximum size of a chunk in words.
Defaults to
250
. This value cannot be higher than300
or lower than20
(forsentence
strategy) or10
(forword
strategy). -
overlap
-
(Optional, integer)
Only for
word
chunking strategy. Specifies the number of overlapping words for chunks. Defaults to100
. This value cannot be higher than the half ofmax_chunking_size
. -
sentence_overlap
-
(Optional, integer)
Only for
sentence
chunking strategy. Specifies the numnber of overlapping sentences for chunks. It can be either1
or0
. Defaults to1
. -
strategy
-
(Optional, string)
Specifies the chunking strategy.
It could be either
sentence
orword
.
-
-
service
-
(Required, string)
The type of service supported for the specified task type. In this case,
anthropic
. -
service_settings
-
(Required, object) Settings used to install the inference model.
These settings are specific to the
anthropic
service.-
api_key
- (Required, string) A valid API key for the Anthropic API.
-
model_id
- (Required, string) The name of the model to use for the inference task. You can find the supported models at Anthropic models.
-
rate_limit
-
(Optional, object) By default, the
anthropic
service sets the number of requests allowed per minute to50
. This helps to minimize the number of rate limit errors returned from Anthropic. To modify this, set therequests_per_minute
setting of this object in your service settings:"rate_limit": { "requests_per_minute": <<number_of_requests>> }
-
-
task_settings
-
(Required, object) Settings to configure the inference task. These settings are specific to the
<task_type>
you specified.task_settings
for thecompletion
task type-
max_tokens
- (Required, integer) The maximum number of tokens to generate before stopping.
-
temperature
-
(Optional, float) The amount of randomness injected into the response.
For more details about the supported range, see the Anthropic messages API.
-
top_k
-
(Optional, integer) Specifies to only sample from the top K options for each subsequent token.
Recommended for advanced use cases only. You usually only need to use
temperature
.For more details, see the Anthropic messages API.
-
top_p
-
(Optional, float) Specifies to use Anthropic’s nucleus sampling.
In nucleus sampling, Anthropic computes the cumulative distribution over all the options for each subsequent token in decreasing probability order and cut it off once it reaches a particular probability specified by
top_p
. You should either altertemperature
ortop_p
, but not both.Recommended for advanced use cases only. You usually only need to use
temperature
.For more details, see the Anthropic messages API.
-
Anthropic service example
editThe following example shows how to create an inference endpoint called
anthropic_completion
to perform a completion
task type.
resp = client.inference.put( task_type="completion", inference_id="anthropic_completion", inference_config={ "service": "anthropic", "service_settings": { "api_key": "<api_key>", "model_id": "<model_id>" }, "task_settings": { "max_tokens": 1024 } }, ) print(resp)
const response = await client.inference.put({ task_type: "completion", inference_id: "anthropic_completion", inference_config: { service: "anthropic", service_settings: { api_key: "<api_key>", model_id: "<model_id>", }, task_settings: { max_tokens: 1024, }, }, }); console.log(response);
PUT _inference/completion/anthropic_completion { "service": "anthropic", "service_settings": { "api_key": "<api_key>", "model_id": "<model_id>" }, "task_settings": { "max_tokens": 1024 } }