Cohere inference service
editCohere inference service
editCreates an inference endpoint to perform an inference task with the cohere
service.
Request
editPUT /_inference/<task_type>/<inference_id>
Path parameters
edit-
<inference_id>
- (Required, string) The unique identifier of the inference endpoint.
-
<task_type>
-
(Required, string) The type of the inference task that the model will perform.
Available task types:
-
completion
, -
rerank
, -
text_embedding
.
-
Request body
edit-
chunking_settings
-
(Optional, object) Chunking configuration object. Refer to Configuring chunking to learn more about chunking.
-
max_chunking_size
-
(Optional, integer)
Specifies the maximum size of a chunk in words.
Defaults to
250
. This value cannot be higher than300
or lower than20
(forsentence
strategy) or10
(forword
strategy). -
overlap
-
(Optional, integer)
Only for
word
chunking strategy. Specifies the number of overlapping words for chunks. Defaults to100
. This value cannot be higher than the half ofmax_chunking_size
. -
sentence_overlap
-
(Optional, integer)
Only for
sentence
chunking strategy. Specifies the numnber of overlapping sentences for chunks. It can be either1
or0
. Defaults to1
. -
strategy
-
(Optional, string)
Specifies the chunking strategy.
It could be either
sentence
orword
.
-
-
service
-
(Required, string)
The type of service supported for the specified task type. In this case,
cohere
. -
service_settings
-
(Required, object) Settings used to install the inference model.
These settings are specific to the
cohere
service.-
api_key
-
(Required, string) A valid API key of your Cohere account. You can find your Cohere API keys or you can create a new one on the API keys settings page.
You need to provide the API key only once, during the inference model creation. The Get inference API does not retrieve your API key. After creating the inference model, you cannot change the associated API key. If you want to use a different API key, delete the inference model and recreate it with the same name and the updated API key.
-
rate_limit
-
(Optional, object) By default, the
cohere
service sets the number of requests allowed per minute to10000
. This value is the same for all task types. This helps to minimize the number of rate limit errors returned from Cohere. To modify this, set therequests_per_minute
setting of this object in your service settings:"rate_limit": { "requests_per_minute": <<number_of_requests>> }
More information about Cohere’s rate limits can be found in Cohere’s production key docs.
service_settings
for thecompletion
task type-
model_id
-
(Optional, string)
The name of the model to use for the inference task.
To review the available
completion
models, refer to the Cohere docs.
service_settings
for thererank
task type-
model_id
-
(Optional, string)
The name of the model to use for the inference task.
To review the available
rerank
models, refer to the Cohere docs.
service_settings
for thetext_embedding
task type-
embedding_type
-
(Optional, string) Specifies the types of embeddings you want to get back. Defaults to
float
. Valid values are:-
byte
: use it for signed int8 embeddings (this is a synonym ofint8
). -
float
: use it for the default float embeddings. -
int8
: use it for signed int8 embeddings.
-
-
model_id
-
(Optional, string)
The name of the model to use for the inference task.
To review the available
text_embedding
models, refer to the Cohere docs. The default value fortext_embedding
isembed-english-v2.0
. -
similarity
-
(Optional, string)
Similarity measure. One of
cosine
,dot_product
,l2_norm
. Defaults based on theembedding_type
(float
→dot_product
,int8/byte
→cosine
).
-
-
-
task_settings
-
(Optional, object) Settings to configure the inference task. These settings are specific to the
<task_type>
you specified.task_settings
for thererank
task type-
return_documents
- (Optional, boolean) Specify whether to return doc text within the results.
-
top_n
-
(Optional, integer)
The number of most relevant documents to return, defaults to the number of the documents.
If this inference endpoint is used in a
text_similarity_reranker
retriever query andtop_n
is set, it must be greater than or equal torank_window_size
in the query.
task_settings
for thetext_embedding
task type-
input_type
-
(Optional, string) Specifies the type of input passed to the model. Valid values are:
-
classification
: use it for embeddings passed through a text classifier. -
clusterning
: use it for the embeddings run through a clustering algorithm. -
ingest
: use it for storing document embeddings in a vector database. -
search
: use it for storing embeddings of search queries run against a vector database to find relevant documents.The
input_type
field is required when using embedding modelsv3
and higher.
-
-
truncate
-
(Optional, string) Specifies how the API handles inputs longer than the maximum token length. Defaults to
END
. Valid values are:-
NONE
: when the input exceeds the maximum input token length an error is returned. -
START
: when the input exceeds the maximum input token length the start of the input is discarded. -
END
: when the input exceeds the maximum input token length the end of the input is discarded.
-
-
Cohere service examples
editThe following example shows how to create an inference endpoint called
cohere-embeddings
to perform a text_embedding
task type.
resp = client.inference.put( task_type="text_embedding", inference_id="cohere-embeddings", inference_config={ "service": "cohere", "service_settings": { "api_key": "<api_key>", "model_id": "embed-english-light-v3.0", "embedding_type": "byte" } }, ) print(resp)
const response = await client.inference.put({ task_type: "text_embedding", inference_id: "cohere-embeddings", inference_config: { service: "cohere", service_settings: { api_key: "<api_key>", model_id: "embed-english-light-v3.0", embedding_type: "byte", }, }, }); console.log(response);
PUT _inference/text_embedding/cohere-embeddings { "service": "cohere", "service_settings": { "api_key": "<api_key>", "model_id": "embed-english-light-v3.0", "embedding_type": "byte" } }
The following example shows how to create an inference endpoint called
cohere-rerank
to perform a rerank
task type.
resp = client.inference.put( task_type="rerank", inference_id="cohere-rerank", inference_config={ "service": "cohere", "service_settings": { "api_key": "<API-KEY>", "model_id": "rerank-english-v3.0" }, "task_settings": { "top_n": 10, "return_documents": True } }, ) print(resp)
const response = await client.inference.put({ task_type: "rerank", inference_id: "cohere-rerank", inference_config: { service: "cohere", service_settings: { api_key: "<API-KEY>", model_id: "rerank-english-v3.0", }, task_settings: { top_n: 10, return_documents: true, }, }, }); console.log(response);
PUT _inference/rerank/cohere-rerank { "service": "cohere", "service_settings": { "api_key": "<API-KEY>", "model_id": "rerank-english-v3.0" }, "task_settings": { "top_n": 10, "return_documents": true } }
For more examples, also review the Cohere documentation.