- Introducing Elasticsearch Service
- Adding data to Elasticsearch
- Migrating data
- Ingesting data from your application
- Ingest data with Node.js on Elasticsearch Service
- Ingest data with Python on Elasticsearch Service
- Ingest data from Beats to Elasticsearch Service with Logstash as a proxy
- Ingest data from a relational database into Elasticsearch Service
- Ingest logs from a Python application using Filebeat
- Ingest logs from a Node.js web application using Filebeat
- Configure Beats and Logstash with Cloud ID
- Best practices for managing your data
- Configure index management
- Enable cross-cluster search and cross-cluster replication
- Access other deployments of the same Elasticsearch Service organization
- Access deployments of another Elasticsearch Service organization
- Access deployments of an Elastic Cloud Enterprise environment
- Access clusters of a self-managed environment
- Enabling CCS/R between Elasticsearch Service and ECK
- Edit or remove a trusted environment
- Migrate the cross-cluster search deployment template
- Manage data from the command line
- Preparing a deployment for production
- Securing your deployment
- Monitoring your deployment
- Monitor with AutoOps
- Configure Stack monitoring alerts
- Access performance metrics
- Keep track of deployment activity
- Diagnose and resolve issues
- Diagnose unavailable nodes
- Why are my shards unavailable?
- Why is performance degrading over time?
- Is my cluster really highly available?
- How does high memory pressure affect performance?
- Why are my cluster response times suddenly so much worse?
- How do I resolve deployment health warnings?
- How do I resolve node bootlooping?
- Why did my node move to a different host?
- Snapshot and restore
- Managing your organization
- Your account and billing
- Billing Dimensions
- Billing models
- Using Elastic Consumption Units for billing
- Edit user account settings
- Monitor and analyze your account usage
- Check your subscription overview
- Add your billing details
- Choose a subscription level
- Check your billing history
- Update billing and operational contacts
- Stop charges for a deployment
- Billing FAQ
- Elasticsearch Service hardware
- Elasticsearch Service GCP instance configurations
- Elasticsearch Service GCP default provider instance configurations
- Elasticsearch Service AWS instance configurations
- Elasticsearch Service AWS default provider instance configurations
- Elasticsearch Service Azure instance configurations
- Elasticsearch Service Azure default provider instance configurations
- Change hardware for a specific resource
- Elasticsearch Service regions
- About Elasticsearch Service
- RESTful API
- Release notes
- Enhancements and bug fixes - December 2024
- Enhancements and bug fixes - November 2024
- Enhancements and bug fixes - Late October 2024
- Enhancements and bug fixes - Early October 2024
- Enhancements and bug fixes - September 2024
- Enhancements and bug fixes - Late August 2024
- Enhancements and bug fixes - Early August 2024
- Enhancements and bug fixes - July 2024
- Enhancements and bug fixes - Late June 2024
- Enhancements and bug fixes - Early June 2024
- Enhancements and bug fixes - Early May 2024
- Bring your own key, and more
- AWS region EU Central 2 (Zurich) now available
- GCP region Middle East West 1 (Tel Aviv) now available
- Enhancements and bug fixes - March 2024
- Enhancements and bug fixes - January 2024
- Enhancements and bug fixes
- Enhancements and bug fixes
- Enhancements and bug fixes
- Enhancements and bug fixes
- AWS region EU North 1 (Stockholm) now available
- GCP regions Asia Southeast 2 (Indonesia) and Europe West 9 (Paris)
- Enhancements and bug fixes
- Enhancements and bug fixes
- Bug fixes
- Enhancements and bug fixes
- Role-based access control, and more
- Newly released deployment templates for Integrations Server, Master, and Coordinating
- Enhancements and bug fixes
- Enhancements and bug fixes
- Enhancements and bug fixes
- Enhancements and bug fixes
- Enhancements and bug fixes
- Enhancements and bug fixes
- Enhancements and bug fixes
- Enhancements and bug fixes
- Enhancements and bug fixes
- Enhancements and bug fixes
- Cross environment search and replication, and more
- Enhancements and bug fixes
- Enhancements and bug fixes
- Azure region Canada Central (Toronto) now available
- Azure region Brazil South (São Paulo) now available
- Azure region South Africa North (Johannesburg) now available
- Azure region Central India (Pune) now available
- Enhancements and bug fixes
- Azure new virtual machine types available
- Billing Costs Analysis API, and more
- Organization and billing API updates, and more
- Integrations Server, and more
- Trust across organizations, and more
- Organizations, and more
- Elastic Consumption Units, and more
- AWS region Africa (Cape Town) available
- AWS region Europe (Milan) available
- AWS region Middle East (Bahrain) available
- Enhancements and bug fixes
- Enhancements and bug fixes
- GCP Private Link, and more
- Enhancements and bug fixes
- GCP region Asia Northeast 3 (Seoul) available
- Enhancements and bug fixes
- Enhancements and bug fixes
- Native Azure integration, and more
- Frozen data tier and more
- Enhancements and bug fixes
- Azure region Southcentral US (Texas) available
- Azure region East US (Virginia) available
- Custom endpoint aliases, and more
- Autoscaling, and more
- Cross-region and cross-provider support, warm and cold data tiers, and more
- Better feature usage tracking, new cost and usage analysis page, and more
- New features, enhancements, and bug fixes
- AWS region Asia Pacific (Hong Kong)
- Enterprise subscription self service, log in with Microsoft, bug fixes, and more
- SSO for Enterprise Search, support for more settings
- Azure region Australia East (New South Wales)
- New logging features, better GCP marketplace self service
- Azure region US Central (Iowa)
- AWS region Asia Pacific (Mumbai)
- Elastic solutions and Microsoft Azure Marketplace integration
- AWS region Pacific (Seoul)
- AWS region EU West 3 (Paris)
- Traffic management and improved network security
- AWS region Canada (Central)
- Enterprise Search
- New security setting, in-place configuration changes, new hardware support, and signup with Google
- Azure region France Central (Paris)
- Regions AWS US East 2 (Ohio) and Azure North Europe (Ireland)
- Our Elasticsearch Service API is generally available
- GCP regions Asia East 1 (Taiwan), Europe North 1 (Finland), and Europe West 4 (Netherlands)
- Azure region UK South (London)
- GCP region US East 1 (South Carolina)
- GCP regions Asia Southeast 1 (Singapore) and South America East 1 (Sao Paulo)
- Snapshot lifecycle management, index lifecycle management migration, and more
- Azure region Japan East (Tokyo)
- App Search
- GCP region Asia Pacific South 1 (Mumbai)
- GCP region North America Northeast 1 (Montreal)
- New Elastic Cloud home page and other improvements
- Azure regions US West 2 (Washington) and Southeast Asia (Singapore)
- GCP regions US East 4 (N. Virginia) and Europe West 2 (London)
- Better plugin and bundle support, improved pricing calculator, bug fixes, and more
- GCP region Asia Pacific Southeast 1 (Sydney)
- Elasticsearch Service on Microsoft Azure
- Cross-cluster search, OIDC and Kerberos authentication
- AWS region EU (London)
- GCP region Asia Pacific Northeast 1 (Tokyo)
- Usability improvements and Kibana bug fix
- GCS support and private subscription
- Elastic Stack 6.8 and 7.1
- ILM and hot-warm architecture
- Elasticsearch keystore and more
- Trial capacity and more
- APM Servers and more
- Snapshot retention period and more
- Improvements and snapshot intervals
- SAML and multi-factor authentication
- Next generation of Elasticsearch Service
- Branding update
- Minor Console updates
- New Cloud Console and bug fixes
- What’s new with the Elastic Stack
Upload custom plugins and bundles
editUpload custom plugins and bundles
editThere are several cases where you might need your own files to be made available to your Elasticsearch cluster’s nodes:
- Your own custom plugins, or third-party plugins that are not amongst the officially available plugins.
- Custom dictionaries, such as synonyms, stop words, compound words, and so on.
- Cluster configuration files, such as an Identity Provider metadata file used when you secure your clusters with SAML.
To facilitate this, we make it possible to upload a ZIP file that contains the files you want to make available. Uploaded files are stored using Amazon’s highly-available S3 service. This is necessary so we do not have to rely on the availability of third-party services, such as the official plugin repository, when provisioning nodes.
Custom plugins and bundles are collectively referred to as extensions.
Before you begin
editThe selected plugins/bundles are downloaded and provided when a node starts. Changing a plugin does not change it for nodes already running it. Refer to Updating Plugins and Bundles.
With great power comes great responsibility: your plugins can extend your deployment with new functionality, but also break it. Be careful. We obviously cannot guarantee that your custom code works.
You cannot edit or delete a custom extension after it has been used in a deployment. To remove it from your deployment, you can disable the extension and update your deployment configuration.
Uploaded files cannot be bigger than 20MB for most subscription levels, for Platinum and Enterprise the limit is 8GB.
It is important that plugins and dictionaries that you reference in mappings and configurations are available at all times. For example, if you try to upgrade Elasticsearch and de-select a dictionary that is referenced in your mapping, the new nodes will be unable to recover the cluster state and function. This is true even if the dictionary is referenced by an empty index you do not actually use.
Prepare your files for upload
editPlugins are uploaded as ZIP files. You need to choose whether your uploaded file should be treated as a plugin or as a bundle. Bundles are not installed as plugins. If you need to upload both a custom plugin and custom dictionaries, upload them separately.
To prepare your files, create one of the following:
- Plugins
-
A plugin is a ZIP file that contains a plugin descriptor file and binaries.
The plugin descriptor file is called either
stable-plugin-descriptor.properties
for plugins built against the stable plugin API, orplugin-descriptor.properties
for plugins built against the classic plugin API. A plugin ZIP file should only contain one plugin descriptor file.Elasticsearch assumes that the uploaded ZIP file contains binaries. If it finds any source code, it fails with an error message, causing provisioning to fail. Make sure you upload binaries, and not source code.
Plugins larger than 5GB should have the plugin descriptor file at the top of the archive. This order can be achieved by specifying at time of creating the ZIP file:
zip -r name-of-plugin.zip name-of-descriptor-file.properties *
- Bundles
-
The entire content of a bundle is made available to the node by extracting to the Elasticsearch container’s
/app/config
directory. This is useful to make custom dictionaries available. Dictionaries should be placed in a/dictionaries
folder in the root path of your ZIP file.Here are some examples of bundles:
Script
$ tree . . └── scripts └── test.js
The script
test.js
can be referred in queries as"script": "test"
.Dictionary of synonyms
$ tree . . └── dictionaries └── synonyms.txt
The dictionary
synonyms.txt
can be used assynonyms.txt
or using the full path/app/config/synonyms.txt
in thesynonyms_path
of thesynonym-filter
.To learn more about analyzing with synonyms, check Synonym token filter and Formatting Synonyms.
GeoIP database bundle
$ tree . . └── ingest-geoip └── MyGeoLite2-City.mmdb
Note that the extension must be
-(City|Country|ASN).mmdb
, and it must be a different name than the original file nameGeoLite2-City.mmdb
which already exists in Elasticsearch Service. To use this bundle, you can refer it in the GeoIP ingest pipeline asMyGeoLite2-City.mmdb
underdatabase_file
.
Add your extension
editYou must upload your files before you can apply them to your cluster configuration:
- Log in to the Elasticsearch Service Console.
- Find your deployment on the home page in the Elasticsearch Service card and select Manage to access it directly. Or, select Hosted deployments to go to the deployments page to view all of your deployments.
- Under Features, select Extensions.
- Select Upload extension.
-
Complete the extension fields, including the Elasticsearch version.
-
Plugins must use full version notation down to the patch level, such as
7.10.1
. You cannot use wildcards. This version notation should match the version in your plugin’s plugin descriptor file. For classic plugins, it should also match the target deployment version. -
Bundles should specify major or minor versions with wildcards, such as
7.*
or*
. Wildcards are recommended to ensure the bundle is compatible across all versions of these releases.
-
Plugins must use full version notation down to the patch level, such as
- Select the extension Type.
- Under Plugin file, choose the file to upload.
- Select Create extension.
After creating your extension, you can enable them for existing Elasticsearch deployments or enable them when creating new deployments.
Creating extensions larger than 200MB should be done through the extensions API.
Refer to Managing plugins and extensions through the API for more details.
Update your deployment configuration
editAfter uploading your files, you can select to enable them when creating a new Elasticsearch deployment. For existing deployments, you must update your deployment configuration to use the new files:
- Log in to the Elasticsearch Service Console.
-
Find your deployment on the home page in the Elasticsearch Service card and select Manage to access it directly. Or, select Hosted deployments to go to the deployments page to view all of your deployments.
On the deployments page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list.
- From the Actions dropdown, select Edit deployment.
- Select Manage user settings and extensions.
- Select the Extensions tab.
- Select the custom extension.
- Select Back.
- Select Save. The Elasticsearch cluster is then updated with new nodes that have the plugin installed.
Update your extension
editWhile you can update the ZIP file for any plugin or bundle, these are downloaded and made available only when a node is started.
You should be careful when updating an extension. If you update an existing extension with a new file, and if the file is broken for some reason, all the nodes could be in trouble, as a restart or move node could make even HA clusters non-available.
If the extension is not in use by any deployments, then you are free to update the files or extension details as much as you like. However, if the extension is in use, and if you need to update it with a new file, it is recommended to create a new extension rather than updating the existing one that is in use.
By following this method, only the one node would be down even if the extension file is faulty. This would ensure that HA clusters remain available.
This method also supports having a test/staging deployment to test out the extension changes before applying them on a production deployment.
You may delete the old extension after updating the deployment successfully.
To update an extension with a new file version,
- Prepare a new plugin or bundle.
- On the Extensions page, upload a new extension.
- Make your new files available by uploading them.
-
Find your deployment on the home page in the Elasticsearch Service card and select Manage to access it directly. Or, select Hosted deployments to go to the deployments page to view all of your deployments.
On the deployments page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list.
- From the Actions dropdown, select Edit deployment.
- Select Manage user settings and extensions.
- Select the Extensions tab.
- Select the new extension and de-select the old one.
- Select Back.
- Select Save.
How to use the extensions API
editFor a full set of examples, check Managing plugins and extensions through the API.
If you don’t already have one, create an API key
There are ways that you can use the extensions API to upload a file.
Method 1: Use HTTP POST
to create metadata and then upload the file using HTTP PUT
editStep 1: Create metadata
curl -XPOST \ -H "Authorization: ApiKey $EC_API_KEY" \ -H 'content-type:application/json' \ https://api.elastic-cloud.com/api/v1/deployments/extensions \ -d'{ "name" : "synonyms-v1", "description" : "The best synonyms ever", "extension_type" : "bundle", "version" : "7.*" }'
Step 2: Upload the file
curl -XPUT \ -H "Authorization: ApiKey $EC_API_KEY" \ "https://api.elastic-cloud.com/api/v1/deployments/extensions/$extension_id" \ -T /tmp/synonyms.zip
If you are using a client that does not have native application/zip
handling like curl
, be sure to use the equivalent of the following with content-type: multipart/form-data
:
curl -XPUT \ -H 'Expect:' \ -H 'content-type: multipart/form-data' \ -H "Authorization: ApiKey $EC_API_KEY" \ "https://api.elastic-cloud.com/api/v1/deployments/extensions/$extension_id" -F "file=@/tmp/synonyms.zip"
For example, using the Python requests
module, the PUT
request would be as follows:
import requests files = {'file': open('/tmp/synonyms.zip','rb')} r = requests.put('https://api.elastic-cloud.com/api/v1/deployments/extensions/{}'.format(extension_id), files=files, headers= {'Authorization': 'ApiKey {}'.format(EC_API_KEY)})
Method 2: Single step. Use a download_url
so that the API server downloads the object at the specified URL
editcurl -XPOST \ -H "Authorization: ApiKey $EC_API_KEY" \ -H 'content-type:application/json' \ https://api.elastic-cloud.com/api/v1/deployments/extensions \ -d'{ "name" : "anylysis_icu", "description" : "Helpful description", "extension_type" : "plugin", "version" : "7.13.2", "download_url": "https://artifacts.elastic.co/downloads/elasticsearch-plugins/analysis-icu/analysis-icu-7.13.2.zip" }'
Please refer to the Extensions API reference for the complete set of HTTP methods and payloads.
On this page
- Before you begin
- Prepare your files for upload
- Add your extension
- Update your deployment configuration
- Update your extension
- How to use the extensions API
- Method 1: Use HTTP
POST
to create metadata and then upload the file using HTTPPUT
- Method 2: Single step. Use a
download_url
so that the API server downloads the object at the specified URL
ElasticON events are back!
Learn about the Elastic Search AI Platform from the experts at our live events.
Register now