The Go client for Elasticsearch: Introduction
The official Go client for Elasticsearch is one of the latest additions to the family of clients developed, maintained, and supported by Elastic. The initial version was published early in 2019 and has matured over the past year, gaining features such as retrying requests, discovering cluster nodes, and various helper components. We also provide comprehensive examples to facilitate using the client.
In this series, we’ll explore the architecture and design of the Go client, highlight specific implementation details, and provide examples and guidance for its usage.
In this blog post, we’ll focus on the overall architecture of the client and the package and repository layout.
The client architecture
Every Elasticsearch client library packs a lot of functionality inside. Therefore, it is imperative to separate the concerns properly for easy maintenance and development. From a bird’s-eye view, an Elasticsearch client has two major concerns:
- Exposing the Elasticsearch APIs in the respective programming language
- Sending and receiving data from the cluster
Naturally, the picture is more complicated in the details (How exactly is the data being sent and received? How is the data exposed to the calling code?) but the overall picture is simple.
The elasticsearch package makes this separation of concerns transparent: it physically separates the two layers into different packages, with an umbrella package that ties them together. (This pattern is also implemented by the Ruby client.)
$ go list github.com/elastic/go-elasticsearch/v7/...
github.com/elastic/go-elasticsearch/v7
github.com/elastic/go-elasticsearch/v7/esapi
github.com/elastic/go-elasticsearch/v7/estransport
...
The esapi and estransport packages directly correspond to the two concerns mentioned above.
This has significant consequences for the maintainability, extensibility, and flexibility of the client. First of all, it is straightforward to separate code, tests, and other experiments related to only a single concern; if you look into the estransport unit and integration tests, it is apparent that they are not concerned with the Elasticsearch API at all. Second, it is straightforward to use only a single package, in isolation: for example, only the API-related package. A common question is: “why would somebody do that?” The right answer takes a step back: the package doesn't want to make decisions that prevent users from achieving a specific goal, no matter how rare or unusual.
This trajectory of thinking illustrates the overall approach of the official Elasticsearch clients: provide the best out-of-the-box experience, but don't prevent users from twisting the libraries in unforeseen ways. Over the years, we have loved to see people using the various client extension points to achieve a specific goal in an esoteric environment. For example, earlier this year The Guardian published an article on a custom component for cluster node discovery for the Java client.
Note: You may wonder why the packages are not named simply api and transport. The motivation is to prevent “stealing” a legitimate package or variable name from the user — after all, a package or variable named api is something that occurs quite often.
How are these packages tied together, then? That's the responsibility of the umbrella elasticsearch package.
$ go doc -short github.com/elastic/go-elasticsearch/v7
...
type Client struct{ ... }
func NewClient(cfg Config) (*Client, error)
func NewDefaultClient() (*Client, error)
type Config struct{ ... }
The footprint of this package is intentionally small, and its most important components are the Client and Config types; the latter provides a way to configure the client, and the former embeds the Elasticsearch APIs and the HTTP transport.
$ go doc -short github.com/elastic/go-elasticsearch/v7.Client
type Client struct {
*esapi.API // Embeds the API methods
Transport estransport.Interface
}
func NewClient(cfg Config) (*Client, error)
func NewDefaultClient() (*Client, error)
...
The package exports the client initialization methods: NewDefaultClient() and NewClient(). We'll focus on the configuration and customization of the client in the next blog post.
The esapi package
The esapi package provides access to the Elasticsearch APIs through the Go programming language data types. To index a document, for example, you call the corresponding method on the client:
res, err := client.Index(
"my-index",
strings.NewReader(`{"title":"Test"}`),
client.Index.WithDocumentID("1"))
fmt.Println(res, err)
The Go package provides the same API as clients in other languages, allowing for a consistent user experience across programming languages and facilitating communication in polyglot teams. Consequently, the various namespaces of the Elasticsearch API are available as namespaces on the client. For example, to check the cluster health, you call the Cluster.Health() method on the client; to create an index, you call the Indices.Create() method.
The method returns esapi.Response and error. The latter is returned whenever the request fails; for instance, when the endpoint is unreachable or the request times out. The esapi.Response type is a lightweight wrapper around *http.Response. Apart from exposing the response status, headers, and body, it provides a handful of helper methods, such as IsError(). Note that a 500 Internal Server Error response is still a valid response, and therefore no error is returned in that case — the calling code has to inspect the response status to handle the response correctly. The esapi.Response type also implements the fmt.Stringer interface to allow printing the response during development and debugging, as seen in the example above.
If you look closely, you'll discover that the method call creates a new instance of the IndexRequest struct, and calls its Do() method. It is entirely possible to create the instance and call the method by yourself — the corresponding code would look like this:
req := esapi.IndexRequest{
Index: "my-index",
DocumentID: "1",
Body: strings.NewReader(`{"title":"Test"}`),
}
req.Do(context.Background(), client)
Both variants are functionally equivalent. The method-oriented API has almost negligible overhead, but also has certain advantages.
First of all, it is arguably easier to read and write for humans, as the code flow is more fluent, with a more succinct naming and a tighter interface.
Equally important, it clearly defines which API parameters are required (in the example above, those are the index name and the JSON payload) and which are optional (the document ID). Compare with the Create() API, which makes it clear, in the method signature, that a document ID is required:
$ go doc -short github.com/elastic/go-elasticsearch/v7/esapi.Create
type Create func(index string, id string, body io.Reader, o ...func(*CreateRequest)) (*Response, error)
...
This is especially useful for developers using code completion in their IDEs and editors.
Another convenience provided by the method-oriented API is related to parameters accepting boolean and numerical values. Because all types have a default value in Go, the package wouldn't be able to tell if a value of false is set by the calling code or if it's just the default value for the bool type; a similar problem exists with the int type and a value of 0. This is commonly solved in Go by accepting a pointer to the value as an argument, but that makes the calling code rather verbose.
Therefore, instead of having to declare a variable and pass a pointer to it, the calling code can simply pass the value to the appropriate method, and it will create the pointer automatically:
client.Reindex(
strings.NewReader(`{...}`),
client.Reindex.WithRequestsPerSecond(100),
client.Reindex.WaitForCompletion(true),
)
Note: The package declares the esapi.BoolPtr() and esapi.IntPtr() helper functions to make the use of fields accepting a pointer easier when using the request structs directly.
With over 300 different, constantly evolving Elasticsearch APIs, it is practically impossible to manage the codebase by hand — the only sustainable maintenance mode is a fully generated codebase. Luckily, the Elasticsearch repository contains a comprehensive specification for each and every API, as a collection of JSON documents. This is the primary tool for ensuring consistency across the clients, and for keeping up with the Elasticsearch API evolution.
The Go generator, which is part of a collection of internal packages, translates the API definition from JSON into Go source code, generates the individual files, formats them with gofmt, and generates the file with the esapi.API type with a “map” of all the APIs, using reflection in Go. This type is then embedded into the client.
The Elasticsearch repository contains a corresponding resource as well: a collection of integration tests, encoded as YAML files. Yet another internal package of the Go client translates the YAML definitions into regular Go test files. These tests are run periodically in the continuous integration environment, catching any missing APIs or parameters.
The last internal package generates the examples in the Elasticsearch documentation: to see the output, visit the page for the master version and switch the language of the examples to Go.
As noted above, one of the concerns of an Elasticsearch client is to send and receive data, by default a JSON payload. The Go client exposes both request and response body simply as an io.Reader, leaving any encoding and decoding to the calling code. There are multiple reasons for this implementation. First of all, there's no formal specification for the exceedingly variable API payloads, and thus code generation is out of the question.
Note: There are multiple efforts going on to improve this situation, with the goal of making a fully generated API possible, e.g., the search results, the Query DSL components, and so on.
Another reason has to do with performance and extensibility. By leaving the encoding and decoding to the calling code, there's a clear boundary between it and the client, making reasoning about performance easier. Empirically, JSON encoding and decoding has the biggest influence on the performance of the client, usually higher than the cost of network transfer. From yet another point of view, by offloading the encoding and decoding to the calling code, it is straightforward to use third-party JSON packages, which in most cases surpass the standard library by a large margin. To see examples and benchmarks for various JSON packages, see the _examples/encoding folder in the repository.
Note: To make passing custom payloads to the API easier, the client provides the esutil.JSONReader type. We’ll focus on the esutil package in one of the next posts in this series.
The estransport package
So far, we have been concerned with the first responsibility of the client: exposing the Elasticsearch APIs as types in the Go programming language. Let's switch focus to the component responsible for sending and receiving data over the network: the estransport package.
The main type exposed by the package is estransport.Client, which implements estransport.Interface. It defines a single method, Perform(), which accepts an *http.Request and returns an *http.Response:
$ go doc -short github.com/elastic/go-elasticsearch/v7/estransport.Interface
type Interface interface {
Perform(*http.Request) (*http.Response, error)
}
The default implementation of this method is the heart of the transport component, dealing not only with sending and receiving data, but also with managing the pool of connections, retrying requests, storing client metrics, logging the operations, and so on. Let's have a closer look.
Before the client can send a request to the cluster, it first needs to know where to send it. For local development, that's pretty simple: you just keep the default (http://localhost:9200), or configure the client with a single address. In this case, a "single" connection pool is used, which returns only a single connection. This is also the case when using Elasticsearch Service on Elastic Cloud, which provides only a single endpoint for the cluster because it has its own load balancing logic. And the same applies, naturally, for any cluster behind a load balancer or a proxy.
On-premises production clusters run with multiple nodes, though, and it wouldn't be optimal to send requests only to a single node, making it the bottleneck. That's why the package exports an estransport.ConnectionPool interface, which allows for choosing a connection from a list:
$ go doc -short github.com/elastic/go-elasticsearch/v7/estransport.ConnectionPool
type ConnectionPool interface {
Next() (*Connection, error) // Next returns the next available connection.
OnSuccess(*Connection) error // OnSuccess reports that the connection was successful.
OnFailure(*Connection) error // OnFailure reports that the connection failed.
URLs() []*url.URL // URLs returns the list of URLs of available connections.
}
...
Whenever the client is configured with multiple cluster addresses, the "status" connection pool implementation is used, which keeps a list of healthy and unhealthy connections, with a mechanism to check if an unhealthy connection has become healthy again. In line with the general extensibility of the client, a custom implementation of the connection pool can be passed in configuration when needed.
The Next() method of the "status" connection pool delegates to yet another interface: estransport.Selector, with a default implementation of a round-robin selector, which is usually the most effective way to spread the load across cluster nodes. Again, a custom implementation of the selector can be passed in the configuration, when a more sophisticated connection choosing mechanism is needed in complicated network topologies.
For example, a simplified "hostname" selector implementation could look like this:
func (s *HostnameSelector) Select(conns []*estransport.Connection) (*estransport.Connection, error) {
// Access locking ommited
var filteredConns []*estransport.Connection
for _, c := range conns {
if strings.Contains(c.URL.String(), "es1") {
filteredConns = append(filteredConns, c)
}
}
if len(filteredConns) > 0 {
s.current = (s.current + 1) % len(filteredConns)
return filteredConns[s.current], nil
}
return nil, errors.New("No connection with hostname [es1] available")
}
However, a custom selector is more useful with another feature of the client: the ability to discover nodes in the cluster, colloquially known as "sniffing." It uses the Nodes Info API to retrieve information about nodes in the cluster and updates the client configuration dynamically based on the cluster state. This allows you, for example, to point the client only to the cluster coordinating nodes and lets it discover the data or ingest nodes automatically. (Read more about the feature in the corresponding blog post.)
The Go client exposes a method, DiscoverNodes(), to perform the operation manually, and the DiscoverNodesInterval and DiscoverNodesOnStart configuration options to perform the operation when the client is initialized and in periodic intervals. Apart from getting a list of nodes and making the client configuration dynamic, it also stores the metadata attached to the nodes, such as roles or attributes.
To return briefly to the use case reported by The Guardian mentioned above, solving the issue would mean providing a custom selector that would filter the Attributes field of the connection for a specific value, corresponding to the specific client instances running in a specific availability zone:
func (s *AttributeSelector) Select(conns []*estransport.Connection) (*estransport.Connection, error) {
// ...
for _, c := range conns {
if _, ok := c.Attributes["attribute-name"]; ok {
if c.Attributes["attribute-name"] == "attribute-value" {
filteredConns = append(filteredConns, c)
}
}
}
// ...
}
Note: It needs to be emphasized again that node discovery is useful only when the client is connected to the cluster directly, not when the cluster is behind a proxy, which is also the case when using Elasticsearch Service on Elastic Cloud.
Another useful feature of the transport component is the ability to retry a failed request, which is especially important in a distributed system such as Elasticsearch. By default, it retries the request up to three times when it receives a network error, or an HTTP response with status code 502, 503, or 504. The number of retry attempts and HTTP response codes is configurable, along with the optional backoff delay. This feature can be disabled altogether.
Next steps
As we’ve seen, the simple operation of sending a request and getting a response is actually quite complicated underneath. In order to understand what exactly is happening, the client provides several types of logging components, ranging from a colorized logger useful in local development to a JSON-based logger suitable for production, as well as a standardized metrics output.
We'll have a look at how to use these components, along with all the other client configuration options, in the next blog post: The Go client for Elasticsearch: Configuration and customization.