Release notes v8.0.0
editRelease notes v8.0.0
editDue to the extensive changes in the new Elasticsearch .NET Client, we highly recommend reviewing this documentation in full, before proceeding with migration to v8.
After many months of work, eleven alphas, six betas and two release candidates, we are pleased to announce the GA release of the Elasticsearch .NET Client v8.0.0.
The overall themes of this release have been based around redesigning the client for the future, standardizing serialization, performance improvements, codebase simplification, and code-generation.
The following release notes explain the current state of the client, missing features, breaking changes and our rationale for some of the design choices we have introduced.
Version 8 is a refresh
editIt is important to highlight that v8 of the Elasticsearch .NET Client represents a new start for the client design. It is important to review how this may affect your code and usage.
Mature code becomes increasingly hard to maintain over time, and our ability to make timely releases has diminished as code complexity has increased. Major releases allow us to simplify and better align our language clients with each other in terms of design. Here, it is crucial to find the right balance between uniformity across programming languages and the idiomatic concerns of each language. For .NET, we will typically compare and contrast with Java and Go to make sure that our approach is equivalent for each of these. We also take heavy inspiration from Microsoft framework design guidelines and the conventions of the wider .NET community.
New Elastic.Clients.Elasticsearch NuGet package
editWe have shipped the new code-generated client as a
new NuGet package
with a new root namespace, Elastic.Clients.Elasticsearch
.
The new v8 client is built upon the foundations of the v7 NEST
client, but there
are changes. By shipping as a new package, the expectation is that migration can
be managed with a phased approach.
While this is a new package, we have aligned the major version (v8.x.x) with the supported Elasticsearch server version to clearly indicate the client/server compatibility. The v8 client is designed to work with version 8 of Elasticsearch.
The v7 NEST
client continues to be supported but will not gain new features or
support for new Elasticsearch endpoints. It should be considered deprecated in favour of
the new client.
Limited feature set
editThe 8.0.0 Elasticsearch .NET Client does not have feature parity with the previous v7 NEST
high-level client.
For the initial 8.0.0 release we have limited the features we are shipping. Over the course of the 8.x releases, we will reintroduce features. Therefore, if something is missing, it may not be permanently omitted. We will endeavour to communicate our plans as and when they become available.
If a feature you depend on is missing (and not explicitly documented below as a feature that we do not plan to reintroduce), please open an issue or comment on a relevant existing issue to highlight your need to us. This will help us prioritise our roadmap.
Code generation
editGiven the size of the Elasticsearch API surface today, it is no longer practical to maintain thousands of types (requests, responses, queries, aggregations, etc.) by hand. To ensure consistent, accurate and timely alignment between language clients and Elasticsearch, the 8.x clients, and many of the associated types are now automatically code-generated from a shared specification. This is a common solution to maintaining alignment between client and server among SDKs and libraries, such as those for Azure, AWS and the Google Cloud Platform.
Code-generation from a specification has inevitably led to some differences
between the existing v7 NEST
types and those available in the new v7 Elasticsearch .NET Client.
For the 8.0.0 release, we generate strictly from the specification, special
casing a few areas to improve usability or to align with language idioms.
The base type hierarchy for concepts such as Properties
, Aggregations
and
Queries
is no longer present in generated code, as these arbitrary groupings do
not align with concrete concepts of the public server API. These considerations
do not preclude adding syntactic sugar and usability enhancements to types in future
releases on a case-by-case basis.
Elastic.Transport
editThe .NET client includes a transport layer responsible for abstracting HTTP concepts and to provide functionality such as our request pipeline. This supports round-robin load-balancing of requests to nodes, pinging failed nodes and sniffing the cluster for node roles.
In v7, this layer shipped as Elasticsearch.Net
and was considered our low-level
client which could be used to send and receive raw JSON bytes between the client
and server.
As part of the work for 8.0.0, we have moved the transport layer out into
a new dedicated package and
repository, named
Elastic.Transport
. This supports reuse across future clients and allows
consumers with extremely high-performance requirements to build upon this foundation.
System.Text.Json for serialization
editThe v7 NEST
high-level client used an internalized and modified version of
Utf8Json for request and response
serialization. This was introduced for its performance improvements
over Json.NET, the more common JSON framework at
the time.
While Utf8Json provides good value, we have identified minor bugs and performance issues that have required maintenance over time. Some of these are hard to change without more significant effort. This library is no longer maintained, and any such changes cannot easily be contributed back to the original project.
With .NET Core 3.0, Microsoft shipped new System.Text.Json APIs
that are included in-the-box with current versions of .NET. We have adopted
System.Text.Json
for all serialization. Consumers can still define and register
their own Serializer
implementation for their document types should they prefer
to use a different serialization library.
By adopting System.Text.Json
, we now depend on a well-maintained and supported
library from Microsoft. System.Text.Json
is designed from the ground up to support
the latest performance optimizations in .NET and, as a result, provides both fast and low-allocation serialization.
Mockability of ElasticsearchClient
editTesting code is an important part of software development. We recommend that consumers prefer introducing an abstraction for their use of the Elasticsearch .NET Client as the prefered way to decouple consuming code from client types and support unit testing.
In order to support user testing scenarios, we have unsealed the ElasticsearchClient
type and made its methods virtual. This supports mocking the type directly for unit
testing. This is an improvement over the original IElasticClient
interface from
NEST
(v7) which only supported mocking of top-level client methods.
We have also introduced a TestableResponseFactory
in Elastic.Transport
to
make it easier to create response instances with specific status codes and validity
that can be used during unit testing.
These changes are in addition to our existing support for testing with an
InMemoryConnection
, virtualized clusters and with our
Elastic.Elasticsearch.Managed
library for integration
testing against real Elasticsearch instances. We will introduce more documentation on testing methodologies in the near future.
Migrating to Elastic.Clients.Elasticsearch
editThe 8.0.0 release does not currently have full-feature parity with NEST
. The
client primary use case is for application developers communitating with Elasticsearch.
The 8.0.0 release focuses on core endpoints, more specifically for common CRUD
scenarios. We intend to reduce the feature gap in subsequent versions. We anticipate
that this initial release will best suit new applications and may not yet be migration-ready for all existing consumers. We recommend reviewing this documentation carefully to learn about the missing features and reduced API surface details before migrating from the v7 NEST
client.
The choice to code-generate a new evolution of the Elasticsearch .NET Client introduces some significant breaking changes. We consciously took the opportunity to refactor and reconsider historic design choices as part of this major release, intending to limit future breaking changes.
The v8 client is shipped as a new NuGet package
which can be installed alongside v7 NEST
. We
anticipate that some consumers may prefer a phased migration with both
packages side-by-side for a short period of time to manage complex migrations. In addition, NEST
7.17.x can continue to be used in
compatibility mode
with Elasticsearch 8.x servers until the v8 Elasticsearch .NET Client features
align with application requirements.
We will continue to prioritize the feature roadmap and code-generation work based on feedback from consumers who may rely on features that are initially unavailable.
Breaking Changes
editAs a result of code-generating a majority of the client types, this version of the client includes multiple breaking changes.
We have strived to keep the core foundation reasonably similar, but types emitted
through code-generation are subject to change between NEST
(v7) and the new
Elastic.Clients.Elasticsearch
(v8) package.
Namespaces
editWe have renamed the package and top-level namespace for the v8 client to
Elastic.Clients.Elasticsearch
. All types belong to this namespace. When
necessary, to avoid potential conflicts, types are generated into suitable
sub-namespaces based on the Elasticsearch specification. Additional using
directives may be required to access such types
when using the Elasticsearch .NET Client.
Transport layer concepts have moved to the new Elastic.Transport
NuGet package
and related types are defined under its namespace. Some configuration and low-level transport functionality may require a using
directive for the Elastic.Transport
namespace.
Type names
editType names may have changed from previous versions. We are not listing these explicitly due to the potentially vast number of subtle differences. Type names will now more closely align to those used in the JSON and as documented in the Elasticsearch documentation.
Class members
editTypes may include renamed properties based on the Elasticsearch specification,
which differ from the original NEST
property names. The types used for properties
may also have changed due to code-generation. If you identify missing or
incorrectly-typed properties, please open an issue to alert us.
Sealing classes
editOpinions on "sealing by default" within the .NET ecosystem tend to be quite polarized. Microsoft seal all internal types for potential performance gains and we see a benefit in starting with that approach for the Elasticsearch .NET Client, even for our public API surface.
While it prevents inheritance and, therefore, may inhibit a few consumer scenarios,
sealing by default is intended to avoid the unexpected or invalid
extension of types that could inadvertently be broken in the future. That said,
sealing is not necessarily a final choice for all types; but it is clearly
easier for a future release to unseal a previously-sealed class than
vice versa. We can therefore choose to unseal as valid scenarios arise,
should we determine that doing so is the best solution for those scenarios, such
as with mockability of the ElasticsearchClient
. This goes back to our clean-slate concept for this new client.
Removed features
editAs part of the clean-slate redesign of the new client, we have opted to remove certain features from the v8.0 client. These are listed below:
Attribute mappings
editIn previous versions of the NEST
client, attributes could be used to configure
the mapping behaviour and inference for user types. We have removed support for
these attributes and recommend that mapping be completed via the fluent API when
configuring client instances. System.Text.Json
attributes may be used to rename
and ignore properties during source serialization.
CAT APIs
editThe CAT APIs of Elasticsearch are intended for human-readable usage and will no longer be supported via the v8 Elasticsearch .NET Client.
Interface removal
editWe have removed several interfaces that previously shipped as part of NEST
. This
is a design decision to simplify the library and avoid interfaces where only a
single implementation of that interface is expected to exist, such as
IElasticClient
in NEST
. We have also switched to prefer abstract base classes
over interfaces across the library, as this allows us to add enhancements more
easily without introducing breaking changes for derived types.
Missing features
editWhile not an exhaustive list, the following are some of the main features which have not been re-implemented for this initial 8.0.0 GA release. These remain on our roadmap and will be reviewed and prioritized for inclusion in future releases.
- Query DSL operators for combining queries.
- Scroll Helper.
- Fluent API for union types.
-
AutoMap
for field datatype inference. -
Visitor pattern support for types such as
Properties
. -
Support for
JoinField
which affectsChildrenAggregation
. - Conditionless queries.
-
DiagnosticSources have been removed in
Elastic.Transport
to provide a clean-slate for an improved diagnostics story. The Elasticsearch .NET Client emits OpenTelemetry compatibleActivity
spans which can be consumed by APM agents such as the Elastic APM Agent for .NET. - Documentation is a work in progress, and we will expand on the documented scenarios in future releases.
Reduced API surface
editIn this first release of the code-generated .NET client, we have specifically focused on supporting commonly used endpoints. We have also skipped specific queries and aggregations which need further work to generate code correctly. Before migrating, please refer to the lists below for the endpoints, queries and aggregations currently generated and available in the 8.0.0 GA release to ensure that the features you are using are currently supported.
Supported Elasticsearch endpoints
editThe following are Elasticsearch endpoints currently generated and available in the 8.0.0 Elasticsearch .NET Client.
- AsyncSearch.Delete
- AsyncSearch.Get
- AsyncSearch.Status
- AsyncSearch.Submit
- Bulk
- ClearScroll
- ClosePointInTime
- Cluster.Health
- Count
- Create
- Delete
- DeleteByQuery
- DeleteByQueryRethrottle
- DeleteScript
- EQL.Delete
- EQL.Get
- EQL.Search
- EQL.Status
- Exists
- ExistsSource
- Explain
- FieldCaps
- Get
- GetScript
- GetScriptContext
- GetScriptLanguages
- GetSource
- Index
- Indices.Clone
- Indices.Close
- Indices.Create
- Indices.CreateDataStream
- Indices.Delete
- Indices.DeleteAlias
- Indices.DeleteDataStream
- Indices.DeleteIndexTemplate
- Indices.DeleteTemplate
- Indices.Exists
- Indices.ExistsIndexTemplate
- Indices.ExistsTemplate
- Indices.Flush
- Indices.ForceMerge
- Indices.Get
- Indices.GetAlias
- Indices.GetDataStream
- Indices.GetFieldMapping
- Indices.GetIndexTemplate
- Indices.GetMapping
- Indices.GetTemplate
- Indices.Indices.SimulateTemplate
- Indices.MigrateToDataStream
- Indices.Open
- Indices.PutAlias
- Indices.PutIndexTemplate
- Indices.PutMapping
- Indices.PutTemplate
- Indices.Refresh
- Indices.Rollover
- Indices.Shrink
- Indices.SimulateIndexTemplate
- Indices.Split
- Indices.Unfreeze
- Info
- MGet
- MSearch
- MSearchTemplate
- OpenPointInTime
- Ping
- PutScript
- RankEval
- Reindex
- ReindexRethrottle
- Scroll
- Search
- SearchShards
- SQL.ClearCursor
- SQL.DeleteAsync
- SQL.GetAsync
- SQL.GetAsyncStatus
- SQL.Query
- TermsEnum
- Update
- UpdateByQuery
- UpdateByQueryRethrottle
Supported queries
editThe following are query types currently generated and available in the 8.0.0 Elasticsearch .NET Client.
- Bool
- Boosting
- CombinedFields
- ConstantScore
- DisMax
- Exists
- FunctionScore
- Fuzzy
- HasChild
- HasParent
- Ids
- Intervals
- Match
- MatchAll
- MatchBool
- MatchNone
- MatchPhrase
- MatchPhrasePrefix
- MoreLikeThis
- MultiMatch
- Nested
- ParentId
- Percolate
- Prefix
- QueryString
- RankFeature
- Regexp
- Script
- ScriptScore
- Shape
- SimpleQueryString
- SpanContaining
- SpanFirst
- SpanMultiTerm
- SpanNear
- SpanNot
- SpanOr
- SpanTerm
- SpanWithin
- Term
- Terms
- TermsSet
- Wildcard
- Wrapper
Supported aggregations
editThe following are aggregation types currently generated and available in the 8.0.0 Elasticsearch .NET Client.
- AdjacencyMatrix
- AutoDateHistogram
- Avg
- Boxplot
- Cardinality
- Children
- Composite
- CumulativeCardinality
- DateHistogram
- DateRange
- Derivative
- ExtendedStats
- Filters
- Global
- Histogram
- Inference
- IpRange
- MatrixStats
- Max
- MedianAbsoluteDeviation
- Min
- Missing
- MultiTerms
- Nested
- Parent
- PercentilesBucket
- Range
- Rate
- ReverseNested
- Sampler
- ScriptedMetric
- Stats
- StatsBucket
- StringStats
- Sum
- Terms
- TopHits
- TopMetrics
- TTest
- ValueCount
- VariableWidthHistogram
- WeightedAvg
In closing
editPlease give the new Elastic.Clients.Elasticsearch
client a try in your .NET
applications. If you run into any problems, please open an issue to raise those
with us. Please let us know how you get on and if you have any questions,
reach out on the Discuss forums.