Source serialization
editSource serialization
editSource serialization refers to the process of (de)serializing POCO types in consumer applications as source documents indexed and retrieved from Elasticsearch. A source serializer implementation handles serialization, with the default implementation using the System.Text.Json
library. As a result, you may use System.Text.Json
attributes and converters to control the serialization behavior.
Modeling documents with types
editElasticsearch provides search and aggregation capabilities on the documents that it is sent and indexes. These documents are sent as JSON objects within the request body of a HTTP request. It is natural to model documents within the Elasticsearch .NET client using POCOs (Plain Old CLR Objects).
This section provides an overview of how types and type hierarchies can be used to model documents.
Default behaviour
editThe default behaviour is to serialize type property names as camelcase JSON object members.
We can model documents using a regular class (POCO).
public class MyDocument { public string StringProperty { get; set; } }
We can then index the an instance of the document into Elasticsearch.
using System.Threading.Tasks; using Elastic.Clients.Elasticsearch; var document = new MyDocument { StringProperty = "value" }; var indexResponse = await Client .IndexAsync(document, "my-index-name");
The index request is serialized, with the source serializer handling the MyDocument
type, serializing the POCO property named StringProperty
to the JSON object member named stringProperty
.
{ "stringProperty": "value" }
Customizing source serialization
editThe built-in source serializer handles most POCO document models correctly. Sometimes, you may need further control over how your types are serialized.
The built-in source serializer uses the Microsoft System.Text.Json
library internally. You can apply System.Text.Json
attributes and converters to control the serialization of your document types.
Using System.Text.Json
attributes
editSystem.Text.Json
includes attributes that can be applied to types and properties to control their serialization. These can be applied to your POCO document types to perform actions such as controlling the name of a property or ignoring a property entirely. Visit the Microsoft documentation for further examples.
We can model a document to represent data about a person using a regular class (POCO), applying System.Text.Json
attributes as necessary.
using System.Text.Json.Serialization; public class Person { [JsonPropertyName("forename")] public string FirstName { get; set; } [JsonIgnore] public int Age { get; set; } }
The |
|
The |
We can then index an instance of the document into Elasticsearch.
using System; using System.Text.Json; using System.Text.Json.Serialization; using System.Threading.Tasks; using Elastic.Transport; using Elastic.Clients.Elasticsearch; using Elastic.Clients.Elasticsearch.Serialization; var person = new Person { FirstName = "Steve", Age = 35 }; var indexResponse = await Client.IndexAsync(person, "my-index-name");
The index request is serialized, with the source serializer handling the Person
type, serializing the POCO property named FirstName
to the JSON object member named forename
. The Age
property is ignored and does not appear in the JSON.
{ "forename": "Steve" }
Configuring custom JsonSerializerOptions
editThe default source serializer applies a set of standard JsonSerializerOptions
when serializing source document types. In some circumstances, you may need to override some of our defaults. This is achievable by creating an instance of DefaultSourceSerializer
and passing an Action<JsonSerializerOptions>
, which is applied after our defaults have been set. This mechanism allows you to apply additional settings or change the value of our defaults.
The DefaultSourceSerializer
includes a constructor that accepts the current IElasticsearchClientSettings
and a configureOptions
Action
.
public DefaultSourceSerializer(IElasticsearchClientSettings settings, Action<JsonSerializerOptions> configureOptions);
Our application defines the following Person
class, which models a document we will index to Elasticsearch.
public class Person { public string FirstName { get; set; } }
We want to serialize our source document using Pascal Casing for the JSON properties. Since the options applied in the DefaultSouceSerializer
set the PropertyNamingPolicy
to JsonNamingPolicy.CamelCase
, we must override this setting. After configuring the ElasticsearchClientSettings
, we index our document to Elasticsearch.
using System; using System.Text.Json; using System.Text.Json.Serialization; using System.Threading.Tasks; using Elastic.Transport; using Elastic.Clients.Elasticsearch; using Elastic.Clients.Elasticsearch.Serialization; static void ConfigureOptions(JsonSerializerOptions o) => o.PropertyNamingPolicy = null; var nodePool = new SingleNodePool(new Uri("http://localhost:9200")); var settings = new ElasticsearchClientSettings( nodePool, sourceSerializer: (defaultSerializer, settings) => new DefaultSourceSerializer(settings, ConfigureOptions)); var client = new ElasticsearchClient(settings); var person = new Person { FirstName = "Steve" }; var indexResponse = await client.IndexAsync(person, "my-index-name");
A local function can be defined, accepting a |
|
When creating the |
The Person
instance is serialized, with the source serializer serializing the POCO property named FirstName
using Pascal Case.
{ "FirstName": "Steve" }
As an alternative to using a local function, we could store an Action<JsonSerializerOptions>
into a variable instead, which can be passed to the DefaultSouceSerializer
constructor.
Action<JsonSerializerOptions> configureOptions = o => o.PropertyNamingPolicy = null;
Registering custom System.Text.Json
converters
editIn certain more advanced situations, you may have types which require further customization during serialization than is possible using System.Text.Json
property attributes. In these cases, the recommendation from Microsoft is to leverage a custom JsonConverter
. Source document types serialized using the DefaultSourceSerializer
can leverage the power of custom converters.
For this example, our application has a document class that should use a legacy JSON structure to continue operating with existing indexed documents. Several options are available, but we’ll apply a custom converter in this case.
Our class is defined, and the JsonConverter
attribute is applied to the class type, specifying the type of a custom converter.
using System.Text.Json.Serialization; [JsonConverter(typeof(CustomerConverter))] public class Customer { public string CustomerName { get; set; } public CustomerType CustomerType { get; set; } } public enum CustomerType { Standard, Enhanced }
The |
When serializing this class, rather than include a string value representing the value of the CustomerType
property, we must send a boolean property named isStandard
. This requirement can be achieved with a custom JsonConverter implementation.
using System; using System.Text.Json; using System.Text.Json.Serialization; public class CustomerConverter : JsonConverter<Customer> { public override Customer Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) { var customer = new Customer(); while (reader.Read() && reader.TokenType != JsonTokenType.EndObject) { if (reader.TokenType == JsonTokenType.PropertyName) { if (reader.ValueTextEquals("customerName")) { reader.Read(); customer.CustomerName = reader.GetString(); continue; } if (reader.ValueTextEquals("isStandard")) { reader.Read(); var isStandard = reader.GetBoolean(); if (isStandard) { customer.CustomerType = CustomerType.Standard; } else { customer.CustomerType = CustomerType.Enhanced; } continue; } } } return customer; } public override void Write(Utf8JsonWriter writer, Customer value, JsonSerializerOptions options) { if (value is null) { writer.WriteNullValue(); return; } writer.WriteStartObject(); if (!string.IsNullOrEmpty(value.CustomerName)) { writer.WritePropertyName("customerName"); writer.WriteStringValue(value.CustomerName); } writer.WritePropertyName("isStandard"); if (value.CustomerType == CustomerType.Standard) { writer.WriteBooleanValue(true); } else { writer.WriteBooleanValue(false); } writer.WriteEndObject(); } }
When reading, this converter reads the |
|
When writing, this converter translates the |
We can then index a customer document into Elasticsearch.
using System; using System.Text.Json; using System.Text.Json.Serialization; using System.Threading.Tasks; using Elastic.Transport; using Elastic.Clients.Elasticsearch; using Elastic.Clients.Elasticsearch.Serialization; var customer = new Customer { CustomerName = "Customer Ltd", CustomerType = CustomerType.Enhanced }; var indexResponse = await Client.IndexAsync(customer, "my-index-name");
The Customer
instance is serialized using the custom converter, creating the following JSON document.
{ "customerName": "Customer Ltd", "isStandard": false }
Creating a custom SystemTextJsonSerializer
editThe built-in DefaultSourceSerializer
includes the registration of JsonConverter
instances which apply during source serialization. In most cases, these provide the proper behavior for serializing source documents, including those which use Elastic.Clients.Elasticsearch
types on their properties.
An example of a situation where you may require more control over the converter registration order is for serializing enum
types. The DefaultSourceSerializer
registers the System.Text.Json.Serialization.JsonStringEnumConverter
, so enum values are serialized using their string representation. Generally, this is the preferred option for types used to index documents to Elasticsearch.
In some scenarios, you may need to control the string value sent for an enumeration value. That is not directly supported in System.Text.Json
but can be achieved by creating a custom JsonConverter
for the enum
type you wish to customize. In this situation, it is not sufficient to use the JsonConverterAttribute
on the enum
type to register the converter. System.Text.Json
will prefer the converters added to the Converters
collection on the JsonSerializerOptions
over an attribute applied to an enum
type. It is, therefore, necessary to either remove the JsonStringEnumConverter
from the Converters
collection or register a specialized converter for your enum
type before the JsonStringEnumConverter
.
The latter is possible via several techniques. When using the Elasticsearch .NET library, we can achieve this by deriving from the abstract SystemTextJsonSerializer
class.
Here we have a POCO which uses the CustomerType
enum as the type for a property.
using System.Text.Json.Serialization; public class Customer { public string CustomerName { get; set; } public CustomerType CustomerType { get; set; } } public enum CustomerType { Standard, Enhanced }
To customize the strings used during serialization of the CustomerType
, we define a custom JsonConverter
specific to our enum
type.
using System.Text.Json.Serialization; public class CustomerTypeConverter : JsonConverter<CustomerType> { public override CustomerType Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) { return reader.GetString() switch { "basic" => CustomerType.Standard, "premium" => CustomerType.Enhanced, _ => throw new JsonException( $"Unknown value read when deserializing {nameof(CustomerType)}."), }; } public override void Write(Utf8JsonWriter writer, CustomerType value, JsonSerializerOptions options) { switch (value) { case CustomerType.Standard: writer.WriteStringValue("basic"); return; case CustomerType.Enhanced: writer.WriteStringValue("premium"); return; } writer.WriteNullValue(); } }
When reading, this converter translates the string used in the JSON to the matching enum value. |
|
When writing, this converter translates the |
We create a serializer derived from SystemTextJsonSerializer
to give us complete control of converter registration order.
using System.Text.Json; using Elastic.Clients.Elasticsearch.Serialization; public class MyCustomSerializer : SystemTextJsonSerializer { private readonly JsonSerializerOptions _options; public MyCustomSerializer(IElasticsearchClientSettings settings) : base(settings) { var options = DefaultSourceSerializer.CreateDefaultJsonSerializerOptions(false); options.Converters.Add(new CustomerTypeConverter()); _options = DefaultSourceSerializer.AddDefaultConverters(options); } protected override JsonSerializerOptions CreateJsonSerializerOptions() => _options; }
Inherit from |
|
In the constructor, use the factory method |
|
Register our |
|
To apply any default converters, call the |
|
Implement the |
Because we have registered our CustomerTypeConverter
before the default converters (which include the JsonStringEnumConverter
), our converter takes precedence when serializing CustomerType
instances on source documents.
The base SystemTextJsonSerializer
class handles the implementation details of binding, which is required to ensure that the built-in converters can access the IElasticsearchClientSettings
where needed.
We can then index a customer document into Elasticsearch.
using System; using System.Text.Json; using System.Text.Json.Serialization; using System.Threading.Tasks; using Elastic.Transport; using Elastic.Clients.Elasticsearch; using Elastic.Clients.Elasticsearch.Serialization; var customer = new Customer { CustomerName = "Customer Ltd", CustomerType = CustomerType.Enhanced }; var indexResponse = await client.IndexAsync(customer, "my-index-name");
The Customer
instance is serialized using the custom enum
converter, creating the following JSON document.
Creating a custom Serializer
editSuppose you prefer using an alternative JSON serialization library for your source types. In that case, you can inject an isolated serializer only to be called for the serialization of _source
, _fields
, or wherever a user-provided value is expected to be written and returned.
Implementing Elastic.Transport.Serializer
is technically enough to create a custom source serializer.
using System; using System.IO; using System.Threading; using System.Threading.Tasks; using Elastic.Transport; public class VanillaSerializer : Serializer { public override object Deserialize(Type type, Stream stream) => throw new NotImplementedException(); public override T Deserialize<T>(Stream stream) => throw new NotImplementedException(); public override ValueTask<object> DeserializeAsync(Type type, Stream stream, CancellationToken cancellationToken = default) => throw new NotImplementedException(); public override ValueTask<T> DeserializeAsync<T>(Stream stream, CancellationToken cancellationToken = default) => throw new NotImplementedException(); public override void Serialize<T>(T data, Stream stream, SerializationFormatting formatting = SerializationFormatting.None) => throw new NotImplementedException(); public override Task SerializeAsync<T>(T data, Stream stream, SerializationFormatting formatting = SerializationFormatting.None, CancellationToken cancellationToken = default) => throw new NotImplementedException(); }
Registering up the serializer is performed in the ConnectionSettings
constructor.
using System; using System.Text.Json; using System.Text.Json.Serialization; using System.Threading.Tasks; using Elastic.Transport; using Elastic.Clients.Elasticsearch; using Elastic.Clients.Elasticsearch.Serialization; var nodePool = new SingleNodePool(new Uri("http://localhost:9200")); var settings = new ElasticsearchClientSettings( nodePool, sourceSerializer: (defaultSerializer, settings) => new VanillaSerializer()); var client = new ElasticsearchClient(settings);
There are various cases where you might have a POCO type that contains an Elastic.Clients.Elasticsearch
type as one of its properties. The SourceSerializerFactory
delegate provides access to the default built-in serializer so you can access it when necessary. For example, consider if you want to use percolation; you need to store Elasticsearch queries as part of the _source
of your document, which means you need to have a POCO that looks like this.
using Elastic.Clients.Elasticsearch.QueryDsl; public class MyPercolationDocument { public Query Query { get; set; } public string Category { get; set; } }
A custom serializer would not know how to serialize Query
or other Elastic.Clients.Elasticsearch
types that could appear as part of
the _source
of a document. Therefore, your custom Serializer
would need to store a reference to our built-in serializer and delegate serialization of Elastic types back to it.