ElasticSearch index works from REST API, but not C# code

The 6.x Elasticsearch high level client, NEST, internalized the Json.NET dependency by

• IL-merging Json.NET assembly
• converting all types to internal
• renamespacing them under Nest.*

What this means in practice is that the client does not have a direct dependency on Json.NET (have a read of the release blog post to understand why we did this) and does not know about Json.NET types, including JsonPropertyAttribute or JsonConverter.

There are several ways of solving this. To begin, the following setup may be helpful during development

var defaultIndex = "default-index";var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));var settings = new ConnectionSettings(pool)    .DefaultMappingFor<DataEntity>(m => m        .IndexName(defaultIndex)        .TypeName("_doc")    )    .DisableDirectStreaming()    .PrettyJson()    .OnRequestCompleted(callDetails =>    {        if (callDetails.RequestBodyInBytes != null)        {            Console.WriteLine(                $"{callDetails.HttpMethod} {callDetails.Uri} \n" +                $"{Encoding.UTF8.GetString(callDetails.RequestBodyInBytes)}");        }        else        {            Console.WriteLine($"{callDetails.HttpMethod} {callDetails.Uri}");        }        Console.WriteLine();        if (callDetails.ResponseBodyInBytes != null)        {            Console.WriteLine($"Status: {callDetails.HttpStatusCode}\n" +                     $"{Encoding.UTF8.GetString(callDetails.ResponseBodyInBytes)}\n" +                     $"{new string('-', 30)}\n");        }        else        {            Console.WriteLine($"Status: {callDetails.HttpStatusCode}\n" +                     $"{new string('-', 30)}\n");        }    });var client = new ElasticClient(settings);

This will write all requests and responses out to the console, so you can see what the client is sending and receiving from Elasticsearch. .DisableDirectStreaming() buffers the request and response bytes in memory, to make them available to the delegate passed to .OnRequestCompleted(), so it's useful for development but you'll probably don't want it in production as it comes at a performance cost.

Now, the solutions:

1. Use PropertyNameAttribute

Instead of using JsonPropertyAttribute, you can use PropertyNameAttribute to name the properties for serialization

public sealed class GeoLocationEntity{    public GeoLocationEntity(        double latitude,        double longitude)    {        this.Latitude = latitude;        this.Longitude = longitude;    }    [PropertyName("lat")]    public double Latitude { get; }    [PropertyName("lon")]    public double Longitude { get; }}public sealed class DataEntity{    public DataEntity(        GeoLocationEntity location)    {        this.Location = location;    }    [PropertyName("location")]    public GeoLocationEntity Location { get; }}

and to use

if (client.IndexExists(defaultIndex).Exists)    client.DeleteIndex(defaultIndex);var createIndexResponse = client.CreateIndex(defaultIndex, c => c     .Mappings(m => m        .Map<DataEntity>(mm => mm            .AutoMap()            .Properties(p => p                .GeoPoint(g => g                    .Name(n => n.Location)                )            )        )    ));var indexResponse = client.Index(    new DataEntity(new GeoLocationEntity(88.59, -98.87)),     i => i.Refresh(Refresh.WaitFor));var searchResponse = client.Search<DataEntity>(s => s    .Query(q => q        .MatchAll()    ));

PropertyNameAttribute acts similarly to how you would normally use JsonPropertAttribute with Json.NET.

2. Use DataMemberAttribute

This will work the same as PropertyNameAttribute in this instance, if you'd prefer your POCOs to not be attributed with NEST types (although I'd argue that the POCOs are tied to Elasticsearch, so tying them to .NET Elasticsearch types is probably not an issue).

3. Use Geolocation type

You could replace GeoLocationEntity type with Nest's GeoLocation type, that maps to geo_point field datatype mapping. In using this, it's one less POCO, and the correct mapping can be inferred from the property type

public sealed class DataEntity{    public DataEntity(        GeoLocation location)    {        this.Location = location;    }    [DataMember(Name = "location")]    public GeoLocation Location { get; }}// ---if (client.IndexExists(defaultIndex).Exists)    client.DeleteIndex(defaultIndex);var createIndexResponse = client.CreateIndex(defaultIndex, c => c     .Mappings(m => m        .Map<DataEntity>(mm => mm            .AutoMap()        )    ));var indexResponse = client.Index(    new DataEntity(new GeoLocation(88.59, -98.87)),     i => i.Refresh(Refresh.WaitFor));var searchResponse = client.Search<DataEntity>(s => s    .Query(q => q        .MatchAll()    ));

4. Hooking up JsonNetSerializer

NEST allows a custom serializer to be hooked up, to take care of serializing your types. A separate nuget package, NEST.JsonNetSerializer, allows you to use Json.NET to serialize your types, with the serializer delegating back to the internal serializer for properties that are NEST types.

First, you need to pass the JsonNetSerializer into ConnectionSettings constructor

var settings = new ConnectionSettings(pool, JsonNetSerializer.Default)

Then your original code will work as expected, without the custom JsonConverter

public sealed class GeoLocationEntity{    public GeoLocationEntity(        double latitude,        double longitude)    {        this.Latitude = latitude;        this.Longitude = longitude;    }    [JsonProperty("lat")]    public double Latitude { get; }    [JsonProperty("lon")]    public double Longitude { get; }}public sealed class DataEntity{    public DataEntity(        GeoLocationEntity location)    {        this.Location = location;    }    [JsonProperty("location")]    public GeoLocationEntity Location { get; }}// ---if (client.IndexExists(defaultIndex).Exists)    client.DeleteIndex(defaultIndex);var createIndexResponse = client.CreateIndex(defaultIndex, c => c     .Mappings(m => m        .Map<DataEntity>(mm => mm            .AutoMap()            .Properties(p => p                .GeoPoint(g => g                    .Name(n => n.Location)                )            )        )    ));var indexResponse = client.Index(    new DataEntity(new GeoLocationEntity(88.59, -98.87)),     i => i.Refresh(Refresh.WaitFor));var searchResponse = client.Search<DataEntity>(s => s    .Query(q => q        .MatchAll()    ));

I listed this option last because internally, there is a performance and allocation overhead in handing off serialization to Json.NET in this manner. It is included to provide flexibility, but I would advocate using it only when you really need to, for example, complete custom serialization of a POCO where the serialized structure is not conventional. We're working on much faster serialization that will see this overhead diminish in the future.