Unrecoverable exceptions

Unrecoverable exceptions are expected exceptions that are grounds to exit the client pipeline immediately.

What do we mean by expected exceptions? Aren’t all exceptions exceptional?

Well, there a some exceptions that can be thrown in the course of a request that may be expected, some of which can be retried on another node in the cluster, and some that cannot. For example, an exception thrown when pinging a node throws an exception, but this is an exception that the client expects can happen, and can handle by trying a ping on another node. On the contrary, a bad authentication response from a node will throw an exception, and the client understands that an exception under these circumstances should be handled by not retrying, but by exiting the pipeline.

By default, the client won’t throw on any ElasticsearchClientException but instead return an invalid response that can be detected by checking the .IsValid property on the response. You can change this behaviour with by using ThrowExceptions() on ConnectionSettings.

var failures = Enum.GetValues(typeof(PipelineFailure)).Cast<PipelineFailure>();
foreach (var failure in failures)
{
    switch (failure)
    {
        /** The followinig pipeline failures are recoverable and will be retried */
        case PipelineFailure.PingFailure:
        case PipelineFailure.BadRequest:
        case PipelineFailure.BadResponse:
            var recoverable = new PipelineException(failure);
            recoverable.Recoverable.Should().BeTrue(failure.GetStringValue());
            break;

        /** The followinig pipeline failures are NOT recoverable and won't be retried */
        case PipelineFailure.BadAuthentication:
        case PipelineFailure.SniffFailure:
        case PipelineFailure.CouldNotStartSniffOnStartup:
        case PipelineFailure.MaxTimeoutReached:
        case PipelineFailure.MaxRetriesReached:
        case PipelineFailure.Unexpected:
        case PipelineFailure.NoNodesAttempted:
        case PipelineFailure.FailedProductCheck:
            var unrecoverable = new PipelineException(failure);
            unrecoverable.Recoverable.Should().BeFalse(failure.GetStringValue());
            break;
        default:
            throw new ArgumentOutOfRangeException(failure.GetStringValue());
    }
}

As an example, let’s use our Virtual cluster test framework to set up a 10 node cluster that always succeeds when pinged but fails with a 401 response when making client calls

var audit = new Auditor(() => VirtualClusterWith
    .Nodes(10)
    .Ping(r => r.SucceedAlways()) 
    .ClientCalls(r => r.FailAlways(401)) 
    .StaticConnectionPool()
    .AllDefaults()
);

	Always succeed on ping
	…but always fail on calls with a 401 Bad Authentication response

Now, let’s make a client call. We’ll see that the first audit event is a successful ping followed by a bad response as a result of the 401 bad authentication response

audit = await audit.TraceElasticsearchException(
    new ClientCall {
        { AuditEvent.PingSuccess, 9200 }, 
        { AuditEvent.BadResponse, 9200 }, 
    },
    exception =>
    {
        exception.FailureReason
            .Should().Be(PipelineFailure.BadAuthentication); 
    }
);

	First call results in a successful ping
	Second call results in a bad response
	The reason for the bad response is Bad Authentication

When a bad authentication response occurs, the client attempts to deserialize the response body returned;

In some setups you might be running behind a proxy and you might need to prevent the client from trying to deserialize bad json. In the following example an HTML response is return but with an application/json content type. If the proxy is not under your control you would need to be able to fix this in the client. Here we make the client aware that 401 responses should never be deserialized by calling SkipDeserializationForStatusCodes() on ConnectionSettings.

var audit = new Auditor(() => VirtualClusterWith
    .Nodes(10)
    .Ping(r => r.SucceedAlways())
    .ClientCalls(r => r.FailAlways(401).ReturnByteResponse(HtmlNginx401Response, "application/json")) 
    .StaticConnectionPool()
    .Settings(s => s.SkipDeserializationForStatusCodes(401))
);

audit = await audit.TraceElasticsearchException(
    new ClientCall {
        { AuditEvent.PingSuccess, 9200 },
        { AuditEvent.BadResponse, 9201 },
    },
    (e) =>
    {
        e.FailureReason.Should().Be(PipelineFailure.BadAuthentication);
        e.Response.HttpStatusCode.Should().Be(401);
        e.Response.ResponseBodyInBytes.Should().BeNull(); 
    }
);

	Always return a 401 bad response with a HTML response on client calls
	Assert that the response body bytes are null

Now in this example, by turning on DisableDirectStreaming() on ConnectionSettings, we see the same behaviour exhibited as before, but this time however, the response body bytes are captured in the response and can be inspected. Also note that in this example the 401 returns the correct mime type for html so the client wont try to deserialize to json and we no longer need to set SkipDeserializationForStatusCodes()

var audit = new Auditor(() => VirtualClusterWith
    .Nodes(10)
    .Ping(r => r.SucceedAlways())
    .ClientCalls(r => r.FailAlways(401).ReturnByteResponse(HtmlNginx401Response, "text/html"))
    .StaticConnectionPool()
    .Settings(s => s.DisableDirectStreaming())
);

audit = await audit.TraceElasticsearchException(
    new ClientCall {
        { AuditEvent.PingSuccess, 9200 },
        { AuditEvent.BadResponse, 9200 },
    },
    (e) =>
    {
        e.FailureReason.Should().Be(PipelineFailure.BadAuthentication);
        e.Response.HttpStatusCode.Should().Be(401);
        e.Response.ResponseBodyInBytes.Should().NotBeNull(); 
        var responseString = Encoding.UTF8.GetString(e.Response.ResponseBodyInBytes);
        responseString.Should().Contain("nginx/"); 
        e.DebugInformation.Should().Contain("nginx/");
    }
);

	Response bytes are set on the response
	Assert that the response contains `"nginx/"`

« Unexpected exceptions Transports and pipelines »