Unrecoverable exceptions

edit

Unrecoverable exceptions are expected exceptions that are grounds to exit the client pipeline immediately.

What do we mean by expected exceptions? Aren’t all exceptions exceptional?

Well, there a some exceptions that can be thrown in the course of a request that may be expected, some of which can be retried on another node in the cluster, and some that cannot. For example, an exception thrown when pinging a node throws an exception, but this is an exception that the client expects can happen, and can handle by trying a ping on another node. On the contrary, a bad authentication response from a node will throw an exception, and the client understands that an exception under these circumstances should be handled by not retrying, but by exiting the pipeline.

By default, the client won’t throw on any ElasticsearchClientException but instead return an invalid response that can be detected by checking the .IsValid property on the response. You can change this behaviour with by using ThrowExceptions() on ConnectionSettings.

The following is a collection of unrecoverable exceptions

var unrecoverableExceptions = new[]
{
    new PipelineException(PipelineFailure.CouldNotStartSniffOnStartup),
    new PipelineException(PipelineFailure.SniffFailure),
    new PipelineException(PipelineFailure.Unexpected),
    new PipelineException(PipelineFailure.BadAuthentication),
    new PipelineException(PipelineFailure.MaxRetriesReached),
    new PipelineException(PipelineFailure.MaxTimeoutReached)
};

unrecoverableExceptions.Should().OnlyContain(e => !e.Recoverable);

As an example, let’s use our Virtual cluster test framework to set up a 10 node cluster that always succeeds when pinged but fails with a 401 response when making client calls

var audit = new Auditor(() => Framework.Cluster
    .Nodes(10)
    .Ping(r => r.SucceedAlways()) 
    .ClientCalls(r => r.FailAlways(401)) 
    .StaticConnectionPool()
    .AllDefaults()
);

Always succeed on ping

…​but always fail on calls with a 401 Bad Authentication response

Now, let’s make a client call. We’ll see that the first audit event is a successful ping followed by a bad response as a result of the 401 bad authentication response

audit = await audit.TraceElasticsearchException(
    new ClientCall {
        { AuditEvent.PingSuccess, 9200 }, 
        { AuditEvent.BadResponse, 9200 }, 
    },
    exception =>
    {
        exception.FailureReason
            .Should().Be(PipelineFailure.BadAuthentication); 
    }
);

First call results in a successful ping

Second call results in a bad response

The reason for the bad response is Bad Authentication

When a bad authentication response occurs, the client does not attempt to deserialize the response body returned; The response may or may not have a body and even when it does, it may not even be JSON.

In the following couple of examples, we set up a cluster that always returns a typical nginx HTML response body with 401 response to client calls. In this first example, we assert that the failure is because of a 401 Bad Authentication response but the response body is not captured on the response

var audit = new Auditor(() => Framework.Cluster
    .Nodes(10)
    .Ping(r => r.SucceedAlways())
    .ClientCalls(r => r.FailAlways(401).ReturnResponse(HtmlNginx401Response)) 
    .StaticConnectionPool()
    .AllDefaults()
);

audit = await audit.TraceElasticsearchException(
    new ClientCall {
        { AuditEvent.PingSuccess, 9200 },
        { AuditEvent.BadResponse, 9200 },
    },
    (e) =>
    {
        e.FailureReason.Should().Be(PipelineFailure.BadAuthentication);
        e.Response.HttpStatusCode.Should().Be(401);
        e.Response.ResponseBodyInBytes.Should().BeNull(); 
    }
);

Always return a 401 bad response with a HTML response on client calls

Assert that the response body bytes are null

Now in this example, by turning on DisableDirectStreaming() on ConnectionSettings, we see the same behaviour exhibited as before, but this time however, the response body bytes are captured in the response and can be inspected.

var audit = new Auditor(() => Framework.Cluster
    .Nodes(10)
    .Ping(r => r.SucceedAlways())
    .ClientCalls(r => r.FailAlways(401).ReturnResponse(HtmlNginx401Response))
    .StaticConnectionPool()
    .Settings(s => s.DisableDirectStreaming())
);

audit = await audit.TraceElasticsearchException(
    new ClientCall {
        { AuditEvent.PingSuccess, 9200 },
        { AuditEvent.BadResponse, 9200 },
    },
    (e) =>
    {
        e.FailureReason.Should().Be(PipelineFailure.BadAuthentication);
        e.Response.HttpStatusCode.Should().Be(401);
        e.Response.ResponseBodyInBytes.Should().NotBeNull(); 
        var responseString = Encoding.UTF8.GetString(e.Response.ResponseBodyInBytes);
        responseString.Should().Contain("nginx/"); 
        e.DebugInformation.Should().Contain("nginx/");
    }
);

Response bytes are set on the response

Assert that the response contains "nginx/"