Round robin behaviour

Sniffing and Static connection pools round robin over the live nodes to evenly distribute requests over all known nodes.

This is the method on an IConnectionPool that creates a view of all the live nodes in the cluster that the client knows about. Different connection pool implementations can decide on the view to return, for example,

CreateView is implemented in a lock free thread safe fashion, meaning each callee gets returned its own cursor to advance over the internal list of nodes. This to guarantee each request that needs to fall over tries all the nodes without suffering from noisy neighbours advancing a global cursor.

Here we have setup a Static connection pool seeded with 10 nodes. We force randomization OnStartup to false so that we can test the nodes being returned are in the order we expect them to be.

var uris = Enumerable.Range(9200, NumberOfNodes).Select(p => new Uri("http://localhost:" + p));
var staticPool = new StaticConnectionPool(uris, randomize: false);
var sniffingPool = new SniffingConnectionPool(uris, randomize: false);

this.AssertCreateView(staticPool);
this.AssertCreateView(sniffingPool);

So what order do we expect? Imagine the following:

var startingPositions = Enumerable.Range(0, NumberOfNodes)
    .Select(i => pool.CreateView().First())
    .Select(n => n.Uri.Port)
    .ToList();

var expectedOrder = Enumerable.Range(9200, NumberOfNodes);
startingPositions.Should().ContainInOrder(expectedOrder);

What the above code just proved is that each call to CreateView() gets assigned the next available node.

Lets up the ante:

We’ll validate that each thread sees all the nodes and that they wrap over, for example, after node 9209 comes 9200 again

var threadedStartPositions = new ConcurrentBag<int>();
var threads = Enumerable.Range(0, 20)
    .Select(i => CreateThreadCallingCreateView(pool, threadedStartPositions))
    .ToList();

foreach (var t in threads) t.Start();
foreach (var t in threads) t.Join();

Each thread reported the first node it started off. Let’s make sure we see each node twice because we started NumberOfNodes * 2 threads

var grouped = threadedStartPositions.GroupBy(p => p).ToList();
grouped.Count.Should().Be(NumberOfNodes);
grouped.Select(p => p.Count()).Should().OnlyContain(p => p == 2);