For Small Clusters - Generation Clock

Problem

Possibility of the leader being temporarily disconnected from the followers. There might be a garbage collection pause in the leader process, or a temporary network disruption which disconnects the leader from the follower.

The old leader itself should also be able to detect that it was temporarily disconnected from the cluster and take necessary corrective action to step down from leadership.

Solution

Maintain a monotonically increasing number indicating the generation of the server. Every time a new leader election happens, it should be marked by increasing the generation. The generation needs to be available beyond a server reboot, so it is stored with every entry in the Write-Ahead Log.

At startup, the server reads the last known generation from the log.

class ReplicatedLog…
    this.replicationState = new ReplicationState(config, wal.getLastLogEntryGeneration());

With Leader and Followers servers increment the generation every time there's a new leader election.

class ReplicatedLog…

  private void startLeaderElection() {
      replicationState.setGeneration(replicationState.getGeneration() + 1);
      registerSelfVote();
      requestVoteFrom(followers);
  }

The servers send the generation to other servers as part of the vote requests. This way, after a successful leader election, all the servers have the same generation. Once the leader is elected, followers are told about the new generation

follower (class ReplicatedLog...)

  private void becomeFollower(int leaderId, Long generation) {
      replicationState.reset();
      replicationState.setGeneration(generation);
      replicationState.setLeaderId(leaderId);
      transitionTo(ServerRole.FOLLOWING);
  }

Thereafter, the leader includes the generation in each request it sends to the followers. It includes it in every HeartBeat message as well as the replication requests sent to followers.

Leader persists the generation along with every entry in its Write-Ahead Log.

leader (class ReplicatedLog...)

  Long appendToLocalLog(byte[] data) {
      Long generation = replicationState.getGeneration();
      return appendToLocalLog(data, generation);
  }

  Long appendToLocalLog(byte[] data, Long generation) {
      var logEntryId = wal.getLastLogIndex() + 1;
      var logEntry = new WALEntry(logEntryId, data, EntryType.DATA, generation);
      return wal.writeEntry(logEntry);
  }

If a follower gets a message from a deposed leader, the follower can tell because its generation is too low. The follower then replies with a failure response.

follower (class ReplicatedLog...)

  Long currentGeneration = replicationState.getGeneration();
  if (currentGeneration > request.getGeneration()) {
      return new ReplicationResponse(FAILED, serverId(), currentGeneration, wal.getLastLogIndex());
  }

When a leader gets such a failure response, it becomes a follower and expects communication from the new leader.

Old leader (class ReplicatedLog...)

  if (!response.isSucceeded()) {
      if (response.getGeneration() > replicationState.getGeneration()) {
          becomeFollower(LEADER_NOT_KNOWN, response.getGeneration());
          return;
      }