Base class for user-provided Raft state machines.
Users should extend this class to create a state machine for use within a
CopycatServer.
State machines are responsible for handling
Operation submitted to the Raft cluster and
filtering
Commit operations out of the Raft log. The most important rule of state machines is
that state machines must be deterministic in order to maintain Copycat's consistency guarantees. That is,
state machines must not change their behavior based on external influences and have no side effects. Users should
never use
System time to control behavior within a state machine.
When
Command and
Query (i.e. operations) are submitted to the Raft cluster,
the
CopycatServer will log and replicate them as necessary and, once complete, apply them to the configured
state machine.
State machine operations
State machine operations are implemented as methods on the state machine. Operations can be automatically detected
by the state machine during setup or can be explicitly registered by overriding the
#configure(StateMachineExecutor)method. Each operation method must take a single
Commit argument for a specific operation type.
public class MapStateMachine extends StateMachine finally
previous.close();
}
}
return null;
}
public Object get(Commit commit)
try
Commit current = map.get(commit.operation().key());
return current != null ? current.operation().value() : null;
} finally
commit.close();
}
}
}
}
When operations are applied to the state machine they're wrapped in a
Commit object. The commit provides the
context of how the command or query was committed to the cluster, including the log
Commit#index(), the
Session from which the operation was submitted, and the approximate wall-clock
Commit#time() at which
the commit was written to the Raft log. Note that the commit time is guaranteed to progress monotonically, but it may
not be representative of the progress of actual time. See the
Commit documentation for more information.
State machine operations are guaranteed to be executed in the order in which they were submitted by the client,
always in the same thread, and thus always sequentially. State machines do not need to be thread safe, but they must
be deterministic. That is, state machines are guaranteed to see
Commands in the same order on all servers,
and given the same commands in the same order, all servers' state machines should arrive at the same state with the
same output (return value). The return value of each operation callback is the response value that will be sent back
to the client.
Deterministic scheduling
The
StateMachineExecutor is responsible for executing state machine operations sequentially and provides an
interface similar to that of
java.util.concurrent.ScheduledExecutorService to allow state machines to schedule
time-based callbacks. Because of the determinism requirement, scheduled callbacks are guaranteed to be executed
deterministically as well. The executor can be accessed via the
#executor field.
See the
StateMachineExecutor documentation for more information.
public void putWithTtl(Commit commit) );
}
}
During command or scheduled callbacks,
Sessions can be used to send state machine events back to the client.
For instance, a lock state machine might use a client's
Session to send a lock event to the client.
public void unlock(Commit commit) } finally
commit.close();
}
}
}
Attempts to
io.atomix.copycat.server.session.ServerSession#publish(String,Object)events during the execution will result in an
IllegalStateException.
As with other operations, state machines should ensure that the publishing of session events is deterministic.
Messages published via a
Session will be managed according to the
Command.ConsistencyLevelof the command being executed at the time the event was
io.atomix.copycat.server.session.ServerSession#publish(String,Object) ) published}. Each command may
publish zero or many events. For events published during the execution of a
Command.ConsistencyLevel#LINEARIZABLEcommand, the state machine executor will transparently await responses from the client(s) before completing the command.
For commands with lower consistency levels, command responses will be immediately sent. Session events are always guaranteed
to be received by the client in the order in which they were published by the state machine.
Even though state machines on multiple servers may appear to publish the same event, Copycat's protocol ensures that only
one server ever actually sends the event. Still, it's critical that all state machines publish all events to ensure
consistency and fault tolerance. In the event that a server fails after publishing a session event, the client will transparently
reconnect to another server and retrieve lost event messages.
Log cleaning
As operations are logged, replicated, committed, and applied to the state machine, the underlying
io.atomix.copycat.server.storage.Log grows. Without freeing unnecessary commits from the log it would eventually
consume all available disk or memory. Copycat uses a log cleaning algorithm to remove
Commits that no longer
contribute to the state machine's state from the log. To aid in this process, it's the responsibility of state machine
implementations to indicate when each commit is no longer needed by calling
Commit#close().
public class ValueStateMachine extends StateMachine public void delete(Commit commit)
if (value != null)
value.close();
value = null;
}
commit.close();
}
}
}
State machines should hold on to the
Commit object passed to operation callbacks for as long as the commit
contributes to the state machine's state. Once a commit is no longer needed, commits should be
Commit#close().
Closing a commit notifies the log compaction algorithm that it's safe to remove the commit from the internal
commit log. Copycat will guarantee that
Commits are persisted in the underlying
io.atomix.copycat.server.storage.Log as long as is necessary (even after a commit is closed) to
ensure all operations are applied to a majority of servers and to guarantee delivery of
io.atomix.copycat.server.session.ServerSession#publish(String,Object) session events}
published as a result of specific operations. State machines only need to specify when it's safe to
remove each commit from the log.
Note that if commits are not properly closed and are instead garbage collected, a warning will be logged.
Failure to
Commit#close() a command commit should be considered a critical bug since instances of the
command can eventually fill up disk.
Snapshotting
On top of Copycat's log cleaning algorithm mentioned above, Copycat provides a mechanism for storing and loading
snapshots of a state machine's state. Snapshots are images of the state machine's state stored at a specific
point in logical time (an
index). To support snapshotting, state machine implementations should implement
the
Snapshottable interface.
public class ValueStateMachine extends StateMachine implements Snapshottable public void snapshot(SnapshotWriter writer)
writer.writeObject(value);
}
}
}
For snapshottable state machines, Copycat will periodically request a
io.atomix.copycat.server.storage.snapshot.Snapshotof the state machine's state by calling the
Snapshottable#snapshot(SnapshotWriter) method. Once the state
machine has written a snapshot of its state, Copycat will automatically remove all commands from the underlying log
marked with the
Command.CompactionMode#SNAPSHOT compaction mode. Note that
state machines should still ensure that snapshottable commits are
Commit#close() once they've been
applied to the state machine, but state machines are free to immediately close all snapshottable commits.