A demo 1-to-1 implementation with high availability in Golang for Raft Consensus algorithm. In all tests run in parallel, this implement has more than 99.999% availability. It passed more than 200,000 parallel batch tests with default configuration.
The implement is written according to the paper, based on 6.824's raft labs (and tests). All commits are automatically tested (in small scale) by GitHub Actions, real-time testing results is at here.
The implement contains the following core features.
- Leader Election
- Log Replication
- Persistence (opt-out)
The implement also has the following extra features.
- Using mutex lock shortly.
- Using timer channel to avoid busy-waiting.
- Gracefully killing.
- Enabling big-step next-index descreasing (opt-out).
- Fully logging (opt-in).
- Detecting disconnecting to convert to follower (opt-in).
- Every important code part has comments from the related sentences in the raft paper (so called 1-to-1).
The opt-in/opt-out features can be configured by
DEBUGflags, which are described in the following section.
The project detects a runtime environment variable DEBUG. If it exists and it is not empty (any non-empty values is OK), the debug mode and logging will enable. The following flags can be used in the variable.
| Flag | Effect |
|---|---|
H |
Enable heartbeat logging |
T |
Enable timer logging |
L |
Enable lock logging |
D |
Enable disconnection detecting |
p |
Disable persisting |
b |
Disable the optimization for decreasing next-index in big-step |
Using Go 1.17.5 with go-module enabled.
Run the following script.
cd src/raft
go testRun in GO111MODULE=off mode.
# Lab 1: Leader Election
pwsh -c ./tests/lab1.ps1
# Lab 2: Log Replication
pwsh -c ./tests/lab2.ps1
# Lab 3: Persistence
pwsh -c ./tests/lab3.ps1
# All tests
pwsh -c ./tests/all.ps1python ./batch_test.py "test collection name"
[-c <count of runs=10>]
[-f <DEBUG Flags="HTL">]
[-w <parallelism=the number of CPU cores>]Space charaters in -f option will be striped. It exits with 0 if and only if all tests are passed.
The result will be under the directory logs. All failed tests' logs will be recorded.
check.sh use batch test script in 4 running stages (divided into the following 2 axes) to check correctness.
| Logging | No Logging | |
|---|---|---|
| Serial (100 runs, 1 worker) | Stage 1 | Stage 2 |
| Parallel (1000 runs, 1x-processor workers) | Staget 3 | Stage 4 |
./check.shtest.sh run all tests in parallel (10x-processor workers) and no-logging mode.
./test.sh <count of runs>The simplified output from the command
go test -v.
Test: initial election ...
--- PASS: TestInitialElection (2.50s)
Test: election after network failure ...
--- PASS: TestReElection (5.00s)
Test: basic agreement ...
--- PASS: TestBasicAgree (0.64s)
Test: agreement despite follower failure ...
--- PASS: TestFailAgree (5.44s)
Test: no agreement if too many followers fail ...
--- PASS: TestFailNoAgree (3.58s)
Test: concurrent Start()s ...
--- PASS: TestConcurrentStarts (0.60s)
Test: rejoin of partitioned leader ...
--- PASS: TestRejoin (4.43s)
Test: leader backs up quickly over incorrect follower logs ...
--- PASS: TestBackup (13.12s)
Test: RPC counts aren't too high ...
--- PASS: TestCount (2.14s)
Test: basic persistence ...
--- PASS: TestPersist1 (3.93s)
Test: more persistence ...
--- PASS: TestPersist2 (15.80s)
Test: partitioned leader and one follower crash, leader restarts ...
--- PASS: TestPersist3 (1.61s)
Test: Figure 8 ...
--- PASS: TestFigure8 (26.19s)
Test: unreliable agreement ...
--- PASS: TestUnreliableAgree (3.14s)
Test: Figure 8 (unreliable) ...
--- PASS: TestFigure8Unreliable (25.96s)
Test: churn ...
--- PASS: TestReliableChurn (16.46s)
Test: unreliable churn ...
--- PASS: TestUnreliableChurn (16.12s)
PASS
ok raft 146.675s
The simplified output from the command
./test.sh 100000.
Start Large Scale (100000 cases, 120 workers) (Parallel (Disabled Logging))
Backup: 100.0% (100000/100000)
BasicAgree: 100.0% (100000/100000)
ConcurrentStarts: 100.0% (100000/100000)
Count: 100.0% (100000/100000)
FailAgree: 100.0% (100000/100000)
FailNoAgree: 100.0% (100000/100000)
Figure8: 100.0% (100000/100000)
Figure8Unreliable: 100.0% (100000/100000)
InitialElection: 100.0% (100000/100000)
Persist1: 100.0% (100000/100000)
Persist2: 100.0% (100000/100000)
Persist3: 100.0% (100000/100000)
ReElection: 100.0% (100000/100000)
Rejoin: 100.0% (100000/100000)
ReliableChurn: 100.0% (100000/100000)
UnreliableAgree: 100.0% (100000/100000)
UnreliableChurn: 100.0% (100000/100000)