Updated documentation

Fixes #307 Added Kubernetes docs page Updated vault docs Updated events & data streams docs Added fsm docs Added local setup & startup guide Added grafana to cluster architecture image Updated events chart to correctly reflect historization events Added service catalog to main readme Updated readme for each service
Fortify-Labs · Feb 16, 2021 · 7987c70 · 7987c70
1 parent f274d6c
commit 7987c70
Show file tree

Hide file tree

Showing 41 changed files with 960 additions and 170 deletions.
diff --git a/.github/workflows/cicd.yml b/.github/workflows/cicd.yml
@@ -30,7 +30,7 @@ jobs:
           echo JOBS_VERSION=$(jq -r .version services/jobs/package.json) >> $GITHUB_ENV
           echo HISTORIZATION_VERSION=$(jq -r .version services/historization/package.json) >> $GITHUB_ENV
 
-          echo SENTRY_DISCORD_WEBHOOK_VERSION=1.2.0 >> $GITHUB_ENV
+          echo SENTRY_DISCORD_WEBHOOK_VERSION=$(cat services/sentry-discord-webhook/version) >> $GITHUB_ENV
 
           echo SENTRY_ORG='thomas-kosiewski' >> $GITHUB_ENV
           echo SENTRY_AUTH_TOKEN=${{ secrets.SENTRY_AUTH_TOKEN }} >> $GITHUB_ENV

diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -6,10 +6,13 @@
 	"Dota",
 	"Envs",
 	"Fluentd",
+	"Fortify's",
 	"GHCR",
+	"Grafana",
 	"Persistor",
 	"Scaleway",
 	"TWITCHBOT",
+	"Underlord's",
 	"Underlords",
 	"akhq",
 	"allowlist",

diff --git a/README.md b/README.md
@@ -1,25 +1,65 @@
 # Fortify
 
-## Overview
-
-This project aims to develop a platform, that is aggregating and processing game state integration logs in order to extract live information from matches and collect stats from played matches.
+![Fortify banner](docs/assets/Fortify_Banner.png)
 
-## Architecture
+## Overview
 
-![Image of Architecture](resources/architecture.png)
+This project is a big data processing platform, that is aggregating and processing Dota Underlord's game state integration logs in order to extract live information from matches and collect stats from played matches.
 
 ## Getting started
 
 ### Local Setup
 
-All necessary databases and systems can be started using `docker-compose up -d`.
+All necessary containers to for a development environment can be pulled & started using `docker-compose up -d`.
+
+### Starting already build images
+
+All fortify docker images (that are running in production) can also be downloaded locally.
+
+This can be done, using the following commands:
+
+- Pulling all images
+
+  ```bash
+  source ./scripts/versions.sh
+  docker-compose -f build.docker-compose.yml pull
+  ```
+
+- Pulling a specific image
+
+  ```bash
+  source ./scripts/versions.sh
+  docker-compose -f build.docker-compose.yml pull <service name>
+  ```
 
-The corresponding microservices are located in the `services/` directory.
-Those can be started locally by navigating into the service's directory and running `npm run dev`. (Each microservice will have a dedicated Readme explaining necessary setup steps.)
+## Services
 
-### Kubernetes Setup
+Fortify contains of many smaller services, that each have their own documentation.
 
-TODO: Write Kubernetes setup guide
+- [17kmmrbot](services/17kmmrbot/)
+- [backend](services/backend/)
+- [frontend](services/frontend/)
+- [fsm](services/fsm/)
+- [gsi-receiver](services/gsi-receiver/)
+- [historization](services/historization/)
+- [jobs](services/jobs/)
+- [sentry-discord-webhook](services/sentry-discord-webhook/)
+
+And shared libraries:
+
+- [shared](services/shared/)
+
+## Production deployment
+
+[See Kubernetes docs](./docs/kubernetes.md)
+
+[See Vault docs](./docs/vault.md)
+
+## Implementation details
+
+[GSI parsing finite state machine](./docs/fsm.md)
+
+[Fortify internal events](./docs/events.md)
 
 ## Resources
 
@@ -29,6 +69,10 @@ TODO: Write Kubernetes setup guide
 
 ## Disclaimer
 
-Fortify (© 2020 Thomas Kosiewski) is a community project and is not affiliated with Valve or Steam.
+Fortify (© 2021 Thomas Kosiewski) is a community project and is not affiliated with Valve or Steam or Twitch.
+
+Valve, the Valve logo, Steam, the Steam logo, Source, the Source logo, Dota, the Dota 2 logo, Defense of the Ancients, Dota Underlords and the Dota Underlords logo are trademarks and/or registered trademarks of Valve Corporation.
+
+TWITCH, the TWITCH Logo, the Glitch Logo, and/or TWITCHTV are trademarks of Twitch Interactive, Inc. or its affiliates.
 
-Dota Underlords is a registered trademark of Valve Corporation.
+The Fortify logo and Fortify banner are intellectual property of John "johnxfire" Nathan Fernandez.
diff --git a/architecture.drawio b/architecture.drawio
diff --git a/devops/kubernetes/.env.example b/devops/kubernetes/.env.example
@@ -1,17 +1,34 @@
+# Docker Registry
 REGISTRY=ghcr.io/fortify-labs/fortify/
 
+# Vault Auth Token; Will be reworked into a Kubernetes service account based auth method
 VAULT_TOKEN=
 
+# Production Environment? prod / dev / staging
 ENVIRONMENT=
+# Basic auth for dev & monitoring services (htpasswd string)
 CLUSTER_BASIC_AUTH=
 
+# Production domain
 DOMAIN=
+# Google analytics tracking id
 GA_TRACKING_ID=
 
+# Sentry.io DSNs for each service
 SENTRY_DSN=
 TWITCH_BOT_SENTRY_DSN=
 GSI_RECEIVER_SENTRY_DSN=
 FSM_SENTRY_DSN=
 HISTORIZATION_SENTRY_DSN=
 JOBS_SENTRY_DSN=
 BACKEND_SENTRY_DSN=
+
+# Grafana SSO
+GRAFANA_GITHUB_CLIENT_ID=
+GRAFANA_GITHUB_CLIENT_SECRET=
+
+# S3 for backups
+S3_ENDPOINT=
+S3_REGION=
+S3_ACCESS_KEY_ID=
+S3_SECRET_ACCESS_KEY=
diff --git a/devops/kubernetes/main.test.ts b/devops/kubernetes/main.test.ts
@@ -1,6 +1,6 @@
 import { Testing } from "cdk8s";
 
-import { ClusterSetup } from "./src/cluster";
+import { Cluster } from "./src/cluster";
 import { Fortify } from "./src/fortify";
 
 describe("Placeholder", () => {
@@ -13,7 +13,7 @@ describe("Placeholder", () => {
 
 	test("Empty Clean Cluster", () => {
 		const app = Testing.app();
-		const chart = new ClusterSetup(app, "test-cluster");
+		const chart = new Cluster(app, "test-cluster");
 		const results = Testing.synth(chart);
 		expect(results).toMatchSnapshot();
 	});

diff --git a/devops/kubernetes/main.ts b/devops/kubernetes/main.ts
@@ -3,12 +3,35 @@ config();
 
 import { App } from "cdk8s";
 
-import { ClusterSetup } from "./src/cluster";
+import { Cluster } from "./src/cluster";
 import { Fortify } from "./src/fortify";
+import { AkhqChart } from "./src/charts/akhq/akhq";
+import { CICDUser } from "./src/charts/cicd-user/cicdUser";
+import { GrafanaChart } from "./src/charts/grafana/grafana";
+import { KowlChart } from "./src/charts/kowl/kowl";
+import { PostgresOperator } from "./src/charts/postgres-operator/postgresOperator";
+import { PrometheusChart } from "./src/charts/prometheus/prometheus";
+import { RedisOperatorChart } from "./src/charts/redis-operator/redis-operator";
+import { TraefikChart } from "./src/charts/traefik/traefik";
+import { VaultChart } from "./src/charts/vault/vault";
+
+const { CLUSTER_SETUP } = process.env;
 
 const app = new App();
 
-new ClusterSetup(app, "cluster");
+if (CLUSTER_SETUP) {
+	new AkhqChart(app, "akhq");
+	new CICDUser(app, "cicd");
+	new GrafanaChart(app, "grafana");
+	new KowlChart(app, "kowl");
+	new PostgresOperator(app, "postgres");
+	new PrometheusChart(app, "prometheus");
+	new RedisOperatorChart(app, "redis");
+	new TraefikChart(app, "traefik");
+	new VaultChart(app, "vault");
+}
+
+new Cluster(app, "cluster");
 new Fortify(app, "fortify");
 
 app.synth();
diff --git a/devops/kubernetes/src/cluster.ts b/devops/kubernetes/src/cluster.ts
@@ -63,7 +63,7 @@ export interface CustomKafkaTopicProps extends KafkaTopicProps {
 	metadata?: ObjectMeta;
 }
 
-export class ClusterSetup extends Chart {
+export class Cluster extends Chart {
 	constructor(scope: Construct, name: string) {
 		super(scope, name);
 

diff --git a/docs/Events.md b/docs/Events.md
diff --git a/docs/assets/Fortify_Banner.png b/docs/assets/Fortify_Banner.png
diff --git a/docs/assets/architecture.png b/docs/assets/architecture.png
diff --git a/docs/assets/event_driven_architecture.png b/docs/assets/event_driven_architecture.png
diff --git a/docs/assets/events.png b/docs/assets/events.png
diff --git a/docs/assets/fsm.png b/docs/assets/fsm.png
diff --git a/docs/events.md b/docs/events.md
@@ -0,0 +1,48 @@
+# Events
+
+Fortify is designed with an event driven architecture in mind.
+
+This means that events can be generated from any service at any time and can be processed by any service subscribing to them at any time with variable delays and lags.
+
+Kafka is being utilized as message queue, serving as a common interface for all services & languages, enabling the decoupling of services and offering flexibility for future implementations.
+
+## Event Types
+
+In total there are 3 different type of events:
+
+- [Game](#Game-Events)
+- [Generic](#Generic-Events)
+- [System](#System-Events)
+
+Each event type is written to its corresponding Kafka topic, which in turn has custom retention rules.
+
+All events have a `type` and a `timestamp`, with `timestamp` allowing delayed event processing in case of failures, unforeseen burst loads or batch processing.
+
+The following is a rough chart of data streams inside of Fortify:
+
+![data streams chart](assets/events.png)
+
+## Game Events
+
+- `MATCH_STARTED`: Event created when a new match is detected (FSM generated event)
+- `FINAL_PLACE`: Event created when a player got eliminated with a final placing (FSM generated event)
+- `MATCH_ENDED`: Event created when a match has concluded with a first place winner (FSM generated event)
+- `RANK_TIER_UPDATE`: Event created when a match starts, includes a players current rank tier (FSM generated event)
+- `SMURF_DETECTED`: Event created when a smurf account has been detected (FSM generated event)
+- `UNIT_STATS`: Event created after each combat, tracking unit win/loss in conjunction with active alliances and equipped item (FSM generated event)
+- `ITEM_STATS`: Event created after each combat, tracking item win/loss in conjunction with active alliances (FSM generated event)
+- `ALLIANCE_STATS`: Event created after each combat, tracking alliance win/loss in conjunction with active alliances (FSM generated event)
+- `COMBINED_STATS`: Event combining all 3 stats event for easier batch processing for time series / wide column databases (FSM generated event)
+
+## Generic Events
+
+Currently unused
+
+## System Events
+
+- `FSM_RESET_REQUEST`: RPC event used to reset the finite state machines user state. (Twitch bot / manually generated event)
+- `TWITCH_LINKED`: Event created when a Twitch account has been linked with Fortify (backend generated event)
+- `TWITCH_UNLINKED`: Event created when a Twitch account has been unlinked with Fortify (backend generated event)
+- `TWITCH_MESSAGE_BROADCAST`: Event used to broadcast a message to all twitch channels the bot is currently joined in (manually generated event)
+- `IMPORT_COMPLETED`: Event announcing a successful import of the Underlords leaderboard (import jobs generated event)
+- `HISTORIZATION_COMPLETED`: Event announcing successful storing individual ranks of the imported leaderboard (historization generated event)
diff --git a/docs/fsm.md b/docs/fsm.md
@@ -0,0 +1,63 @@
+# Finite State Machine
+
+Fortify's finite state machine is the heart of the entire operation, successfully processing millions of GSI messages per month.
+
+Initially, the finite state machine was hastily written together and started to show it's shortcomings. These shortcomings included slow processing under load and incorrectly detected game modes, rendering any future use for stats collection impossible.
+
+This lead to a complete redesign & rewrite from scratch, with many new features and fixes baked into it.
+
+A rough, high-level view of the finite state machine can be seen bellow:
+
+![fsm chart](assets/fsm.png)
+
+## Features
+
+- Detect Game Modes
+- Correct unit pool sizes & odds for each game mode
+- Game state tracking
+- Rough round number estimations
+- Rank progression tracking
+- Player lobby processing
+- Consolidated unit pool
+- Individual board states
+- Processing of private player states
+- Smurf detection
+- Unit / Item / Alliance (or to be technically correct: Synergy) performance tracking
+
+## Consolidated match states
+
+With multiple player potentially being in the same match, it's not only wasteful to keep separate match states for each player, but information might also get lost, if the transmitted of one player's GSI information has gotten lost. This might lead to incorrect calculations and stats tracking.
+
+In order to prevent this from happening, all players from one match are all sourced together into one combined match state.
+
+This is introducing a new data inconsistency scenario. As in one player's GSI data is sent with a major lag compared to other players, thus overriding newer data with old data.
+
+In order to avoid this scenarios of potential race conditions and data inconsistencies, the finite state machine is only performing updates on each public and private player state individually, if the `sequence_number` is higher than the previously stored `sequence_number`.
+
+## Shortcomings
+
+Two minor shortcomings of the above finite state machine can be broken down into missing information and information having to be guessed / estimated.
+
+### Unknown match ID
+
+GSI is not forwarding any information about the actual match itself (unlike it's counterpart in Dota and CS GO).
+
+In order to still be able to track matches, the finite state machine is creating it's own match ID based of player account IDs and their slots inside of the lobby, which is then piped into a hash function.
+
+In case of "collisions" (generated match ID has already been used and the original match identified by said ID already has player slots with a final_place set), the above match id generation will have a nonce appended, which will gradually increase until a match ID has been generated, that has not been used before.
+
+### Unknown round number
+
+Like with the unknown match ID, round numbers are also not present in GSI data.
+
+This unknown variable can be guesstimated by taking the Underlords levels into consideration, as Underlord levels are tied to round numbers & game modes.
+
+## Happy Little Accidents
+
+When designing the new version of the finite state machine, I have not considered handling the duos game mode.
+
+Thus the finite state machine was originally designed to primarily work with 8 player lobbies, yet this number has not been capped anywhere in the final implementation.
+
+For some reason, the finite state machine managed to correctly detect duos lobby and wait for all 16 players public player states.
+
+So technically a desired feature, that was not planed at all and manifested itself as a bug. So quite literally a bug turning into a feature. (Probably one of my best & proudest bugs.)