Skip to content

VES is a high performance, highly configurable and easy-to-understand observability data platform. Collecting, processing, vectorizing and storing observability data.

License

Notifications You must be signed in to change notification settings

H3IMD3LL-Labs-Inc/VES

Code Coverage LOC GitHub contributors PRs Welcome GitHub releases GitHub commit activity GitHub open issues GitHub closed issues GitHub stars GitHub forks

Docs - X/Twitter - Roadmap - Why VES? - Changelog

VES is a high performance, highly configurable and easy-to-understand observability data agent-aggregator

VES is an easy to understand, high performance log collection and observability agent-aggregator that makes it easy to add observability to your software at any scale. VES primarily focuses on three core aspects: high performance, ease of use/understanding and configurability.

You no longer need to read hundreds of docs from different tools just to add observability and log collection to your stack with almost undifferentiated performance and ever increasing complexity the deeper you integrate.

Table of Contents

Getting started with VES

Self-hosting the open-source Beta

Currently, the only way to use the latest development version of VES is self-hosting on linux. Their are no pre-built binaries or docker images for the current latest version of VES. We're working on adding these to get you started with VES faster.

See Building VES

VES Cloud (Coming Soon)

The fastest and easiest way to get started with VES will be signing up for free to VES Cloud.

See VES Cloud(Coming Soon)

Setting up VES

Before you've built a VES binary containing the latest development of VES, Build VES. Configure various setting in the configuration file to your preferences.

Currently, VES does not support on-the-fly configuration settings. Work is being done to implement this by v1.0.0

Example configuration file:

[general]
enable_local_mode = true                                        # whether VES is watching log files locally (on the same node as logs are produced)
enable_network_mode = true                                      # whether VES is watching log files over a network (logs are produced on a different node)

[watcher]
enabled = true
checkpoint_path = "path/to/checkpoint.json"                     # path to the VES checkpoint file, required to support resuming on crashes or restarts
log_dir = "path/to/log/file(s)/directory"                       # path VES will use to watch and tail log files it's working on local mode
poll_interval_ms = 5000                                         # poll interval local log file watcher will use to check for any updates to the log_dir
recursive = false                                               # whether the local log watcher will watch files in log_dir recursively

[parser]
# Currently, the parser module is non-configurable, work is being done to implement this

[buffer]
capacity_option = "unbounded"                                   # Options: "bounded", "unbounded" determines what InMemoryBuffer capacity is set to at runtime
buffer_capacity = 10000                                         # determines the capacity InMemoryBuffer will be created with at runtime
batch_size = 200                                                # determine InMemoryBuffer batch size at runtime
batch_timeout_ms = 500                                          # determine InMemoryBuffer batch size based on the size at a point in time at runtime
overflow_policy = "drop_oldest"                                 # determine how to handle InMemoryBuffer overflow at runtime
flush_policy = "batch_size"                                     # determine InMemoryBuffer flush trigger at runtime
drain_policy = "batch_size"                                     # determine InMemoryBuffer draining trigger at runtime

[buffer.durability]
type = "s-q-lite"                                               # determine the durability settings for InMemoryBuffer at runtime. RECOMMENDED s-q-lite to allow graceful restarts without losing data during a crash
path = "/var/ves-sqlite-db/parsed_log_buffer.db"                # determine location of the SQLite DB providing persistence to InMemoryBuffer at runtime

[shipper]
embedder_target_addr = "https://127.0.0.1:50051"                # determine the gRPC address for the Embedding engine. Required to send aggregated/normalized logs to be converted to vector embeddings
connection_timeout_ms = 500                                     # determine time to wait before failing a new connection attempt to the Embedding engine
max_reconnect_attempts = 10                                     # determine reconnection to Embedding engine limit before declaring reconnection failure
initial_retry_delay_ms = 500                                    # determine delay before reconnection retry attempts to Embedding engine
max_retry_delay_ms = 30000                                      # determine ceiling for exponential backoff while retrying reconnections to Embedding engine
backoff_factor = 2.0                                            # determine multiplier for reconnection to Embedding engine retry growth
retry_jitter = 0.2                                              # determine random reconnection to Embedding engine jitter percentage to avoid thundering herd problem on successful reconnection to Embedding engine
send_timeout_ms = 3000                                          # determine max timeout to push one aggregated/normalized logs batch into Embedding engine gRPC stream
response_timeout_ms = 10000                                     # determine max timeout to wait for response from Embedding engine before declaring gRPC stream unhealthy

The above example configuration file can be used as a base to start using VES.

It is recommended to configure the configuration file before compiling a VES binary. VES currently does not support on-the-fly configuration.

Learning more about VES

Curious about how to make the most of VES? See our docs for anything that you think is not in the README.

Roadmap

VES is current roadmap is at getting the current v0.1.0 Beta version to v1.0.0 Stable version. The roadmap is whatever is currently opened in Issues.

The goal is to release a stable version v1.0.0 prior to the new year.

Contributing

All contributions make VES better and help achieve the goals set out for the project. See How can I contribute and star the repo 🌟

Star History Chart

LICENSE

VES is free and the source is available. Currently, VES is released under the AGPL license. See individual files for details which will specify the license applicable to each file.

For more information see the VES License.

About

VES is a high performance, highly configurable and easy-to-understand observability data platform. Collecting, processing, vectorizing and storing observability data.

Topics

Resources

License

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Packages

No packages published