Docs - X/Twitter - Roadmap - Why VES? - Changelog
VES is a high performance, highly configurable and easy-to-understand observability data agent-aggregator
VES is an easy to understand, high performance log collection and observability agent-aggregator that makes it easy to add observability to your software at any scale. VES primarily focuses on three core aspects: high performance, ease of use/understanding and configurability.
You no longer need to read hundreds of docs from different tools just to add observability and log collection to your stack with almost undifferentiated performance and ever increasing complexity the deeper you integrate.
- VES is a high performance, highly configurable and easy-to-understand observability data agent-aggregator
- Guiding principles
- Use cases
- Comparisons
- Getting started with VES
- Setting up VES
- Learning more about VES
- Roadmap
- Contributing
- License
Currently, the only way to use the latest development version of VES is self-hosting on linux. Their are no pre-built binaries or docker images for the current latest version of VES. We're working on adding these to get you started with VES faster.
See Building VES
The fastest and easiest way to get started with VES will be signing up for free to VES Cloud.
Before you've built a VES binary containing the latest development of VES, Build VES. Configure various setting in the configuration file to your preferences.
Currently, VES does not support on-the-fly configuration settings. Work is being done to implement this by v1.0.0
Example configuration file:
[general]
enable_local_mode = true # whether VES is watching log files locally (on the same node as logs are produced)
enable_network_mode = true # whether VES is watching log files over a network (logs are produced on a different node)
[watcher]
enabled = true
checkpoint_path = "path/to/checkpoint.json" # path to the VES checkpoint file, required to support resuming on crashes or restarts
log_dir = "path/to/log/file(s)/directory" # path VES will use to watch and tail log files it's working on local mode
poll_interval_ms = 5000 # poll interval local log file watcher will use to check for any updates to the log_dir
recursive = false # whether the local log watcher will watch files in log_dir recursively
[parser]
# Currently, the parser module is non-configurable, work is being done to implement this
[buffer]
capacity_option = "unbounded" # Options: "bounded", "unbounded" determines what InMemoryBuffer capacity is set to at runtime
buffer_capacity = 10000 # determines the capacity InMemoryBuffer will be created with at runtime
batch_size = 200 # determine InMemoryBuffer batch size at runtime
batch_timeout_ms = 500 # determine InMemoryBuffer batch size based on the size at a point in time at runtime
overflow_policy = "drop_oldest" # determine how to handle InMemoryBuffer overflow at runtime
flush_policy = "batch_size" # determine InMemoryBuffer flush trigger at runtime
drain_policy = "batch_size" # determine InMemoryBuffer draining trigger at runtime
[buffer.durability]
type = "s-q-lite" # determine the durability settings for InMemoryBuffer at runtime. RECOMMENDED s-q-lite to allow graceful restarts without losing data during a crash
path = "/var/ves-sqlite-db/parsed_log_buffer.db" # determine location of the SQLite DB providing persistence to InMemoryBuffer at runtime
[shipper]
embedder_target_addr = "https://127.0.0.1:50051" # determine the gRPC address for the Embedding engine. Required to send aggregated/normalized logs to be converted to vector embeddings
connection_timeout_ms = 500 # determine time to wait before failing a new connection attempt to the Embedding engine
max_reconnect_attempts = 10 # determine reconnection to Embedding engine limit before declaring reconnection failure
initial_retry_delay_ms = 500 # determine delay before reconnection retry attempts to Embedding engine
max_retry_delay_ms = 30000 # determine ceiling for exponential backoff while retrying reconnections to Embedding engine
backoff_factor = 2.0 # determine multiplier for reconnection to Embedding engine retry growth
retry_jitter = 0.2 # determine random reconnection to Embedding engine jitter percentage to avoid thundering herd problem on successful reconnection to Embedding engine
send_timeout_ms = 3000 # determine max timeout to push one aggregated/normalized logs batch into Embedding engine gRPC stream
response_timeout_ms = 10000 # determine max timeout to wait for response from Embedding engine before declaring gRPC stream unhealthyThe above example configuration file can be used as a base to start using VES.
It is recommended to configure the configuration file before compiling a VES binary. VES currently does not support on-the-fly configuration.
Curious about how to make the most of VES? See our docs for anything that you think is not in the README.
VES is current roadmap is at getting the current v0.1.0 Beta version to v1.0.0 Stable version. The roadmap is whatever is currently opened in Issues.
The goal is to release a stable version v1.0.0 prior to the new year.
All contributions make VES better and help achieve the goals set out for the project. See How can I contribute and star the repo 🌟
VES is free and the source is available. Currently, VES is released under the AGPL license. See individual files for details which will specify the license applicable to each file.
For more information see the VES License.