A tool for developing in a secure, isolated and reproducible environment
This is a democratic project not a dictatorship, so you have the freedom to join to it and build the new features you need for the benefit of everyone
Click on the menu right before README.md as shown in the image below.
This is a tool for developing applications in a secure, isolated and reproducible environment with virtual machines using a Docker-like workflow.
Many apps are made for desktop environments, others apps rely on systemd, and others technologies that doesn't work well in an OCI-compliant container. The software we are developing needs to be tested on a system as close as possible to the one where it is going to be executed, Sometimes it is very difficult to satisfy the requirements with Docker and we have to use virtual machines missing the docker workflow. This is why I started the development of vedv. I hope you find it useful.
👉 This tool does not work with nested virtualization, it must be installed and executed on a real machine. It's tested on the following OS:
Runtime Dependencies:
# SYSTEM: Linux yuniel-pc 6.1.44-1-MANJARO #1 SMP PREEMPT_DYNAMIC Wed Aug 9 09:02:26 UTC 2023 x86_64 GNU/Linux
# YAY
bash 5.1.016-2
python 3.11.3-2
python-pip 23.2.1-1
virtualbox 7.0.10-1
openbsd-netcat 1.225_1-1
sshpass 1.10-1
python-pip 23.2.1-1
libxml2 2.11.4-1
# PIP
dockerfile-parse 2.0.0
petname 2.6Runtime Dependencies:
# SYSTEM: Linux user-MS-7D43 6.2.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 13 16:27:29 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
# APT
bash 5.1-6ubuntu1
curl 7.81.0-1ubuntu1.13
libxml2-utils 2.9.13+dfsg-1ubuntu0.3
netcat-openbsd 1.218-4ubuntu1
python3-pip 22.0.2+dfsg-1ubuntu0.3
python3 3.10.6-1~22.04
sshpass 1.09-1
virtualbox-7.0 7.0.10-158379~Ubuntu~jammy
# PIP
dockerfile-parse 2.0.1
petname 2.6
sudo pacman -Syu --noconfirm --needed yay &&
yay -Sy --noconfirm --needed vedv-git &&
sudo pacman -Syu --noconfirm --needed "$(mhwd-kernel -li | grep -Po '\(\Klinux\d+\S*(?=\))')-virtualbox-host-modules" &&
sudo vboxreloadFor installation from source code is required to have installed git and make.
It's recommended to install bmon to see the download/upload activity and vim
as text editor.
Clone the repository and switch to vedv directory
- from github
git clone https://github.com/yunielrc/vedv.git && cd vedvSelect the last stable version
git checkout "$(git tag --sort='version:refname' | grep -Po '^v\d+\.\d+\.\d+$' | tail -n 1)"make install-deps-manjaro && sudo make installmake install-deps-ubuntu && sudo make installInstall runtime dependencies manually and run the command below:
sudo make installIt's highly recommended for any other linux distro than manjaro and ubuntu to import and run an image from virtualbox gui before running vedv to guarantee that virtualbox is working properly.
Download an alpine image
wget --header='User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36' 'https://onedrive.live.com/download?resid=DBA0B75F07574EAA%21272&authkey=!AP8U5cI4V7DusSg' -qO alpine-inv.ovaImport the image from virtualbox gui:
- Open Virtualbox
- Click on File menu > Import Appliance
- Import the image
alpine-inv.ova - Start the
alpine-x86_64virtual machine
If you have any problem running vedv on your linux distro, it can be a missing dependency, please open a Bug issue.
👉 If you don't see the registration mail in your inbox, check the spam folder.
Register at https://registry.vedv.dev
-
Scroll to the end up to the section: Devices & sessions
-
Write on App Name: vedv
-
Click on Create new app password
-
Copy the generated password
Copy the config to your home directory
cp /etc/skel/.vedv.env ~Edit the config, set the registry credentials, save and exit
cd && vim .vedv.envShow the help
vedv --help# command output:
Usage:
vedv COMMAND
A tool for developing in a secure and reproducible environment
Flags:
-h, --help show this help
Management Commands:
container manage containers
image manage images
builder manage builder
registry manage registry
Run 'vedv COMMAND --help' for more information on a command.Download an image with custom name, create a container and start it
- To see the download activity, install
bmon, open itbmon -band select the network interface
vedv image pull -n alpine admin@alpine/alpine-3.17.3-x86_64
# 10.399s 90Mbps --> download + import + creation time
vedv container create -n alpine alpine
# 1.660s
# starting a container can take up to 1 minute the first time or more
# deppending on your hardware and the image os
vedv container start -w alpine
# 13.395s --> startup + sshd service time to be readyOr download an image and create a container, then start it
vedv container create -n alpine admin@alpine/alpine-3.17.3-x86_64
# starting a container can take up to 1 minute the first time or more
# deppending on your hardware and the image os
vedv container start alpineShow running container
vedv container lsIf you see the error: Connection timed out during banner exchange,
the container is starting, wait around 10 seconds and try again.
To avoid that you can starts the container with the --wait flag
If you run the login command and the container is stopped, it is started automatically
# login to a container can take up to 1 minute the first time or more
# deppending on your hardware and the image os
vedv container login alpinePlay with it :)
Exit the container and remove it
vedv container rm --force alpineShow the image and copy its name to remove
vedv image lsRemove the image
vedv image rm <your-image-name>This is the sample app from https://www.docker.com/101-tutorial/. Thanks to Docker.
We will be working with a simple todo list manager that is running in Node.js.
Clone the app repository and switch to its directory
- from github:
git clone https://github.com/yunielrc/todo-101.git && cd todo-101- Create a file named
Vedvfilein the same folder as the file package.json with the following contents.
# Download the image from the registry and import it
FROM admin@alpine/alpine-3.17.3-x86_64
# Set the user, create it if doesn't exist and change the owner
# of WORKDIR recursively
USER root
# Set the working directory and create it if doesn't exist
WORKDIR /app
# Run commands on the current working directory
RUN apk add -U nodejs~18 npm yarn
# Copy package.json to the current working directory
COPY package.json .
# Copy yarn.lock to the current working directory
COPY yarn.lock .
# Install dependencies
RUN yarn install --production
# Copy the service file and set the permissions
COPY --chmod 0755 ./root/etc/init.d/todo-101 /etc/init.d/todo-101
# Add the service to the default runlevel (auto-start on boot)
RUN rc-update add todo-101
# Copy the source code to the current working directory
COPY . .
# Start the service
RUN rc-service todo-101 start
# Expose port 3000
EXPOSE 3000/tcpVedvfile syntax is highly inspired by Dockerfile, but with some differences
-
Save the file and exit
-
Build it
vedv image build -n todo-101-alpine-1.0.0-x86_64
# 55.360s --> [download] + [import] + creation + startup + build + stop time- Create a container from the image
vedv container create -p 3000:3000/tcp -n todo-101 todo-101-alpine-1.0.0-x86_64
# 1.766s
# When the ports are the same like `3000:3000` the command below is equivalent
# vedv container create -p 3000/tcp -n todo-101 todo-101-alpine-1.0.0-x86_64
vedv container start -w todo-101
# 13.631s-
Open your browser and go to http://localhost:3000
You should see the todo list manager app
vedv container rm --force todo-101
# 2.829s- In the src/static/js/app.js file, update line 56 to use the new empty text.
- <p className="text-center">No items yet! Add one above!</p>
+ <p className="text-center">You have no todo items yet! Add one above!</p>- Let's build our updated version of the image, using the same command we used before.
vedv image build -n todo-101-alpine-1.0.0-x86_64
# 30.420s --> [download] + [import] + [creation] + startup + build + stop timeEach line like the one below is a deleted layer:
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
In this case 4 layers were deleted, starting from COPY . . to the end of the file,
and the layer POWEROFF added during the build process.
- Let's start a new container using the updated code.
vedv container create -p 3000/tcp -n todo-101 todo-101-alpine-1.0.0-x86_64
# 1.811s
vedv container start -w todo-101
# 13.698s-
Refresh your browser on http://localhost:3000 and you should see your updated help text!
-
Remove the container
vedv container rm --force todo-101
# 3.207s👉 There is only 200MB of free space for each user at the moment, so use it for testing purposes only.
vedv image push <your_user_id>@alpine/todo-101-alpine-1.0.0-x86_64
# 1m 31.28s 12Mbps
# If you want to push the image with a different name run the command below:
# vedv image push -n todo-101-alpine-1.0.0-x86_64 <your_user_id>@alpine/<your_image_name>-1.0.0-x86_64- On finish open your browser and go to https://registry.vedv.dev/apps/files/?dir=/00-user-images
You should see the collection:
<your_user_id>@alpine
- Click on the collection and you should see the image file an its .sha256sum file:
todo-101-alpine-1.0.0-x86_64.ova
todo-101-alpine-1.0.0-x86_64.ova.sha256sum
The uploaded image size is around 130MB, to give chance other users to test vedv, please delete the image from the registry and push an image link instead. Thanks.
👍 Congratulations, you have builded and uploaded your first image to the registry.
By default the images are private, only you can see them.
To make your image public, you need to share it with the group public.
-
Open your browser and go to https://registry.vedv.dev/apps/files/?dir=/00-user-images
-
Click the Share button on the image you want to share
- Or click the Share folder menu item
- On the Sharing panel search for
publicgroup and select it
- If the image is shared the collection must have the Shared icon
👍 Congratulations, your image is Public and other users can download and use it
If your registry has images that weights MB, to give chance other users to test vedv, please delete that images and push image links instead. Thanks.
Due to the limited storage space at the moment, you should push image links to the registry.
- Export the image to a file with the .ova extension
vedv image export todo-101-alpine-1.0.0-x86_64 todo-101-alpine-1.0.0-x86_64.ova
# 8.157s- Upload the image and the .sha256sum to OneDrive or Google Drive or any http server,
-
Open the image file menu and Click on Embed as shown in the image below
- Click on Generate button
- Reaload onedrive page and Click on Shared
- Click on Links Tab and Copy it
- Open your terminal and save the image link to a variable
Replace <image_link> with the link you have copied
image='onedrive=<image_link>'-
Repeat the steps 1-5 for sharing the .sha256sum file
-
On the same terminal save the .sha256sum link to a variable
Replace <sum_link> with the link you have copied
sum='onedrive=<sum_link>'- On the same terminal execute the command below
vedv image push-link --image-address "$image" --checksum-address "$sum" \
<your_user_id>@alpine/todo-101-alpine-1.0.0-x86_64
# 2.706s- Remove the image to download it again to test the link
vedv image rm todo-101-alpine-1.0.0-x86_64
# 0.995s- Pull the image from the link
Use the flag --no-cache to download the image again
vedv image pull --no-cache -n todo-101-alpine-1.0.0-x86_64 \
<your_user_id>@alpine/todo-101-alpine-1.0.0-x86_64
# 17.334s 90Mbps👍 Congratulations, you have uploaded your image link to the registry and saved space for other users.
-
Open the image file menu and Click on Share as shown in the image below
- Click on General access and select Anyone with the link
- Copy the link and click on Done
- Open your terminal and save the image link to a variable
Replace <image_link> with the link you have copied
image='gdrive-big=<image_link>'-
Repeat the steps 1-5 for sharing the .sha256sum file
-
On the same terminal save the .sha256sum link to a variable
Replace <sum_link> with the link you have copied
sum='gdrive-small=<sum_link>'- On the same terminal execute the command below
vedv image push-link --image-address "$image" --checksum-address "$sum" \
<your_user_id>@alpine/todo-101-alpine-1.0.0-x86_64If you see the error below:
error downloading file from https://drive.google.com/file/d/......
Execute the command below at least 5 times:
wget --no-check-certificate \
--connect-timeout 10 \
--header "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" \
-O "alpine-3.18.3-x86_64-fat-inv.ova.sha256sum" "https://docs.google.com/uc?export=download&id=1X5v6DYZeEo3zLLd2ZYIEbRO4hI1IQ9gY"If google drive respond with ERROR 500 or ERROR 403,
and you make a right copy of the url, the error is due google drive it's having problems
or it can be doing some nasty things to make it hard to download.
- Remove the image to download it again to test the link
vedv image rm todo-101-alpine-1.0.0-x86_64- Pull the image from the link
Use the flag --no-cache to download the image again
vedv image pull --no-cache -n todo-101-alpine-1.0.0-x86_64 \
<your_user_id>@alpine/todo-101-alpine-1.0.0-x86_64- Remove the image
vedv image rm --force todo-101-alpine-1.0.0-x86_64👍 Congratulations, you have uploaded your image link to the registry and saved space for other users.
# --------------
# Text surrounded by `` on the comments here is for emphasize.
# Environment variables are always expanded in the Vedvfile keeping its quotes,
# even those surrounded by ''. To avoid a variable expansion on Vedvfile escape
# it, e.g.: \$VAR
# The scope of environment variables is Vedvfile and image os.
# Globbing is disabled in the Vedvfile, it is only enabled on the image os.
# All instructions are evaluated in the Vedvfile, except:
# RUN, to send the command to the image os as it is.
# ENV, to send the variable to the image os as it is, with its quotes.
# (unescaped environment variables are expanded in RUN and ENV instructions too)
# Host os environment variables aren't available in the Vedvfile
# -------------
# Download the image from the registry and import it
FROM admin@alpine/alpine-3.17.3-x86_64
# Declare an environment variable
# image os evals `DEST=dest`
ENV DEST=dest
# Image os evals `GREETINGS='hello world'`
ENV GREETINGS='hello world'
# Set the shell
SHELL /bin/sh
# Copy all files from `src` to `dest`. `COPY src/* dest` doesn't work because
# globbing is disabled in the Vedvfile.
COPY src/ dest
# Copy as root user
COPY --root ./src ./dest
# Copy and change user and group of `dest` to vedv recursively
COPY --chown vedv:vedv src dest
# Copy `src` to `WORKDIR` as vedv user
COPY --user vedv src .
# Variable `DEST` is expanded in the Vedvfile
COPY . $DEST
# Run command as root user
RUN --root id
# Run command as vedv user
RUN --user vedv id
# Variable is expanded in the Vedvfile
# image os evals `echo "hello world"`
RUN echo "$GREETINGS"
# Variable is expanded in the image os, the output is the same as the previous
# command, image os evals `echo "$GREETINGS"`
RUN echo "\$GREETINGS"
# Variable is expanded in the Vedvfile
# image os evals `echo 'hello world'`
RUN echo '$GREETINGS'
# Change the system cpu cores and memory
# Never use this instruction at the end of the Vedvfile, because it poweroff
# the image and don't save the state, so a rebuild will take more time.
# It is a very time consuming instruction, to change cpu cores and memory,
# the image needs to be turned off, the instruction is executed, and the image
# is turned on again by the next instruction.
SYSTEM --cpus 2 --memory 512Transparency and security are very important to us, so all official images are created from the scratch, here are the steps:
- The OS iso is downloaded from the official source and verified
- A vm is created from the iso on Virtualbox and started
- The vm is configured for vedv. e.g.: For alpine linux 3.17.3-x86_64
image configuration:
- 3.1. The specific CaC script
icac/images/alpine-linux.vm.prod.cacis copied and executed in the vm. - 3.2. The common linux CaC script
icac/images/common-linux.vm.prod.cacis copied and executed in de the vm.
- 3.1. The specific CaC script
- The vm is securely powered off and exported to an
alpine.ovafile from Virtualbox - The
alpine.ovafile is imported with the command:vedv image import -n alpine-3.17.3-x86_64 ./alpine.ova - The image is uploaded to the registry with the command:
vedv image push admin@alpine/alpine-3.17.3-x86_64 - The image is tagged as
Officialon registry web interface - The image is shared with the group
publicon registry web interface
We encourage you to read the CaC scripts to see what they do inside the vm.
The registry is a Nextcloud instance, you can host your own registry, Nextcloud is free and open source software.
The scripts for provision and configuration are located at
icac/nextcloud/vultr.prod/nextcloud-aio.
If you want to host your own registry on Vultr you must have a vultr account (I am not associated with vultr), then follow the steps below:
- Copy and edit the .env file, inside the .env look for the section
icac/nextcloud/vultr
cp .env.sample .env
vim .env- Provision and Configure the registry
make registry-prod-setupIf you want to host your own registry on any other cloud provider, you can reuse the CaC script and write your IaC script for provisioning.
ydf is a dotfiles manager+ that uses vedv for developing.
ydf is the kind of tool that make changes to your system, install apps and write config files, so it needs to be tested in a secure, isolated and reproducible environment, to avoid damaging your system and to guarantee that the changes made by ydf are always reproducible on a clean system during testing. Many apps that are installed by ydf are made for a desktop environments, others apps rely on systemd, and others technologies that doesn't work in docker containers.
ydf is developed initially for Manjaro so the vedv image used is admin@manjaro/manjaro-gnome-22.1.3-x86_64
You should open and see the 4 files below available on ydf project to understand how to use vedv
during development:
Makefile, tools/ct-clean-exec, Vedvfile,.vedvfileignore
Start with the Makefile at: https://github.com/yunielrc/ydf/blob/master/Makefile
Good luck.
Contributions, issues and feature requests are welcome!
# SYSTEM: Linux yuniel-pc 6.1.44-1-MANJARO #1 SMP PREEMPT_DYNAMIC Wed Aug 9 09:02:26 UTC 2023 x86_64 GNU/Linux
# YAY
make 4.4.1-2
davfs2 1.7.0-1
python-pre-commit 2.20.0-3
shfmt 3.7.0-1
shellcheck 0.9.0-45
nodejs 20.5.0-1
npm 9.8.1-1
bash-bats 1.9.0-1
bash-bats-assert-git 2.1.0-1
bash-bats-file 0.4.0-2
bash-bats-support-git 0.3.0-1
vultr-cli 2.17.0-1
# NPM
@commitlint/[email protected]
@commitlint/[email protected]
[email protected]
[email protected]
# SYSTEM: Linux user-MS-7D43 6.2.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 13 16:27:29 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
# APT
davfs2 1.6.1-1
make 4.3-4.1build1
pre-commit 2.17.0-1
# SNAP
node 18.17.1
shellcheck v0.9.0
shfmt 3.5.1
# NPM
@commitlint/[email protected]
@commitlint/[email protected]
[email protected]
[email protected]
[email protected]
# GITHUB
bats-support
bats-assert
bats-file
vultr-cli
Execute the command below to copy config samples to the product root directory
cp .env.sample .env
cp .vedv.env.sample .vedv.envFor Manjaro:
make OS=manjaro configureFor Ubuntu:
make OS=ubuntu configureExecute the command below to provision and configure the registry development service on local machine
make registry-dev-setupWrite your code
If functional and integration tests that work with virtualbox are failing,
delete all virtual machines and hard disks from Virtualbox and delete the
Virtualbox VMs directory usually located at ~/VirtualBox\ VMs and run the tests again.
Before testing, the registry development service must be started. Check the status with the command below.
make registry-dev-statusStart it if stopped
make registry-dev-startRun Unit Testing for one component
make test-suite u="$(fd registry-service.bats)"
# 4.899sRun Unit Testing for one function
make test-name n='::push_link\(\)' u="$(fd registry-service.bats)"
# 0.676sRun Integration Testing for one function
make test-name n='download_file()' u="$(fd registry-nextcloud-api-client.i.bats)"
# 3.105sRun Functional Testing for one function
make test-name n="vedv registry push[^-]" u="$(fd registry-command.f.bats)"
# 13.85sRun All Unit Tests
make test-unit
# 1m 4.09sRun All Integration Tests
make test-integrationRun All Functional Tests
make test-functionalRun All tests
On Manjaro
make OS=manjaro test-all
# 45m 12.79sOn Ubuntu
make OS=ubuntu test-allThis project uses Conventional Commits.
For commiting use the command below
make commitGive a ⭐️ if this project helped you!
![@github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=64&v=4){kind=small}