You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
How To Install Nginx, MySQL, PHP, Redis and more on a Debian 10 Virtual Machine
This series of documents will configure and setup a Nginx, MySQL, and
PHP (LEMP) server on a pretty feature-rich VM running Debian. I used a
VM with 8GB RAM and 4 CPU Cores so YMML with some of the settings below.
This will also install other useful packages and configurations like,
Redis, Memcached, Node, NPM, Composer, and a fully automated SSL
service using certbot for Let's Encrypt.
These documents take you through each step of the configuration from:
$ sudo apt install mariadb-server mariadb-client
$ sudo systemctl start mariadb
$ sudo systemctl enable mariadb
$ sudo mysql_secure_installation # answer N to validate password plugin
$ sudo mysql
Configure MySQL users
mysql>SELECT user,authentication_string,plugin,host FROMmysql.user;
mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password';
mysql> FLUSH PRIVILEGES;
mysql>SELECT user,authentication_string,plugin,host FROMmysql.user;
mysql> exit
Step 3 - Install PHP-FPM
I've chosen to install PHP 8.0 but you can change any of the commands from php8.0 to php7.4 as needed. For example any reference to php8.0-fpm can be adjusted simply to php7.4-fpm.
These are the different configuration files for setting up the servers.
Configure php.ini
$ sudo nano /etc/php/8.0/fpm/php.ini
OR for PHP 7.4:
$ sudo nano /etc/php/7.4/fpm/php.ini
Change these values:
cgi.fix_pathinfo=0
post_max_size=256M
upload_max_filesize=256M
date.timezone=Europe/London
[opcache]opcache.memory_consumption=256
opcache.max_accelerated_files=50000
opcache.validate_timestamps=0
opcache.revalidate_freq=120
opcache.error_log=/var/log/opcache-errors.log
[apc]apc.shm_size=128M
apc.stat=1
; Relative to the number of cached files (you may need to watch your stats for; a day or two to find out a good number).apc.num_files_hint=7000
; The number of seconds a cache entry is allowed to idle in a slot before APC; dumps the cache.apc.ttl=7200
apc.user_ttl=7200
apc.include_once_override=0
; Allow 2 seconds after a file is created before it is cached to prevent users; from seeing half-written/weird pages.apc.file_update_protection=2
; Ignore filesapc.filters = "/var/www/html/apc.php"apc.cache_by_default=1
apc.use_request_time=1
apc.slam_defense=0
apc.stat_ctime=0
apc.canonicalize=1
apc.write_lock=1
apc.report_autofilter=0
apc.rfc1867=0
apc.rfc1867_prefix=upload_
apc.rfc1867_name=APC_UPLOAD_PROGRESS
apc.rfc1867_freq=0
apc.rfc1867_ttl=3600
apc.lazy_classes=0
apc.lazy_functions=0
This section outlines how to go about configuring your php-fpm www.conf pool file. It may seem complicated and overwhelming and you can totally skip this if you want to. That said, overlooking this step is like buying a sports car and never going above 20 mph; you're going to kill the engine by not using the car correctly. You really do want to maximise your server as much as possible to avoid bottlenecks and crashes later on.
An Overview
You are probably going to want to read this article on FPM tuning which I used to determine the settings below. The article will help you scientifically calculate the exact settings you should use for max_children, start_servers, min_spare_servers, and max_spare_servers. The most complicated one to calculate is max_children because there is a formula he uses to do this which is (Total RAM - Memory used on server) / Process Size = max_children. I've tried my best to give you the succinct version of what to calculate, as well as how, but depending on your level of skill, you may need to read the article fully.
There is also reference to a script called ps_mem.py which, when run on your server, will give you the total Process Size needed in the max_children formula above.
Calculations
Here are the settings and calculations. The Value column has details on how to calculate the Setting value.
Setting
Value
max_children
(Total RAM - Memory used on server) / Process Size
start_servers
Number of CPU cores x 4
min_spare_servers
Number of CPU cores x 2
max_spare_servers
Equal to same value as start_servers
How to calculate max_children
Let's assume, for this example, you have a server with 8GB RAM and 4 CPU Cores. With this in mind we can continue to work this formula out:
(Total RAM - Memory used on server) / Process Size = max_children
First let's run some commands on the server to get the data we need for the formula.
Memory Usage
We need to know how much total memory we have as well as how much of the memory is used. The rest is irrelevant.
$ sudo free -hl total used free shared buff/cache availableMem: 7.8Gi 331Mi 6.0Gi 27Mi 1.5Gi 7.1Gi
Process Memory
You're now going to want to download the ps_mem.py file and run the command. When done we are only interested in the total amount the sum calculates; i.e. the total amount in MiB after the = sign.
We've got everything we need to apply these values to the formula as follows:
Total RAM = 7800 is the total column value (7.8Gi) when running sudo free -hl
Memory Used = 331 is the used column value (331Mi) when running sudo free -hl
Process Size = 33.7 is the total value (33.7 MiB) after executing sudo python ps_mem.py | grep php-fpm
Pull this all together and run the formula of (Total RAM - Memory used on server) / Process Size which will give you something similar to this:
(7800 - 331) / 33.7 = 221.63
Therefore, we now know that max_children = 222 👍🏿
The other calculations are now easy because they are all based on how many CPU Cores your have. Open up the www.conf file and begin editing with your newly calculated values.
Edit the Settings
Change the www pool so it now has the following settings.
$ sudo nano /etc/php/8.0/fpm/pool.d/www.conf
OR for PHP 7.4:
$ sudo nano /etc/php/7.4/fpm/pool.d/www.conf
[www]user = www-data
group = www-data
; listen = /run/php/php7.4-fpm.socklisten = /run/php/php8.0-fpm.sock
listen.owner = www-data
listen.group = www-data
; Server Resources: 8G RAM | 4x CPU Cores; `free -hl` output: Total 7.8Gi Used 331Mi Free 5.8Gi; `ps_mem.py` output: 24.8 MiB + 9.0 MiB = 33.7 MiBpm = dynamic
; (Mem - Used) / ps_mem.py = max_children; (7800 - 331) / 33.7 = 221.63pm.max_children = 222
; CPU * 4 = 16pm.start_servers = 16
; CPU * 2 = 8pm.min_spare_servers = 8
; Same as start_serverspm.max_spare_servers = 16
pm.max_requests = 500
Reload PHP:
$ sudo systemctl reload php8.0-fpm # or replace with php7.4-fpm
To ensure good server performance, the total number of client connections, database files, and log files must not exceed the maximum file descriptor limit on the operating system ulimit -n. Linux systems limit the number of file descriptors that any one process may open to 1,024 per process. On active database servers (especially production ones) it can easily reach the default system limit.
To increase this, edit /etc/security/limits.conf and specify or add the following:
# Added to increase server limit of 1024 per process.
mysql soft nofile 65535
mysql hard nofile 65535
# Uncomment this if ulimit doesn't change from 1024#* soft nofile 65535
This requires a system restart. Once rebooted, check the limit with:
$ ulimit -Sn65535
$ ulimit -Hn65535
Alternatively, you could set system-wide file and process limits with the following configuration:
# /etc/security/limits.conf# Added to increase server limit of 1024 per process.
* soft nofile 102400
* hard nofile 102400
* soft nproc 10240
* hard nproc 10240
NB: Your milage may vary so please only use the above with caution!
Most database setups do not need to record file access time. You might want to disable this when mounting the volume into the system. To do this, edit your file /etc/fstab. For example, on a volume named /dev/sda1, this how it looks like
/dev/sda1 / ext4 noatime,errors=remount-ro 0 1
Tuneup MariaDB To Utilise Memory Efficiently
You have these settings to consideration when fine-tuning your server:
innodb_buffer_pool_size: You can set this from 70-80% of the total available memory on a dedicated database server with only or primarily InnoDB tables. If set to 2 GB or more, you will probably want to adjust innodb_buffer_pool_instances as well. You can set this dynamically if you are using MariaDB >= 10.2.2 version.
innodb_flush_log_at_trx_commit: This setting offers a significant trade-off between performance and reliability. When set to 0, database performance is greatly increased. However, up to 1 second of transactions can be lost in a crash. The default value for this setting is 1.
innodb_flush_method: Set this to O_DIRECT in order to avoid double buffering data.
innodb_log_file_size: If you are preferring more performance gains especially during and handling your InnoDB transactions, setting the variable innodb_log_file_size to a larger value such as 5Gib or even 10GiB is reasonable. Increasing means that the larger transactions can run without needing to perform disk I/O before committing.
max_connections: If you are application requires a lot of concurrent connections, you can start setting this to 500.
max_allowed_packet: The max_allowed_packet directive defines the maximum size of packet that can be sent. If the client sends more data than max_allowed_packet size, the socket will be closed. Ideally, especially on most application demands today, you can start setting this to 512M.
back_log: Increase if you expect short bursts of connections. Cannot be set higher than the operating system limit. To calculate use (50 + max_connections/5) to get total value.
Query Cache: Preferably you should disable query cache in all of your MariaDB setup. You need to ensure that query_cache_type=OFF and query_cache_size=0 to complete disable query cache. Unlike MySQL, MariaDB is still completely supporting query cache and do not have any plans on withdrawing its support to use query cache.
Turn on slow_query_log and se the long_query_time to 1 second to log queries to /var/lib/mysql/hostname-slow.log.
There are now also some issues with systemd limiting file processes. Have a look at the following:
$ sudo systemctl status redis-server● redis-server.service - Advanced key-value store Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2021-08-01 23:24:48 BST; 13min ago Docs: http://redis.io/documentation, man:redis-server(1) Main PID: 571 (redis-server) Tasks: 4 (limit: 4915) Memory: 9.7M CGroup: /system.slice/redis-server.service └─571 /usr/bin/redis-server 127.0.0.1:6379
If you see the Tasks: 4 (limit: 4915) has a limit set of 4915. This is how the Linux Kernel works and it defaults to 15%,
which equals 4915 with the kernel's defaults on the host.
In order to check what your system is capable of it's a rather easy calculation.
$ more /proc/sys/kernel/pid_max32768
You now take this number and work out 15% of the value, i.e. 32768*15/100 = 4915.2 and you can see how this is calculated.
That said, you can override this for certain Debian-based services and here's how we change them for those services.
$ sudo apt-get install python-certbot-nginx
$ sudo openssl dhparam -out /etc/nginx/dhparam.pem 2048
$ sudo mkdir -p /var/www/_letsencrypt
$ sudo chown www-data /var/www/_letsencrypt
$ sudo certbot certonly --webroot -d example.com -d www.example.com --email [email protected] -w /var/www/_letsencrypt -n --agree-tos --force-renewalSaving debug log to /var/log/letsencrypt/letsencrypt.logPlugins selected: Authenticator webroot, Installer NoneObtaining a new certificatePerforming the following challenges:http-01 challenge for example.comUsing the webroot path /var/www/_letsencrypt for all unmatched domains.Waiting for verification...Cleaning up challengesIMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/example.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/example.com/privkey.pem Your cert will expire on 2022-02-22. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew"
You will now need to create a post hook script for certbot so that it can reload Nginx once a domain has been renewed. Luckily I have created a script for you to do this and you can access the script over here. Setting up this script as is a simple 3 step process.
You are going to want to copy the code of the script by clicking here and head on over and paste it in your /etc/letsencrypt/renewal-hooks/post/nginx-reload.sh file.
Step 3
Once you have the file saved, make it executable and reload nginx.
Done. The script is now setup to begin working automatically in the background. Certbot runs a cronjob twice daily and if a certificate is renewed your newly created post hook script will be executed and run.
Success Output
Executing the nginx-reload.sh file without any errors in any of your nginx config or virtual host files will output the following success messages to your console:
$ sudo /etc/letsencrypt/renewal-hooks/post/nginx-reload.shRemoving old log file.Success: Reloading nginx server.
Similarly, when the certbot cronjob executes and runs this post hook script the following successful output will be seen.
Executing the nginx-reload.sh file with an error in either your nginx config or virtual hosts files will output the following error and log file:
$ sudo /etc/letsencrypt/renewal-hooks/post/nginx-reload.shRemoving old log file.Fail: Error with config file.Aborting nginx restart.Log file output:nginx: [emerg] "worker_connections" directive is not allowed here in /etc/nginx/nginx.conf:7nginx: configuration file /etc/nginx/nginx.conf test failed
Similarly, when the certbot cronjob executes and runs this post hook script the following output will be seen.
This uses public/private SSH key for authentication.
$ sudo apt-get install openssh-server
$ sudo useradd -m USERNAME -g www-data # add user to group they can write to www folder
$ sudo passwd USERNAME # set password for login
$ sudo chmod -R g+w /var/www # set group write permissions
$ sudo mkdir -p /home/USERNAME/.ssh
$ sudo nano /home/USERNAME/.ssh/authorized_keys # copy your `~/.ssh/id_rsa.pub` key from computer and paste here
$ sudo chmod -R go= /home/USERNAME/.ssh
$ sudo chown -R USERNAME:www-data /home/USERNAME
$ sudo usermod -a -G sudo USERNAME # optionally, add user to sudoers file
$ sudo service ssh restart
Now login to the SSH server with your private key and test:
$ ssh -i ~/.ssh/id_rsa [email protected]Linux server.com 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64The programs included with the Debian GNU/Linux system are free software;the exact distribution terms for each program are described in theindividual files in /usr/share/doc/*/copyright.Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extentpermitted by applicable law.Last login: Tue Dec 8 23:12:02 2020 from 104.245.101.122USERNAME@server:~#
If all goes well you won't be prompted for a password and you will just login straight away. If you are asked for a password you have done something wrong. Try again.
Turn off sudo password authentication
With the above configured you will still be prompted for your password when running a sudo command on the server. To disable the prompting of passwords, do the following.
$ sudo visudo
Add your username to the end of the file so it looks similar to:
# Allow members of group sudo to execute any command
%sudo ALL=(ALL:ALL) ALL
#includedir /etc/sudoers.d
USERNAME ALL=(ALL) NOPASSWD: ALL
NB: replace USERNAME with the username you created, don't leave it as is. Then save the file and restart the ssh server.
supervised systemd # use Debian's systemd
bind 127.0.0.1 ::1
# If you want to secure your Redis server set a password below.# requirepass change-this-password
Restart Redis
$ sudo systemctl restart redis-server
$ sudo systemctl status redis-server
# Start with a cap of 64 megs of memory. It's reasonable, and the daemon default# Note that the daemon will grow to this size, but does not start out holding this much# memory
-m 256
Composer will be installed at /usr/local/bin/composer and you can test the installation as follows:
$ /usr/local/bin/composer -VComposer version 2.1.6 2021-08-19 17:11:08
If you get an error output stating Installer corrupt, it means the version of composer has been updated and you'll need a new hash. Visit the official getcomposer.org download page to get the latest hash_file string and/or install via that method.
Install Node & NPM
Debian 10 comes with Node 10 and the following instructions will install Node 12 and NPM 6 by installing from source. If you would like to install Node 14 simply replace setup_12.x in the command below with setup_14.x. If that's not high enough and you'd like Node 15, replace the command with setup_15.x instead.
Optionally, you can install the Debian build-essential packages as certain node packages require the source code of certain packages when building them at runtime. You probably won't need to do this but I've included it here for those who think they will need it moving forward.