CentrioHost Blog

Introduction

It has been a long time since my first vps handbook published on Linux Today . So I am writing this new version VPS Hadnbook ,V2 ,for VPS/ Cloud users and System engineers. It will be a good checklist for optimizing and securing a vps node.
The aim of this book is to optimize and harden a VPS node.

Terms used :

The Following technical terms are used in this article,

Node – Represent the Physical / Hardware Node

VPS , Container , Guest – Represent the virtual servers hosted inside the node

Requirement :

This article is based on the Openvz virtualization Platform . You can easily adopt this procedure for most of the Linux VPS/Cloud technologies and Nodes. I used the following environment for doing this task

OS – Centos 6 64 bit

Vps technology – Openvz

The VPS node optimization and Security have two parts as follows,

a) Node Optimization & Security
b) container Optimization & Security
In this article I am referring only to the node optimization and security part. If you have any type of custom application and services running on your vps node ,then you have to think twice before applying the patches and steps mentioned below.

Before Installing VPS Platform:

You need to install and OS along with proper disk partition for building a proper production vps node. These are some tips to you.

a) Slow File Systems : Don’t use software RAID in Nodes. Software is running over the host kernel , so which have to use the CPU and RAM . It is slower than hardware RAID. So I recommend to use Hardware RAID instead of software RAID . Also don’t use LVM partitions. LVM is slower than generic raid ext3/ext4 partitions . But still some control panels prefer this. LVM have the ability to resize , but if your are planning for ultimate performance, then don’t use it. Using the ext4 filesystem is a better choice.

b) Use Faster HDD’s : It is better to use 15K SAS HDD’s or SSDs . The disk I/O problem is a biggest enemy for VPS hosters . So if you chose faster HDD’s then you can add three times the number of containers in a node. These HDD’s are little costly than normal SATA HDD’s , but you can save it from selling more vps from a single node.

c) Chose CPU Wisely : You may need to chose a CPU for VPS node based on the performance and your budget. You need to plan the total cost that you are spending on a vps node and you are going to get i from selling the containers. There is a best place , that you can compare the CPU model. It is from http://www.cpubenchmark.net/

d) Choosing RAM : Well you may need to remember only one thing . Don’t allow the node to use massive amount of SWAP. So if you see using large SWAP , it is a time to do the RAM upgrade. RAM is not so costly . You can add 32 GB or 64 GB RAM according to your need.

Optimization

In this sections we will do some sort of optimization. It is a generic optimization technique that can be usefull for your node.

HDD Optimization :

As I suggested you may need to chose a good Physical HDD initially with a proper disk partition. It is better to use a disk partition as follows for an openvz node.

 /  =  20 GB
 /tmp = 5 GB
 /vz  =  Big space for VPS
 /backup (optional ) for backups

After installing the OS and Openvz . You may need to adjust the above disk partitions mount options in /etc/fstab as follows ,

/dev/sdb3 	/vz	ext4	defaults,noatime  0 0 

The noatime flag will stop disk timestamps and increase the I/O speed

Service Optimization:

You can remove all useless services from the containers. It is better to use the hostnode only for hosting the containers. If you are using the node for apache / mysql / email services ,that will result in performance degrade , because the host node have to share its resources for all these services. It is also better to remove services like FTP , Rlogin ,etc,. from the nodes too.

Removal Of Useless Software:

If you minimize the softwares installed in the node, then you minimize the security vulnerability and increase the performance too. You can list the following commands for finding the installed packages.

# yum list installed
# yum list PACKAGE-NAME
# yum remove PACKAGE_NAME
# rpm -qa 
# rpm -e PACKAGE-Name

Removal of useless Services:

You can disable all useless services that running in the main node. The services like xfs, cups , bluetooth must be disabled. You can check it as follows,

# chkconfig --list | grep '3:on'
crond              0:off    1:off    2:on    3:on    4:on    5:on    6:off
ip6tables          0:off    1:off    2:on    3:on    4:on    5:on    6:off
iptables           0:off    1:off    2:on    3:on    4:on    5:on    6:off
lighttpd           0:off    1:off    2:on    3:on    4:on    5:on    6:off
messagebus         0:off    1:off    2:on    3:on    4:on    5:on    6:off
network            0:off    1:off    2:on    3:on    4:on    5:on    6:off
rsyslog            0:off    1:off    2:on    3:on    4:on    5:on    6:off
solusvm-ipsecure    0:off    1:off    2:on    3:on    4:on    5:on    6:off
solusvm-netspeed    0:off    1:off    2:on    3:on    4:on    5:on    6:off
sshd               0:off    1:off    2:on    3:on    4:on    5:on    6:off
sysstat            0:off    1:on    2:on    3:on    4:on    5:on    6:off
udev-post          0:off    1:on    2:on    3:on    4:on    5:on    6:off
vpsmon-agent       0:off    1:off    2:off    3:on    4:on    5:on    6:off
vpsmon-enforce     0:off    1:off    2:off    3:on    4:on    5:on    6:off
vpsmon-io          0:off    1:off    2:off    3:on    4:on    5:on    6:off
vz                 0:off    1:off    2:on    3:on    4:on    5:on    6:off
vzeventd           0:off    1:off    2:on    3:on    4:on    5:on    6:off

You can disable a service as follows,

# service stop servicename
# chkconfig  servvicename off

Kernel Optimization:

You can use an uptodate kernel with a good sysctl.conf parameter for tuning it. It will be discussed deeply in the comming sections.

Uptodate System Software:

It is better to keep the system software always uptodate . You can update this node as follows,

# yum update

Fix System Clock:

It is better to keep the server clock correctly. You can update it using rdate and set a daily cron for it as follows,

# yum install rdate
# rdate -s rdate.cpanel.net 

Set cron as follows under root,

0 1 * * * /usr/bin/rdate -s rdate.cpanel.net && hwclock --systohc

Optimize the System Iptable:

In an openvps vps , there is always issues with iptables and loaded firewall moduels. Well you can add a set of generic iptable modules in the main node configuration file /etc/vz/vz.conf . You can update it as follows,

## IPv4 iptables kernel modules to be enabled in CTs by default
IPTABLES="ipt_REJECT ipt_tos ipt_limit ipt_multiport iptable_filter iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length ip_tables iptable_filter ipt_LOG   ipt_state  ipt_owner iptable_nat ipt_REDIRECT ip_conntrack iptable_nat ip_nat_ftp ip_conntrack_ftp"
## IPv4 iptables kernel modules to be loaded by init.d/vz script
IPTABLES_MODULES="$IPTABLES"

The above iptables loaded modules will help your container to avoid issues related with CSF firewall installations and other iptable firewall scripts installations inside the containers.

VE0 CPUUnits:

It is better to increase the cpu cycle from default value 1000 to 10000 at least in /etc/vz/vz.conf .

VE0CPUUNITS=1000

You may need to reboot the vps service . But please note it will result in reboot of all containers. So better schedule a single reboot at the end of all the optimization task.

CPUUNITS Optimizations:

You can see some of the containers will be slow and you are not properly using the cpu cycle from the node. Please run the following command,

#  vzcpucheck  -v
VEID		CPUUNITS
-------------------------
0		1000
1002		166666
1003		166666
1004		166666
1007		10000
1001		166666
1006		10000
Current CPU utilization: 687664
Power of the node: 1360012

In the above result , you can see a total utilization and the power of the node. You can allocate more cpu . If you see a 1000 Units in the result for each container , then that is the default value of the containers and your vps containers are not tunned properly as per the node cpu cycle utilization. Adjusting the CPU units will increase stability of the containers and make your clients Happy. But improper adjustment will also result in performance abuse of a single container too. So if yo are planing to optimize the container based on the openvz configuration , it is better to find an expert in this task for doing this job. Host SEO Support Team have a lot of hands on experience in this area too.

Change Memory Setting To VSWAP:

The option VSWAP only available for RHEL6 containers. It takes openvz to a next level memory management. Now you can set two parameters with this value RAM and SWAP. You can use vzctl command to set the VSWAP in openvz as follows,

# vzctl set 1000 --swappages 0:1024M --save

See the swap result below,

[root@node ~]#  vzctl exec 1000 free -m
             total       used       free     shared    buffers     cached
Mem:          4096       1814       2281          0          0       1814
-/+ buffers/cache:          0       4096
Swap:         1024          2       1021

Please see more about VSWAP from https://centriohost.com//enable-swap-in-openvz-vps/ and http://openvz.org/VSwap

CPULIMIT Tuning:

You can setup proper CPULIMIT for each containers . It is the limite of the CPU usage for the container. The default value will be 0. That means there is no limit in CPU . Suppose if you have 4 CPU cores in the node , then then total cpu limit will be 400%. If you only need to use two core inside a container , you can set the value 200 using the command vzctl

# vzctl set 1000 --cpulimit 200 --save

Limit the number of CPU cores:

You can also limit the number of CPU cores that using inside a vps node. Suppose if you have 64 cores inside a node and you only need to allow 4 cores for a cpu. You can set it as follows,

# vzctl set 1000 --cpu 4 --save

Security & Hardening

Now we can add some basic security and analyze tests in the following sections.

Secure tmp folders:

This is one of the basic security that need to implement in all productions servers. First of all you need to make the following soft links

# rm -fr /var/tmp
# ln  -s /tmp /var/tmp
# rm -rf /usr/tmp
# ln -s /tmp /usr/tmp

Now need to secure /tmp . If you a separate HDD partition , then please add the following mount flag options to /tmp

/dev/sdb3	/tmp	ext4	defaults,noatime,noexec,nosuid 	0 0

Now remount /tmp as follows,

# mount -o remount /tmp

If you don’t have separate HDD partition . Then you can secure it as follows,

Create a raw file of size 5 GB as follows

dd if=/dev/zero of=/usr/tmpfile bs=1M count=5000

Format it using ext4 file system

# mkfs.ext4 /usr/tmpfile

Add the mount entry in /etc/fstab as follows

/usr/tmpfile              /tmp                    ext3 loop,nosuid,noexec,rw 0 0
mount -a

Now set the proper permission for /tmp

chmod  1777 /tmp

SHM Security:

You may need to secure /dev/shm mount options in /etc/fstab as follows,

tmpfs                   /dev/shm                tmpfs   defaults,noatime,noexec,nosuid        0 0

Loop Backup Resolver Security :

Most of those vps nodes will be configured as localhost resolver. That is not good. You may need to check the resolver configuration file , /etc/resolv.conf and remove the following lines

nameserver localhost
nameserver 127.0.0.1
search localhost 

You may need to ask to your server provider or ISP for getting the proper resolver IPs , otherwise you can use the following google resolvers,

nameserver 8.8.8.8
nameserver 8.8.4.4

Kernel Firmware Security:

I have seen a lot of Openvz nodes using the wrong kernel firmware package. You must chose the package “vzkernel-firmware”. You can install it as follows,

# yum install vzkernel-firmware

You can test it as follows too:

# rpm -qa | grep -i vzkernel-firmware

Sysctl Security:

You can secure the /etc/sysctl.conf by adjusting some of the following parameters below,

1) Disabled Proxy ARP:

# Disable proxy arp

net.ipv4.conf.default.proxy_arp = 0

Enabled Magic Sys Request Key:

# Enables the magic-sysrq key
kernel.sysrq = 1

Enabled the default interface to send redirects:

# We do not want all our interfaces to send redirects
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0

Enabled IP spoofing Protection:

# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

Disabled ICMP Redirect acceptance:

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

Enabled Log Spoofed Packets, Source Routed Packets, Redirect Packets:

# Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets net.ipv4.conf.all.log_martians = 0 net.ipv4.conf.lo.log_martians = 0

7) Disabled TCP Time stamps:

# Turn off the tcp_timestamps
net.ipv4.tcp_timestamps = 0

8) Enabled TCP synchronize cookie protection:

# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1

Enabled Ignoring Broadcast requests:

# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1

Enabled bad error message Protection:

# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 1

A sample sysctl.conf file will be as follows after applying all the above settings and some basic security tools,

# Enable packet forwarding enabled
net.ipv4.ip_forward = 1
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.all.forwarding = 1
# Disable proxy arp
net.ipv4.conf.default.proxy_arp = 0
# Enables source route verification
net.ipv4.conf.all.rp_filter = 1
# Enables the magic-sysrq key
kernel.sysrq = 1
# We do not want all our interfaces to send redirects
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.lo.log_martians = 0
# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time = 1800

# Turn off the tcp_timestamps
net.ipv4.tcp_timestamps = 0

# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1

# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 1

# Increases the size of the socket queue (effectively, q0).
net.ipv4.tcp_max_syn_backlog = 1024

# Increase the tcp-time-wait buckets pool size
net.ipv4.tcp_max_tw_buckets = 1440000

#tcp time stamp disable
net.ipv4.tcp_timestamps = 0

Secure Openssh Servers :

This is one of the important security changes. You need to disable direct ssh login of root user and the default ssh port too. You may need to edit /etc/ssh/sshd_config and make the following changes,

a) Enable only protocol 2 in sshd

# Protocol 2

b) Limit users to ssh . Only allow some users as follows,

# AllowUsers root centriohost supportadmin

c) Configure Idle Log Out Timeout Interval

User can login to server via ssh and you can set an idle timeout interval to avoid unattended ssh session. Open sshd_config and make sure following values are configured:

# ClientAliveInterval 300
# ClientAliveCountMax 0

d) Disable Hostbase authentication

# HostbasedAuthentication no

e) Disable rhost

# IgnoreRhosts yes

f) Disable Empty Passwords

# PermitEmptyPasswords no

g) Turn on privilege separation

# UsePrivilegeSeparation yes

h) Prevent the use of insecure home directory and key file permissions

# StrictModes yes

compiler Security:

You can disable all compilers that are using in node.

eg : chmod 700 /usr/bin/gcc

SHELL Fork Bomb Protection:

You can add the following to bashrc file for protecting the local shell users from fork bomb attack.

#unlimit so we can run the whoami
ulimit -n 4096 -u 14335 -m unlimited -d unlimited -s 8192 -c 1000000 -v unlimited 2>/dev/null

LIMITUSER=$USER
if [ -e "/usr/bin/whoami" ]; then
        LIMITUSER=`/usr/bin/whoami`
fi
if [ "$LIMITUSER" != "root" ]; then
        ulimit -n 100 -u 35 -m 200000 -d 200000 -s 8192 -c 200000 -v unlimited 2>/dev/null
else
        ulimit -n 4096 -u 14335 -m unlimited -d unlimited -s 8192 -c 1000000 -v unlimited 2>/dev/null
fi

History Time STAMP :

This will help the server administrator to track the VPS node shell operations . Add the following line to bashrc

HISTTIMEFORMAT='%F %T '

Identify Local Openports:

You can find all open ports as follows,

# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 127.0.0.2:25                0.0.0.0:*                   LISTEN      142797/master       
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      142797/master       
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      101481/sshd         
tcp        0      0 127.0.0.1:587               0.0.0.0:*                   LISTEN      238905/sendmail: MT
tcp        0      0 127.0.0.1:3306              0.0.0.0:*                   LISTEN      142409/mysqld       
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      340045/httpd        
tcp        0      0 0.0.0.0:6767                0.0.0.0:*                   LISTEN      106680/lighttpd     
tcp        0      0 127.0.0.1:53                0.0.0.0:*                   LISTEN      649832/named        
tcp        0      0 127.0.0.1:5557              0.0.0.0:*                   LISTEN      8824/vpsmon-agent l
tcp        0      0 127.0.0.1:953               0.0.0.0:*                   LISTEN      649832/named        
tcp        0      0 0.0.0.0:21                  0.0.0.0:*                   LISTEN      858317/vsftpd       
tcp        0      0 ::1:25                      :::*                        LISTEN      142797/master       
tcp        0      0 :::22                       :::*                        LISTEN      12070/sshd          
tcp        0      0 :::4925                     :::*                        LISTEN      567946/(squid)      
tcp        0      0 ::1:53                      :::*                        LISTEN      649832/named        
tcp        0      0 ::1:953                     :::*                        LISTEN      649832/named        
tcp        0      0 :::80                       :::*                        LISTEN      12125/httpd         
udp        0      0 127.0.0.1:53                0.0.0.0:*                               649832/named        
udp        0      0 162.216.155.101:1194        0.0.0.0:*                               238790/openvpn      
udp        0      0 0.0.0.0:35346               0.0.0.0:*                               567946/(squid)      
udp        0      0 199.48.164.93:123           0.0.0.0:*                               6698/ntpd           
udp        0      0 127.0.0.1:123               0.0.0.0:*                               6698/ntpd           
udp        0      0 0.0.0.0:123                 0.0.0.0:*                               6698/ntpd           
udp        0      0 :::36810                    :::*                                    567946/(squid)      
udp        0      0 ::1:53                      :::*                                    649832/named        
udp        0      0 fe80::2a01ff:feed:e878:123 :::*                                    6698/ntpd           
udp        0      0 ::1:123                     :::*                                    6698/ntpd           
udp        0      0 fe80::1:123                 :::*                                    6698/ntpd           
udp        0      0 :::123                      :::*                                    6698/ntpd           

Chrootkit Configuration:

You may need to install and configure the chrootkit tool in the server. You can setup a daily cron too for scanning it for rootkits.

cd /usr/local/src
rm -rf chkrootkit*
wget http://files.centriohost.com/src/chkrootkit.tar.gz
tar -xzf chkrootkit.tar.gz
cd chkrootkit-0.49/
make

Now setup a daily cron scripts, /etc/cron.daily/chkrootkit_cron.sh , as follows,

#!/bin/bash
EMAIL="youremail@yourdomain.tld"
cd /usr/local/bin/; ./chkrootkit 2>&1 | mail -s "Host SEO : CHKROOTKIT Scan Details" $EMAIL

Rkhunder:

This is a rootkit hunter tool. You can install it and configure as a daily cron. This will also help to do a daily scanning for rootkits.

cd /usr/local/src
rm -rf rkhunter*
wget http://files.centriohost.com/src/rkhunter-1.4.0.tar.gz
tar -xzf rkhunter-1.4.0.tar.gz
cd rkhunter-1.4.0/
sh installer.sh  --install

You can setup a cron as follows,

#!/bin/bash
EMAIL="youremail@yourdomain.tld"
/usr/local/bin/rkhunter --update 
/usr/local/bin/rkhunter -sk -c --nocolors  --versioncheck  | mail -s "Host SEO: RKhunter Scan Details" $EMAIL

LogWatch:

Logwatch will watch the logs. You can install and configure it for sending daily log reports to your email address too. For installing logwatch , do it as follows,

# yum install logwatch

Now edit the file /etc/cron.daily/00logwatch and modify the line begin with “logwatch” as follows,

/usr/sbin/logwatch --mailto  alertemail@yourdomain.tld

Please also change the email address in /usr/share/logwatch/default.conf/logwatch.conf . After that you will receive the daily alerts from this.

Analyze System Users With Empty Passwords:

It is better to find out which users are using empty passwords in your server and remove them. To do so you can try the following command,

# awk -F: '($2 == "") {print}' /etc/shadow

Locate zeron ‘0’ uid owners:

Only root account have UID 0 with full permissions to access the system.It is better to find out some one already have this. Type the following command to display all:

# awk -F: '($3 == "0") {print}' /etc/passwd	

Remove X window Group Ware System Software :

It is better to remove all X-window software from your Nodes. Normally no one install and use desk tops in Node. Desktop applications will have a lot of bugs and security holes. So please remove it as follows,

# yum groupremove "X Window System"

Find all SUID & SGID Binaries :

Find out all SUID and SGID binaries . And remove all the useless binaries or change the permissions of these binaries. You can run the following command and check it,

# find /{bin,usr} ( -perm -4000 -o -perm -2000 ) -print
/bin/ping
/bin/cgexec
/bin/ping6
/bin/mount
/bin/su
/bin/umount
/usr/libexec/openssh/ssh-keysign
/usr/libexec/pt_chown
/usr/libexec/polkit-1/polkit-agent-helper-1
/usr/libexec/abrt-action-install-debuginfo-to-abrt-cache
/usr/libexec/utempter/utempter
/usr/bin/chsh
/usr/bin/sudo
/usr/bin/chage
/usr/bin/gpasswd
/usr/bin/ssh-agent
/usr/bin/pkexec
/usr/bin/wall
/usr/bin/passwd
/usr/bin/crontab
/usr/bin/solusvmconsolevz
/usr/bin/solusvmconsolexen
/usr/bin/locate
/usr/bin/chfn
/usr/bin/write
/usr/bin/newgrp
/usr/bin/at
/usr/bin/staprun
/usr/local/solusvm/core/solusvmc-tc
/usr/local/solusvm/core/solusvmc-xen
/usr/local/solusvm/core/solusvmc-h
/usr/local/solusvm/core/solusvmc-vz
/usr/local/solusvm/core/solusvmc-node
/usr/sbin/postqueue
/usr/sbin/userhelper
/usr/sbin/postdrop
/usr/sbin/usernetctl
/usr/sbin/suexec
#

Find world writable Files:

It is important to find all world writable files and remove it or change the permissions . You can run the following command and check it,

# find /{bin,usr,lib,scripts,var,home} -xdev -type d ( -perm -0002 -a ! -perm -1000 ) -print

Find No-owner Files:

Files not owned by any user or group can pose a security problem. It is better to remove those those files or change the ownerships . You can find it as follows,

# find  /{bin,usr,lib,scripts,var,home} ( -nouser -o -nogroup ) -print

Install and configure CLAMD:

Clamd is an antivirus software. It will help to scan your node and find out all windows virus files uploaded from your clients desktops.

Conclusion:

This document only provide a minimal and standard set of procedures that your can do it your self for securing and optimizing a vps nodes. It is always better to consult with and experienced admin too.