This article details all operations needed to setup a NAS based on FreeBSD 9.1 and ZFS in a jailed environnement.
This is a updated version of this article, with a separate SSD for the OS (for lower noise and power consumption).
System setup
FreeBSD 9.1 is a release of choice for this kind of installation, it brings support of TRIM and SHA-512 hash for passwords. ECC based protocols are included in OpenSSH since 9.0.
Before the setup, start the installation media in single user mode, then :
- Check if TRIM is listed on the tunables of the SSD drive :
/sbin/tunefs -p /dev/<drive_id> - If required, enable TRIM :
tunefs -t enable /dev/<drive_id>
The system can be installed including src and ports.
If required, new user can be assiged to wheel (for su access) and operator (for shutdown / restart) groups.
At first boot, perform the classic configuration operations :
# echo 'ifconfig_<interface_name>="inet <host_IP> netmask <host_mask>"' >>
/etc/rc.conf
# echo 'ifconfig_<interface_name>="defaultrouter="<router_IP>""' >>
/etc/rc.conf
# echo 'nameserver <DNS_IP>' >> /etc/resolv.conf
For SSH, check if root login is disabled and restrict connections to host IP only to avoid conflicts with jails. In /etc/ssh/sshd.conf :
PermitRootLogin no ListenAddress <host_IP>
Enable password for single user mode in /etc/ttys :
console none unknown off insecure
Enable SHA-512 for system passwords (/etc/login.conf) :
default:\
:passwd_format=sha512:\
# cap_mkdb /etc/login.conf
# passwd
# passwd <existing_users>
Then, configure the build environment :
# cp /usr/share/examples/etc/make.conf /etc/
And adapt /etc/make.conf :
CPUTYPE?=native #'?=' allows to buildworld for a different CPUTYPE MAKE_JOBS_NUMBER=<number_of_CPU_cores>
Install subversion :
# portsnap fetch extract && cd /usr/ports/devel/subversion
# make install clean
Then configure it and update the local source [1] :
# svn co https://svn0.eu.FreeBSD.org/base/releng/9.1 /usr/src
And rebuild the system :
# make buildworld
# make buildkernel
# make installworld
# make installkernel
# reboot
Configure tcsh (~/.login_conf) :
me:\
:charset=UTF-8:\
:lang=fr_FR.UTF-8:
This settings can be checked with the locale command.
Install some essentials :
# portsnap fetch extract
# cd /usr/ports/ports-mgt/portmaster && make install clean
# portmaster editors/vim-lite
# portmaster sysutils/ataidle
Enable power-savings options in /etc/rc.conf :
powerd_flags="-b min" powerd_enable="YES" ataidle_enable="YES" ataidle_devices="ada<id> ada<id> ada<id>" ataidle_ada<id>="-I 150" ataidle_ada<id>="-I 150" ataidle_ada<id>="-I 150"
Some kernel configuration is also necessary (/boot/loader.conf) :
- Force ZFS prefetch, which is disabled by default if the host have less than 4096MB of RAM.
- Enable AHCI for NCQ support.
- Enable thermal monitoring for AMD CPU (
$ sysctl dev.cpu.0.temperature).
loader_logo="beastiebw" vfs.zfs.prefetch_disable=0 ahci_load="YES" amdtemp_load="YES"
And finally, import the existing pool without mounting it (it will be mounted inside a jail) :
zpool import -N
At this point, the host system is configured and all other features will be running in separate jail environments (in my case, in /usr/jails).
Jails configuration
Create the jail for the NAS environment :
# cd /usr/src
# make buildworld #If not already done
# make installworld DESTDIR=/usr/jails/
# make distribution DESTDIR=/usr/jails/
# mount -t devfs devfs /usr/jails//dev
Next, configure this jail in /etc/rc.conf :
ifconfig_<interface_ID>_alias<jail_id>="inet <IP>/<mask_prefix>" jail_<jail_name>_hostname="<hostname>" jail_<jail_name>_rootdir="/usr/jails/<path>" jail_<jail_name>_devfs_enable="YES" jail_<jail_name>_ip="<IP>" jail_<jail_name>_exec_poststart0="/usr/jails/config/<configuration_script>.sh"
The scripts located in /usr/jails/config and declared with exec_poststart0 are launched at the jail startup, and could be used to set no-persistent configuration.
Finally, enable the jail :
jail_enable="YES" jail_list="<jail_name>"
Generic jail configuration
After the jail is started (jls), start a shell inside it :
# jexec <jail_id> tcsh
By default in jails, there is no password for the root user, therefore the first thing to do is to set one :
# passwd
Or disable the user by adding a ‘*’ character in the second field of /etc/passwd (using vipw).
Then, basic configuration for jails :
# tzsetup
# echo 'nameserver <DNS_IP>' >> /etc/resolv.conf
NAS specific configuration
First, enable SSH in /etc/rc.conf :
sshd_enable="YES"
In jails, SSH server must be binded to the IP alias of the jail, using the listenadress directive in /etc/ssh/sshd_config :
ListenAddress <jail_IP>
and I chose to disable all authentication methods except for certificates :
PasswordAuthentication no UsePAM no ChallengeResponseAuthentication no PermitRootLogin no
SSH will be used with sshfs for NAS files access.
Now, existing ZFS pool can be imported into the jail. First, make sure there is no zfs_enable="YES" directive in the /etc/rc.conf file of the host so the pool will not be mounted automatically at host level during the boot, but only when the NAS jail will start and explicitly mount the filesystems by calling the zfs command.
The ZFS datasets must be explicitly configured to be allow mouting in a jail with the jailed property (which is permanent) :
# zfs set jailed=on <pool_or_dataset>
Then, create one or more users with the same uids than the owner of the files in the pool (in wheel group if necessary).
The operations required to mount the pool in the jail are performed in this script, which must be called at jail startup using the jail_exec_poststart0 directive.
Once the ZFS filesystem are mounted inside the jail, the ZFS mountpoints will remain the same in zfs list, but on the host, those mountpoints will be mounted under /usr/jails/<jail_name> from the df view.
Jail for remote access
On my host, I also made a other dedicated jail for remote SSH access with port-knocking.
Once logged into this jail, the other hosts / jails are reachable through another SSH connection on the local network.
After the generic configuration of the jail, SSH was also configured to accept only certificates :
ListenAddress <jail_IP> PasswordAuthentication no UsePAM no ChallengeResponseAuthentication no PermitRootLogin no
Then SSH was configured to accept only connections from specific usernames and source IPs :
AllowUsers <username1>@<remote_IP1> <username2>@<remote_IP2>
sshd is only started on demand via a port-knocking mechanism, via knockd :
# /usr/ports/security/knock
# make config #only server part
# make install clean
This feature require a specific devfs ruleset to allow bpf* devices to be accessible from the jail. If the configuration file does not already exists, copy the default file :
# cp /etc/defaults/devfs.rules /etc/devfs.rules
Then, in /etc/devfs.rules, add a specific section :
# Knockd [devfsrules_unhide_knockd=5] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add path 'bpf*' unhide
This ruleset must be declared in /etc/rc.conf :
jail_<jail_name>_devfs_ruleset="devfsrules_unhide_knockd"
The knockd configuration is available here. It start the sshd, and modify the sshd_config file to accept connections from the knocking IP.
It must be enabled with :
echo 'knockd_enable="YES"' >> /etc/rc.conf
sshd is automatically shutdown every five minutes via crond in order to refuse new connections and hide the service :
*/5 * * * * root /etc/rc.d/sshd onestop &> /dev/null
But sshd keeps existing connections until they are intentionally closed, so this shutdown is transparent for existing sessions.
Updating
Source tree can be updated via svn update, and ports with portmasters -Da.
In order to recompile only the required parts and save time recompile existing kernel and world with the following options :
# make -DNO_KERNELCLEAN buildkernel
# make -DNO_CLEAN -j<NUMBER_OF_CORES> buildworld
Jails can be updated with the usual installworld target :
make installworld DESTDIR=/usr/jails/<jail>