MakeItZone FreeNas Setup

FreeNAS

  • basic FreeNAS server
  • mirrored ZFS across two mirrors of two drives
  • after reboot:
    • must log into GUI and unlock storage pool
    • must restart Duplicacy (backup) jail

Datasets

  • Backups
    • for stand-alone windows machine OS backups
  • Creations
    • for member/student content
  • Infrastructure
    • software installers, etc
    • only for admins
  • Public
    • in-secure, quick access, public (for intranet) file share
  • VMWare
    • network file store for ESXi and other VMWare software
  • iocages
    • for local jails & VMs

Most datasets have a top level structure of LocalOnly and BackedUp. The later is backed up online (best effort, no guarantees.)

Setup

Hardware

  • counter to iXSystems and community recomendations/suggestions, installed to (small) internal SSD. Found external USB drives became unreliable after 6-12 months. Possible cause- reporting being written to system pool (see below.)

Users & Groups

  • an internal user for infrastructure use
  • long term members
  • user group for users
  • shared group for shares
  • add users to user as supplementary group
  • add users that need access to share to shared
  • NOTE - there maybe some more fine-grained setup of groups to separate public and user access

System Pool

  • can only be on unencrypted, or non-passphrase protected pools that are available at boot
  • will block setting/resetting storage key passphrase
  • must be set to use the boot pool if you want encrypted drives, or add another drive for this pool
  • DISABLE reporting database! It creates a lot of writes on the boot drive (possibly why USB flash drivers failed.) Disabling this buffers writes in memory and does an hourly dump. Turn it on if the system is crashing/needs to be analyzed.

ACLs

FreeNAS is designed for sharing to different client OSs. As such, it has several sub-systems for controlling file access: Unix (simple oga-rwx access control), and ACLs for Windows (SMB), AFS.

When you setup a share, it will (silently) change the access control type on the datastore/directory.

Unfortunately ACLs aren’t easy to use/understand from the command line(!) The FreeNAS videos recommend using a windows PC(!). NOTE: FreeNAS 11.3 has a GUI for managing ACLS.

When ACLs are enabled, chmod will (often/usually?) fail. Even if the Unix permissions look right, the ACLs may block what you want to do. You can see if ACLs are present by ls -l: you’ll see a + at the end of the permissions.

You can revert a datastore’s permission types back to Unix BUT changing the share will reset it. Best to leave it as windows and get used to ACLs…

However, ACLs can make it very hard to mount a path (that’s also shared) into a jail.

Easiest solution found is to remove the ACLs on the specific directory that needs to be (read-only) mounted in the jail:

setfacl -b <directory to remove ACLs>
find <directory to remove ACLs> | setfacl -b # as setfacl doesn't have a recursive option.

chmod o+rx <directory to remove ACLs>
chmod -R o+r <directory to remove ACLs>
find <directory to remove ACLs>/ -type d -exec chmod o+x {} \;

References:

Snapshots

Thanks to ZFS being a COW (copy on write) system, it takes minimal space, and is extremely fast, to record the state of all files as a read-only record, with updates only recording the changes.

Furthermore, periodic tasks can be set up to regularly take these snapshots and make them available- sort of file level ‘undo’, somewhat similar to Apple’s time machine.

Periodic snapshots have been enabled (see Tasks section for details).

Make them accessible to windows versioning (file->properties->versions) by setting the “snap shot task” in the share definition.

Outline info re accessing the snapshots is in the official docs.

This forum post has the real details for implementing it- to each share add:

zfsacl:expose_snapdir=True
veto files = /.windows/.mac/

to the auxiliary parameters of each share.

Remove the following if you need to change a volume/share permission type - you’ll slow things down immensely as setfacl will be called on, and fail, for every item in the snapshots!

Then from the FreeNAS shell, cd to each dataset (and share) and:

ln -s .zfs/snapshot snapshot

This will make it easier for all clients (OS X, Windows, Linux) to find the snapshots.

As well, a snapshot was set on backups after the first full-system backups of the non-vm windows instances.

Shares

  • CIFS (Samba) used for majority of shares
  • NFS is available only to ESXi server, for VMWare dataset via dedicated link
    • don’t forget to set the IP of devices connected to this network to be on the storage network!
  • Most windows machines have a P:\ drive mapped to the public share.
  • users can manually, short term, map/access member shares
  • see #snapshots above for adding snapshot access to shares
  • add fruit to vfs_options of all shares that could be simultaneously accessed by OS X clients (OS X only initializes AAPL SMB extensions the first time connecting to a server.)

EMail

TODO - need to setup a means for alerts to be emailed

UPS

TODO

Online Backup

  • jail set up to run duplicacy
  • backed up content (listed above) encrypted before being sent to Backblaze

Backup Jail

Create a duplicacy jail.
Mount the backup directories into the Jail read-only, eg to /mnt/backup/BackedUp.

Install Duplicacy In Jail

pkg install wget
pkg install security/ca_root_nss  #to install certificates for wget
pkg install bash

wget <url of duplicacy executable>

Move duplicacy to /usr/local/bin.

Initialize Duplicacy

cd /mnt/backup/
duplicacy init <snapshot id> b2://<bucket id> -e

Store Settings

Save the required credentials.

duplicacy set -storage b2://<bucket id> -key b2_id -value <b2 account>
duplicacy set -storage b2://<bucket id> -key b2_key -value <b2 app key>
duplicacy set -storage b2://<bucket id> -key password -value <encryption pass phrase>

References:

Create Filters

Exclude .zfs and snapshot directories- add .duplicacy\filters:

e:\.zfs/.*$
e:snapshot/.*$

Test filters (doesn’t send to/touch repository) with duplicacy -d -log backup -enum-only.

Runner Script

Create a script to run duplicacy with the correct parameters etc, and make sure only a single copy is running:

/root/backup.sh:

#!/usr/local/bin/bash

# https://stackoverflow.com/a/185473/1388019
lockfile="/tmp/duplicacy.lock"

if [ -e ${lockfile} ] && kill -0 `cat ${lockfile}`; then
    echo "duplicacy already running"
    exit
fi

# make sure the lockfile is removed when we exit and then claim it
trap "rm -f ${lockfile}; exit" INT TERM EXIT
echo $$ > ${lockfile}

# run the backup with default settings
cd /mnt/backups
/usr/local/bin/duplicacy -log backup -threads 4

# Purge old snapshots after backup
# Keep none after a year, and one for every 7 days after 60 days
/usr/local/bin/duplicacy -log prune -threads 4 -all -keep 0:365 -keep 7:60

# clean up lockfile
rm -f ${lockfile}

(based on this article.)

Manually run backup.sh and check it works ok.

Virus Checking (ClamAV)

Added via plugins. Will appear as a jail.

Add read only mounts to the datasets that have regular files.

Add the avscan.sh script from here.

Comment out deletion of log files - until email is working.

Check the log files in the jail.

Tasks

S.M.A.R.T. Tests

Add a daily short SMART test and a weekly long.

Periodic Snapshots

Frequency Retention
every 15 minutes 2 hours
hourly 1 day
daily 1 week

Cron Jobs

Try to balance times for when things are run!

To run commands in jails:

/bin/sh -c "/usr/local/bin/iocage exec <jail name> <command including path in jail>`

Duplicacy Cloud Backup

Daily. Command:

/bin/sh -c "/usr/local/bin/iocage exec -f duplicacy /backups/backup.sh"

ClamAV Scan

Daily. Command:

/bin/sh -c "/usr/local/bin/iocage exec -f clamav /root/avscan.sh"

Other Notes

  • For FreeNAS 11.3 (maybe before?) need to add -f to iocage exec to force the jail to start
  • if you can’t add a passphrase to an encrypted pool, check system -> system dataset. Make sure it’s set to use a different pool!
  • Apple is deprecating AFP; they’re moving to SMB
  • FreeNas SMB can now support Apples extensions, including TimeMachine. Instructions.
  • RealTek NICs are really not supported in FreeBSD. A short-term work-around seems to be to turn off as many features as possible. I added to the list: -rxcsum -txcsum -rxcsum6 -txcsum6 -tso -lro -vlanmtu -vlanhwtag -vlanhwfilter -vlanhwtso. Only run with this until your new Intel NIC arrives! Note, does seem to be stable (survived a couple of machines doing backups) but speed is reduced to ~3-4% (ie ~3-4MB/sec transfer speeds of gigabit ethernets approximate 125MB/s (ref)
  • data vol share type determines how permissions are managed - both from terminal (file system) and via shares. Easiest is to stick to Unix file permissions. If you set it to Windows it uses ACLs. See this for more info.

How FreeNAS and GELI Encryption Are Configured

Best explanation is here:

More details on how this works:
Every geli provider (“disk drive”) has its own AES master key that can never be changed – you would have to rewrite (reencrypt) all data on the drive to change it. The master key is stored in the geli metadata on the drive itself. This key is, of course, encrypted. There are two “slots” for the encrypted master key in the geli metadata, which allows you to encrypt the master key with two user keys (0 and 1). Any of the user keys (if set) can be used to decrypt the master key and thus access the data. You can change the user keys as often as you want provided that you have at least one of them – you decrypt the master key with the old user key and encrypt it with the new one. An user key can be created from two components – a passphrase and/or a key file – you can use one, the other, or both.

FreeNAS uses the user keys as follows:

  • user key 0 (the “main” key) always has the key file component which is stored in /data/geli. It optionally can also have the passphrase component. If the passphrase is set then both the key file and the passphrase are needed to unlock the pool (decrypt the master key).
  • user key 1 (the “recovery” key) is optional and can only have the key file component (the middleware actually supports a passphrase here as well, but the GUI doesn’t use that functionality).

This is how all this maps to FreeNAS operations and GUI buttons (when I say “sets a key” it means that the master key is encrypted with the key and stored in the respective user key slot in the geli metadata):

  • When you create an encrypted pool the master keys are initialized on all drives, a random key file is generated, stored in /data/geli and set as the key file component of user key 0. There is no passphrase component and the pool will automount after reboot.
  • The Create passsphrase button adds the passphrase component to user key 0. The pool will no longer automount after reboot as the key file stored in /data/geli is not enough any more (it is now just one component of user key 0). You now need to provide the passphrase via the Unlock button.
  • The Change passphrase button changes the passphrase component of user key 0.
    The Download key button gives you the key file component of user key 0 (it just lets you download the file from /data/geli).
  • The Encryption Re-key button generates a new key file, stores it in /data/geli and sets it as the key file component of user key 0.
  • The Add recovery key button generates a new key file, sets it as the user key 1 (overwriting anything that was there before) and allows you to download it. This key file is not kept stored anywhere in FreeNAS.
  • The Remove recovery key button removes the user key 1 from the geli metadata.

Note- the UI is sometimes confusing, calling key 0 the encryption key, but then calling it the recovery key in the download dialog.

Manual key management can be done with:

zpool status # to get list of all active, encrypted drives - drop the .eli to get the underlying encrypted partition

cd /data/geli

geli setkey -n 0 -K <encrytionkey.key> /dev/gptid/<each of the partition ids found above>

More details, including hints for adding/removing drives to an encrypted pool here