HowGreyholeWorks

How Greyhole Works

Basics

• The user defines it's storage pool by listing all the paths of the different disks he wants to use.
• When files are added on Greyhole shares, the Greyhole daemon moves those files into 1+ of the paths defined as the storage pool. It then replaces the original file (on the share) by a symbolic link pointing to one of the copy created in the pool.
• Samba is configured to use opaque symbolic links, i.e. what are in fact symbolic links in the shares appear as normal files to clients.

In Details

• Samba logs all writes, renames & deletes happening in Greyhole shares, using the Greyhole VFS module. /var/spool/greyhole is where the logged operations are kept: the VFS module creates a new file in that folder for each logged operation.
• The Greyhole daemon consumes those logs, and inserts tasks in a custom database (MySQL).
• The Greyhole daemon acts on each tasks in sequential order:
  • For writes, it checks the (user-specified) number of copies to keep, and will create as many copies of the file on the different disk included in the storage pool. Once it created the first copy, it will replace the original file by a symbolic link pointing to the new copy created.
  • For renames & deletes, it will replicate the operation on all directories part of the storage pool.

Example

What's better than a good example to understand a system!

• /mnt/hdd0: 1TB empty hard-drive
• /mnt/hdd1: 2TB empty hard-drive

smb.conf:

[Music]
	path = /shares/Music
	dfree command = /usr/bin/greyhole-dfree
	vfs objects = greyhole

[RecordedTV]
	path = /shares/RecordedTV
	dfree command = /usr/bin/greyhole-dfree
	vfs objects = greyhole

greyhole.conf:

# Storage pool directories
storage_pool_directory = /mnt/hdd0/gh, min_free: 50gb
storage_pool_directory = /mnt/hdd1/gh, min_free: 10gb

# Shares
num_copies[Backups] = 2
num_copies[RecordedTV] = 1

Example:

• Client:

mount_smb //server/Music /mnt/Music
echo 1 > /mnt/Music/file1

• Server (Greyhole daemon):

rsync /shares/Music/file1 /mnt/hdd1/gh/Music/file1
rm /shares/Music/file1
ln -s /mnt/hdd1/gh/Music/file1 /shares/Music/file1
rsync /mnt/hdd1/gh/Music/file1 /mnt/hdd0/gh/Music/file1
# Writes metadata about the above in /mnt/hdd[0-1]/gh/.gh_metastore/Music/file1

• Client:

mv /mnt/Music/file1 /mnt/Music/file2

• Server (Greyhole daemon):

mv /mnt/hdd1/gh/Music/file1 /mnt/hdd1/gh/Music/file2
rm /shares/Music/file2
ln -s /mnt/hdd1/gh/Music/file2 /shares/Music/file2
mv /mnt/hdd0/gh/Music/file1 /mnt/hdd0/gh/Music/file2
rm /mnt/hdd[0-1]/gh/.gh_metastore/Music/file1
# Writes metadata about the above in /mnt/hdd[0-1]/gh/.gh_metastore/Music/file2

• Client:

rm /mnt/Music/file2

• Server (Greyhole daemon):

rm /mnt/hdd1/gh/Music/file2
rm /mnt/hdd0/gh/Music/file2
rm /mnt/hdd[0-1]/gh/.gh_metastore/Music/file2