In this article by Karan Singh, author of the book, Learning Ceph, we will cover the following topics:

• Creating a sandbox environment with VirtualBox
• From zero to Ceph – deploying your first Ceph cluster
• Scaling up your Ceph cluster – monitor and OSD addition

(For more resources related to this topic, see here.)

Creating a sandbox environment with VirtualBox

We can test deploy Ceph in a sandbox environment using Oracle VirtualBox virtual machines. This virtual setup can help us discover and perform experiments with Ceph storage clusters as if we are working in a real environment. Since Ceph is an open source software-defined storage deployed on top of commodity hardware in a production environment, we can imitate a fully functioning Ceph environment on virtual machines, instead of real-commodity hardware, for our testing purposes.

Oracle VirtualBox is a free software available at http://www.virtualbox.org for Windows, Mac OS X, and Linux. We must fulfil system requirements for the VirtualBox software so that it can function properly during our testing. We assume that your host operating system is a Unix variant; for Microsoft windows, host machines use an absolute path to run the VBoxManage command, which is by default c:Program FilesOracleVirtualBoxVBoxManage.exe.

The system requirement for VirtualBox depends upon the number and configuration of virtual machines running on top of it. Your VirtualBox host should require an x86-type processor (Intel or AMD), a few gigabytes of memory (to run three Ceph virtual machines), and a couple of gigabytes of hard drive space. To begin with, we must download VirtualBox from http://www.virtualbox.org/ and then follow the installation procedure once this has been downloaded. We will also need to download the CentOS 6.4 Server ISO image from http://vault.centos.org/6.4/isos/.

To set up our sandbox environment, we will create a minimum of three virtual machines; you can create even more machines for your Ceph cluster based on the hardware configuration of your host machine. We will first create a single VM and install OS on it; after this, we will clone this VM twice. This will save us a lot of time and increase our productivity. Let’s begin by performing the following steps to create the first virtual machine:

The VirtualBox host machine used throughout in this demonstration is a Mac OS X which is a UNIX-type host. If you are performing these steps on a non-UNIX machine that is, on Windows-based host then keep in mind that virtualbox hostonly adapter name will be something like VirtualBox Host-Only Ethernet Adapter #<adapter number>. Please run these commands with the correct adapter names. On windows-based hosts, you can check VirtualBox networking options in Oracle VM VirtualBox Manager by navigating to File | VirtualBox Settings | Network | Host-only Networks.

1. After the installation of the VirtualBox software, a network adapter is created that you can use, or you can create a new adapter with a custom IP:For UNIX-based VirtualBox hosts

   # VBoxManage hostonlyif remove vboxnet1
   # VBoxManage hostonlyif create
   # VBoxManage hostonlyif ipconfig vboxnet1 --ip 192.168.57.1 --netmask 255.255.255.0
   

   For Windows-based VirtualBox hosts

   # VBoxManage.exe hostonlyif remove "VirtualBox Host-Only Ethernet Adapter"
   # VBoxManage.exe hostonlyif create
   # VBoxManage hostonlyif ipconfig "VirtualBox Host-Only 
   Ethernet Adapter" --ip 192.168.57.1 --netmask 255.255.255.

2. VirtualBox comes with a GUI manager. If your host is running Linux OS, it should have the X-desktop environment (Gnome or KDE) installed. Open Oracle VM VirtualBox Manager and create a new virtual machine with the following specifications using GUI-based New Virtual Machine Wizard, or use the CLI commands mentioned at the end of every step:
   • 1 CPU
   • 1024 MB memory
   • 10 GB X 4 hard disks (one drive for OS and three drives for Ceph OSD)
   • 2 network adapters
   • CentOS 6.4 ISO attached to VM

   The following is the step-by-step process to create virtual machines using CLI commands:

   1. Create your first virtual machine:

      # VBoxManage createvm --name ceph-node1 --ostype RedHat_64 --register
      # VBoxManage modifyvm ceph-node1 --memory 1024 
      --nic1 nat --nic2 hostonly --hostonlyadapter2 vboxnet1

      For Windows VirtualBox hosts:

      # VBoxManage.exe modifyvm ceph-node1 --memory 1024 --nic1 nat 
      --nic2 hostonly --hostonlyadapter2 "VirtualBox Host-Only Ethernet Adapter"

   2. Create CD-Drive and attach CentOS ISO image to first virtual machine:

      # VBoxManage storagectl ceph-node1 --name "IDE Controller" 
      --add ide --controller PIIX4 --hostiocache on --bootable on
      # VBoxManage storageattach ceph-node1 --storagectl "IDE Controller" 
      --type dvddrive --port 0 --device 0 --medium CentOS-6.4-x86_64-bin-DVD1.iso

      Make sure you execute the preceding command from the same directory where you have saved CentOS ISO image or you can specify the location where you saved it.

   3. Create SATA interface, OS hard drive and attach them to VM; make sure the VirtualBox host has enough free space for creating vm disks. If not, select the host drive which have free space:

      # VBoxManage storagectl ceph-node1 --name "SATA Controller" 
      --add sata --controller IntelAHCI --hostiocache on --bootable on
      # VBoxManage createhd --filename OS-ceph-node1.vdi --size 10240
      # VBoxManage storageattach ceph-node1 --storagectl 
      "SATA Controller" --port 0 --device 0 --type hdd --medium OS-ceph-node1.vdi

   4. Create SATA interface, first ceph disk and attach them to VM:

      # VBoxManage createhd --filename ceph-node1-osd1.vdi --size 10240
      # VBoxManage storageattach ceph-node1 --storagectl "SATA Controller" 
      --port 1 --device 0 --type hdd --medium ceph-node1-osd1.vdi

   5. Create SATA interface, second ceph disk and attach them to VM:

      # VBoxManage createhd --filename ceph-node1-osd2.vdi --size 10240
      # VBoxManage storageattach ceph-node1 --storagectl "SATA Controller" 
      --port 2 --device 0 --type hdd --medium ceph-node1-osd2.vdi

   6. Create SATA interface, third ceph disk and attach them to VM:

      # VBoxManage createhd --filename ceph-node1-osd3.vdi --size 10240
      # VBoxManage storageattach ceph-node1 --storagectl "SATA Controller" 
      --port 3 --device 0 --type hdd --medium ceph-node1-osd3.vdi

3. Now, at this point, we are ready to power on our ceph-node1 VM. You can do this by selecting the ceph-node1 VM from Oracle VM VirtualBox Manager, and then clicking on the Start button, or you can run the following command:

   # VBoxManage startvm ceph-node1 --type gui

4. As soon as you start your VM, it should boot from the ISO image. After this, you should install CentOS on VM. If you are not already familiar with Linux OS installation, you can follow the documentation at https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/index.html.
5. Once you have successfully installed the operating system, edit the network configuration of the machine:
   • Edit /etc/sysconfig/network and change the hostname parameter HOSTNAME=ceph-node1
   • Edit the /etc/sysconfig/network-scripts/ifcfg-eth0 file and add:

     ONBOOT=yes
     BOOTPROTO=dhcp

   • Edit the /etc/sysconfig/network-scripts/ifcfg-eth1 file and add:<

     ONBOOT=yes
     BOOTPROTO=static
     IPADDR=192.168.57.101
     NETMASK=255.255.255.0

   • Edit the /etc/hosts file and add:

     192.168.57.101 ceph-node1
     192.168.57.102 ceph-node2
     192.168.57.103 ceph-node3

6. Once the network settings have been configured, restart VM and log in via SSH from your host machine. Also, test the Internet connectivity on this machine, which is required to download Ceph packages:

   # ssh [email protected]

7. Once the network setup has been configured correctly, you should shut down your first VM so that we can make two clones of your first VM. If you do not shut down your first VM, the cloning operation might fail.
   1. Create clone of ceph-node1 as ceph-node2:

      # VBoxManage clonevm --name ceph-node2 ceph-node1 --register
      

   2. Create clone of ceph-node1 as ceph-node3:

      # VBoxManage clonevm --name ceph-node3 ceph-node1 --register

8. After the cloning operation is complete, you can start all three VMs:

   # VBoxManage startvm ceph-node1
   # VBoxManage startvm ceph-node2
   # VBoxManage startvm ceph-node3

9. Set up VM ceph-node2 with the correct hostname and network configuration:
   • Edit /etc/sysconfig/network and change the hostname parameter:

     HOSTNAME=ceph-node2

   • Edit the /etc/sysconfig/network-scripts/ifcfg-<first interface name> file and add:

     DEVICE=<correct device name of your first network interface, check ifconfig -a>
     ONBOOT=yes
     BOOTPROTO=dhcp
     HWADDR= <correct MAC address of your first network interface, check ifconfig -a >

   • Edit the /etc/sysconfig/network-scripts/ifcfg-<second interface name> file and add:

     DEVICE=<correct device name of your second network interface, check ifconfig -a>
     ONBOOT=yes
     BOOTPROTO=static
     IPADDR=192.168.57.102
     NETMASK=255.255.255.0
     HWADDR= <correct MAC address of your second network interface, check ifconfig -a >

   • Edit the /etc/hosts file and add:

     192.168.57.101 ceph-node1
     192.168.57.102 ceph-node2
     192.168.57.103 ceph-node3

   After performing these changes, you should restart your virtual machine to bring the new hostname into effect. The restart will also update your network configurations.

10. Set up VM ceph-node3 with the correct hostname and network configuration:
    • Edit /etc/sysconfig/network and change the hostname parameter:HOSTNAME=ceph-node3
    • Edit the /etc/sysconfig/network-scripts/ifcfg-<first interface name> file and add:

      DEVICE=<correct device name of your first network interface, check ifconfig -a>
      ONBOOT=yes
      BOOTPROTO=dhcp
      HWADDR= <correct MAC address of your first network interface, check ifconfig -a >

    • Edit the /etc/sysconfig/network-scripts/ifcfg-<second interface name> file and add:

      DEVICE=<correct device name of your second network interface, check ifconfig -a>
      ONBOOT=yes
      BOOTPROTO=static
      IPADDR=192.168.57.103
      NETMASK=255.255.255.0
      HWADDR= <correct MAC address of your second network interface, check ifconfig -a >

    • Edit the /etc/hosts file and add:

      192.168.57.101 ceph-node1
      192.168.57.102 ceph-node2
      192.168.57.103 ceph-node3

    After performing these changes, you should restart your virtual machine to bring a new hostname into effect; the restart will also update your network configurations.

At this point, we prepare three virtual machines and make sure each VM communicates with each other. They should also have access to the Internet to install Ceph packages.

From zero to Ceph – deploying your first Ceph cluster

To deploy our first Ceph cluster, we will use the ceph-deploy tool to install and configure Ceph on all three virtual machines. The ceph-deploy tool is a part of the Ceph software-defined storage, which is used for easier deployment and management of your Ceph storage cluster.

Since we created three virtual machines that run CentOS 6.4 and have connectivity with the Internet as well as private network connections, we will configure these machines as Ceph storage clusters as mentioned in the following diagram:

1. Configure ceph-node1 for an SSH passwordless login to other nodes. Execute the following commands from ceph-node1:
   • While configuring SSH, leave the paraphrase empty and proceed with the default settings:

     # ssh-keygen

   • Copy the SSH key IDs to ceph-node2 and ceph-node3 by providing their root passwords. After this, you should be able to log in on these nodes without a password:

     # ssh-copy-id ceph-node2

2. Installing and configuring EPEL on all Ceph nodes:
   1. Install EPEL which is the repository for installing extra packages for your Linux system by executing the following command on all Ceph nodes:

      # rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

   2. Make sure the baserul parameter is enabled under the /etc/yum.repos.d/epel.repo file. The baseurl parameter defines the URL for extra Linux packages. Also make sure the mirrorlist parameter must be disabled (commented) under this file. Problems been observed during installation if the mirrorlist parameter is enabled under epel.repo file. Perform this step on all the three nodes.
3. Install ceph-deploy on the ceph-node1 machine by executing the following command from ceph-node1:

   # yum install ceph-deploy

4. Next, we will create a Ceph cluster using ceph-deploy by executing the following command from ceph-node1:

   # ceph-deploy new ceph-node1
   ## Create a directory for ceph
   # mkdir /etc/ceph
   # cd /etc/ceph

   The new subcommand of ceph-deploy deploys a new cluster with ceph as the cluster name, which is by default; it generates a cluster configuration and keying files. List the present working directory; you will find the ceph.conf and ceph.mon.keyring files.

   In this testing, we will intentionally install the Emperor release (v0.72) of Ceph software, which is not the latest release. Later in this book, we will demonstrate the upgradation of Emperor to Firefly release of Ceph.

5. To install Ceph software binaries on all the machines using ceph-deploy; execute the following command from ceph-node1:ceph-deploy install –release emperor ceph-node1 ceph-node2 ceph-node3
   The ceph-deploy tool will first install all the dependencies followed by the Ceph Emperor binaries. Once the command completes successfully, check the Ceph version and Ceph health on all the nodes, as follows:

   # ceph –v

6. Create your first monitor on ceph-node1:

   # ceph-deploy mon create-initial

   Once monitor creation is successful, check your cluster status. Your cluster will not be healthy at this stage:

   # ceph status

7. Create an object storage device (OSD) on the ceph-node1 machine, and add it to the Ceph cluster executing the following steps:
   1. List the disks on VM:

      # ceph-deploy disk list ceph-node1

      From the output, carefully identify the disks (other than OS-partition disks) on which we should create Ceph OSD. In our case, the disk names will ideally be sdb, sdc, and sdd.

   2. The disk zap subcommand will destroy the existing partition table and content from the disk. Before running the following command, make sure you use the correct disk device name.

      # ceph-deploy disk zap ceph-node1:sdb ceph-node1:sdc ceph-node1:sdd

   3. The osd create subcommand will first prepare the disk, that is, erase the disk with a filesystem, which is xfs by default. Then, it will activate the disk’s first partition as data partition and second partition as journal:

      # ceph-deploy osd create ceph-node1:sdb ceph-node1:sdc ceph-node1:sdd

   4. Check the cluster status for new OSD entries:

      # ceph status

      At this stage, your cluster will not be healthy. We need to add a few more nodes to the Ceph cluster so that it can set up a distributed, replicated object storage, and hence become healthy.

Scaling up your Ceph cluster – monitor and OSD addition

Now we have a single-node Ceph cluster. We should scale it up to make it a distributed, reliable storage cluster. To scale up a cluster, we should add more monitor nodes and OSD. As per our plan, we will now configure ceph-node2 and ceph-node3 machines as monitor as well as OSD nodes.

Adding the Ceph monitor

A Ceph storage cluster requires at least one monitor to run. For high availability, a Ceph storage cluster relies on an odd number of monitors that’s more than one, for example, 3 or 5, to form a quorum. It uses the Paxos algorithm to maintain quorum majority. Since we already have one monitor running on ceph-node1, let’s create two more monitors for our Ceph cluster:

1. The firewall rules should not block communication between Ceph monitor nodes. If they do, you need to adjust the firewall rules in order to let monitors form a quorum. Since this is our test setup, let’s disable firewall on all three nodes. We will run these commands from the ceph-node1 machine, unless otherwise specified:

   # service iptables stop
   # chkconfig iptables off
   # ssh ceph-node2 service iptables stop
   # ssh ceph-node2 chkconfig iptables off
   # ssh ceph-node3 service iptables stop
   # ssh ceph-node3 chkconfig iptables off

2. Deploy a monitor on ceph-node2 and ceph-node3:

   # ceph-deploy mon create ceph-node2
   # ceph-deploy mon create ceph-node3

3. The deploy operation should be successful; you can then check your newly added monitors in the Ceph status:
4. You might encounter warning messages related to clock skew on new monitor nodes. To resolve this, we need to set up Network Time Protocol (NTP) on new monitor nodes:

   # chkconfig ntpd on
   # ssh ceph-node2 chkconfig ntpd on
   # ssh ceph-node3 chkconfig ntpd on
   # ntpdate pool.ntp.org
   # ssh ceph-node2 ntpdate pool.ntp.org
   # ssh ceph-node3 ntpdate pool.ntp.org
   # /etc/init.d/ntpd start
   # ssh ceph-node2 /etc/init.d/ntpd start
   # ssh ceph-node3 /etc/init.d/ntpd start

Adding the Ceph OSD

At this point, we have a running Ceph cluster with three monitors OSDs. Now we will scale our cluster and add more OSDs. To accomplish this, we will run the following commands from the ceph-node1 machine, unless otherwise specified.

We will follow the same method for OSD addition:

# ceph-deploy disk list ceph-node2 ceph-node3
# ceph-deploy disk zap ceph-node2:sdb ceph-node2:sdc ceph-node2:sdd
# ceph-deploy disk zap ceph-node3:sdb ceph-node3:sdc ceph-node3:sdd
# ceph-deploy osd create ceph-node2:sdb ceph-node2:sdc ceph-node2:sdd
# ceph-deploy osd create ceph-node3:sdb ceph-node3:sdc ceph-node3:sdd
# ceph status

Check the cluster status for a new OSD. At this stage, your cluster will be healthy with nine OSDs in and up:

Summary

The software-defined nature of Ceph provides a great deal of flexibility to its adopters. Unlike other proprietary storage systems, which are hardware dependent, Ceph can be easily deployed and tested on almost any computer system available today. Moreover, if getting physical machines is a challenge, you can use virtual machines to install Ceph, as mentioned in this article, but keep in mind that such a setup should only be used for testing purposes.

In this article, we learned how to create a set of virtual machines using the VirtualBox software, followed by Ceph deployment as a three-node cluster using the ceph-deploy tool. We also added a couple of OSDs and monitor machines to our cluster in order to demonstrate its dynamic scalability. We recommend you deploy a Ceph cluster of your own using the instructions mentioned in this article.

Resources for Article:

Further resources on this subject:

1. • Linux Shell Scripting – various recipes to help you [article]
   • GNU Octave: Data Analysis Examples [article]
   • What is Kali Linux [article]