Home
This documentation is for lrbd 1.9 and 2.1. Documentation for lrbd 1.0 is here.
You must have the following:
- a working Ceph cluster
- a host to act as a gateway (iSCSI target)
- a host to act as a client (iSCSI initiator)
- lrbd installed on the gateway
For Adding a Gateway and Adding a Target
- another host to act as a second or redundant gateway
The lrbd configuration is a global configuration for all gateways and clients. The gateway sections include the Ceph pools, RBD images, portals (i.e. network interfaces), static targets and authentication. Within these sections are specific access definitions and restrictions for initiators.
Imagine the physical layout below with a single client acting as an initiator connecting via a gateway to a Ceph cluster.
For a first configuration, find the simple.json file located in /usr/share/doc/packages/lrbd/samples. Copy this file to another location such as your home directory. The file contains four sections which are targets, auth, portals and pools. The purpose of the separate sections is to accommodate the myriad of configurations that iSCSI can support.
For your convenience, the contents of simple.json are displayed below.
{
"targets": [
{
"host": "igw1",
"target": "iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk"
}
],
"auth": [
{
"host": "igw1",
"authentication": "tpg",
"tpg": {
"userid": "common1",
"password": "pass1"
}
}
],
"portals": [
{
"name": "portal1",
"addresses": [ "172.16.1.16" ]
}
],
"pools": [
{
"pool": "rbd",
"gateways": [
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": "archive"
}
]
}
]
}
]
}
An experienced storage administrator may ask "Where's the Lun?". Essentially, the mapped RBD image is the LUN. Lrbd uses the Ceph terminology of pool and image, but uses the iSCSI terms target, initiator and portal. The following diagram shows the relationship.
Note that this configuration is one portal within one target on one gateway accessing one image within one pool. Before exploring more complex configurations, follow these instructions for customizing and applying the configuration.
If you wish to use the default pool and RBD image, verify the following:
- the rbd pool exists
# rados lspools - the archive image exists
# rbd -p rbd ls
Otherwise, create your own pool and RBD image. Update the pool and image in the pools section.
Update the host to match the hostname of the gateway.
"pools": [
{
"pool": "rbd",
"gateways": [
{
"host": igw1,
"tpg": [
{
"portal": "portal1",
"image": "archive"
}
]
}
]
}
] Under the targets section, update the host to match the hostname of the gateway. Replace the target with the output of iscsi-name.
"targets": [
{
"host": igw1,
"target": iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk,
}
]Replace the address with a valid network address of the gateway host.
"portals": [
{
"name": "portal1",
"addresses": [ 172.16.1.16 ]
}
]The authentication is set to tpg+identified. All initiators will use the same credentials to connect to this target. Replace the host with the hostname of the gateway. Set the desired userid and password for CHAP authentication.
"auth": [
{
"host": igw1,
"authentication": "tpg+identified",
"tpg": {
"userid": common1,
"password": pass1
}
}
]Add comments if you wish using a '#'. Any comments will be stripped when importing the configuration.
Import the configuration and then query the configuration from a gateway host.
# lrbd -f ~/simple.json
# lrbd -o
The output will be similar to the following:
{
"auth": [
{
"host": "igw1",
"authentication": "tpg+identified",
"tpg": {
"userid": "common1",
"password": "pass1"
}
}
],
"targets": [
{
"host": "igw1",
"target": "iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk"
}
],
"portals": [
{
"name": "portal1",
"addresses": [
"172.16.1.16"
]
}
],
"pools": [
{
"pool": "rbd",
"gateways": [
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
}
]
}
]
}
Another option to display the configuration is
# lrbd -l
In this case, both outputs will be the same. The -o displays all hosts while -l shows only entries relevant to this host. This is most useful when determining whether a configuration is correct for a gateway.
Inspect the current configuration.
# targetcli ls
The output will show nothing configured.
Loaded vhost_scsi kernel module.
Created '/sys/kernel/config/target/vhost'.
Loaded iscsi_target_mod kernel module.
Created '/sys/kernel/config/target/iscsi'.
Loaded tcm_loop kernel module.
Created '/sys/kernel/config/target/loopback'.
Loaded tcm_fc kernel module.
Created '/sys/kernel/config/target/fc'.
Loaded ib_srpt kernel module.
Created '/sys/kernel/config/target/srpt'.
Loaded tcm_qla2xxx kernel module.
Created '/sys/kernel/config/target/qla2xxx'.
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- fileio ............................................... [0 Storage Object]
| o- iblock ............................................... [0 Storage Object]
| o- pscsi ................................................ [0 Storage Object]
| o- rbd .................................................. [0 Storage Object]
| o- rd_mcp ............................................... [0 Storage Object]
o- ib_srpt ....................................................... [0 Targets]
o- iscsi ......................................................... [0 Targets]
o- loopback ...................................................... [0 Targets]
o- qla2xxx ....................................................... [0 Targets]
o- tcm_fc ........................................................ [0 Targets]
o- vhost ......................................................... [0 Targets]Verify that the hostname matches the configuration. Note the defaults to value for the -H option below.
# lrbd -h
usage: lrbd [-h] [-e] [-E editor] [-c name] [--ceph ceph] [-H host] [-o] [-l]
[-f file] [-a file] [-u] [-v] [-d] [-I] [-R] [-W] [-C]
optional arguments:
-h, --help show this help message and exit
-e, --edit edit the rbd configuration for iSCSI
-E editor, --editor editor
use editor to edit the rbd configuration for iSCSI
-c name, --config name
use name for object, defaults to "lrbd.conf"
--ceph ceph specify the ceph configuration file
-H host, --host host specify the hostname, defaults to "igw1"
-o, --output display the configuration
-l, --local display the host configuration
-f file, --file file import the configuration from file
-a file, --add file add the configuration from file
-u, --unmap unmap the rbd images
-v, --verbose print INFO messages
-d, --debug print DEBUG messages
-I, --iblock set the backstore to iblock, defaults to rbd
-R, --rbd set the backstore to rbd
-W, --wipe wipe the configuration objects from all pools
-C, --clear clear the targetcli configuration Run the lrbd command to apply the saved configuration.
# lrbd
If the hostname in the configuration does not match the hostname in the help message, run the following command to apply the configuration instead.
# lrbd -H configured_hostname
The lrbd command outputs each rbd and targetcli command with its output.
targetcli /backstores/rbd create name=archive dev=/dev/rbd/rbd/archive
Loaded vhost_scsi kernel module.
Created '/sys/kernel/config/target/vhost'.
Loaded iscsi_target_mod kernel module.
Created '/sys/kernel/config/target/iscsi'.
Loaded tcm_loop kernel module.
Created '/sys/kernel/config/target/loopback'.
Loaded tcm_fc kernel module.
Created '/sys/kernel/config/target/fc'.
Loaded ib_srpt kernel module.
Created '/sys/kernel/config/target/srpt'.
Loaded tcm_qla2xxx kernel module.
Created '/sys/kernel/config/target/qla2xxx'.
Created RBD storage object archive using /dev/rbd/rbd/archive.
targetcli /iscsi create iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk
Created target iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk.
Selected TPG Tag 1.
Created TPG 1.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk/tpg1/luns create /backstores/rbd/archive
Selected LUN 0.
Created LUN 0.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk/tpg1/portals create 172.16.1.16
Using default IP port 3260
Created network portal 172.16.1.16:3260.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk/tpg1/acls create iqn.1996-04.de.suse:01:e6ca28cc9f20
Created Node ACL for iqn.1996-04.de.suse:01:e6ca28cc9f20
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f20 create 0 0
Created Mapped LUN 0.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f20 set auth userid=common1 password=pass1
Parameter password is now 'pass1'.
Parameter userid is now 'common1'.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk/tpg1 set attribute authentication=1 demo_mode_write_protect=0 generate_node_acls=0
Parameter authentication is now '1'.
Parameter demo_mode_write_protect is now '0'.
Parameter generate_node_acls is now '0'.Inspect the current configuration again.
# targetcli ls
The output will look similar to the following:
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- fileio ............................................... [0 Storage Object]
| o- iblock ............................................... [0 Storage Object]
| o- pscsi ................................................ [0 Storage Object]
| o- rbd .................................................. [1 Storage Object]
| | o- archive .............................. [/dev/rbd/rbd/archive activated]
| o- rd_mcp ............................................... [0 Storage Object]
o- ib_srpt ....................................................... [0 Targets]
o- iscsi .......................................................... [1 Target]
| o- iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk .......... [1 TPG]
| o- tpg1 ........................................................ [enabled]
| o- acls ........................................................ [1 ACL]
| | o- iqn.1996-04.de.suse:01:e6ca28cc9f20 ................ [1 Mapped LUN]
| | o- mapped_lun0 ......................................... [lun0 (rw)]
| o- luns ........................................................ [1 LUN]
| | o- lun0 ......................... [rbd/archive (/dev/rbd/rbd/archive)]
| o- portals .................................................. [1 Portal]
| o- 172.16.1.16:3260 .............................. [OK, iser disabled]
o- loopback ...................................................... [0 Targets]
o- qla2xxx ....................................................... [0 Targets]
o- tcm_fc ........................................................ [0 Targets]
o- vhost ......................................................... [0 Targets]The import, editing or querying can be done on any Ceph client with lrbd installed. Applying the configuration will only make changes to the current host.
Host specific configurations can be queried from another host. For example, running the following from host igw1
# lrbd -H igw2 -l
will display the configuration of host igw2. Applying the configuration of another host is similar. From a host test1, running
# lrbd -H igw2
applies the configuration from host igw2. This allows investigation from separate hardware or temporary replacement of failed hardware without modifying the configuration.
Growing the configuration to match your requirements is straightforward. Add the necessary entries often by copying an existing entry and making simple changes. Be aware that a valid JSON structure needs a comma between elements but not after the last element in a list.
The following examples reference the simple.json in the previous section.
Create an image within the same pool.
# rbd -p rbd create data --size=2048
Copy the tpg entry containing image, portal and initiator. Change the image in the new entry, save your changes and apply. An initiator can access two RBD images from the same pool on the same gateway over the same network path.
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
},
{
"portal": "portal1",
"image": data,
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}Nearly identical to the adding an image, but change the initiator in the new entry. Two different initiators can access the same RBD image on the same gateway over the same network path.
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
},
{
"portal": "portal1",
"image": "archive",
"initiator": iqn.1991-05.com.microsoft:client.example.com
}In the portals section, duplicate the portal entry and change the name and address.
{
"name": "portal1",
"addresses": [ "172.16.1.16" ]
},
{
"name": portal2,
"addresses": [ 172.16.2.16 ]
}Use the new portal name. In this example, each initiator will use a dedicated network interface.
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
},
{
"portal": portal2,
"image": "archive",
"initiator": "iqn.1991-05.com.microsoft:client.example.com"
}In the portals section, provide a second address to an existing portal. Any entry using this portal will provide redundancy across two network paths.
{
"name": "portal1",
"addresses": [ "172.16.1.16", 172.16.2.16 ]
}Copy an existing gateway entry. Maintain the same host and change any or all of portal, image and initiator. This allows isolated paths from initiator to image.
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
},
{
"host": "igw1",
"tpg": [
{
"portal": portal2,
"image": data,
"initiator": iqn.1991-05.com.microsoft:client.example.com
}
]
} Similar to adding a tpg, copy an existing gateway entry. Change the host of the new entry to the hostname of your second gateway. Do not use this for redundancy. This configuration is for two gateways with independent images.
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
},
{
"host": igw2,
"tpg": [
{
"portal": portal3,
"image": data,
"initiator": iqn.1991-05.com.microsoft:client.example.com
}
]
} Add another portal entry for the address of the second gateway.
{
"name": "portal1",
"addresses": [ "172.16.1.16" ]
},
{
"name": portal3,
"addresses": [ 172.16.1.17 ]
}Duplicate a pool entry. Update the pool and image. Each image is accessed via the same network path.
{
"pool": "rbd",
"gateways": [
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
}
]
},
{
"pool": car,
"gateways": [
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": cement,
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
}
]
}More than one gateway can advertise the same target providing redundancy. This configuration requires a kernel that supports the tpg_enabled_sendtargets (formerly discoverable_while_disabled) and a targetcli that allows portal creation with non-existent addresses.
Copy an existing gateway entry. Change the host keyword to target. Update the hostname to an iqn value. Consider using iscsi-name as a template. Change the embedded hostname (e.g. igw1 to any). Remove the portal.
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": "archive",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
},
{
target: iqn.2003-01.org.linux-iscsi.any.x86:sn.db7fc86c644,
"tpg": [
{
"image": "data",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
} Update the targets section. Add the new target with a list of hosts and portals.
{
"host": "igw1",
"target": "iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk"
},
{
hosts: [
{ "host": igw1, "portal": portal1 },
{ "host": igw2, "portal": portal4 }
],
"target": "iqn.2003-01.org.linux-iscsi.any.x86:sn.db7fc86c644"
}Update the portals section. Add the new portal with the address list from both gateways.
{
"name": "portal1",
"addresses": [ "172.16.1.16" ]
}
{
"name": portal4,
"addresses": [ 172.16.4.16 ]
}When applied, each gateway will have two TPGs for the common target. Note that a TPG is disabled on each gateway and contains the portal of the other gateway.
Iscsi authentication is flexible resulting in many possibilities. The five possible top level settings are no authentication, tpg, acls, tpg+identified and identified.
No authentication means no initiator will require a username and password to access any luns for a specified host or target. No authentication can be set explicitly or implicitly. Specify a value of none for authentication to be set explicitly. For example,
{
"host": "igw1",
"authentication": none
}Removing the entire auth section from the configuration will use no authentication implicitly.
For common credentials or a shared username/password, set authentication to tpg. This setting will apply to all initiators for the associated host or target. In this example, the same username and password are used for the redundant target and a target local to igw1.
{
"target": "iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant",
"authentication": tpg,
"tpg": {
"userid": "common1",
"password": "pass1"
}
},
{
"host": "igw1",
"authentication": tpg,
"tpg": {
"userid": "common1",
"password": "pass1"
}
}Redundant configurations will have the same credentials across gateways but are independent of other configurations. In other words, luns configured specifically for a host and multiple redundant configurations can have a unique username and password for each.
One caveat to note is that any initiator setting will be ignored when using tpg authentication. Using common credentials does not restrict which initiators may connect. This configuration may be suitable in isolated network environments.
For unique credentials for each initiator, set authentication to acls. Additionally, only defined initiators are allowed to connect.
{
"host": "igw1",
"authentication": acls,
"acls": [
{
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20",
"userid": "initiator1",
"password": "pass1",
}
]
},The previous two authentication settings each have a use, but pair two independent features. TPG pairs common credentials with unidentified initiators. Acls pair unique credentials with identified initiators.
This authentication setting pairs common credentials with identified initiators. This can be imitated by choosing acls and repeating the same credentials with each initiator, but the configuration would grow unnecessarily and annoy the maintainers. For this configuration, set authentication to tpg+identified.
This configuration uses the tpg configuration with only the authentication keyword changing.
{
"target": "iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant",
"authentication": tpg+identified,
"tpg": {
"userid": "common1",
"password": "pass1"
}
},
{
"host": "igw1",
"authentication": tpg+identified,
"tpg": {
"userid": "common1",
"password": "pass1"
}
}The list of initiators is gathered from those defined in the pools for the given hosts and targets in the authentication section.
The setting is an oxymoron since the configuration lacks any credentials. In secure environments where only assignment of initiators is needed, set the authentication to identified. All initiators will connect but only have access to the images listed in the pools section.
{
"target": "iqn.2003-01.org.linux-iscsi:igw.x86:sn.redundant",
"authentication": "identified",
},
{
"host": "igw1",
"authentication": "identified",
}Independent of these four authentication settings is discovery authentication. Discovery authentication requires credentials for browsing.
Authentication for tpg, tpg+identified, acls and discovery support mutual authentication. Specifiying the mutual settings requires that the target authenticates against the initiator.
Discovery and mutual authentication are optional. These options can be present, but disabled allowing experimentation with a particular configuration. Once decided, disabled entries can be removed without harming the configuration.
For several possibilities, please see the examples in /usr/share/doc/packages/lrbd/samples. Excerpts from one file can be combined with others to create unique configurations.
The following settings may be useful for some environments. For images, they are uuid, lun, retries, sleep, retry_errors and backstore attributes. The first two allow hardcoding of the uuid or lun for a specific image. Either can be set for an image. The retries, sleep and retry_errors affect the attempts to map an rbd image. The backstore attributes allow tuning of kernel parameters. Use with caution.
"pools": [
{
"pool": "rbd",
"gateways": [
{
"host": "igw1",
"tpg": [
{
"image": "archive",
"uuid": "12345678-abcd-9012-efab-345678901234",
"lun": "2",
"retries": "3",
"sleep": "4",
"retry_errors": [ 95 ],
"backstore_block_size": "512",
"backstore_emulate_3pc": "1",
"backstore_emulate_caw": "1",
"backstore_emulate_dpo": "0",
"backstore_emulate_fua_read": "0",
"backstore_emulate_fua_write": "1",
"backstore_emulate_model_alias": "0",
"backstore_emulate_rest_reord": "0",
"backstore_emulate_tas": "1",
"backstore_emulate_tpu": "0",
"backstore_emulate_tpws": "0",
"backstore_emulate_ua_intlck_ctrl": "0",
"backstore_emulate_write_cache": "0",
"backstore_enforce_pr_isids": "1",
"backstore_fabric_max_sectors": "8192",
"backstore_hw_block_size": "512",
"backstore_hw_max_sectors": "8192",
"backstore_hw_pi_prot_type": "0",
"backstore_hw_queue_depth": "128",
"backstore_is_nonrot": "1",
"backstore_max_unmap_block_desc_count": "1",
"backstore_max_unmap_lba_count": "8192",
"backstore_max_write_same_len": "65535",
"backstore_optimal_sectors": "8192",
"backstore_pi_prot_format": "0",
"backstore_pi_prot_type": "0",
"backstore_queue_depth": "128",
"backstore_unmap_granularity": "8192",
"backstore_unmap_granularity_alignment": "4194304"
}
]
}
]
}
]For targets, the tpg attributes allow tuning of kernel parameters. Use with caution.
"targets": [
{
"host": "igw1",
"target": "iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk",
"tpg_default_cmdsn_depth": "64",
"tpg_default_erl": "0",
"tpg_login_timeout": "10",
"tpg_netif_timeout": "2",
"tpg_prod_mode_write_protect": "0",
"tpg_t10_pi": "0"
}For initiators, the attrib and param settings allow the tuning of kernel parameters. Use with caution. These are set in the authentication section. If the authentication is tpg+identified or identified, then the subsection is identifed.
"auth": [
{
"authentication": "tpg+identified",
"identified": [
{
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20",
"attrib_dataout_timeout": "3",
"attrib_dataout_timeout_retries": "5",
"attrib_default_erl": "0",
"attrib_nopin_response_timeout": "30",
"attrib_nopin_timeout": "15",
"attrib_random_datain_pdu_offsets": "0",
"attrib_random_datain_seq_offsets": "0",
"attrib_random_r2t_offsets": "0",
"param_DataPDUInOrder": "1",
"param_DataSequenceInOrder": "1",
"param_DefaultTime2Retain": "0",
"param_DefaultTime2Wait": "2",
"param_ErrorRecoveryLevel": "0",
"param_FirstBurstLength": "65536",
"param_ImmediateData": "1",
"param_InitialR2T": "1",
"param_MaxBurstLength": "262144",
"param_MaxConnections": "1",
"param_MaxOutstandingR2T": "1"
}
]
}
]if the authentication is acls, then the settings are included in the acls subsection.
"auth": [
{
"host": "igw1",
"authentication": "acls",
"acls": [
{
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20",
"userid": "initiator1",
"password": "pass1",
"attrib_dataout_timeout": "3",
"attrib_dataout_timeout_retries": "5",
"attrib_default_erl": "0",
"attrib_nopin_response_timeout": "30",
"attrib_nopin_timeout": "15",
"attrib_random_datain_pdu_offsets": "0",
"attrib_random_datain_seq_offsets": "0",
"attrib_random_r2t_offsets": "0",
"param_DataPDUInOrder": "1",
"param_DataSequenceInOrder": "1",
"param_DefaultTime2Retain": "0",
"param_DefaultTime2Wait": "2",
"param_ErrorRecoveryLevel": "0",
"param_FirstBurstLength": "65536",
"param_ImmediateData": "1",
"param_InitialR2T": "1",
"param_MaxBurstLength": "262144",
"param_MaxConnections": "1",
"param_MaxOutstandingR2T": "1"
}
]
},
]One caveat is that settings are only applied for active initiators. If an initiator is absent from the pools section, the acl entry is not created and settings cannot be applied.
The rbd_name is a backward compatibility option for setting the backstore name to only use the name of the image. The current default uses pool name, hyphen and image name. Do not use this option for new installations.
"pools": [
{
"pool": "rbd",
"gateways": [
{
"host": "igw1",
"tpg": [
{
"image": "archive",
"rbd_name": "simple"
}
]
}
]
}
]Likewise, the wwn_generate will use the original scheme of target and image name for setting the vpn_unit_serial. The current default uses pool name, target and image name. Do not use this option for new installations.
"targets": [
{
"host": "igw1",
"target": "iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk",
"wwn_generate": "original"
}The lrbd utility includes a service file. Systemd will run lrbd on boot when the service is enabled. To enable and start the service, run
# systemctl enable lrbd
# systemctl start lrbd
The service is a one-shot service. Any changes to the lrbd configuration do not propogate automatically. To apply the changes without clearing the current configuration, run
# systemctl reload lrbd
The rationale for this behavior allows administrators to control when the configuration is applied to each gateway.
The configuration may be updated directly with lrbd -e. The entire configuration can be output with lrbd -o > filename and maintained under your choice of version control.
Alternatively, the global configuration can be re-imported after making changes. Running lrbd -f filename will replace any configuration stored within Ceph. The advantage of this method is the input file may contain comments for complex configurations.
Lastly, configurations may be added cumulatively. For instance, if two groups within an organization each have independent gateways and clients but wish to share the same Ceph cluster, multiple configurations can be maintained independently. Each configuration must be valid by itself, but could be divided into separate files by pool or gateway. One file could contain gateways A, B and C while another contains D and E.
The first configuration is imported as normal with lrbd -f ABC.config. The second configuration is added with lrbd -a DE.config. The pools and authentication sections will be merged. The targets and portals sections are replaced. Depending on the configuration, either remove the targets and portals section from the DE.config or duplicate these sections in both files.
The above example could also be executed with lrbd -W which wipes the configuration from Ceph and then executes lrbd -a for each file.
The multipath configuration is maintained on the clients or initiators and is independent of any lrbd configuration. Select a strategy prior to using block storage. After editing the /etc/multipath.conf, restart multipathd with
# systemctl restart multipathd
For an active-passive configuration with friendly names, add
defaults {
user_friendly_names yes
}to your /etc/multipath.conf. After connecting to your targets successfully, run
# multipath -ll
mpathd (36001405dbb561b2b5e439f0aed2f8e1e) dm-0 SUSE,RBD
size=2.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 2:0:0:3 sdl 8:176 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 3:0:0:3 sdj 8:144 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 4:0:0:3 sdk 8:160 active ready runningNote the status of each link. For an active-active configuration, add
defaults {
user_friendly_names yes
}
devices {
device {
vendor "(LIO-ORG|SUSE)"
product "RBD"
path_grouping_policy "multibus"
path_checker "tur"
features "0"
hardware_handler "1 alua"
prio "alua"
failback "immediate"
rr_weight "uniform"
no_path_retry 12
rr_min_io 100
}
}to your /etc/multipath.conf. Restart multipathd and run
# multipath -ll
mpathd (36001405dbb561b2b5e439f0aed2f8e1e) dm-3 SUSE,RBD
size=2.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 4:0:0:3 sdj 8:144 active ready running
|- 3:0:0:3 sdk 8:160 active ready running
`- 2:0:0:3 sdl 8:176 active ready runningThe following examples are located in /usr/share/doc/packages/lrbd/samples. Some are admittedly contrived, but show combinations that may more closely match your requirements.
The requirement is to provide a highly available solution for an application accessing Ceph storage. In this example, the image city is advertised from two gateways. Each gateway has a single nic for iSCSI access. The physical hardware could be the following:
The expected configuration would look like the following:
The sample is called 2gateways+no_authentication.json, which is shown below:
{
"auth": [
{
"authentication": "none",
"target": "iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant"
}
],
"targets": [
{
"hosts": [
{ "host": "igw1", "portal": "portal1" },
{ "host": "igw2", "portal": "portal2" }
],
"target": "iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant"
}
],
"portals": [
{
"name": "portal1",
"addresses": [ "172.16.1.16"]
},
{
"name": "portal2",
"addresses": [ "172.16.1.17" ]
}
],
"pools": [
{
"pool": "rbd",
"gateways": [
{
"target": "iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant",
"tpg": [
{
"image": "city"
}
]
}
]
}
]
}
Running lrbd on the first gateway, igw1, will output the commands and respective outputs:
modprobe vhost_scsi
modprobe iscsi_target_mod
modprobe tcm_loop
modprobe tcm_fc
modprobe ib_srpt
modprobe tcm_qla2xxx
modprobe target_core_rbd
targetcli /backstores/rbd create name=city dev=/dev/rbd/rbd/city
Created '/sys/kernel/config/target/vhost'.
Created '/sys/kernel/config/target/iscsi'.
Created '/sys/kernel/config/target/loopback'.
Created '/sys/kernel/config/target/fc'.
Created '/sys/kernel/config/target/srpt'.
Created '/sys/kernel/config/target/qla2xxx'.
Created RBD storage object city using /dev/rbd/rbd/city.
targetcli /iscsi create iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant
Created target iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant.
Selected TPG Tag 1.
Created TPG 1.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant create 2
Created TPG 2.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant/tpg2 disable
The TPG has been disabled.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant/tpg1/luns create /backstores/rbd/city
Selected LUN 0.
Created LUN 0.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant/tpg2/luns create /backstores/rbd/city
Selected LUN 0.
Created LUN 0.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant/tpg1/portals create 172.16.1.16
Using default IP port 3260
Created network portal 172.16.1.16:3260.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant/tpg2/portals create 172.16.1.17
Using default IP port 3260
IP address 172.16.1.17 does not exist on this host.
Created network portal 172.16.1.17:3260.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant/tpg1 set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1
Parameter authentication is now '0'.
Parameter demo_mode_write_protect is now '0'.
Parameter generate_node_acls is now '1'.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant/tpg2 set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1
Parameter authentication is now '0'.
Parameter demo_mode_write_protect is now '0'.
Parameter generate_node_acls is now '1'.
To inspect the configuration, run
# targetcli ls
The output will be
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- fileio ............................................... [0 Storage Object]
| o- iblock ............................................... [0 Storage Object]
| o- pscsi ................................................ [0 Storage Object]
| o- rbd .................................................. [1 Storage Object]
| | o- city .................................... [/dev/rbd/rbd/city activated]
| o- rd_mcp ............................................... [0 Storage Object]
o- ib_srpt ....................................................... [0 Targets]
o- iscsi .......................................................... [1 Target]
| o- iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant ............... [2 TPGs]
| o- tpg1 ........................................................ [enabled]
| | o- acls ....................................................... [0 ACLs]
| | o- luns ........................................................ [1 LUN]
| | | o- lun0 ............................... [rbd/city (/dev/rbd/rbd/city)]
| | o- portals .................................................. [1 Portal]
| | o- 172.16.1.16:3260 .............................. [OK, iser disabled]
| o- tpg2 ....................................................... [disabled]
| o- acls ....................................................... [0 ACLs]
| o- luns ........................................................ [1 LUN]
| | o- lun0 ............................... [rbd/city (/dev/rbd/rbd/city)]
| o- portals .................................................. [1 Portal]
| o- 172.16.1.17:3260 .............................. [OK, iser disabled]
o- loopback ...................................................... [0 Targets]
o- qla2xxx ....................................................... [0 Targets]
o- tcm_fc ........................................................ [0 Targets]
o- vhost ......................................................... [0 Targets]
Note that tpg2 is disabled since this portal does not exist on gateway igw1. Applying the same configuration on igw2 will result in the reverse. Tpg1 will be disabled and tpg2 will be enabled.
From the initiator, a discovery of igw1 will output the following:
# iscsiadm -m discovery -t st -p 172.16.1.16
172.16.1.16:3260,1 iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant
172.16.1.17:3260,2 iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant
A discovery of igw2 will give the same results.
Once discovered, both gateways can be logged into simultaneously with
# iscsiadm -m node -L all
And multipath will list both paths as separate devices.
# multipath -ll
3600140569accc1f6b1f3853bb38f8deb dm-0 SUSE,RBD
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 8:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 9:0:0:0 sda 8:0 active ready running
Rather than see a trivial example of iSCSI access controls, this example shows access lists on two independent gateways each accessing different images within Ceph and restricting access to two clients. Additionally, each gateway has redundant network interfaces.
Ceph is configured for five images across three pools. The pool rbd contains city, writers and wood, car contains cement and whirl contains cheese. The rationale is to simply demonstrate that multiple images per pool and multiple pools work. A typical configuration would be a subset.
Each client has an initiator iqn. Each gateway has a target iqn with a single portal of two network interfaces.
The sample is acls.json and is shown below:
{
"targets": [
{
"host": "igw1",
"target": "iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141"
},
{
"host": "igw2",
"target": "iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579"
}
],
"auth": [
{
"host": "igw1",
"authentication": "acls",
"acls": [
{
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20",
"userid": "initiator1",
"password": "pass1"
}
]
},
{
"host": "igw2",
"authentication": "acls",
"acls": [
{
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20",
"userid": "initiator1",
"password": "pass1"
},
{
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f24",
"userid": "initiator2",
"password": "pass5"
}
]
}
],
"portals": [
{
"name": "portal1",
"addresses": [ "172.16.1.16", "172.16.2.16" ]
},
{
"name": "portal2",
"addresses": [ "172.16.1.17", "172.16.2.17" ]
}
],
"pools": [
{
"pool": "rbd",
"gateways": [
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"image": "city",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20"
}
]
},
{
"host": "igw2",
"tpg": [
{
"portal": "portal2",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20",
"image": "writers"
},
{
"portal": "portal2",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f24",
"image": "writers"
},
{
"portal": "portal2",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f24",
"image": "wood"
}
]
}
]
},
{
"pool": "car",
"gateways": [
{
"host": "igw1",
"tpg": [
{
"portal": "portal1",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20",
"image": "cement"
}
]
}
]
},
{
"pool": "whirl",
"gateways": [
{
"host": "igw2",
"tpg": [
{
"portal": "portal2",
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20",
"image": "cheese"
}
]
}
]
}
]
}
Unlike the previous example where both gateways are configured essentially identically, these gateways are independent. Running lrbd on igw1 results in the following
modprobe vhost_scsi
modprobe iscsi_target_mod
modprobe tcm_loop
modprobe tcm_fc
modprobe ib_srpt
modprobe tcm_qla2xxx
modprobe target_core_rbd
targetcli /backstores/rbd create name=city dev=/dev/rbd/rbd/city
Created '/sys/kernel/config/target/vhost'.
Created '/sys/kernel/config/target/iscsi'.
Created '/sys/kernel/config/target/loopback'.
Created '/sys/kernel/config/target/fc'.
Created '/sys/kernel/config/target/srpt'.
Created '/sys/kernel/config/target/qla2xxx'.
Created RBD storage object city using /dev/rbd/rbd/city.
targetcli /backstores/rbd create name=cement dev=/dev/rbd/car/cement
Created RBD storage object cement using /dev/rbd/car/cement.
targetcli /iscsi create iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141
Created target iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141.
Selected TPG Tag 1.
Created TPG 1.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141/tpg1/luns create /backstores/rbd/cement
Selected LUN 0.
Created LUN 0.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141/tpg1/luns create /backstores/rbd/city
Selected LUN 1.
Created LUN 1.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141/tpg1/portals create 172.16.1.16
Using default IP port 3260
Created network portal 172.16.1.16:3260.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141/tpg1/portals create 172.16.2.16
Using default IP port 3260
Created network portal 172.16.2.16:3260.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141/tpg1/portals create 172.16.1.16
Using default IP port 3260
This NetworkPortal already exists in configFS.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141/tpg1/portals create 172.16.2.16
Using default IP port 3260
This NetworkPortal already exists in configFS.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141/tpg1/acls create iqn.1996-04.de.suse:01:e6ca28cc9f20
Created Node ACL for iqn.1996-04.de.suse:01:e6ca28cc9f20
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f20 create 1 1
Created Mapped LUN 1.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f20 create 0 0
Created Mapped LUN 0.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f20 set auth userid=initiator1 password=pass1
Parameter password is now 'pass1'.
Parameter userid is now 'initiator1'.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141/tpg1 set attribute authentication=1 demo_mode_write_protect=0 generate_node_acls=0
Parameter authentication is now '1'.
Parameter demo_mode_write_protect is now '0'.
Parameter generate_node_acls is now '0'.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f20 set auth userid=initiator1 password=pass1
Parameter password is now 'pass1'.
Parameter userid is now 'initiator1'.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141/tpg1 set attribute authentication=1 demo_mode_write_protect=0 generate_node_acls=0
Parameter authentication is now '1'.
Parameter demo_mode_write_protect is now '0'.
Parameter generate_node_acls is now '0'.
While running lrbd on igw2 outputs
modprobe vhost_scsi
modprobe iscsi_target_mod
modprobe tcm_loop
modprobe tcm_fc
modprobe ib_srpt
modprobe tcm_qla2xxx
modprobe target_core_rbd
targetcli /backstores/rbd create name=mental dev=/dev/rbd/whirl/mental
Created '/sys/kernel/config/target/vhost'.
Created '/sys/kernel/config/target/iscsi'.
Created '/sys/kernel/config/target/loopback'.
Created '/sys/kernel/config/target/fc'.
Created '/sys/kernel/config/target/srpt'.
Created '/sys/kernel/config/target/qla2xxx'.
Created RBD storage object mental using /dev/rbd/whirl/mental.
targetcli /backstores/rbd create name=writers dev=/dev/rbd/rbd/writers
Created RBD storage object writers using /dev/rbd/rbd/writers.
targetcli /backstores/rbd create name=wood dev=/dev/rbd/rbd/wood
Created RBD storage object wood using /dev/rbd/rbd/wood.
targetcli /iscsi create iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579
Created target iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579.
Selected TPG Tag 1.
Created TPG 1.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/luns create /backstores/rbd/wood
Selected LUN 0.
Created LUN 0.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/luns create /backstores/rbd/writers
Selected LUN 1.
Created LUN 1.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/luns create /backstores/rbd/mental
Selected LUN 2.
Created LUN 2.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/portals create 172.16.1.17
Using default IP port 3260
Created network portal 172.16.1.17:3260.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/portals create 172.16.2.17
Using default IP port 3260
Created network portal 172.16.2.17:3260.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/portals create 172.16.1.17
Using default IP port 3260
This NetworkPortal already exists in configFS.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/portals create 172.16.2.17
Using default IP port 3260
This NetworkPortal already exists in configFS.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/portals create 172.16.1.17
Using default IP port 3260
This NetworkPortal already exists in configFS.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/portals create 172.16.2.17
Using default IP port 3260
This NetworkPortal already exists in configFS.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/acls create iqn.1996-04.de.suse:01:e6ca28cc9f24
Created Node ACL for iqn.1996-04.de.suse:01:e6ca28cc9f24
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/acls create iqn.1996-04.de.suse:01:e6ca28cc9f20
Created Node ACL for iqn.1996-04.de.suse:01:e6ca28cc9f20
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f20 create 1 1
Created Mapped LUN 1.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f24 create 1 1
Created Mapped LUN 1.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f24 create 0 0
Created Mapped LUN 0.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f20 create 2 2
Created Mapped LUN 2.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f20 set auth userid=initiator1 password=pass1
Parameter password is now 'pass1'.
Parameter userid is now 'initiator1'.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f24 set auth userid=initiator2 password=pass5
Parameter password is now 'pass5'.
Parameter userid is now 'initiator2'.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1 set attribute authentication=1 demo_mode_write_protect=0 generate_node_acls=0
Parameter authentication is now '1'.
Parameter demo_mode_write_protect is now '0'.
Parameter generate_node_acls is now '0'.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f20 set auth userid=initiator1 password=pass1
Parameter password is now 'pass1'.
Parameter userid is now 'initiator1'.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f24 set auth userid=initiator2 password=pass5
Parameter password is now 'pass5'.
Parameter userid is now 'initiator2'.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1 set attribute authentication=1 demo_mode_write_protect=0 generate_node_acls=0
Parameter authentication is now '1'.
Parameter demo_mode_write_protect is now '0'.
Parameter generate_node_acls is now '0'.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f20 set auth userid=initiator1 password=pass1
Parameter password is now 'pass1'.
Parameter userid is now 'initiator1'.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1/acls/iqn.1996-04.de.suse:01:e6ca28cc9f24 set auth userid=initiator2 password=pass5
Parameter password is now 'pass5'.
Parameter userid is now 'initiator2'.
targetcli /iscsi/iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579/tpg1 set attribute authentication=1 demo_mode_write_protect=0 generate_node_acls=0
Parameter authentication is now '1'.
Parameter demo_mode_write_protect is now '0'.
Parameter generate_node_acls is now '0'.
The first gateway will share the two images, cement and city with only one initiator. This initiator can connect to either image over either interface, but must login with the credentials specified.
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- fileio ............................................... [0 Storage Object]
| o- iblock ............................................... [0 Storage Object]
| o- pscsi ................................................ [0 Storage Object]
| o- rbd ................................................. [2 Storage Objects]
| | o- cement ................................ [/dev/rbd/car/cement activated]
| | o- city .................................... [/dev/rbd/rbd/city activated]
| o- rd_mcp ............................................... [0 Storage Object]
o- ib_srpt ....................................................... [0 Targets]
o- iscsi .......................................................... [1 Target]
| o- iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141 ............ [1 TPG]
| o- tpg1 ........................................................ [enabled]
| o- acls ........................................................ [1 ACL]
| | o- iqn.1996-04.de.suse:01:e6ca28cc9f20 ............... [2 Mapped LUNs]
| | o- mapped_lun0 ......................................... [lun0 (rw)]
| | o- mapped_lun1 ......................................... [lun1 (rw)]
| o- luns ....................................................... [2 LUNs]
| | o- lun0 ........................... [rbd/cement (/dev/rbd/car/cement)]
| | o- lun1 ............................... [rbd/city (/dev/rbd/rbd/city)]
| o- portals ................................................. [2 Portals]
| o- 172.16.1.16:3260 .............................. [OK, iser disabled]
| o- 172.16.2.16:3260 .............................. [OK, iser disabled]
o- loopback ...................................................... [0 Targets]
o- qla2xxx ....................................................... [0 Targets]
o- tcm_fc ........................................................ [0 Targets]
o- vhost ......................................................... [0 Targets]
The second gateway shares three images - cheese, wood and writers. The first initiator can access writers and cheese, but not wood. The second initiator can access wood and writers, but not cheese.
Note that although writers is accessible by both initiators, two independently acting applications can still corrupt the data. Consider the example of mounting a filesystem (e.g. ext4, xfs) to two different servers and writing both simultaneously. Corruption will ensue.
However, an administrator may have an application requiring a cold spare where this solution may be suitable.
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- fileio ............................................... [0 Storage Object]
| o- iblock ............................................... [0 Storage Object]
| o- pscsi ................................................ [0 Storage Object]
| o- rbd ................................................. [3 Storage Objects]
| | o- cheese .............................. [/dev/rbd/whirl/cheese activated]
| | o- wood .................................... [/dev/rbd/rbd/wood activated]
| | o- writers .............................. [/dev/rbd/rbd/writers activated]
| o- rd_mcp ............................................... [0 Storage Object]
o- ib_srpt ....................................................... [0 Targets]
o- iscsi .......................................................... [1 Target]
| o- iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579 ............ [1 TPG]
| o- tpg1 ........................................................ [enabled]
| o- acls ....................................................... [2 ACLs]
| | o- iqn.1996-04.de.suse:01:e6ca28cc9f20 ............... [2 Mapped LUNs]
| | | o- mapped_lun1 ......................................... [lun1 (rw)]
| | | o- mapped_lun2 ......................................... [lun2 (rw)]
| | o- iqn.1996-04.de.suse:01:e6ca28cc9f24 ............... [2 Mapped LUNs]
| | o- mapped_lun0 ......................................... [lun0 (rw)]
| | o- mapped_lun1 ......................................... [lun1 (rw)]
| o- luns ....................................................... [3 LUNs]
| | o- lun0 ............................... [rbd/wood (/dev/rbd/rbd/wood)]
| | o- lun1 ......................... [rbd/writers (/dev/rbd/rbd/writers)]
| | o- lun2 ......................... [rbd/cheese (/dev/rbd/whirl/cheese)]
| o- portals ................................................. [2 Portals]
| o- 172.16.1.17:3260 .............................. [OK, iser disabled]
| o- 172.16.2.17:3260 .............................. [OK, iser disabled]
o- loopback ...................................................... [0 Targets]
o- qla2xxx ....................................................... [0 Targets]
o- tcm_fc ........................................................ [0 Targets]
o- vhost ......................................................... [0 Targets]
From the first initiator, a discovery of igw1 will output the following:
# iscsiadm -m discovery -t st -p 172.16.1.16
172.16.1.16:3260,1 iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141
172.16.2.16:3260,1 iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141
Before logging in, the credentials must be configured.
# iscsiadm -m node -o update --name=node.session.auth.username --value=initiator1
# iscsiadm -m node -o update --name=node.session.auth.password --value=pass1
# iscsiadm -m node -o update --name=node.session.auth.authmethod --value=CHAP
# iscsiadm -m node -L all
These values can be verified with
# iscsiadm -m node -o show
Next, login from the first initiator to the igw1 with
# iscsiadm -m node -L all
If the login fails, consider running journalctl -f on igw1 to help diagnose the issue.
Once the login is successful, four devices will be available.
# multipath -ll
360014052d4012252b4e4f69849df36eb dm-1 SUSE,RBD
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 7:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 6:0:0:0 sda 8:0 active ready running
360014050c2b99be7c3c421f8ef1d34df dm-0 SUSE,RBD
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 6:0:0:1 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 7:0:0:1 sdd 8:48 active ready running
Now the first initiator can log into igw2 as well. After running a discovery,
# iscsiadm -m discovery -t st -p 172.16.1.17
both paths for each gateway will be displayed from
# iscsiadm -m node
172.16.1.16:3260,1 iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141
172.16.2.16:3260,1 iqn.2003-01.org.linux-iscsi.igw1.x86:sn.1a2b6ae09141
172.16.2.17:3260,1 iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579
172.16.1.17:3260,1 iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579
Initialize the credentials if necessary and login.
# iscsiadm -m node -o update --name=node.session.auth.username --value=initiator1
# iscsiadm -m node -o update --name=node.session.auth.password --value=pass1
# iscsiadm -m node -o update --name=node.session.auth.authmethod --value=CHAP
# iscsiadm -m node -L all
The last command will complain that two sessions are already active. The devices for igw2 have been added. Verify with
# multipath -ll
360014052d4012252b4e4f69849df36eb dm-1 SUSE,RBD
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 7:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 6:0:0:0 sda 8:0 active ready running
360014050c2b99be7c3c421f8ef1d34df dm-0 SUSE,RBD
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 6:0:0:1 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 7:0:0:1 sdd 8:48 active ready running
360014054267d0d72a5447d3bc42c50f9 dm-2 SUSE,RBD
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 10:0:0:1 sde 8:64 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 11:0:0:1 sdg 8:96 active ready running
36001405f2fc087d51114b909e0cfcfd7 dm-3 SUSE,RBD
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 10:0:0:2 sdf 8:80 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 11:0:0:2 sdh 8:112 active ready running
The second initiator can log into igw2 following the same steps as the first initiator.
# iscsiadm -m discovery -t st -p 172.16.1.17
172.16.1.17:3260,1 iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579
172.16.2.17:3260,1 iqn.2003-01.org.linux-iscsi.igw2.x86:sn.b611a553e579
Note the different username and password.
# iscsiadm -m node -o update --name=node.session.auth.username --value=initiator2
# iscsiadm -m node -o update --name=node.session.auth.password --value=pass5
# iscsiadm -m node -o update --name=node.session.auth.authmethod --value=CHAP
# iscsiadm -m node -L all
The devices added on the second initiator are
# multipath -ll
3600140576c55e60dd7d4ae2982bce328 dm-1 SUSE,RBD
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 13:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 12:0:0:0 sda 8:0 active ready running
36001405f39addb925d44472951de78da dm-0 SUSE,RBD
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 12:0:0:1 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 13:0:0:1 sdd 8:48 active ready running
A potentially useful configuration concerns different requirements for images. Imagine customizing network buffers or having greater bandwidth for one network interface but not another.
The requirement is to dedicate one network interface for the image city and the other network interface to wood. The administrator wishes to have the image archive accessible from one gateway for experimentation. Additionally, the images city and wood must be restricted to their respective initiators, but archive is not.
The physical layout is identical to the previous example.
The organization is different. Gateway igw1 has three targets and gateway igw2 has two. Each network interface is associated with a portal.
The configuration sample is plain+2gateways+2portals+2images+isolated+combined.json and shown here.
{
"auth": [
{
"target": "iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant",
"authentication": "acls",
"acls": [
{
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20",
"userid": "common1",
"password": "pass1"
}
]
},
{
"target": "iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant-too",
"authentication": "acls",
"acls": [
{
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f24",
"userid": "common2",
"password": "pass2"
}
]
},
{
"host": "igw1",
"authentication": "tpg",
"tpg": {
"userid": "common1",
"password": "pass1"
}
}
],
"targets": [
{
"hosts": [
{ "host": "igw1", "portal": "portal1" },
{ "host": "igw2", "portal": "portal2" }
],
"target": "iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant"
},
{
"hosts": [
{ "host": "igw1", "portal": "portal3" },
{ "host": "igw2", "portal": "portal4" }
],
"target": "iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant-too"
},
{
"host": "igw1",
"target": "iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk"
}
],
"portals": [
{
"name": "portal1",
"addresses": [ "172.16.1.16" ]
},
{
"name": "portal2",
"addresses": [ "172.16.1.17" ]
},
{
"name": "portal3",
"addresses": [ "172.16.2.16" ]
},
{
"name": "portal4",
"addresses": [ "172.16.2.17" ]
}
],
"pools": [
{
"pool": "rbd",
"gateways": [
{
"host": "igw1",
"tpg": [
{
"image": "archive",
"portal": "portal1"
}
]
},
{
"target": "iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant",
"tpg": [
{
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20",
"image": "city"
}
]
},
{
"target": "iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant-too",
"tpg": [
{
"initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f24",
"image": "wood"
}
]
}
]
}
]
}
The output of the lrbd commands is similar on both gateways. Since neither demonstrates anything unique from the previous examples, these are omitted. Inspecting both gateways will show the differences from the first example.
On igw1, run
# targetcli ls
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- fileio ............................................... [0 Storage Object]
| o- iblock ............................................... [0 Storage Object]
| o- pscsi ................................................ [0 Storage Object]
| o- rbd ................................................. [3 Storage Objects]
| | o- archive .............................. [/dev/rbd/rbd/archive activated]
| | o- city .................................... [/dev/rbd/rbd/city activated]
| | o- wood .................................... [/dev/rbd/rbd/wood activated]
| o- rd_mcp ............................................... [0 Storage Object]
o- ib_srpt ....................................................... [0 Targets]
o- iscsi ......................................................... [3 Targets]
| o- iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk .......... [1 TPG]
| | o- tpg1 ........................................................ [enabled]
| | o- acls ....................................................... [0 ACLs]
| | o- luns ........................................................ [1 LUN]
| | | o- lun0 ......................... [rbd/archive (/dev/rbd/rbd/archive)]
| | o- portals .................................................. [1 Portal]
| | o- 172.16.1.16:3260 .............................. [OK, iser disabled]
| o- iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant ............... [2 TPGs]
| | o- tpg1 ........................................................ [enabled]
| | | o- acls ........................................................ [1 ACL]
| | | | o- iqn.1996-04.de.suse:01:e6ca28cc9f20 ................ [1 Mapped LUN]
| | | | o- mapped_lun0 ......................................... [lun0 (rw)]
| | | o- luns ........................................................ [1 LUN]
| | | | o- lun0 ............................... [rbd/city (/dev/rbd/rbd/city)]
| | | o- portals .................................................. [1 Portal]
| | | o- 172.16.1.16:3260 .............................. [OK, iser disabled]
| | o- tpg2 ....................................................... [disabled]
| | o- acls ........................................................ [1 ACL]
| | | o- iqn.1996-04.de.suse:01:e6ca28cc9f20 ................ [1 Mapped LUN]
| | | o- mapped_lun0 ......................................... [lun0 (rw)]
| | o- luns ........................................................ [1 LUN]
| | | o- lun0 ............................... [rbd/city (/dev/rbd/rbd/city)]
| | o- portals .................................................. [1 Portal]
| | o- 172.16.1.17:3260 .............................. [OK, iser disabled]
| o- iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant-too ........... [2 TPGs]
| o- tpg1 ........................................................ [enabled]
| | o- acls ........................................................ [1 ACL]
| | | o- iqn.1996-04.de.suse:01:e6ca28cc9f24 ................ [1 Mapped LUN]
| | | o- mapped_lun0 ......................................... [lun0 (rw)]
| | o- luns ........................................................ [1 LUN]
| | | o- lun0 ............................... [rbd/wood (/dev/rbd/rbd/wood)]
| | o- portals .................................................. [1 Portal]
| | o- 172.16.2.16:3260 .............................. [OK, iser disabled]
| o- tpg2 ....................................................... [disabled]
| o- acls ........................................................ [1 ACL]
| | o- iqn.1996-04.de.suse:01:e6ca28cc9f24 ................ [1 Mapped LUN]
| | o- mapped_lun0 ......................................... [lun0 (rw)]
| o- luns ........................................................ [1 LUN]
| | o- lun0 ............................... [rbd/wood (/dev/rbd/rbd/wood)]
| o- portals .................................................. [1 Portal]
| o- 172.16.2.17:3260 .............................. [OK, iser disabled]
o- loopback ...................................................... [0 Targets]
o- qla2xxx ....................................................... [0 Targets]
o- tcm_fc ........................................................ [0 Targets]
o- vhost ......................................................... [0 Targets]
Note the nonredundant target shares the archive image. Both redundant targets have two TPGs each. The first is enabled and the second disabled.
The configuration for igw2 shows
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- fileio ............................................... [0 Storage Object]
| o- iblock ............................................... [0 Storage Object]
| o- pscsi ................................................ [0 Storage Object]
| o- rbd ................................................. [2 Storage Objects]
| | o- city .................................... [/dev/rbd/rbd/city activated]
| | o- wood .................................... [/dev/rbd/rbd/wood activated]
| o- rd_mcp ............................................... [0 Storage Object]
o- ib_srpt ....................................................... [0 Targets]
o- iscsi ......................................................... [2 Targets]
| o- iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant ............... [2 TPGs]
| | o- tpg1 ....................................................... [disabled]
| | | o- acls ........................................................ [1 ACL]
| | | | o- iqn.1996-04.de.suse:01:e6ca28cc9f20 ................ [1 Mapped LUN]
| | | | o- mapped_lun0 ......................................... [lun0 (rw)]
| | | o- luns ........................................................ [1 LUN]
| | | | o- lun0 ............................... [rbd/city (/dev/rbd/rbd/city)]
| | | o- portals .................................................. [1 Portal]
| | | o- 172.16.1.16:3260 .............................. [OK, iser disabled]
| | o- tpg2 ........................................................ [enabled]
| | o- acls ........................................................ [1 ACL]
| | | o- iqn.1996-04.de.suse:01:e6ca28cc9f20 ................ [1 Mapped LUN]
| | | o- mapped_lun0 ......................................... [lun0 (rw)]
| | o- luns ........................................................ [1 LUN]
| | | o- lun0 ............................... [rbd/city (/dev/rbd/rbd/city)]
| | o- portals .................................................. [1 Portal]
| | o- 172.16.1.17:3260 .............................. [OK, iser disabled]
| o- iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant-too ........... [2 TPGs]
| o- tpg1 ....................................................... [disabled]
| | o- acls ........................................................ [1 ACL]
| | | o- iqn.1996-04.de.suse:01:e6ca28cc9f24 ................ [1 Mapped LUN]
| | | o- mapped_lun0 ......................................... [lun0 (rw)]
| | o- luns ........................................................ [1 LUN]
| | | o- lun0 ............................... [rbd/wood (/dev/rbd/rbd/wood)]
| | o- portals .................................................. [1 Portal]
| | o- 172.16.2.16:3260 .............................. [OK, iser disabled]
| o- tpg2 ........................................................ [enabled]
| o- acls ........................................................ [1 ACL]
| | o- iqn.1996-04.de.suse:01:e6ca28cc9f24 ................ [1 Mapped LUN]
| | o- mapped_lun0 ......................................... [lun0 (rw)]
| o- luns ........................................................ [1 LUN]
| | o- lun0 ............................... [rbd/wood (/dev/rbd/rbd/wood)]
| o- portals .................................................. [1 Portal]
| o- 172.16.2.17:3260 .............................. [OK, iser disabled]
o- loopback ...................................................... [0 Targets]
o- qla2xxx ....................................................... [0 Targets]
o- tcm_fc ........................................................ [0 Targets]
o- vhost ......................................................... [0 Targets]
Only the redundant targets are present. For each target, tpg1 is disabled and tpg2 is enabled. Essentially, the reverse of the first gateway.
Discovery and authentication is the same as the previous example. Both redundant targets are restricted to their respective initiators. The exception is the archive image may be accessed by any initiator with the correct credentials.