OpenIndiana as ISCSI Target
When one of your servers has a ton of storage and the others all the RAM, then comes the time to share that storage via the network.
NFS is the first choice in most cases but as backing store of zones it has some drawbacks compared to ZFS. Unfortunately ZFS is not network shared like Ceph. But no problem ZFS is just a Filesystem and Volume Manager, it can be used ontop of any block storage. Thankfully exporting Block storage is easy in illumos with “scsi target mode framework (smtf)”. With it you can export any Block storage including of cource our beloved ZFS Volumes over the network. smtf also supports iSCSI. As iSCSI on OpenIndiana lacks guides I figured I would write up this post to guide people through the Process of setting up smtf with iSCSI.
Overview
We will have two nodes. A Target (Host/Server) with all the storage we could need and a Initiator (Client) where we want to run the zones.
Target Setup
Step 1 Create a LUN (Exported Disk)
First we need a block storage to export. Create a ZFS Volume for that. Please note that we will have to sync block sizes between stmf and zfs. Zfs by default uses 8K for Performance but stmf can only handle up to 4K. Thus we need to specify -o volblocksize=4K manually here. Otherwise you will get poor performance. I am planing on putting zones on the Lun so compression and deduplication here will count for the whole pool on the consumer/initiator side. Compression and dedup are up to you however. Use what you need. Only volblocksize is important.
zfs create -V 1000G -o volblocksize=4K -o compression=lz4 -o dedup=on rpool/zonelun0
This will create a Volume of one Terrabyte on the Rootpool with the name zonelun0. Now we need to let smtf know of the Block storage. Note the blk argument. this must be equal to the volblksize in the zfs volume.
stmfadm create-lu -p blk=4096 /dev/zvol/dsk/rpool/zonelun0
Now we can have a look at what smtf sees with list-lu. We should see something like the following.
~# stmfadm list-lu -v
LU Name: 600144F0AD85453600005B9188B40001
Operational Status: Offline
Provider Name : sbd
Alias : /dev/zvol/dsk/rpool/zonelun0
View Entry Count : 0
Data File : /dev/zvol/dsk/rpool/zonelun0
Meta File : not set
Size : 1073741824000
Block Size : 512
Management URL : not set
Vendor ID : SUN
Product ID : COMSTAR
Serial Num : not set
Write Protect : Disabled
Writeback Cache : Disabled
Access State : Active
Note that our new LUN has a name of 600144F0AD85453600005B9188B40001. This will be used later on.
Step 2 Make LUN visible (View)
Now that we have a LUN we need to define who can see it. As we have no security needs atm we make it visible to everybody.
stmfadm add-view 600144F0AD85453600005B9188B40001 #Note the LUN name from step 1
~# stmfadm list-view -l 600144F0AD85453600005B9188B40001
View Entry: 0
Host group : All
Target group : All
LUN : 0
Step 3 Create the Target
Now that we have the basic LUN setup we can create the iSCSI Target itself
First check if STMF service is running:
~# svcs -l stmf
fmri svc:/system/stmf:default
name STMF
enabled true
state online
next_state none
state_time September 7, 2018 at 12:09:11 AM CEST
logfile /var/svc/log/system-stmf:default.log
restarter svc:/system/svc/restarter:default
dependency require_all/none svc:/system/filesystem/local:default (online)
if not enable it
svcadm enable stmf
Install and enable the iSCSI Target Service
pkg install network/iscsi/target
svcadm enable -r svc:/network/iscsi/target:default
create a new target
itadm create-target
~# itadm list-target -v
TARGET NAME STATE SESSIONS
iqn.2010-08.org.illumos:02:6343e9ea-0184-cb7c-be14-eaa7fbaad11a online 0
alias: -
auth: none (defaults)
targetchapuser: -
targetchapsecret: unset
tpg-tags: default
create target portal group (TGP) and stmf target group
itadm create-tpg iscsi01 192.168.1.100:3260 #Note use your ip here
stmfadm create-tg iscsi-tg01
svcadm disable stmf
stmfadm add-tg-member -g iscsi-tg01 iqn.2010-08.org.illumos:02:6343e9ea-0184-cb7c-be14-eaa7fbaad11a
svcadm enable stmf
Initiator Setup
Step 1
Now that we have the target we just need to login into the target and get the LUN mapped.
iscsiadm add discovery-address 192.168.1.100
iscsiadm modify discovery -t enable
Now we have a list of targets exposed by our Server
~# iscsiadm list target
Target: iqn.2010-08.org.illumos:02:6343e9ea-0184-cb7c-be14-eaa7fbaad11a
Alias: -
TPGT: 1
ISID: 4000002a0000
Connections: 1
And we see our LUN
~# iscsiadm list target -S -v
Target: iqn.2010-08.org.illumos:02:6343e9ea-0184-cb7c-be14-eaa7fbaad11a
Alias: -
TPGT: 1
ISID: 4000002a0000
Connections: 1
CID: 0
IP address (Local): 192.168.1.5:36981
IP address (Peer): 192.168.1.100:3260
Discovery Method: SendTargets
Login Parameters (Negotiated):
Data Sequence In Order: yes
Data PDU In Order: yes
Default Time To Retain: 20
Default Time To Wait: 2
Error Recovery Level: 0
First Burst Length: 65536
Immediate Data: yes
Initial Ready To Transfer (R2T): yes
Max Burst Length: 262144
Max Outstanding R2T: 1
Max Receive Data Segment Length: 32768
Max Connections: 1
Header Digest: NONE
Data Digest: NONE
LUN: 0
Vendor: SUN
Product: COMSTAR
OS Device Name: /dev/rdsk/c0t600144F0AD85453600005B9188B40001d0s2
Step Format and Profit
zpool create zonespool c0t600144F0AD85453600005B9188B40001d0
And now have fun with your iSCSI backed ZFS Pool