OpenIndiana as ISCSI Target
When one of your servers has a ton of storage and the others all the RAM, then comes the time to share that storage via the network.
NFS is the first choice in most cases but as backing store of zones it has some drawbacks compared to ZFS. Unfortunately ZFS is not network shared like Ceph. But no problem ZFS is just a Filesystem and Volume Manager, it can be used ontop of any block storage. Thankfully exporting Block storage is easy in illumos with “scsi target mode framework (smtf)”. With it you can export any Block storage including of cource our beloved ZFS Volumes over the network. smtf also supports iSCSI. As iSCSI on OpenIndiana lacks guides I figured I would write up this post to guide people through the Process of setting up smtf with iSCSI.
We will have two nodes. A Target (Host/Server) with all the storage we could need and a Initiator (Client) where we want to run the zones.
Step 1 Create a LUN (Exported Disk)
First we need a block storage to export. Create a ZFS Volume for that. Please note that we will have to sync block sizes between stmf and zfs. Zfs by default uses 8K for Performance but stmf can only handle up to 4K. Thus we need to specify -o volblocksize=4K manually here. Otherwise you will get poor performance. I am planing on putting zones on the Lun so compression and deduplication here will count for the whole pool on the consumer/initiator side. Compression and dedup are up to you however. Use what you need. Only volblocksize is important.
zfs create -V 1000G -o volblocksize=4K -o compression=lz4 -o dedup=on rpool/zonelun0
This will create a Volume of one Terrabyte on the Rootpool with the name zonelun0. Now we need to let smtf know of the Block storage. Note the blk argument. this must be equal to the volblksize in the zfs volume.
stmfadm create-lu -p blk=4096 /dev/zvol/dsk/rpool/zonelun0
Now we can have a look at what smtf sees with list-lu. We should see something like the following.
~# stmfadm list-lu -v LU Name: 600144F0AD85453600005B9188B40001 Operational Status: Offline Provider Name : sbd Alias : /dev/zvol/dsk/rpool/zonelun0 View Entry Count : 0 Data File : /dev/zvol/dsk/rpool/zonelun0 Meta File : not set Size : 1073741824000 Block Size : 512 Management URL : not set Vendor ID : SUN Product ID : COMSTAR Serial Num : not set Write Protect : Disabled Writeback Cache : Disabled Access State : Active
Note that our new LUN has a name of 600144F0AD85453600005B9188B40001. This will be used later on.
Step 2 Make LUN visible (View)
Now that we have a LUN we need to define who can see it. As we have no security needs atm we make it visible to everybody.
stmfadm add-view 600144F0AD85453600005B9188B40001 #Note the LUN name from step 1 ~# stmfadm list-view -l 600144F0AD85453600005B9188B40001 View Entry: 0 Host group : All Target group : All LUN : 0
Step 3 Create the Target
Now that we have the basic LUN setup we can create the iSCSI Target itself
First check if STMF service is running:
~# svcs -l stmf fmri svc:/system/stmf:default name STMF enabled true state online next_state none state_time September 7, 2018 at 12:09:11 AM CEST logfile /var/svc/log/system-stmf:default.log restarter svc:/system/svc/restarter:default dependency require_all/none svc:/system/filesystem/local:default (online)
if not enable it
svcadm enable stmf
Install and enable the iSCSI Target Service
pkg install network/iscsi/target svcadm enable -r svc:/network/iscsi/target:default
create a new target
~# itadm list-target -v TARGET NAME STATE SESSIONS iqn.2010-08.org.illumos:02:6343e9ea-0184-cb7c-be14-eaa7fbaad11a online 0 alias: - auth: none (defaults) targetchapuser: - targetchapsecret: unset tpg-tags: default
create target portal group (TGP) and stmf target group
itadm create-tpg iscsi01 192.168.1.100:3260 #Note use your ip here stmfadm create-tg iscsi-tg01 svcadm disable stmf stmfadm add-tg-member -g iscsi-tg01 iqn.2010-08.org.illumos:02:6343e9ea-0184-cb7c-be14-eaa7fbaad11a svcadm enable stmf
Now that we have the target we just need to login into the target and get the LUN mapped.
iscsiadm add discovery-address 192.168.1.100 iscsiadm modify discovery -t enable
Now we have a list of targets exposed by our Server
~# iscsiadm list target Target: iqn.2010-08.org.illumos:02:6343e9ea-0184-cb7c-be14-eaa7fbaad11a Alias: - TPGT: 1 ISID: 4000002a0000 Connections: 1
And we see our LUN
~# iscsiadm list target -S -v Target: iqn.2010-08.org.illumos:02:6343e9ea-0184-cb7c-be14-eaa7fbaad11a Alias: - TPGT: 1 ISID: 4000002a0000 Connections: 1 CID: 0 IP address (Local): 192.168.1.5:36981 IP address (Peer): 192.168.1.100:3260 Discovery Method: SendTargets Login Parameters (Negotiated): Data Sequence In Order: yes Data PDU In Order: yes Default Time To Retain: 20 Default Time To Wait: 2 Error Recovery Level: 0 First Burst Length: 65536 Immediate Data: yes Initial Ready To Transfer (R2T): yes Max Burst Length: 262144 Max Outstanding R2T: 1 Max Receive Data Segment Length: 32768 Max Connections: 1 Header Digest: NONE Data Digest: NONE LUN: 0 Vendor: SUN Product: COMSTAR OS Device Name: /dev/rdsk/c0t600144F0AD85453600005B9188B40001d0s2
Step Format and Profit
zpool create zonespool c0t600144F0AD85453600005B9188B40001d0
And now have fun with your iSCSI backed ZFS Pool