rather old? - update #

sorry, this wiki-page is rather old. today we don't need self-compiling anymore, so here are 2 links which show the basic setup by using allready distribution-integrated packages:

literature #

  • http://www.openfiler.com/
    • " You bring the hardware, any industry standard server will do, combine it with our Openfiler software and the result is a powerful networked storage solution that exports your data via a full suite of industry standard storage networking protocols. Openfiler lowers deployment and maintenance costs for networked storage without compromising functionality or performance."(http://www.openfiler.com/about)

terminology / basics#

  • iSCSI target : is the server piece of an iSCSI SAN.
  • iSCSI initiator: the client piece/driver is called the initiator


yep, wikipedia knows interesting basics about iSCSI too http://en.wikipedia.org/wiki/ISCSI.

setting it up#

requirements#

for playing you need:

  • iSCSI Target: iSCSI Enterprise Target http://sourceforge.net/project/showfiles.php?group_id=108475
    • The aim of the project is to develop an open source iSCSI target with professional features, that works well in enterprise environment under real workload, and is scalable and versatile enough to meet the challenge of future storage needs and developements.
  • http://www.open-iscsi.org/ most popular iSCSI initiator under linux
    • Open-iSCSI project is a high performance, transport independent, multi-platform implementation of RFC3720.

  • recent kernel-sources (>=2.6.14, in the following i use 2.6.18), gcc etc. if you use debian just do a apt-get install kernel-package build-essential fakeroot to get all you need

Part 1.0 - building a shared-storage with one storage-box and 2 iSCSI-clients#

the following is "typescript" of what i've done in lab so far. it's Part 1 of the "masterplan" ;-)

i begin with a simple setup for building a shared storage between 2 nodes. therefore i need 3 machines (1x storage (target), 2x client (initiator)). of course, using one storage-box is a single-point-of-failure but the HA will be build in a next step.

### my setup ###

all machines with:

  os:     ubuntu 6.06.1
  kernel: linux kernel 2.6.18.1 

  iscsi-tools: iscsitarget-0.4.13 (target), "inofficial" (?) open-iscsi-2.0-711 (initiator)

machines, function, IPs:

target 1:    10.10.59.104 (hostname: ubuntu8) # this is the "backend" / storage-box or "target" in iSCSI terms
initiator 1: 10.10.59.106 (hostname: ubuntu2)
initiator 2: 10.10.59.97  (hostname: ubuntu3) # for playing with shared-storage and cluster-fs

# none of the machines has a free block device, so i just take its swap-partitions (/dev/sda5)
# for iSCSI export on the targets. (swapoff -a; fdisk /dev/sda5 -> changing part-type from swap to linux) :-)

target#

at first we need a box exporting the storage, this one is called "target" in iSCSI-terms.

# make sure you have kernel-sources / headers linked to /usr/src/linux , gcc etc.

# Initiator

wget htp:// ... iscsitarget-0.4.13.tar.gz

cd /usr/local/src/iscsi

make KERNELSRC=/usr/src/linux
make KERNELSRC=/usr/src/linux install

# under debian the above install script puts the iscsi_tgt module
# under /lib/modules/kernel/iscsi what is obviously wrong and
# leads to "iscsi_tgt not found" error...
#
# suiting make output of this "bug":
# [...]
#„etc/initd/initd.debian“ -> „/etc/init.d/iscsi-target“
#install: Verzeichnis „/lib/modules//kernel“ angelegt
#install: Verzeichnis „/lib/modules//kernel/iscsi“ angelegt
#„kernel/iscsi_trgt.ko“ -> „/lib/modules//kernel/iscsi/iscsi_trgt.ko“
#depmod -aq
# [...]
#
# well, doing a `mv /lib/modules/kernel/iscsi/ /lib/modules/$(uname -r)/ && depmod -a`
# helps here :-)

cp /usr/local/src/iscsi/etc/ietd.conf /etc

# setup the ietd.conf file

vi /etc/ietd.conf

  # change the line Lun 0 Path= that it point to your iSCSI disk or partition

  Lun 0 Path=/dev/sda5 # i took a swap-partition here ;-) (swapoff -a) ...
  
  # and give a unique Targetname etc., e.g:

  Target iqn.2006-10.de.storage:storage.disk1.sys1.xyz
        Lun 0 Path=/dev/sda5,Type=fileio
        Alias test1

# start the beast

/etc/init.d/iscsi-target start

# check if module is loaded

# lsmod | grep -i iscs
iscsi_trgt             64092  4

# ok, all done :-) 
# now install an iscsi initiator on one ore more other boxes and have fun -
# for linux open-iSCSI is recommended!
# (yep, ms boxes should also work fine (free download? dunnot use ms ;-))

initiator#

second we need one ore more boxes importing the storage, this is called "initiator" in iSCSI terms.

notice:

but nevertheless, the question of which iSCSI-initiator to use or what's the official release for kernel 2.6.18 shouldn't prevent us from trying it out. it's just too cool! :-) in the following i'm using kernel 2.6.18.1 and the "inofficial" (?) open-iscsi-2.0-711:

### PART1: installing & configuring the initiator
###        - discovering the target, connecting to it and using it


# get and extract the open-iscsi sources

wget ...  open-iscsi-1.0-485.tar.gz # for kernel < 2.6.16 && > 2.6.11
wget ... open-iscsi-2.0-711.tar.gz  # for kernel >= 2.6.16

cp  open-iscsi-2.0-711.tar.gz /usr/local/src/ && cd /usr/local/src/
tar -xvzf  open-iscsi-2.0-711.tar.gz
cd  open-iscsi-2.0-711

# compile it

make KSRC=<kernel-src> # if you want to compile for the actual kernel, simply `make` is alright
                       # btw. you can even cross-compile via
                       # `make KSRC=<kernel-src> KARCH="ARCH=um"`, see README


[...]
make[2]: Leaving directory `/usr/src/linux-2.6.18.1-kf01'
make[1]: Leaving directory `/usr/local/src/open-iscsi-2.0-711/kernel'

Compilation complete                Output file
----------------------------------- ----------------
Built iSCSI Open Interface module:  kernel/scsi_transport_iscsi.ko
Built iSCSI library module:         kernel/libiscsi.ko
Built iSCSI over TCP kernel module: kernel/iscsi_tcp.ko
Built iSCSI daemon:                 usr/iscsid
Built management application:       usr/iscsiadm

Read README file for detailed information.
[...]


make install # things go to /sbin, /etc/init.d/, /lib/modules/<kernel>

# The default configuration file is /etc/iscsi/iscsid.conf. This file contains
# only configuration that could be overwritten by iSCSI Discovery,
# or manualy updated via iscsiadm utility.

vi /etc/init.d/open-iscsi 
  # change the path to iscsid
  #DAEMON=/usr/sbin/iscsid
  DAEMON=/sbin/iscsid

# start the thing...

root@ubuntu2:/etc/iscsi# /etc/init.d/open-iscsi start
Starting iSCSI initiator service: Error: initiatorname file doesn't exist! default [/etc/iscsi/initiatorname.iscsi]

# whoops. we must create a unique initiatorname.iscsi first!

vi /etc/iscsi/initiatorname.iscsi # the following is an example i use, change to suit your needs

root@ubuntu2:/etc/iscsi# cat /etc/iscsi/initiatorname.iscsi
  ## DO NOT EDIT OR REMOVE THIS FILE!
  ## If you remove this file, the iSCSI daemon will not start.
  ## If you change the InitiatorName, existing access control lists
  ## may reject this initiator.  The InitiatorName must be unique
  ## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.

  # - iqn is static, it must be there.
  # - 2006-10 is the year and month where the domain name below became valid.
  # - de.storagebeast the domain name in "reverse order", the domain here is anything.net..
  # - :01 is the initiator number.
  #
  # 000000000001 is an arbitrary value to make the name unique.

  # look at http://en.opensuse.org/Open-iSCSI_and_SUSE_Linux#Add_the_initiatorname_config_file
  # for a good description

  InitiatorName=iqn.2006-10.de.storagebeast:01.000000000001


# next try to start the beast...

root@ubuntu2:/etc/iscsi# /etc/init.d/open-iscsi start
Starting iSCSI initiator service: succeeded.

# test discovery and connectivity

# to discover, connect, edit and delete targets from the initiator the tool iscsiadm
# is used (See man iscsiadm)
# of course, we need to know the target's IP address and iSCSI port (standard is TCP port 3260).
# (see "target" section above)


root@ubuntu2:/etc/iscsi# iscsiadm -m discovery -t st -p 10.10.59.104
10.10.59.104:3260,1 iqn.2006-10.de.storage:storage.disk1.sys1.xyz

# yea!!! seems to work! :-)

# too see the target was put into the initiator's database issue:
root@ubuntu2:/etc/iscsi# iscsiadm -m discovery
10.10.59.104:3260 via sendtargets

root@ubuntu2:/etc/iscsi# iscsiadm -m node
10.10.59.104:3260,1 iqn.2006-10.de.storage:storage.disk1.sys1.xyz

# now we establish a session (login)

iscsiadm --mode node --targetname iqn.2006-10.de.storage:storage.disk1.sys1.xyz --portal 10.10.59.104:3260 --login

# via dmesg or tail -25 /var/log/messages we see the disk

[  417.456000] Loading iSCSI transport class v2.0-711.
[  417.480000] iscsi: registered transport (tcp)
[ 1606.920000] scsi2 : iSCSI Initiator over TCP/IP
[ 1607.180000]   Vendor: IET       Model: VIRTUAL-DISK      Rev: 0
[ 1607.180000]   Type:   Direct-Access                      ANSI SCSI revision: 04
[ 1607.180000] SCSI device sdb: 5863662 512-byte hdwr sectors (3002 MB)
[ 1607.180000] sdb: Write Protect is off
[ 1607.180000] sdb: Mode Sense: 77 00 00 08
[ 1607.180000] SCSI device sdb: drive cache: write back
[ 1607.180000] SCSI device sdb: 5863662 512-byte hdwr sectors (3002 MB)
[ 1607.180000] sdb: Write Protect is off
[ 1607.180000] sdb: Mode Sense: 77 00 00 08
[ 1607.180000] SCSI device sdb: drive cache: write back
[ 1607.180000]  sdb: unknown partition table
[ 1607.208000] sd 2:0:0:0: Attached scsi disk sdb
[ 1607.208000] sd 2:0:0:0: Attached scsi generic sg1 type 0

# and can use it

root@ubuntu2:/etc/iscsi# lsscsi
[0:0:0:0]    disk    ATA      SAMSUNG HD080HJ  WT10  -
[2:0:0:0]    disk    IET      VIRTUAL-DISK     0     -

root@ubuntu2:/etc/iscsi# fdisk -l /dev/sdb

Disk /dev/sdb: 3002 MB, 3002194944 bytes
93 heads, 62 sectors/track, 1016 cylinders
Units = cylinders of 5766 * 512 = 2952192 bytes

Disk /dev/sdb doesn't contain a valid partition table

# it's a good idea now to put a partition on it :-)
# btw. remember "the abstraction" here. what we export IS just
# a partition (the swap we deactivated on the target) - 
# but we see it as a whole scsi disk. think about using 
# LVM etc. as exported target instead of a dull partition - 
# pretty and cool, isn't it?

root@ubuntu2:/etc/iscsi# fdisk -l /dev/sdb

Disk /dev/sdb: 3002 MB, 3002194944 bytes
93 heads, 62 sectors/track, 1016 cylinders
Units = cylinders of 5766 * 512 = 2952192 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1016     2929097   83  Linux

# putting filesystem on it and mounting the disk

root@ubuntu2:/etc/iscsi# mkfs -t ext3 /dev/sdb1 && mnt /dev/sdb1 /mnt

# done! now we're going to put a cluster-filesystem
# on it...

Part 1.2 - setting a ClusterFS on top #

### PART1.2: - building a "shared-storage" by putting a cluster-file-system
               (here ocfs2) on top :-)

setup:

 - 2nd initiator: 10.10.59.97 (ubuntu3); same config as the first (10.10.59.16) with one exception,

# ok, first step is equal to PART1
# same config as for initiator1, except for /etc/iscsi/initiatorname.iscsi (initiatornames must be unique!)

root@ubuntu3:/usr/local/src/open-iscsi-2.0-711# scp 10.10.59.106:/etc/iscsi/initiator* /etc/iscsi/
initiatorname.iscsi                                                                    

# changing the initiator-name to e.g.:

InitiatorName=iqn.2006-10.de.storagebeast:02.000000000002

# after starting, discovering & connecting (see above)
# we are connected to the same storage as initiator1 (10.10.59.106)

[ 1165.436000] scsi2 : iSCSI Initiator over TCP/IP
[ 1165.708000]   Vendor: IET       Model: VIRTUAL-DISK      Rev: 0
[ 1165.708000]   Type:   Direct-Access                      ANSI SCSI revision: 04
[ 1165.708000] SCSI device sdb: 5863662 512-byte hdwr sectors (3002 MB)
[ 1165.708000] sdb: Write Protect is off
[ 1165.708000] sdb: Mode Sense: 77 00 00 08
[ 1165.708000] SCSI device sdb: drive cache: write back
[ 1165.708000] SCSI device sdb: 5863662 512-byte hdwr sectors (3002 MB)
[ 1165.712000] sdb: Write Protect is off
[ 1165.712000] sdb: Mode Sense: 77 00 00 08
[ 1165.712000] SCSI device sdb: drive cache: write back
[ 1165.712000]  sdb: sdb1
[ 1165.712000] sd 2:0:0:0: Attached scsi disk sdb
[ 1165.712000] sd 2:0:0:0: Attached scsi generic sg1 type 0

root@ubuntu3:/usr/local/src/open-iscsi-2.0-711# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: SAMSUNG HD080HJ  Rev: WT10
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: IET      Model: VIRTUAL-DISK     Rev: 0
  Type:   Direct-Access                    ANSI SCSI revision: 04

root@ubuntu3:/usr/local/src/open-iscsi-2.0-711# fdisk -l /dev/sdb

Disk /dev/sdb: 3002 MB, 3002194944 bytes
93 heads, 62 sectors/track, 1016 cylinders
Units = cylinders of 5766 * 512 = 2952192 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1016     2929097   83  Linux

now we're going to stack a cluster-fs on top for being able to mount the storage on both nodes at the same time and to write stuff on it at the same time! for simplicity we use OCFS2 here. a good description of its usage and setup you find in the official user-guide http://oss.oracle.com/projects/ocfs2/dist/documentation/ocfs2_users_guide.pdf. ocfs2 has one central config file: /etc/ocfs2/cluster.conf. for easiest creation and deployment we use the GUI (ocfs2console).


## get the ocfs2 tools

apt-get install ocfs2-tools ocfs2console

## setup & propagate the cluster.conf

# start the ocfs2console on one initiator
# add all cluster-nodes (here: 10.10.59.106 and 10.10.59.97)
#  -> in production environment we should use at least one dedicated interlink for this i think
# distribute the cluster.conf
# add and format the iSCSI-Storage (/dev/sdb1)

# on both nodes the config looks like this now.

root@ubuntu2:~# cat /etc/ocfs2/cluster.conf
node:
        ip_port = 7777
        ip_address = 10.10.59.106
        number = 0
        name = ubuntu2
        cluster = ocfs2

node:
        ip_port = 7777
        ip_address = 10.10.59.97
        number = 1
        name = ubuntu3
        cluster = ocfs2

cluster:
        node_count = 2
        name = ocfs2


## starting the cluster (we do this on each node)

/etc/init.d/o2cb status
/etc/init.d/o2cb load  # load the modules
/etc/init.d/o2cb online ocfs2 # bring cluster online

# on initiator1

root@ubuntu2:~# /etc/init.d/o2cb status
Module "configfs": Not loaded
Filesystem "configfs": Not mounted
Module "ocfs2_nodemanager": Not loaded
Module "ocfs2_dlm": Not loaded
Module "ocfs2_dlmfs": Not loaded
Filesystem "ocfs2_dlmfs": Not mounted
root@ubuntu2:~# /etc/init.d/o2cb load
Loading module "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
root@ubuntu2:~# /etc/init.d/o2cb online ocfs2
Starting cluster ocfs2: OK

# on initiator2

root@ubuntu3:/etc/ocfs2# /etc/init.d/o2cb status
Module "configfs": Not loaded
Filesystem "configfs": Not mounted
Module "ocfs2_nodemanager": Not loaded
Module "ocfs2_dlm": Not loaded
Module "ocfs2_dlmfs": Not loaded
Filesystem "ocfs2_dlmfs": Not mounted
root@ubuntu3:/etc/ocfs2# /etc/init.d/o2cb load
Loading module "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
root@ubuntu3:/etc/ocfs2# /etc/init.d/o2cb online ocfs2
Starting cluster ocfs2: OK


# mounting the shared-volume

mkdir /mnt/shared_iscsi && mount -t ocfs2 /dev/sdb1 /mnt/shared_iscsi

# on initiator 2 (yep, we could also use initiator1 at first, it just doesn't matter)

root@ubuntu3:/etc/ocfs2# mkdir /mnt/shared_iscsi && mount -t ocfs2 /dev/sdb1 /mnt/shared_iscsi

# on initiator1

root@ubuntu2:~#  mkdir /mnt/shared_iscsi && mount -t ocfs2 /dev/sdb1 /mnt/shared_iscsi

root@ubuntu2:~# df -h
Dateisystem            Größe Benut  Verf Ben% Eingehängt auf
/dev/sda1              71G  3,3G   64G   5% /
varrun                490M   92K  490M   1% /var/run
varlock               490M  4,0K  490M   1% /var/lock
udev                  490M   92K  490M   1% /dev
devshm                490M     0  490M   0% /dev/shm
/dev/sdb1             2,8G  134M  2,7G   5% /mnt/shared_iscsi

root@ubuntu2:~# mount
/dev/sda1 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw)
/sys on /sys type sysfs (rw)
varrun on /var/run type tmpfs (rw)
varlock on /var/lock type tmpfs (rw)
udev on /dev type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
devshm on /dev/shm type tmpfs (rw)
configfs on /sys/kernel/config type configfs (rw)
ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
/dev/sdb1 on /mnt/shared_iscsi type ocfs2 (rw,_netdev,heartbeat=local)

root@ubuntu2:~# /etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking cluster ocfs2: Online
Checking heartbeat: Active

# looking in /var/log/messages (here on initiator2 (ubuntu3))

[...]
Oct 18 16:29:40 ubuntu3 kernel: [ 4812.492000] OCFS2 Node Manager 1.3.3
Oct 18 16:29:40 ubuntu3 kernel: [ 4812.512000] OCFS2 DLM 1.3.3
Oct 18 16:29:40 ubuntu3 kernel: [ 4812.528000] OCFS2 DLMFS 1.3.3
Oct 18 16:29:40 ubuntu3 kernel: [ 4812.528000] OCFS2 User DLM kernel interface loaded
Oct 18 16:31:20 ubuntu3 kernel: [ 4912.552000] OCFS2 Node Manager 1.3.3
Oct 18 16:31:20 ubuntu3 kernel: [ 4912.572000] OCFS2 DLM 1.3.3
Oct 18 16:31:20 ubuntu3 kernel: [ 4912.584000] OCFS2 DLMFS 1.3.3
Oct 18 16:31:20 ubuntu3 kernel: [ 4912.584000] OCFS2 User DLM kernel interface loaded
Oct 18 16:32:24 ubuntu3 kernel: [ 4976.772000] OCFS2 1.3.3
Oct 18 16:32:24 ubuntu3 kernel: [ 4976.784000] ocfs2_dlm: Nodes in domain ("0C7185F173904713BBA27964AB03EA18"): 1
Oct 18 16:32:24 ubuntu3 kernel: [ 4976.784000] JBD: Ignoring recovery information on journal
Oct 18 16:32:24 ubuntu3 kernel: [ 4976.968000] kjournald starting.  Commit interval 5 seconds
Oct 18 16:32:24 ubuntu3 kernel: [ 4976.968000] ocfs2: Mounting device (8,17) on (node 1, slot 0) with ordered data mode.
Oct 18 16:32:54 ubuntu3 kernel: [ 5006.796000] o2net: connected to node ubuntu2 (num 0) at 10.10.59.106:7777
Oct 18 16:32:57 ubuntu3 kernel: [ 5010.072000] ocfs2_dlm: Node 0 joins domain 0C7185F173904713BBA27964AB03EA18
Oct 18 16:32:57 ubuntu3 kernel: [ 5010.072000] ocfs2_dlm: Nodes in domain ("0C7185F173904713BBA27964AB03EA18"): 0 1
[...]


# FINISHED! now we can write to the storage from both nodes at the same time (in different files ;-)
# all changes one node makes will occour "on-the-fly" for the other(s) too.
# a stress test via bonnie++ etc. would also be nice...

Part 2.0 - Clustering the Backend etc.#

ideas/options:

  • Host-Based-Mirroring
    • building a soft-raid over 2 storage-boxes on th "client" side. on top of this we can put ocfs etc.
  • DRBD & heartbeat
    • building an active/passive storage-cluster with automatic IP takeover (virtual ip) if one storage fails
  • NIC bonding on each storage-box (target)!
    • better performance
    • improved availabiltity
  • ...what else?

...sorry, TODO

Part 2.1#

Part 3.0 - Authentication & Tuning#

### PART3: Authentication & Tuning

sorry, TODO!

Testing it#

basic tests for Part 1.2#

shutting down one initiator hard#

shutting down initiator2 (ubuntu3) by powering off leads to:

Oct 18 18:57:28 ubuntu2 kernel: [14877.052000] o2net: connection to node ubuntu3 (num 1) at 10.10.59.97:7777 has been idle for 10 seconds, shutting it down.
Oct 18 18:57:28 ubuntu2 kernel: [14877.052000] (0,0):o2net_idle_timer:1297 here are some times that might help debug the situation: (tmr 1161190638.707881 now 1161190648.706481 dr 1161190638.707854 adv 1161190638.707894:1161190638.707898 func (f185710c:2) 1161190618.715914:1161190618.715916)
Oct 18 18:57:28 ubuntu2 kernel: [14877.052000] o2net: no longer connected to node ubuntu3 (num 1) at 10.10.59.97:7777
Oct 18 18:57:36 ubuntu2 kernel: [14885.316000] (12680,0):dlm_get_lock_resource:849 0C7185F173904713BBA27964AB03EA18:M00000000000000000000189cbf0ea3: at least one node (1) torecover before lock mastery can begin
Oct 18 18:57:37 ubuntu2 kernel: [14885.424000] (12148,0):dlm_get_lock_resource:849 0C7185F173904713BBA27964AB03EA18:$RECOVERY: at least one node (1) torecover before lock mastery can begin
Oct 18 18:57:37 ubuntu2 kernel: [14885.424000] (12148,0):dlm_get_lock_resource:876 0C7185F173904713BBA27964AB03EA18: recovery map is not empty, but must master $RECOVERY lock now
Oct 18 18:57:38 ubuntu2 kernel: [14886.528000] (12680,0):ocfs2_replay_journal:1184 Recovering node 1 from slot 0 on device (8,17)
Oct 18 18:57:40 ubuntu2 kernel: [14888.532000] kjournald starting.  Commit interval 5 seconds

on initiator1 (ubuntu2) all seems ok. write and read is possible. neither kernel panic nor any prob. fine :-)

bringing initiator2 back online also works:

root@ubuntu3:~# /etc/init.d/open-iscsi start
Starting iSCSI initiator service: succeeded.

root@ubuntu3:~# iscsiadm --mode node --targetname iqn.2006-10.de.storage:storage.disk1.sys1.xyz --portal 10.10.59.104:3260 --login

root@ubuntu3:~# /etc/init.d/o2cb load
Loading module "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK

root@ubuntu3:~# /etc/init.d/o2cb online ocfs2
Starting cluster ocfs2: OK

root@ubuntu3:~# mount /dev/sdb1 /mnt/shared_iscsi/

root@ubuntu3:~# mount
/dev/sda1 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw)
/sys on /sys type sysfs (rw)
varrun on /var/run type tmpfs (rw)
varlock on /var/lock type tmpfs (rw)
udev on /dev type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
devshm on /dev/shm type tmpfs (rw)
configfs on /sys/kernel/config type configfs (rw)
ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
/dev/sdb1 on /mnt/shared_iscsi type ocfs2 (rw,_netdev,heartbeat=local)

on initiator1 the join is recognized and no prob occours:

root@ubuntu2:/mnt/shared_iscsi# tail -10 /var/log/messages
Oct 18 18:57:28 ubuntu2 kernel: [14877.052000] o2net: no longer connected to node ubuntu3 (num 1) at 10.10.59.97:7777
Oct 18 18:57:36 ubuntu2 kernel: [14885.316000] (12680,0):dlm_get_lock_resource:849 0C7185F173904713BBA27964AB03EA18:M00000000000000000000189cbf0ea3: at least one node (1) torecover before lock mastery can begin
Oct 18 18:57:37 ubuntu2 kernel: [14885.424000] (12148,0):dlm_get_lock_resource:849 0C7185F173904713BBA27964AB03EA18:$RECOVERY: at least one node (1) torecover before lock mastery can begin
Oct 18 18:57:37 ubuntu2 kernel: [14885.424000] (12148,0):dlm_get_lock_resource:876 0C7185F173904713BBA27964AB03EA18: recovery map is not empty, but must master $RECOVERY lock now
Oct 18 18:57:38 ubuntu2 kernel: [14886.528000] (12680,0):ocfs2_replay_journal:1184 Recovering node 1 from slot 0 on device (8,17)
Oct 18 18:57:40 ubuntu2 kernel: [14888.532000] kjournald starting.  Commit interval 5 seconds
Oct 18 19:09:51 ubuntu2 -- MARK --
Oct 18 19:13:37 ubuntu2 kernel: [15846.220000] o2net: accepted connection from node ubuntu3 (num 1) at 10.10.59.97:7777
Oct 18 19:13:41 ubuntu2 kernel: [15850.264000] ocfs2_dlm: Node 1 joins domain 0C7185F173904713BBA27964AB03EA18
Oct 18 19:13:41 ubuntu2 kernel: [15850.264000] ocfs2_dlm: Nodes in domain ("0C7185F173904713BBA27964AB03EA18"): 0 1

masterplan - personal roadmap ;-)#

notes & questions#

  • can any block device be a target (think of loop-devices, lvm volumes etc.)?
    • what about giving a lvm-volume out and resizing it? does it work on the target side on-the-fly??
  • what abaout drbd to have some cheap kind of "srdf" with at least 2 storage-boxes (Backend / Targets)? (like presented in the idea "Xen with DRBD, GNBD and OCFS2 HOWTO" http://xenamo.sourceforge.net/ to GNBD)
    • can drbd handle more than 2 members?
  • what about software raid-5 over a bunch of machines (md)
  • what about nic-binding to get failover capabilites or even more GBit/s in Backend (Targets)??
    • NIC_bonding
    • VLANs / using different switches like in FibreChannel-Environments
  • emulating the whole thing in Xen or vmware? (at least 2 storages and 2 or more "clients")

Notes for Administration #

just some notes...

### NOTES: Administration of the iscsi stuff

### see: man iscsiadm

# discovery, 10.10.59.104 is the target (iSCSI storage box) here

root@ubuntu2:/etc/iscsi# iscsiadm -m discovery -t st -p 10.10.59.104

# login and logout

iscsiadm --mode node --targetname iqn.2006-10.de.storage:storage.disk1.sys1.xyz --portal 10.10.59.104:3260 --login
iscsiadm --mode node --targetname iqn.2006-10.de.storage:storage.disk1.sys1.xyz --portal 10.10.59.104:3260 --logout


# Display all data for a given node record:

root@ubuntu2:/etc/iscsi# iscsiadm --mode node --targetname iqn.2006-10.de.storage:storage.disk1.sys1.xyz --portal 10.10.59.104:3260
node.name = iqn.2006-10.de.storage:storage.disk1.sys1.xyz
node.transport_name = tcp
node.tpgt = 1
node.active_conn = 1
node.startup = manual
node.session.initial_cmdsn = 0
node.session.auth.authmethod = None
node.session.auth.username = <empty>
node.session.auth.password = <empty>
node.session.auth.username_in = <empty>
node.session.auth.password_in = <empty>
node.session.timeo.replacement_timeout = 120
node.session.err_timeo.abort_timeout = 10
node.session.err_timeo.reset_timeout = 30
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 0
node.session.iscsi.MaxConnections = 0
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.conn[0].address = 10.10.59.104
node.conn[0].port = 3260
node.conn[0].startup = manual
node.conn[0].tcp.window_size = 524288
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.active_timeout = 5
node.conn[0].timeo.idle_timeout = 60
node.conn[0].timeo.ping_timeout = 5
node.conn[0].timeo.noop_out_interval = 0
node.conn[0].timeo.noop_out_timeout = 0
node.conn[0].iscsi.MaxRecvDataSegmentLength = 65536
node.conn[0].iscsi.HeaderDigest = None
node.conn[0].iscsi.DataDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No

--category: CategoryStorage
--see also: GNBD NAS CategorySAN

Add new attachment

Only authorized users are allowed to upload new attachments.

List of attachments

Kind Attachment Name Size Version Date Modified Author Change note
png
ocfs2_setup_01.png 170.4 kB 1 18-Oct-2006 16:51 kafisch
png
ocfs2_setup_02.png 160.8 kB 1 18-Oct-2006 16:51 kafisch
« This page (revision-178) was last changed on 22-Jul-2011 09:56 by Sven Hessenmueller