Changes between Initial Version and Version 1 of EuroGate

Timestamp:

05/15/07 17:55:24 (13 years ago)

Author:

sternber

Comment:

--

EuroGate

@@ +1 @@
+= The !EuroGate Tape Backend System =
+
+== Introduction ==
+
+Eurogate is a simple Tertiary Storage System which can be customized to manage Tape Backends starting from manual stackers up to sophisticated robotic systems like the SUN ACSLC. Eurogate may as well be operated in ''simulation mode'' in which it stores data on disk instead of removable media. While doing so, it still simulates the use of tapes and drives.
+
+== !EuroGate and dCache ==
+
+We exclusively offer !EuroGate to enable dCache sysadmins to simulate a tape backend for SRM 2.2 evaluation purposes. The package we provide is prepared to run in simulation mode only. It is preconfigured as follows
+
+  * Only one virtual store is configured, called ''MAIN''.
+  * ''MAIN'' has 10 simulated drives available.
+  * There is only one ''volume set'' (aka ''Storage Group'') available in ''MAIN'' called ''all''.
+  * 200 Tapes are available in ''volume set '' ''all'', with a capacity of 20 GBytes each.
+
+Consequently the system will store up to 4 TBytes on disk. Make sure sufficient space is available.
+
+Let us know if you need a different setup. We currently don't have the capacity to provide reasonable documention on how to setup and configure the system.
+
+== Installation ==
+
+
+=== Overview ===
+
+The Eurogate package provides the HSM simulator (the server) and a client to write, read and remove files. Therefore the package must be installed on a machine which acts as the server (should have plenty of diskspace) and on all HSM-connected pool nodes, which act as clients. Eurogate depends on nothing but Java >= 1.4.2.
+
+  Please find !EuroGate (Version 1.2 {{{eurogate-1-2.noarch.rpm}}}) 
+    * in the DESY dCache repositories under {{{RPMS.tools}}} and 
+    * on the [http://www.dcache.org/downloads/IAgree.shtml dCache.ORG download area] (Checkout for '''!EuroGate''' in the ''Available Packages'' section.)
+
+=== Server install ===
+
+After installing the RPM all files can be found under {{{/opt/eurogate/}}}. Throughout this How-to this path is referred to as {{{$EuHome}}}.
+
+Now we have to create the directory where all files are stored and create a pointer to this area.
+
+{{{
+    mkdir /bigdiskspace/eurogate
+    ln -s /bigdiskspace/eurogate $EuHome/db-2
+}}}
+
+Set the path to the java executable in {{{$EuHome/config/eurogateSetup}}} by modifying the line
+{{{
+java=<path/to/bin/java>
+}}}
+
+Finally, the server can be started by executing
+{{{
+> $EuHome/jobs/eurogate -logArea=/var/log start
+ * Creating new Database
+Starting eurogateDomain  6 5 4 3 2 1 0 Done (pid=27050)
+}}}
+
+=== Accessing and monitoring !EuroGate services ===
+
+The Eurogate server is listening on 3 Tcp/Ip service ports which may be customized in {{{/opt/eurogate/config/eurogateSetup}}}.
+
+{{{
+#!rst
+  ======================= ============== =================== ===============================================================================
+   **Service Type**        **Protocol**   **Default Port**       **Service Details**
+  ----------------------- -------------- ------------------- -------------------------------------------------------------------------------
+   EuroGate Commands       EuroGate       28000                store-/restore-/remove/query - requests (used by EuroGate clients only)
+   Web Pages               http           23080                Monitoring (setup, drive and tape status)
+   Ssh Login               ssh(v1)        23125                ssh access via standard 'ssh shell' or GUI (Experts Only, see below)
+  ======================= ============== =================== ===============================================================================
+}}}
+
+  You may login into the !EuroGate system by either using a standard ssh client
+   {{{ssh -l admin -c blowfish -p 23125 <serverHostName>}}}
+  or by means of the pcells [http://www.dcache.org/downloads/gui/index.shtml GUI].[[BR]]
+  Please note : During regular !EuoGate operations there is no need to access the server via the Ssh Login interface.
+
+=== Testing the server ===
+
+To test the simulator, you can copy files in and out by using the included Eurogate client.
+
+First, the client command needs to be prepared:
+{{{
+> CLIENT="java -cp $EuHome/classes/cells.jar:$EuHome/classes/eurogate.jar  eurogate.gate.EuroSyncClient -host=<eurogateServer> -port=28000"
+
+    Replace <eurogateServer> by the hostname or IP number of your EuroGate server.
+}}}
+
+Now let's store {{{/proc/kcore}}} in the Storage Group {{{all}}}
+
+{{{
+> $CLIENT write /proc/kcore all 
+3434.0.582.1177000854763
+}}}
+
+Upon success, the command returns a so called ''Bitfile ID''. This ID needs to be used in subsequent operations in order to uniquely identify the dataset.
+
+You may use the webinterface to watch the drives mounting a tape, doing the I/O and after a while, dismounting the tape. 
+
+{{{
+> $CLIENT read <Bitfile ID> /dev/null
+> $CLIENT remove <Bitfile ID>
+
+   Replace <BitfileID> by the ID returned by the corresponding write operation.
+}}}
+
+This restores the file and removes it afterwards from the system, freeing up diskspace.
+
+=== Resetting the server in case of insufficient disk space ===
+
+Since Eurogate stores all files physcially on disk, you sooner or later run out of space depending on the usage. When the disk is filled up, the behaviour of the HSM simulator is undefined and it will probably stop functioning. To reset Eurogate and wipe all its content, execute
+{{{
+> $EuHome/jobs/eurogate -logArea=/var/log stop
+> rm -rf $EuHome/db-2/*
+> $EuHome/jobs/eurogate -logArea=/var/log start
+}}}
+
+This operation obviously destroyes all data stored in !EuroGate. Consequently it should only be used if the corresponding dCache test system (including pnfs) is wiped as well. Otherwise dCache restore operations on old files will fail.
+
+=== Connecting dCache pools to the HSM Simulator ===
+
+On each pool node, the Eurogate Package must be installed. It contains the Eurogate Client and a wrapper script suitable for dCache./
+
+After the installation of the RPM, the java path in {{{$EuHome/jobs/euwrapper.sh}}} must be customised:
+
+{{{
+JAVA=/path/to/bin/java
+}}}
+
+All HSM-connected pool nodes must have PNFS mounted in read-write-mode to store the storageinfo back into PNFS after successful flushing. Please note that each poolnode must be trusted by PNFS, which can be achieved by executing on the PNFS-server
+
+{{{
+> echo "15" > /pnfs/fs/admin/etc/exports/trusted/<ipAdressOfPoolnode>
+}}}
+
+To configure the pools, remove the key/value-pair "{{{lfs=precious}}}" from the file {{{/opt/d-cache/config/<poolnodeName>.poollist}}} for pool you wish to connect. The 2nd step consists of modifying the poolsetup-file (found at {{{$PoolHomeDir/pool/setup}}}):
+
+{{{
+# set the shellscript excecuted for each flush operation (replace $EuHome by the real path)
+hsm set osm -command=$EuHome/jobs/euwrapper.sh
+
+# again, replace $EuHome
+hsm set osm -eudir=$EuHome
+hsm set osm -euhost=<EurogateServer>
+hsm set osm -euport=28000
+
+# set you pnfs prefix here
+hsm set osm -pnfs=/pnfs/desy.de
+
+# set the max. number of active flush operations >= 1 (default: 0)
+st set max active 5
+
+#set the max. number of active restore operations >= 1 (default: 0)
+rh set max active 5
+}}}
+
+Finally, all HSM-connected pools must be restarted.
+
+To enable auto-restore on dCache read requests (in case the cached file has been removed from all pools), edit on the admin node the {{{/opt/d-cache/config/PoolManager.conf}}} file and restart core-dCache:
+
+{{{
+rc set stage on
+}}}