Changes between Initial Version and Version 1 of EuroGate


Ignore:
Timestamp:
05/15/07 17:55:24 (11 years ago)
Author:
sternber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • EuroGate

    v1 v1  
     1= The !EuroGate Tape Backend System = 
     2 
     3== Introduction == 
     4 
     5Eurogate is a simple Tertiary Storage System which can be customized to manage Tape Backends starting from manual stackers up to sophisticated robotic systems like the SUN ACSLC. Eurogate may as well be operated in ''simulation mode'' in which it stores data on disk instead of removable media. While doing so, it still simulates the use of tapes and drives. 
     6 
     7== !EuroGate and dCache == 
     8 
     9We exclusively offer !EuroGate to enable dCache sysadmins to simulate a tape backend for SRM 2.2 evaluation purposes. The package we provide is prepared to run in simulation mode only. It is preconfigured as follows 
     10 
     11  * Only one virtual store is configured, called ''MAIN''. 
     12  * ''MAIN'' has 10 simulated drives available. 
     13  * There is only one ''volume set'' (aka ''Storage Group'') available in ''MAIN'' called ''all''. 
     14  * 200 Tapes are available in ''volume set '' ''all'', with a capacity of 20 GBytes each. 
     15 
     16Consequently the system will store up to 4 TBytes on disk. Make sure sufficient space is available. 
     17 
     18Let us know if you need a different setup. We currently don't have the capacity to provide reasonable documention on how to setup and configure the system. 
     19 
     20== Installation == 
     21 
     22 
     23=== Overview === 
     24 
     25The Eurogate package provides the HSM simulator (the server) and a client to write, read and remove files. Therefore the package must be installed on a machine which acts as the server (should have plenty of diskspace) and on all HSM-connected pool nodes, which act as clients. Eurogate depends on nothing but Java >= 1.4.2. 
     26 
     27  Please find !EuroGate (Version 1.2 {{{eurogate-1-2.noarch.rpm}}})  
     28    * in the DESY dCache repositories under {{{RPMS.tools}}} and  
     29    * on the [http://www.dcache.org/downloads/IAgree.shtml dCache.ORG download area] (Checkout for '''!EuroGate''' in the ''Available Packages'' section.) 
     30 
     31=== Server install === 
     32 
     33After installing the RPM all files can be found under {{{/opt/eurogate/}}}. Throughout this How-to this path is referred to as {{{$EuHome}}}. 
     34 
     35Now we have to create the directory where all files are stored and create a pointer to this area. 
     36 
     37{{{ 
     38    mkdir /bigdiskspace/eurogate 
     39    ln -s /bigdiskspace/eurogate $EuHome/db-2 
     40}}} 
     41 
     42Set the path to the java executable in {{{$EuHome/config/eurogateSetup}}} by modifying the line 
     43{{{ 
     44java=<path/to/bin/java> 
     45}}} 
     46 
     47Finally, the server can be started by executing 
     48{{{ 
     49> $EuHome/jobs/eurogate -logArea=/var/log start 
     50 * Creating new Database 
     51Starting eurogateDomain  6 5 4 3 2 1 0 Done (pid=27050) 
     52}}} 
     53 
     54=== Accessing and monitoring !EuroGate services === 
     55 
     56The Eurogate server is listening on 3 Tcp/Ip service ports which may be customized in {{{/opt/eurogate/config/eurogateSetup}}}. 
     57 
     58{{{ 
     59#!rst 
     60  ======================= ============== =================== =============================================================================== 
     61   **Service Type**        **Protocol**   **Default Port**       **Service Details** 
     62  ----------------------- -------------- ------------------- ------------------------------------------------------------------------------- 
     63   EuroGate Commands       EuroGate       28000                store-/restore-/remove/query - requests (used by EuroGate clients only) 
     64   Web Pages               http           23080                Monitoring (setup, drive and tape status) 
     65   Ssh Login               ssh(v1)        23125                ssh access via standard 'ssh shell' or GUI (Experts Only, see below) 
     66  ======================= ============== =================== =============================================================================== 
     67}}} 
     68 
     69  You may login into the !EuroGate system by either using a standard ssh client 
     70   {{{ssh -l admin -c blowfish -p 23125 <serverHostName>}}} 
     71  or by means of the pcells [http://www.dcache.org/downloads/gui/index.shtml GUI].[[BR]] 
     72  Please note : During regular !EuoGate operations there is no need to access the server via the Ssh Login interface. 
     73 
     74=== Testing the server === 
     75 
     76To test the simulator, you can copy files in and out by using the included Eurogate client. 
     77 
     78First, the client command needs to be prepared: 
     79{{{ 
     80> CLIENT="java -cp $EuHome/classes/cells.jar:$EuHome/classes/eurogate.jar  eurogate.gate.EuroSyncClient -host=<eurogateServer> -port=28000" 
     81 
     82    Replace <eurogateServer> by the hostname or IP number of your EuroGate server. 
     83}}} 
     84 
     85Now let's store {{{/proc/kcore}}} in the Storage Group {{{all}}} 
     86 
     87{{{ 
     88> $CLIENT write /proc/kcore all  
     893434.0.582.1177000854763 
     90}}} 
     91 
     92Upon success, the command returns a so called ''Bitfile ID''. This ID needs to be used in subsequent operations in order to uniquely identify the dataset. 
     93 
     94You may use the webinterface to watch the drives mounting a tape, doing the I/O and after a while, dismounting the tape.  
     95 
     96{{{ 
     97> $CLIENT read <Bitfile ID> /dev/null 
     98> $CLIENT remove <Bitfile ID> 
     99 
     100   Replace <BitfileID> by the ID returned by the corresponding write operation. 
     101}}} 
     102 
     103This restores the file and removes it afterwards from the system, freeing up diskspace. 
     104 
     105=== Resetting the server in case of insufficient disk space === 
     106 
     107Since Eurogate stores all files physcially on disk, you sooner or later run out of space depending on the usage. When the disk is filled up, the behaviour of the HSM simulator is undefined and it will probably stop functioning. To reset Eurogate and wipe all its content, execute 
     108{{{ 
     109> $EuHome/jobs/eurogate -logArea=/var/log stop 
     110> rm -rf $EuHome/db-2/* 
     111> $EuHome/jobs/eurogate -logArea=/var/log start 
     112}}} 
     113 
     114This operation obviously destroyes all data stored in !EuroGate. Consequently it should only be used if the corresponding dCache test system (including pnfs) is wiped as well. Otherwise dCache restore operations on old files will fail. 
     115 
     116=== Connecting dCache pools to the HSM Simulator === 
     117 
     118On each pool node, the Eurogate Package must be installed. It contains the Eurogate Client and a wrapper script suitable for dCache./ 
     119 
     120After the installation of the RPM, the java path in {{{$EuHome/jobs/euwrapper.sh}}} must be customised: 
     121 
     122{{{ 
     123JAVA=/path/to/bin/java 
     124}}} 
     125 
     126All HSM-connected pool nodes must have PNFS mounted in read-write-mode to store the storageinfo back into PNFS after successful flushing. Please note that each poolnode must be trusted by PNFS, which can be achieved by executing on the PNFS-server 
     127 
     128{{{ 
     129> echo "15" > /pnfs/fs/admin/etc/exports/trusted/<ipAdressOfPoolnode> 
     130}}} 
     131 
     132To configure the pools, remove the key/value-pair "{{{lfs=precious}}}" from the file {{{/opt/d-cache/config/<poolnodeName>.poollist}}} for pool you wish to connect. The 2nd step consists of modifying the poolsetup-file (found at {{{$PoolHomeDir/pool/setup}}}): 
     133 
     134{{{ 
     135# set the shellscript excecuted for each flush operation (replace $EuHome by the real path) 
     136hsm set osm -command=$EuHome/jobs/euwrapper.sh 
     137 
     138# again, replace $EuHome 
     139hsm set osm -eudir=$EuHome 
     140hsm set osm -euhost=<EurogateServer> 
     141hsm set osm -euport=28000 
     142 
     143# set you pnfs prefix here 
     144hsm set osm -pnfs=/pnfs/desy.de 
     145 
     146# set the max. number of active flush operations >= 1 (default: 0) 
     147st set max active 5 
     148 
     149#set the max. number of active restore operations >= 1 (default: 0) 
     150rh set max active 5 
     151}}} 
     152 
     153Finally, all HSM-connected pools must be restarted. 
     154 
     155To enable auto-restore on dCache read requests (in case the cached file has been removed from all pools), edit on the admin node the {{{/opt/d-cache/config/PoolManager.conf}}} file and restart core-dCache: 
     156 
     157{{{ 
     158rc set stage on 
     159}}}