wiki:dCacheToGo
Last modified 9 years ago Last modified on 07/27/09 14:50:49

dCache to take away

a preinstalled, fully functional dCache virtualised on your Desktop

This page describes how to obtain and use a virtual image containing a dCache server and clients. It's intended to be used as a basis for hands-on workshops on dCache and for trying it out/playing before going to install it (similar to a Linux live-CD). It is currently based on Virtual Box.

The goal is to provide you with a dCache system that works out-of-the-box, which allows testing and practice of various configuration scenarios without effecting real installations.

Download the image

Right-click the link and select "Save link as.." in the popup-menu:

dateversionsha1 hash
6.6.2008 dCacheToGo v.05 6749404efce54ff079985955a1444e471a3326e6

In the meantime, the host and the user certificates of the virtual machine are expired. Please get a new set of certificates from dcache.org and proceed as follows :

cd /tmp
[root@hal9000 tmp]# wget http://www.dcache.org/downloads/virtual/hal9000.certificates.20090721.tar 
[root@hal9000 tmp]# tar xf hal9000.certificates.20090721.tar 
[root@hal9000 tmp]# cd hal9000
[root@hal9000 hal9000]# ls
2f63e9e8.0  2f63e9e8.r0  2f63e9e8.signing_policy  hostcert.pem  hostkey.pem  usercert.pem  userkey.pem
[root@hal9000 hal9000]# cp 2f* /etc/grid-security/certificates/
[root@hal9000 hal9000]# cp host* /etc/grid-security/
[root@hal9000 hal9000]# cp user* /home/ui_user/.globus/
[root@hal9000 hal9000]# chmod 0600 /home/ui_user/.globus/user*
[root@hal9000 hal9000]# chown ui_user:ui_user /home/ui_user/.globus/user*
[root@hal9000 hal9000]# 
The newest dCache 1.9.4-2 release with all certificates updated can be downloaded here:
dateversionmd5 hash
27.7.2009 dCacheToGo_dCache1.9.4-2.vdi.gz c030a20d2777a511fc17fe8f0ac7eba7

Changelog

V04 -> V05

  • new dCache server and client version 1.8.0-15p5
  • user certificate ("Kermit the frog") now valid till 06/2009
  • automatic CRL-fetching at startup

V03 -> V04

  • fixed Chimera's PNFS backwards-compatibility

V02 -> V03

  • dCacheServer 1.8.0-15p1
  • dCacheClient 1.8.0-15
  • an optional pool setup (not enabled by default) which simulates a tape backend for dCache (Online<->Nearline testing)
  • predefined gPlazma mapping, which considers local users 'desy01', 'desy02' and group 'desy'
  • VOMS2gPlazma script for automated vo-rolemap-file creation

What you get

  • SL 4.5 (32bit) without X11
  • dCache 1.8
    • Chimera instead of PNFS as the namespace provider
    • 4 pools
    • all domains running on the same machine (including SRM 2.2)
  • a full gLite userinferface, providing commands like voms-proxy-init, globus-url-copy, etc.
  • dCache clients (dccp, srmcp)
  • a (limited) user grid certificate

Restrictions

  • no infoprovider preinstalled
  • the default network setup is based on NAT. This means the VM is not reachable from outside, but the other way round works.
  • due to the former restriction, data transfers can only happen inside the VM

Prerequisites

  • You need to have Virtual Box installed on your machine to make use of this image!! VirtualBox is a free desktop virtualisation solution available for Linux, Windows and Mac.
  • 4 GB of diskspace for the VM
  • at least 512MB of RAM that you can assign to the VM (the more the better, since all dCache services+ clients are running simultaneously)

Getting started

Download the image and unzip it. Start the VirtualBox GUI and register the .vdi-file by going to File-> Virtual Disk manager-> Add.

Next, create a new VM from the registered image:

  1. Selecting New will launch the a wizard, which will guide you through the creation process.
  2. On the first page, type in a name for the virtual machine (e.g., 'dCacheToGo') and select 'Linux 2.6' for the OS type.
  3. On the next screen, assign some RAM to the VM. This should be at least 512MB, but allocating more will help dCache run more smoothly.
  4. On the next screen, attach the previously registered image by clicking on Existing.
  5. Select Finish followed by Start to boot your virtual machine!

First steps

The root password is .school

Also included is a normal used account, as some Globus tools only work from non-root accounts. This account has

username: ui_user
password: .grid

To shutdown the VM just close the window and then select Send shutdown signal or type halt as the root user. Please remember that all changes you do inside the VM will be persisted in the disk-image. In order to try something dangerous witch could damage the system, make yourself familiar with taking snapshots. To let the VM loose the focus of the mouse arrow, press Right-Ctlr.

Some words about networking: The FQDN is hardcoded to hal9000.dcache.org. The installed host-certificate relies on this. On startup, the VM gets an IP assigned by the Virtual Box DHCP server (which supplies an IP address from the private subnet 10.0.2.0/24). The gateway (which is identical to the host machine) is bound to 10.0.2.2. For example, one could use scp from within the VM like this

scp /some/file someUser@10.0.2.2:/some/path

Enabling ssh access

Console access is OK to begin with, but not very convenient. For example, copy&paste to/from the clipboard doesn't work between the VM window and the host system. The image comes with an auto-started ssh deamon, but unfortunately the VM is hidden behind a NAT from outside. Therefore to get ssh-access from the host machine, we have to enable port-forwarding.

In order to forward sshd port 22 of the VM to an unused port (2022 in this example) on the host system, the VBoxManage command on the host can be used. Please replace "dCacheToGo" by your VM name.

# set the guest port (port 22 for sshd)
VBoxManage setextradata "dCacheToGo" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/sshd/GuestPort" 22

# set the host port (the port where the VirtualBox-process listens on behalf of the VM)
VBoxManage setextradata "dCacheToGo" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/sshd/HostPort" 2022

# set the protocol
VBoxManage setextradata "dCacheToGo" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/sshd/Protocol" TCP

You can double-check what you set by typing the following (again, substitute "dCacheToGo" with your VM name)

VBoxManage getextradata "dCacheToGo" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/sshd/GuestPort"
VBoxManage getextradata "dCacheToGo" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/sshd/HostPort"
VBoxManage getextradata "dCacheToGo" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/sshd/Protocol"

Having configured everything correctly, stop the virtual OS by closing the VM window and selecting Send shutdown signal. As soon as the VM is halted, close the VM window and restart the VM by clicking Start in the VirtualBox control panel. The moment the VM came up again and sshd service is started, you should be able to access the VM from the host via

ssh -p 2022 root@localhost
ssh -p 2022 ui_user@localhost

# mind the capital 'P'
scp -P 2022 /some/file root@localhost:/some/destination

Getting started with dCache

After booting up, dCache needs to be started manually (for now). Don't worry about the PostGreSQL database service, it's started automatically. To start dCache with pools, do a

dcache start

Check that status of the domains with

dcache status

After a few minutes the system should have been finished initialising itself and is ready to go! You can still use the webinterface the check the services:

links http://hal9000:2288

Obviously, using a text browser is not very convenient. A better solution is to forward the port of the dCache web interface analogous to what we did with ssh. This can be achieved via the follwing commands: (note that the HostPort can be any unused port and again, don't forget to replace "dCacheToGo" by your VM name)

VBoxManage setextradata "dCacheToGo" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/dCacheWeb/GuestPort" 2288
VBoxManage setextradata "dCacheToGo" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/dCacheWeb/HostPort" 2288
VBoxManage setextradata "dCacheToGo" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/dCacheWeb/Protocol" TCP

You could now access the dCache web interface on the host by pointing your favourite browser to http://localhost:2288.

Mounting the namespace

Mounting the Chimera namespace does not only provide a convenient way to keep track of the file system tree (using ls, rm from the OS), it is also needed by SRM- and GridFTP-doors to allow them serving commands like ftp-ls.

At startup, a NFS3-server on top of Chimera is run. Unfortunately, the automount procedure is not reliable at the moment. So please check, whether Chimera is mounted by making sure /pnfs/dcache.org/data exists. In case it doesn't, mount it manually:

mount localhost:/pnfs /pnfs

Transferring files

To make use of the gLite UI commands, login as the unprivileged ui_user. The VM image comes with a preinstalled user certificate (located at /home/ui_user/.globus).

Some notes about the certificate

The distinguished name (DN) is /C=DE/O=DESY/CN=Kermit the frog.

It is intended to be used only within this VM! It is signed with a self-made CA and will be useless outside of the VM. The only other grid service that knows this CA is the VOMS server that provides the VO 'desy'. This is required as our grid user is assigned to this VO and has some capabilities like roles.

To obtain a extended proxy, this VOMS server will be contacted to download the extensions. You can do this typing

voms-proxy-init --voms desy

Password is not needed, the key is unprotected for simplicity reasons.

You can replace this test certificate/key with your 'real' certificate/key. This would allow you to obtain a proxy assigned to another VO.

Having obtained a proxy, you should now be able to access the dCache server. The dCache server (gPlazma) is configured such that it will authenticate the DN /C=DE/O=DESY/CN=Kermit the frog.

Try some file transfers

Try some file transfers, using the following commands

srmcp file:////bin/sh srm://hal9000:8443//pnfs/dcache.org/data/file1
globus-url-copy gsiftp://hal9000:2811/pnfs/dcache.org/data/file1 file:///dev/null

# unauhenticated dcap access
dccp /pnfs/dcache.org/data/file1 /dev/null

# list a directory using the mounted namespace
ls -l /pnfs/dcache.org/data

# list a directory as an external user would do
srmls srm://hal9000:8443/pnfs/dcache.org/data

Create your own dCacheToGo version

In order to create an updated version of dCacheToGo, please follow this steps:

First, download the latest version of the dCacheToGo image and register it in Virtual Box as described above. Boot the virtual machine. Now make your changes on the virtual system, like updating RPMS, configuration files, certificates etc. Try to get rid of temporary files, e.g. /tmp/*, yum clean all, logfiles. Shutdown the virtual machine cleanly.

Now, you need to clone the VDI image in order to make it relocatable:

VBoxManage clonevdi         <uuid>|<filename> <outputfile>

Zip the output VDI to reduce the filesize. Voila, have fun with your newly crated image and go distribute it!

How to convert the VDI image from VirtualBox to a VMWare

Download the inoffical vditool from virtualbox.org. You also need to have the program qemu-img installed (ships with qemu). Then do

vditool COPYDD dCacheToGo-V05.vdi dCacheToGo-V05.raw
qemu-img convert -f raw dCacheToGo-V05.raw -O vmdk dCacheToGo-V05.vmdk

The resulting VMDK image file can be understood by VMWare. You then need to create a configuration file (.VMX), for example using this creator. Once you have the VMDK image and the VMX configuration, you import both files in VMware player.

Please note that the networking doesn't work out of the box for VMWare (NAT mode), since it was somewhat tweaked for VirtualBox. This convertion tip was sent in by Olga Levchuk (Olga (dott) Levchuk (At) rz (dot) uni-freiburg (dot) de)