wiki:Migrating Hands on
Last modified 10 years ago Last modified on 03/16/11 17:24:14

Migration to dcache 1.9.12

To practice the migration you will need VirtualBox running on your laptop. This is because we will be using a prepared image to give a controlled environment. The prepared image will be running as a Guest OS inside VirtualBox.

Basics migration.

The basic migration takes a simple but working dCache 1.9.5 instance and migrates it to the corresponding 1.9.12 instance.

Preparation.

If you have not already done so, you must configure VirtualBox to forward various TCP ports from the host machine to the guest OS. The following commands achieve this. Substitute the name of the Guest OS for <your VM name>:

VBoxManage modifyvm <your VM name> --natpf1 "guestssh,tcp,,2222,,22"
VBoxManage modifyvm <your VM name> --natpf1 "guestdcachewebservice,tcp,,2288,,2288"
VBoxManage modifyvm <your VM name> --natpf1 "guestadmin,tcp,,22223,,22223"
VBoxManage modifyvm <your VM name> --natpf1 "guestwebdav,tcp,,22880,,2880"
VBoxManage modifyvm <your VM name> --natpf1 "guestwebdavsec,tcp,,22881,,2881"

In case you made a mistake (and there was a typo before in this section), you have to delete the previously created rule before you can add a new one using the same name. E.g. to replace the rule for the WebDAV-rule you have to execute...

VBoxManage modifyvm <your VM name> --natpf1 delete "guestwebdav"
VBoxManage modifyvm <your VM name> --natpf1 "guestwebdav,tcp,,22880,,2880"

You should now start VirtualBox and start the Guest OS.

Once the Guest OS has finished booting, you should log in as root using the password .school (that is, DOT--s--c--h--o--o--l).

Before we start the migration process let’s do some preparation and take a backup of the current dCache settings. The backup will allow you to roll back to the initial state, should you have any problems.

When the machine has apparently finished booting then it is possible that dCache is still starting. Therefore we must first wait for dCache to finish starting.

[root@dcachetogo ~]# /opt/d-cache/bin/dcache status
Domain                   Service          Status
lmDomain                 lm               running
dCacheDomain             dCache           running
dirDomain                dir              running
adminDoorDomain          admin            stopped
httpdDomain              httpd            stopped
utilityDomain            utility          stopped
gPlazma-dcachetogoDomain gPlazma          stopped
chimeraDomain            chimera          stopped
pool1Domain              pool             stopped
pool2Domain              pool             stopped
srm-dcachetogoDomain     srm              stopped
dcap-dcachetogoDomain    dcap             stopped
gridftp-dcachetogoDomain gridftp          stopped

... a little while later:

[root@dcachetogo ~]# /opt/d-cache/bin/dcache status
Domain                   Service          Status
lmDomain                 lm               running
dCacheDomain             dCache           running
dirDomain                dir              running
adminDoorDomain          admin            running
httpdDomain              httpd            running
utilityDomain            utility          running
gPlazma-dcachetogoDomain gPlazma          running
chimeraDomain            chimera          running
pool1Domain              pool             running
pool2Domain              pool             running
srm-dcachetogoDomain     srm              running
dcap-dcachetogoDomain    dcap             running
gridftp-dcachetogoDomain gridftp          running

Now we stop dCache:

[root@dcachetogo ~]# /etc/init.d/dcache stop
Stopping gridftp-dcachetogoDomain (pid=2485) 0 1 Done
Stopping dcap-dcachetogoDomain (pid=2418) 0 1 Done
Using CATALINA_BASE:   /opt/d-cache/libexec/apache-tomcat-5.5.20
Using CATALINA_HOME:   /opt/d-cache/libexec/apache-tomcat-5.5.20
Using CATALINA_TMPDIR: /opt/d-cache/libexec/apache-tomcat-5.5.20/temp
Using JRE_HOME:       /usr/java/jdk1.6.0_24
Stopping srm-dcachetogo (pid=2107) 0 Done
Stopping pool2Domain (pid=2065) 0 1 2 3 Done
Stopping pool1Domain (pid=1978) 0 1 2 3 Done
Stopping chimeraDomain (pid=1889) 0 1 Done
Stopping gPlazma-dcachetogoDomain (pid=1787) 0 Done
Stopping utilityDomain (pid=1703) 0 1 Done
Stopping httpdDomain (pid=1609) 0 1 Done
Stopping adminDoorDomain (pid=1512) 0 1 Done
Stopping dirDomain (pid=1428) 0 Done
Stopping dCacheDomain (pid=1355) 0 1 Done
Stopping lmDomain (pid=1288) 0 Done

Then unmount and stop the Chimera NFS server:

[root@dcachetogo ~]# umount /pnfs
[root@dcachetogo ~]# /etc/init.d/chimera stop
Shutting down Chimera-NFSv3 interface

There is no separate chimera script in 1.9.12, so we remove the chimera service script from the different run-levels.

[root@dcachetogo ~]# chkconfig --list chimera
chimera         0:off   1:off   2:off   3:on    4:on    5:on    6:off
[root@dcachetogo ~]# chkconfig --del chimera
[root@dcachetogo ~]# chkconfig --list chimera
service chimera supports chkconfig, but is not referenced in any runlevel (run 'chkconfig --add chimera')

Copy the dCache home directory. Usually it is necessary to save only the contents of /opt/d-cache/config and /opt-d-cache/etc directories, but to keep things simple, we copy the whole of /opt/d-cache.

[root@dcachetogo ~]# cp -r /opt/d-cache /opt/d-cache_195

After these basic preparations are completed, we can install (or, in our case, upgrade) the new dCache RPM.

[root@dcachetogo ~]# rpm -Uhv ~/install/dcache-server-1.9.12-r15295.noarch.rpm
Preparing...                ########################################### [100%]
  1:dcache-server          ########################################### [100%]
warning: /opt/d-cache/etc/srm_setup.env saved as /opt/d-cache/etc/srm_setup.env.rpmsave

Migrating

There are a couple of bugs in the migration script we must work-around. These will be fixed with the release of 1.9.12.

[root@dcachetogo ~]# sed -ie 's/tail +4/tail -n +4/' /opt/d-cache/libexec/migrate-from-1.9.5.sh
[root@dcachetogo ~]# mv /opt/d-cache/config/chimera-config.xml{.rpmsave,}

We now run the migration script. This will generate new configuration files: etc/dcache.conf and etc/layouts/dcachetogo.conf

[root@dcachetogo ~]# /opt/d-cache/libexec/migrate-from-1.9.5.sh
Converting /opt/d-cache/config/dCacheSetup to
/opt/d-cache/etc/dcache.conf.
Renaming /opt/d-cache/config/dCacheSetup to
/opt/d-cache/config/dCacheSetup.pre-197

Converting /opt/d-cache/etc/node_config to
/opt/d-cache/etc/layouts/dcachetogo.conf.
Renaming /opt/d-cache/etc/node_config to
/opt/d-cache/etc/node_config.pre-197


DISCLAIMER: Please inspect the generated configuration. The import is
           known to be incomplete; in particular if batch files have
           been modified or the symbolic links to config/dCacheSetup
           have been broken to make domain specific setups. This
           includes the case where services have been moved between
           domains or service specific configuration parameters are
           used in a 1.9.6 style installation.

Checking the migrated configuration for any problems.

[WARNING] dcache.conf:22: Property serviceLocatorHost: use "broker.host"
                          instead; support for serviceLocatorHost will be
                          removed in the future
[ERROR] dcache.conf:23: Property java_options: may not be adjusted; See
                        dcache.java.options or dcache.java.options.extra
[WARNING] dcache.conf:24: Property logArea: use "dcache.log.dir" instead;
                          support for logArea will be removed in the future
Found 1 error and 2 warnings.

At the end of the migration process, the script automatically runs the command /opt/d-cache/bin/dcache check-config. This command checks for problems in configuration. You should be careful as not all problem can be found automatically! Here is the output if the command is run manually:

[root@dcachetogo ~]# /opt/d-cache/bin/dcache check-config
[WARNING] dcache.conf:22: Property serviceLocatorHost: use "broker.host"
                          instead; support for serviceLocatorHost will be
                          removed in the future
[ERROR] dcache.conf:23: Property java_options: may not be adjusted; See
                        dcache.java.options or dcache.java.options.extra
[WARNING] dcache.conf:24: Property logArea: use "dcache.log.dir" instead;
                          support for logArea will be removed in the future
Found 1 error and 2 warnings.

WARNINGs are things that should be fixed, but do not prevent dCache from starting. ERRORs are problems so severe that dCache refuses to start.

We must fix the ERROR, and should fix the WARNINGs. All three problems are located in the dcache.conf file. Inspect the file /opt/d-cache/etc/dcache.conf with your favorite editor.

The dcache.conf file adjusts dCache's behaviour by overriding dCache default properties. It replaces the dCacheSetup file.

To fix the three problems returned by the check-config command we need change the following properties: serviceLocatorHost, logArea and java_options.

Fixing serviceLocatorHost and logArea is easy: these properties have new names broker.host and dcache.log.dir, as described in the warning messages. Make the following changes to the dcache.conf file:

  • change the line serviceLocatorHost=localhost to broker.host=localhost
  • change the line logArea=/var/log/dCache to dcache.log.dir=/var/log/dCache

The error on line 23 ("Property java_options: may not be adjusted") is bit more tricky. In our case, the java_options property was configured to allocate additional memory to the Java process: -Xmx64m means the maximum heap memory size is increased to 64 MiB and -XX:MaxDirectMemorySize=64m means the maximum direct-memory (for direct IO) is also increased to 64 MiB.

With dCache v1.9.12, changing how much memory Java should use is much easier. The two properties dcache.java.memory.heap and dcache.java.memory.direct may be used to adjust the amount of memory Java will use.

Edit the file dcache.conf with your favourite editor and make the following changes:

  • Remove the property line that starts java_options=
  • Add the line dcache.java.memory.heap=64m
  • Add the line dcache.java.memory.direct=64m

You should re-run the check-config command. This should now give no warnings and no errors.

[root@dcachetogo ~]# /opt/d-cache/bin/dcache check-config
No problems found.

The next step is to inspect layout file. You don't need to change anything at this time, but it is useful to familiarise yourself with its structure.

By default, the layout file is single (corresponding to /opt/d-cache/etc/layouts/single.conf). However, the migration script will follow the recommended practise of using the host's name. This is achieved by dcache.conf containing the line: dcache.layout=${host.name}

Looking at the layout file /opt/d-cache/etc/layouts/dcachetogo.conf you should recognise the list of domains with one or more service inside each domain. The layout file can have configuration lines in it too, but the migration script will not generate any.

Here is a fragment from the layout file /opt/d-cache/etc/layouts/dcachetogo.conf

[dCacheDomain]
[dCacheDomain/poolmanager]
[dCacheDomain/broadcast]
[dCacheDomain/loginbroker]
[dCacheDomain/topo]

The line [dCacheDomain] declares a new domain called dCacheDomain.

The line [dCacheDomain/poolmanager] declares that the dCacheDomain domain should start a service of type poolmanager. The [dCacheDomain/broadcast] line means dCacheDomain domain will also host a broadcast service, and so on.

Final step.

Now we start the migrated dCache:

[root@dcachetogo ~]# /etc/init.d/dcache start
Starting dCacheDomain done
Starting dirDomain done
Starting adminDoorDomain done
Starting httpdDomain done
Starting utilityDomain done
Starting gPlazma-dcachetogoDomain done
Starting namespaceDomain done
Starting pool1Domain done
Starting pool2Domain done
Starting srm-dcachetogoDomain done
Starting dcap-dcachetogoDomain done
Starting gridftp-dcachetogoDomain done

Be aware that domains may take a little longer to start once dcache start returns.

We should check some basic functionality.

First, we make sure that the dcap door is ready. If it isn't ready, just wait.

[dc_user@dcachetogo ~]$ netstat -nl|grep 22125
tcp        0      0 0.0.0.0:22125               0.0.0.0:*                   LISTEN

Now we try writing data into dCache and read that data back again.

[root@dcachetogo ~]# su - dc_user
[dc_user@dcachetogo ~]$ /usr/bin/dccp /bin/bash dcap://localhost/pnfs/dcache.org/data/worldwritable/test1
877480 bytes (857 kiB) in 0 seconds
[dc_user@dcachetogo ~]$ /usr/bin/dccp  dcap://localhost/pnfs/dcache.org/data/worldwritable/test1  /dev/null
877480 bytes (857 kiB) in 0 seconds
[dc_user@dcachetogo ~]$ exit
logout

dCache 1.9.12 (with Chimera) doesn't need the namespace to be mounted: dCache will work fine. (For PNFS, the PnfsManager needs the namespace to be mounted but that's the only component that needs it.)

The migration script doesn't configure an NFS server. So, after the migration, there's no mounted namespace:

[root@dcachetogo ~]# mount
/dev/sda1 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

You might want the namespace mounted anyway, for administrative purposes. dCache isn't running an NFS server at the moment.

Let's fix that.

Edit the layout file, /opt/d-cache/etc/layouts/dcachetogo.conf, and add the following line:

[namespaceDomain/nfsv3]

The line can go anywhere after the domain declaration, [namespaceDomain], but it's good practise to put all the service declarations immediately after their corresponding domain declaration.

Once you have edited the layout file, you can verify that dCache will start the NFSv3 service by running the dcache service command:

[root@dcachetogo ~]# /opt/d-cache/bin/dcache services | grep ^namespaceDomain
namespaceDomain          pnfsmanager  PnfsManager      /var/log/dCache/namespaceDomain.log          
namespaceDomain          cleaner      cleaner          /var/log/dCache/namespaceDomain.log          
namespaceDomain          acl          acladmin         /var/log/dCache/namespaceDomain.log          
namespaceDomain          nfsv3        NFSv3-dcachetogo /var/log/dCache/namespaceDomain.log          

Having changed which services are to run in the namespaceDomain domain, we must restart the domain:

[root@dcachetogo ~]# /opt/d-cache/bin/dcache restart namespaceDomain
Stopping namespaceDomain (pid=17545) 0 1 2 3 done
Starting namespaceDomain done

Now that namespaceDomain is hosting the NFS server, we can mount the namespace at the usual mount-point: /pnfs

[root@dcachetogo ~]# mount -o intr,rw,nolock,noac,hard,nfsvers=3 localhost:/pnfs /pnfs

We can verify that the namespace is available by looking at the file test1 that was copied into dCache earlier as a test:

[root@dcachetogo ~]# stat /pnfs/dcache.org/data/worldwritable/test1
  File: `/pnfs/dcache.org/data/worldwritable/test1'
  Size: 877480          Blocks: 1714       IO Block: 32768  regular file
Device: 14h/20d Inode: 18446744072961198379  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (  500/ dc_user)   Gid: (  500/ dc_user)
Access: 2011-03-15 11:49:50.000000000 +0100
Modify: 2011-03-15 11:49:53.000000000 +0100
Change: 2011-03-15 11:49:50.000000000 +0100

Before we finish, let's clean up all the broken symbolic links in /opt/d-cache/config

[root@dcachetogo ~]# find -L /opt/d-cache/config -type l
/opt/d-cache/config/dirSetup
/opt/d-cache/config/infoProviderSetup
/opt/d-cache/config/gridftpdoorSetup
/opt/d-cache/config/gPlazmaSetup
/opt/d-cache/config/xrootdDoorSetup
/opt/d-cache/config/kerberosdcapdoorSetup
/opt/d-cache/config/doorSetup
/opt/d-cache/config/gsidcapdoorSetup
/opt/d-cache/config/poolSetup
/opt/d-cache/config/kerberosftpdoorSetup
/opt/d-cache/config/infoSetup
/opt/d-cache/config/nfsv41Setup
/opt/d-cache/config/chimeraSetup
/opt/d-cache/config/httpdSetup
/opt/d-cache/config/httpdoorSetup
/opt/d-cache/config/utilitySetup
/opt/d-cache/config/weakftpdoorSetup
/opt/d-cache/config/srmSetup
/opt/d-cache/config/replicaSetup
/opt/d-cache/config/adminDoorSetup
/opt/d-cache/config/pnfsSetup
/opt/d-cache/config/authdoorSetup
/opt/d-cache/config/statisticsSetup
/opt/d-cache/config/lmSetup
[root@dcachetogo dCache]# find -L /opt/d-cache/config -type l -exec rm {} ;
[root@dcachetogo dCache]# find -L /opt/d-cache/config -type l

The simple migration scenario is now completed.

More complicated scenario.

Now we try migrating a node based on a real system (from Goettingen) but with some awkward configuration added to stress the migration process.

Preparation.

The VirtualBox image keeps another configuration which is more close to the real setup. To enable it, we have to switch dCache home directories:

[root@dcachetogo ~]# umount /pnfs
[root@dcachetogo ~]# /opt/d-cache/bin/dcache stop
Stopping gridftp-dcachetogoDomain (pid=5142) 0 1 done
Stopping dcap-dcachetogoDomain (pid=5090) 0 done
Stopping srm-dcachetogoDomain (pid=5030) 0 1 2 3 done
Stopping pool2Domain (pid=4962) 0 1 2 done
Stopping pool1Domain (pid=4908) 0 1 2 done
Stopping namespaceDomain (pid=5621) 0 1 2 3 done
Stopping gPlazma-dcachetogoDomain (pid=4812) 0 done
Stopping utilityDomain (pid=4761) 0 done
Stopping httpdDomain (pid=4720) 0 1 done
Stopping adminDoorDomain (pid=4679) 0 1 done
Stopping dirDomain (pid=4642) 0 done
Stopping dCacheDomain (pid=4600) 0 done
[root@dcachetogo ~]# mv /opt/d-cache /opt/d-cache_1912
[root@dcachetogo ~]# mv /opt/d-cache_real/ /opt/d-cache

We also work-around some last-minute problems:

[root@dcachetogo ~]# sed -nie '/^p/p' /opt/d-cache/config/pools.poollist
[root@dcachetogo ~]# sed -ie 's/statistics=yes/statistics=no/' /opt/d-cache/etc/node_config
[root@dcachetogo ~]# sed -ie 's/billingToDb=yes//' /opt/d-cache/config/{dCache,httpd}Setup

Activate it with some magic, by rolling back to 1.9.5.

[root@dcachetogo ~]# rpm -Uhv --force ~/install/dcache-server-1.9.5-24.noarch.rpm
Preparing...                ########################################### [100%]
  1:dcache-server########################################### [100%]

Start Chimera and mount PNFS

[root@dcachetogo ~]# /etc/init.d/chimera start
Starting Chimera-NFSv3 interface
rpcinfo: RPC: Unable to receive; errno = Connection refused
Waiting for portmap port: rpcinfo: RPC: Unable to receive; errno = Connection refused
.rpcinfo: RPC: Unable to receive; errno = Connection refused
.rpcinfo: RPC: Unable to receive; errno = Connection refused
.rpcinfo: RPC: Unable to receive; errno = Connection refused
.rpcinfo: RPC: Unable to receive; errno = Connection refused
.rpcinfo: RPC: Unable to receive; errno = Connection refused
.
[root@dcachetogo ~]# mount -o nolock,intr,rw,noac,hard,nfsvers=3 localhost:/pnfs /pnfs

We copy the previous PoolManager.conf file, as the one supplied is only good for Goettingen machines

[root@dcachetogo ~]# cp /opt/d-cache{_195,}/config/PoolManager.conf
cp: overwrite `/opt/d-cache/config/PoolManager.conf'? y

Start dCache:

[root@dcachetogo ~]# /opt/d-cache/bin/dcache start
Starting lmDomain 6 Done
Starting dCacheDomain 6 Done
Starting dirDomain 6 Done
Starting adminDoorDomain 6 Done
Starting httpdDomain 6 Done
Starting utilityDomain 6 Done
Starting gPlazma-dcachetogoDomain 6 Done
Starting chimeraDomain 6 Done
Starting poolsDomain 6 Done
Using CATALINA_BASE:   /opt/d-cache/libexec/apache-tomcat-5.5.20
Using CATALINA_HOME:   /opt/d-cache/libexec/apache-tomcat-5.5.20
Using CATALINA_TMPDIR: /opt/d-cache/libexec/apache-tomcat-5.5.20/temp
Using JRE_HOME:       /usr/java/jdk1.6.0_24

Pinging srm server to wake it up, will take few seconds ...
VersionInfo : v2.2
backend_type:dCache
backend_version:production-1.9.5-24

Done
httpdDomain might still be running
Starting dcap-dcachetogoDomain 6 Done

Verify that the dcap door is running (just wait until something is LISTENing on port 22125)

[root@dcachetogo ~]# netstat --tcp -nl|grep 22125
tcp        0      0 0.0.0.0:22125               0.0.0.0:*                   LISTEN      

To demonstrate that this dCache instance is working, we'll copy another file:

[root@dcachetogo ~]# su - dc_user
[dc_user@dcachetogo ~]$ /usr/bin/dccp /bin/bash dcap://localhost/pnfs/dcache.org/data/worldwritable/test2
877480 bytes (857 kiB) in 0 seconds
[dc_user@dcachetogo ~]$ /usr/bin/dccp  dcap://localhost/pnfs/dcache.org/data/worldwritable/test2  /dev/null
877480 bytes (857 kiB) in 0 seconds
[dc_user@dcachetogo ~]$ exit
logout

We must stop dCache and Chimera before we migrate to 1.9.12

[root@dcachetogo ~]# /opt/d-cache/bin/dcache stop
Stopping dcap-dcachetogoDomain (pid=16555) 0 Done
Stopping httpdDomain (pid=15465) 0 Done
Using CATALINA_BASE:   /opt/d-cache/libexec/apache-tomcat-5.5.20
Using CATALINA_HOME:   /opt/d-cache/libexec/apache-tomcat-5.5.20
Using CATALINA_TMPDIR: /opt/d-cache/libexec/apache-tomcat-5.5.20/temp
Using JRE_HOME:       /usr/java/jdk1.6.0_24
Stopping srm-dcachetogo (pid=15962) 0 Done
Stopping statisticsDomain (pid=20363) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Done
Stopping poolsDomain (pid=19126) 0 1 2 Done
Stopping chimeraDomain (pid=15744) 0 Done
Stopping gPlazma-dcachetogoDomain (pid=15646) 0 Done
Stopping utilityDomain (pid=15560) 0 1 Done
Stopping adminDoorDomain (pid=15379) 0 Done
Stopping dirDomain (pid=15297) 0 Done
Stopping dCacheDomain (pid=15224) 0 Done
Stopping lmDomain (pid=15156) 0 Done
[root@dcachetogo ~]# umount /pnfs
[root@dcachetogo ~]# /etc/init.d/chimera stop
Shutting down Chimera-NFSv3 interface

Migration.

Next, we update to dCache v1.9.12

[root@dcachetogo ~]# rpm -Uhv ~/install/dcache-server-1.9.12-r15295.noarch.rpm
Preparing...                ########################################### [100%]
  1:dcache-server          ########################################### [100%]

Apply the same patches to avoid bugs in the current migration script (these are already fixed in later release-candidates of 1.9.12).

[root@dcachetogo ~]# sed -ie 's/tail +4/tail -n +4/' /opt/d-cache/libexec/migrate-from-1.9.5.sh
[root@dcachetogo ~]# cp /opt/d-cache{_195,}/config/chimera-config.xml 

Execute the migration script.

[root@dcachetogo ~]# /opt/d-cache/libexec/migrate-from-1.9.5.sh
Converting /opt/d-cache/config/dCacheSetup to
/opt/d-cache/etc/dcache.conf.
Renaming /opt/d-cache/config/dCacheSetup to
/opt/d-cache/config/dCacheSetup.pre-197

Converting /opt/d-cache/etc/node_config to
/opt/d-cache/etc/layouts/dcachetogo.conf.
Renaming /opt/d-cache/etc/node_config to
/opt/d-cache/etc/node_config.pre-197


DISCLAIMER: Please inspect the generated configuration. The import is
            known to be incomplete; in particular if batch files have
            been modified or the symbolic links to config/dCacheSetup
            have been broken to make domain specific setups. This
            includes the case where services have been moved between
            domains or service specific configuration parameters are
            used in a 1.9.6 style installation.

Checking the migrated configuration for any problems.

[WARNING] dcache.conf:22: Property serviceLocatorHost: use "broker.host"
                          instead; support for serviceLocatorHost will be
                          removed in the future
[ERROR] dcache.conf:23: Property java_options: may not be adjusted; See
                        dcache.java.options or dcache.java.options.extra
Found 1 error and 1 warning.

You probably remember these findings, as they already showed up in the previous migration example. In order to fix them, dit the file /opt/d-cache/etc/dcache.conf with your favourite editor and apply the following changes:

  • change the line serviceLocatorHost=localhost to broker.host=localhost
  • Remove the property line that starts java_options=
  • Add the line dcache.java.memory.heap=64m
  • Add the line dcache.java.memory.direct=64m

Now, the check-config command should give no errors.

[root@dcachetogo ~]# /opt/d-cache/bin/dcache check-config
No problems found.

If you're sure that all conflicts are resolved, you may try to start dCache.

[root@dcachetogo d-cache]# /opt/d-cache/bin/dcache start
Starting dCacheDomain done
Starting dirDomain done
Starting adminDoorDomain done
Starting httpdDomain done
Starting utilityDomain done
Starting gPlazma-dcachetogoDomain done
Starting namespaceDomain done
Starting poolsDomain done
Starting srm-dcachetogoDomain done
Starting dcap-dcachetogoDomain done

But looking at the current status for the services, something went wrong with the httpd server.

[root@dcachetogo ~]# /opt/d-cache/bin/dcache status
DOMAIN                   STATUS     PID   USER                          
dCacheDomain             running    12117 root                          
dirDomain                running    12154 root                          
adminDoorDomain          running    12195 root                          
httpdDomain              restarting       [whoever runs "dcache start"] 
utilityDomain            running    12278 root                          
gPlazma-dcachetogoDomain running    12320 root                          
namespaceDomain          running    12371 root                          
poolsDomain              running    12425 root                          
srm-dcachetogoDomain     running    12480 root                          
dcap-dcachetogoDomain    running    12561 root                          

To understand what's happening, look back at the output when running /opt/d-cache/bin/dcache start with dCache v1.9.5. You see a line like:

httpdDomain might still be running

This is because the configuration attempts to start httpdDomain twice. In 1.9.5, the second attempt prints a warning message but doesn't affect the existing httpdDomain. It's working "by accident".

After migrating, the layout file will include two httpdDomain domain declarations with their corresponding collection of services. You should be able to see this in the file /opt/d-cache/etc/layouts/dcachetogo.conf. The effect is that the configuration attempts to instantiate each httpdDomain service twice. Some of these services cannot be instantiated twice with the default configuration (e.g., httpd service), which causes the httpdDomain to fail.

The fix is simple: delete one of the httpdDomain domain declarations along with it's corresponding set of services; then either restart the httpdDomain explicitly or simply wait for the automatic restart.

[root@dcachetogo ~]# /opt/d-cache/bin/dcache restart httpdDomain
Stopping httpdDomain (pid=12231) done
Starting httpdDomain done

Final step.

Let's tidy up the broken symbolic links.

[root@dcachetogo d-cache]# find -L /opt/d-cache/config -type l -exec rm {} ;

Hop-la seems that the http daemon on the old setup has his own settings

[root@dcachetogo d-cache]# find  /opt/d-cache/config/ -name "*Setup"
/opt/d-cache/config/httpdSetup
[root@dcachetogo ~]# diff -u /opt/d-cache/config/{dCacheSetup.pre-197,httpdSetup}
--- /opt/d-cache/config/dCacheSetup.pre-197     2011-03-15 14:59:06.495324515 +0100
+++ /opt/d-cache/config/httpdSetup      2011-03-15 14:59:06.500325150 +0100
@@ -1,7 +1,7 @@
 serviceLocatorHost=localhost
 serviceLocatorPort=11111
 java="/usr/bin/java"
-java_options="-server -Xmx64m -XX:MaxDirectMemorySize=64m 
+java_options="-server -Xmx32m -XX:MaxDirectMemorySize=32m 
               -Dsun.net.inetaddr.ttl=1800 
               -Dorg.globus.tcp.port.range=20000,25000 
               -Djava.net.preferIPv4Stack=true 

The httpdDomain has less memory allocated to it. We can configure this using the dcache.java.memory.heap and dcache.java.memory.direct properties inside the layout file.

[httpdDomain]
dcache.java.memory.heap=32m
dcache.java.memory.direct=32m
[httpdDomain/httpd]
[httpdDomain/billing]
[httpdDomain/srm-loginbroker]

We need to restart of httpdDomain for it to use the reduced memory.

[root@dcachetogo ~]# /opt/d-cache/bin/dcache restart httpdDomain
Stopping httpdDomain (pid=14844) 0 1 done
Starting httpdDomain done

Here we double check that the httpdDomain has been started with the reduced memory

[root@dcachetogo ~]# ps -C java -o cmd|sed 's/-D.*start/[..]/'
CMD
/usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] dCacheDomain
/usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] dirDomain
/usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] adminDoorDomain
/usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] utilityDomain
/usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] gPlazma-dcachetogoDomain
/usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] namespaceDomain
/usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] poolsDomain
/usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] srm-dcachetogoDomain
/usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] dcap-dcachetogoDomain
/usr/bin/java -server -Xmx32m -XX:MaxDirectMemorySize=32m [..] httpdDomain

Congratulations! You have completed the second migration successfully.

Now you may shuffle around services within domains, merge domains and enable new features like NFS4.1 and/or WebDAV.