Migration to dcache 1.9.12
To practice the migration you will need VirtualBox running on your laptop. This is because we will be using a prepared image to give a controlled environment. The prepared image will be running as a Guest OS inside VirtualBox.
Basics migration.
The basic migration takes a simple but working dCache 1.9.5 instance and migrates it to the corresponding 1.9.12 instance.
Preparation.
- Get VirtualBox from here.
- Get the virtual image form here.
- For installation and configuration please follow these instructions.
If you have not already done so, you must configure VirtualBox to forward various TCP ports from the host machine to the guest OS. The following commands achieve this. Substitute the name of the Guest OS for <your VM name>:
VBoxManage modifyvm <your VM name> --natpf1 "guestssh,tcp,,2222,,22" VBoxManage modifyvm <your VM name> --natpf1 "guestdcachewebservice,tcp,,2288,,2288" VBoxManage modifyvm <your VM name> --natpf1 "guestadmin,tcp,,22223,,22223" VBoxManage modifyvm <your VM name> --natpf1 "guestwebdav,tcp,,22880,,2880" VBoxManage modifyvm <your VM name> --natpf1 "guestwebdavsec,tcp,,22881,,2881"
In case you made a mistake (and there was a typo before in this section), you have to delete the previously created rule before you can add a new one using the same name. E.g. to replace the rule for the WebDAV-rule you have to execute...
VBoxManage modifyvm <your VM name> --natpf1 delete "guestwebdav" VBoxManage modifyvm <your VM name> --natpf1 "guestwebdav,tcp,,22880,,2880"
You should now start VirtualBox and start the Guest OS.
Once the Guest OS has finished booting, you should log in as root using the password .school (that is, DOT--s--c--h--o--o--l).
Before we start the migration process let’s do some preparation and take a backup of the current dCache settings. The backup will allow you to roll back to the initial state, should you have any problems.
When the machine has apparently finished booting then it is possible that dCache is still starting. Therefore we must first wait for dCache to finish starting.
[root@dcachetogo ~]# /opt/d-cache/bin/dcache status Domain Service Status lmDomain lm running dCacheDomain dCache running dirDomain dir running adminDoorDomain admin stopped httpdDomain httpd stopped utilityDomain utility stopped gPlazma-dcachetogoDomain gPlazma stopped chimeraDomain chimera stopped pool1Domain pool stopped pool2Domain pool stopped srm-dcachetogoDomain srm stopped dcap-dcachetogoDomain dcap stopped gridftp-dcachetogoDomain gridftp stopped
... a little while later:
[root@dcachetogo ~]# /opt/d-cache/bin/dcache status Domain Service Status lmDomain lm running dCacheDomain dCache running dirDomain dir running adminDoorDomain admin running httpdDomain httpd running utilityDomain utility running gPlazma-dcachetogoDomain gPlazma running chimeraDomain chimera running pool1Domain pool running pool2Domain pool running srm-dcachetogoDomain srm running dcap-dcachetogoDomain dcap running gridftp-dcachetogoDomain gridftp running
Now we stop dCache:
[root@dcachetogo ~]# /etc/init.d/dcache stop Stopping gridftp-dcachetogoDomain (pid=2485) 0 1 Done Stopping dcap-dcachetogoDomain (pid=2418) 0 1 Done Using CATALINA_BASE: /opt/d-cache/libexec/apache-tomcat-5.5.20 Using CATALINA_HOME: /opt/d-cache/libexec/apache-tomcat-5.5.20 Using CATALINA_TMPDIR: /opt/d-cache/libexec/apache-tomcat-5.5.20/temp Using JRE_HOME: /usr/java/jdk1.6.0_24 Stopping srm-dcachetogo (pid=2107) 0 Done Stopping pool2Domain (pid=2065) 0 1 2 3 Done Stopping pool1Domain (pid=1978) 0 1 2 3 Done Stopping chimeraDomain (pid=1889) 0 1 Done Stopping gPlazma-dcachetogoDomain (pid=1787) 0 Done Stopping utilityDomain (pid=1703) 0 1 Done Stopping httpdDomain (pid=1609) 0 1 Done Stopping adminDoorDomain (pid=1512) 0 1 Done Stopping dirDomain (pid=1428) 0 Done Stopping dCacheDomain (pid=1355) 0 1 Done Stopping lmDomain (pid=1288) 0 Done
Then unmount and stop the Chimera NFS server:
[root@dcachetogo ~]# umount /pnfs [root@dcachetogo ~]# /etc/init.d/chimera stop Shutting down Chimera-NFSv3 interface
There is no separate chimera script in 1.9.12, so we remove the chimera service script from the different run-levels.
[root@dcachetogo ~]# chkconfig --list chimera chimera 0:off 1:off 2:off 3:on 4:on 5:on 6:off [root@dcachetogo ~]# chkconfig --del chimera [root@dcachetogo ~]# chkconfig --list chimera service chimera supports chkconfig, but is not referenced in any runlevel (run 'chkconfig --add chimera')
Copy the dCache home directory. Usually it is necessary to save only the contents of /opt/d-cache/config and /opt-d-cache/etc directories, but to keep things simple, we copy the whole of /opt/d-cache.
[root@dcachetogo ~]# cp -r /opt/d-cache /opt/d-cache_195
After these basic preparations are completed, we can install (or, in our case, upgrade) the new dCache RPM.
[root@dcachetogo ~]# rpm -Uhv ~/install/dcache-server-1.9.12-r15295.noarch.rpm Preparing... ########################################### [100%] 1:dcache-server ########################################### [100%] warning: /opt/d-cache/etc/srm_setup.env saved as /opt/d-cache/etc/srm_setup.env.rpmsave
Migrating
There are a couple of bugs in the migration script we must work-around. These will be fixed with the release of 1.9.12.
[root@dcachetogo ~]# sed -ie 's/tail +4/tail -n +4/' /opt/d-cache/libexec/migrate-from-1.9.5.sh
[root@dcachetogo ~]# mv /opt/d-cache/config/chimera-config.xml{.rpmsave,}
We now run the migration script. This will generate new configuration files: etc/dcache.conf and etc/layouts/dcachetogo.conf
[root@dcachetogo ~]# /opt/d-cache/libexec/migrate-from-1.9.5.sh
Converting /opt/d-cache/config/dCacheSetup to
/opt/d-cache/etc/dcache.conf.
Renaming /opt/d-cache/config/dCacheSetup to
/opt/d-cache/config/dCacheSetup.pre-197
Converting /opt/d-cache/etc/node_config to
/opt/d-cache/etc/layouts/dcachetogo.conf.
Renaming /opt/d-cache/etc/node_config to
/opt/d-cache/etc/node_config.pre-197
DISCLAIMER: Please inspect the generated configuration. The import is
known to be incomplete; in particular if batch files have
been modified or the symbolic links to config/dCacheSetup
have been broken to make domain specific setups. This
includes the case where services have been moved between
domains or service specific configuration parameters are
used in a 1.9.6 style installation.
Checking the migrated configuration for any problems.
[WARNING] dcache.conf:22: Property serviceLocatorHost: use "broker.host"
instead; support for serviceLocatorHost will be
removed in the future
[ERROR] dcache.conf:23: Property java_options: may not be adjusted; See
dcache.java.options or dcache.java.options.extra
[WARNING] dcache.conf:24: Property logArea: use "dcache.log.dir" instead;
support for logArea will be removed in the future
Found 1 error and 2 warnings.
At the end of the migration process, the script automatically runs the command /opt/d-cache/bin/dcache check-config. This command checks for problems in configuration. You should be careful as not all problem can be found automatically! Here is the output if the command is run manually:
[root@dcachetogo ~]# /opt/d-cache/bin/dcache check-config
[WARNING] dcache.conf:22: Property serviceLocatorHost: use "broker.host"
instead; support for serviceLocatorHost will be
removed in the future
[ERROR] dcache.conf:23: Property java_options: may not be adjusted; See
dcache.java.options or dcache.java.options.extra
[WARNING] dcache.conf:24: Property logArea: use "dcache.log.dir" instead;
support for logArea will be removed in the future
Found 1 error and 2 warnings.
WARNINGs are things that should be fixed, but do not prevent dCache from starting. ERRORs are problems so severe that dCache refuses to start.
We must fix the ERROR, and should fix the WARNINGs. All three problems are located in the dcache.conf file. Inspect the file /opt/d-cache/etc/dcache.conf with your favorite editor.
The dcache.conf file adjusts dCache's behaviour by overriding dCache default properties. It replaces the dCacheSetup file.
To fix the three problems returned by the check-config command we need change the following properties: serviceLocatorHost, logArea and java_options.
Fixing serviceLocatorHost and logArea is easy: these properties have new names broker.host and dcache.log.dir, as described in the warning messages. Make the following changes to the dcache.conf file:
- change the line serviceLocatorHost=localhost to broker.host=localhost
- change the line logArea=/var/log/dCache to dcache.log.dir=/var/log/dCache
The error on line 23 ("Property java_options: may not be adjusted") is bit more tricky. In our case, the java_options property was configured to allocate additional memory to the Java process: -Xmx64m means the maximum heap memory size is increased to 64 MiB and -XX:MaxDirectMemorySize=64m means the maximum direct-memory (for direct IO) is also increased to 64 MiB.
With dCache v1.9.12, changing how much memory Java should use is much easier. The two properties dcache.java.memory.heap and dcache.java.memory.direct may be used to adjust the amount of memory Java will use.
Edit the file dcache.conf with your favourite editor and make the following changes:
- Remove the property line that starts java_options=
- Add the line dcache.java.memory.heap=64m
- Add the line dcache.java.memory.direct=64m
You should re-run the check-config command. This should now give no warnings and no errors.
[root@dcachetogo ~]# /opt/d-cache/bin/dcache check-config No problems found.
The next step is to inspect layout file. You don't need to change anything at this time, but it is useful to familiarise yourself with its structure.
By default, the layout file is single (corresponding to /opt/d-cache/etc/layouts/single.conf). However, the migration script will follow the recommended practise of using the host's name. This is achieved by dcache.conf containing the line: dcache.layout=${host.name}
Looking at the layout file /opt/d-cache/etc/layouts/dcachetogo.conf you should recognise the list of domains with one or more service inside each domain. The layout file can have configuration lines in it too, but the migration script will not generate any.
Here is a fragment from the layout file /opt/d-cache/etc/layouts/dcachetogo.conf
[dCacheDomain] [dCacheDomain/poolmanager] [dCacheDomain/broadcast] [dCacheDomain/loginbroker] [dCacheDomain/topo]
The line [dCacheDomain] declares a new domain called dCacheDomain.
The line [dCacheDomain/poolmanager] declares that the dCacheDomain domain should start a service of type poolmanager. The [dCacheDomain/broadcast] line means dCacheDomain domain will also host a broadcast service, and so on.
Final step.
Now we start the migrated dCache:
[root@dcachetogo ~]# /etc/init.d/dcache start Starting dCacheDomain done Starting dirDomain done Starting adminDoorDomain done Starting httpdDomain done Starting utilityDomain done Starting gPlazma-dcachetogoDomain done Starting namespaceDomain done Starting pool1Domain done Starting pool2Domain done Starting srm-dcachetogoDomain done Starting dcap-dcachetogoDomain done Starting gridftp-dcachetogoDomain done
Be aware that domains may take a little longer to start once dcache start returns.
We should check some basic functionality.
First, we make sure that the dcap door is ready. If it isn't ready, just wait.
[dc_user@dcachetogo ~]$ netstat -nl|grep 22125 tcp 0 0 0.0.0.0:22125 0.0.0.0:* LISTEN
Now we try writing data into dCache and read that data back again.
[root@dcachetogo ~]# su - dc_user [dc_user@dcachetogo ~]$ /usr/bin/dccp /bin/bash dcap://localhost/pnfs/dcache.org/data/worldwritable/test1 877480 bytes (857 kiB) in 0 seconds [dc_user@dcachetogo ~]$ /usr/bin/dccp dcap://localhost/pnfs/dcache.org/data/worldwritable/test1 /dev/null 877480 bytes (857 kiB) in 0 seconds [dc_user@dcachetogo ~]$ exit logout
dCache 1.9.12 (with Chimera) doesn't need the namespace to be mounted: dCache will work fine. (For PNFS, the PnfsManager needs the namespace to be mounted but that's the only component that needs it.)
The migration script doesn't configure an NFS server. So, after the migration, there's no mounted namespace:
[root@dcachetogo ~]# mount /dev/sda1 on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
You might want the namespace mounted anyway, for administrative purposes. dCache isn't running an NFS server at the moment.
Let's fix that.
Edit the layout file, /opt/d-cache/etc/layouts/dcachetogo.conf, and add the following line:
[namespaceDomain/nfsv3]
The line can go anywhere after the domain declaration, [namespaceDomain], but it's good practise to put all the service declarations immediately after their corresponding domain declaration.
Once you have edited the layout file, you can verify that dCache will start the NFSv3 service by running the dcache service command:
[root@dcachetogo ~]# /opt/d-cache/bin/dcache services | grep ^namespaceDomain namespaceDomain pnfsmanager PnfsManager /var/log/dCache/namespaceDomain.log namespaceDomain cleaner cleaner /var/log/dCache/namespaceDomain.log namespaceDomain acl acladmin /var/log/dCache/namespaceDomain.log namespaceDomain nfsv3 NFSv3-dcachetogo /var/log/dCache/namespaceDomain.log
Having changed which services are to run in the namespaceDomain domain, we must restart the domain:
[root@dcachetogo ~]# /opt/d-cache/bin/dcache restart namespaceDomain Stopping namespaceDomain (pid=17545) 0 1 2 3 done Starting namespaceDomain done
Now that namespaceDomain is hosting the NFS server, we can mount the namespace at the usual mount-point: /pnfs
[root@dcachetogo ~]# mount -o intr,rw,nolock,noac,hard,nfsvers=3 localhost:/pnfs /pnfs
We can verify that the namespace is available by looking at the file test1 that was copied into dCache earlier as a test:
[root@dcachetogo ~]# stat /pnfs/dcache.org/data/worldwritable/test1 File: `/pnfs/dcache.org/data/worldwritable/test1' Size: 877480 Blocks: 1714 IO Block: 32768 regular file Device: 14h/20d Inode: 18446744072961198379 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 500/ dc_user) Gid: ( 500/ dc_user) Access: 2011-03-15 11:49:50.000000000 +0100 Modify: 2011-03-15 11:49:53.000000000 +0100 Change: 2011-03-15 11:49:50.000000000 +0100
Before we finish, let's clean up all the broken symbolic links in /opt/d-cache/config
[root@dcachetogo ~]# find -L /opt/d-cache/config -type l
/opt/d-cache/config/dirSetup
/opt/d-cache/config/infoProviderSetup
/opt/d-cache/config/gridftpdoorSetup
/opt/d-cache/config/gPlazmaSetup
/opt/d-cache/config/xrootdDoorSetup
/opt/d-cache/config/kerberosdcapdoorSetup
/opt/d-cache/config/doorSetup
/opt/d-cache/config/gsidcapdoorSetup
/opt/d-cache/config/poolSetup
/opt/d-cache/config/kerberosftpdoorSetup
/opt/d-cache/config/infoSetup
/opt/d-cache/config/nfsv41Setup
/opt/d-cache/config/chimeraSetup
/opt/d-cache/config/httpdSetup
/opt/d-cache/config/httpdoorSetup
/opt/d-cache/config/utilitySetup
/opt/d-cache/config/weakftpdoorSetup
/opt/d-cache/config/srmSetup
/opt/d-cache/config/replicaSetup
/opt/d-cache/config/adminDoorSetup
/opt/d-cache/config/pnfsSetup
/opt/d-cache/config/authdoorSetup
/opt/d-cache/config/statisticsSetup
/opt/d-cache/config/lmSetup
[root@dcachetogo dCache]# find -L /opt/d-cache/config -type l -exec rm {} ;
[root@dcachetogo dCache]# find -L /opt/d-cache/config -type l
The simple migration scenario is now completed.
More complicated scenario.
Now we try migrating a node based on a real system (from Goettingen) but with some awkward configuration added to stress the migration process.
Preparation.
The VirtualBox image keeps another configuration which is more close to the real setup. To enable it, we have to switch dCache home directories:
[root@dcachetogo ~]# umount /pnfs [root@dcachetogo ~]# /opt/d-cache/bin/dcache stop Stopping gridftp-dcachetogoDomain (pid=5142) 0 1 done Stopping dcap-dcachetogoDomain (pid=5090) 0 done Stopping srm-dcachetogoDomain (pid=5030) 0 1 2 3 done Stopping pool2Domain (pid=4962) 0 1 2 done Stopping pool1Domain (pid=4908) 0 1 2 done Stopping namespaceDomain (pid=5621) 0 1 2 3 done Stopping gPlazma-dcachetogoDomain (pid=4812) 0 done Stopping utilityDomain (pid=4761) 0 done Stopping httpdDomain (pid=4720) 0 1 done Stopping adminDoorDomain (pid=4679) 0 1 done Stopping dirDomain (pid=4642) 0 done Stopping dCacheDomain (pid=4600) 0 done [root@dcachetogo ~]# mv /opt/d-cache /opt/d-cache_1912 [root@dcachetogo ~]# mv /opt/d-cache_real/ /opt/d-cache
We also work-around some last-minute problems:
[root@dcachetogo ~]# sed -nie '/^p/p' /opt/d-cache/config/pools.poollist
[root@dcachetogo ~]# sed -ie 's/statistics=yes/statistics=no/' /opt/d-cache/etc/node_config
[root@dcachetogo ~]# sed -ie 's/billingToDb=yes//' /opt/d-cache/config/{dCache,httpd}Setup
Activate it with some magic, by rolling back to 1.9.5.
[root@dcachetogo ~]# rpm -Uhv --force ~/install/dcache-server-1.9.5-24.noarch.rpm Preparing... ########################################### [100%] 1:dcache-server########################################### [100%]
Start Chimera and mount PNFS
[root@dcachetogo ~]# /etc/init.d/chimera start Starting Chimera-NFSv3 interface rpcinfo: RPC: Unable to receive; errno = Connection refused Waiting for portmap port: rpcinfo: RPC: Unable to receive; errno = Connection refused .rpcinfo: RPC: Unable to receive; errno = Connection refused .rpcinfo: RPC: Unable to receive; errno = Connection refused .rpcinfo: RPC: Unable to receive; errno = Connection refused .rpcinfo: RPC: Unable to receive; errno = Connection refused .rpcinfo: RPC: Unable to receive; errno = Connection refused . [root@dcachetogo ~]# mount -o nolock,intr,rw,noac,hard,nfsvers=3 localhost:/pnfs /pnfs
We copy the previous PoolManager.conf file, as the one supplied is only good for Goettingen machines
[root@dcachetogo ~]# cp /opt/d-cache{_195,}/config/PoolManager.conf
cp: overwrite `/opt/d-cache/config/PoolManager.conf'? y
Start dCache:
[root@dcachetogo ~]# /opt/d-cache/bin/dcache start Starting lmDomain 6 Done Starting dCacheDomain 6 Done Starting dirDomain 6 Done Starting adminDoorDomain 6 Done Starting httpdDomain 6 Done Starting utilityDomain 6 Done Starting gPlazma-dcachetogoDomain 6 Done Starting chimeraDomain 6 Done Starting poolsDomain 6 Done Using CATALINA_BASE: /opt/d-cache/libexec/apache-tomcat-5.5.20 Using CATALINA_HOME: /opt/d-cache/libexec/apache-tomcat-5.5.20 Using CATALINA_TMPDIR: /opt/d-cache/libexec/apache-tomcat-5.5.20/temp Using JRE_HOME: /usr/java/jdk1.6.0_24 Pinging srm server to wake it up, will take few seconds ... VersionInfo : v2.2 backend_type:dCache backend_version:production-1.9.5-24 Done httpdDomain might still be running Starting dcap-dcachetogoDomain 6 Done
Verify that the dcap door is running (just wait until something is LISTENing on port 22125)
[root@dcachetogo ~]# netstat --tcp -nl|grep 22125 tcp 0 0 0.0.0.0:22125 0.0.0.0:* LISTEN
To demonstrate that this dCache instance is working, we'll copy another file:
[root@dcachetogo ~]# su - dc_user [dc_user@dcachetogo ~]$ /usr/bin/dccp /bin/bash dcap://localhost/pnfs/dcache.org/data/worldwritable/test2 877480 bytes (857 kiB) in 0 seconds [dc_user@dcachetogo ~]$ /usr/bin/dccp dcap://localhost/pnfs/dcache.org/data/worldwritable/test2 /dev/null 877480 bytes (857 kiB) in 0 seconds [dc_user@dcachetogo ~]$ exit logout
We must stop dCache and Chimera before we migrate to 1.9.12
[root@dcachetogo ~]# /opt/d-cache/bin/dcache stop Stopping dcap-dcachetogoDomain (pid=16555) 0 Done Stopping httpdDomain (pid=15465) 0 Done Using CATALINA_BASE: /opt/d-cache/libexec/apache-tomcat-5.5.20 Using CATALINA_HOME: /opt/d-cache/libexec/apache-tomcat-5.5.20 Using CATALINA_TMPDIR: /opt/d-cache/libexec/apache-tomcat-5.5.20/temp Using JRE_HOME: /usr/java/jdk1.6.0_24 Stopping srm-dcachetogo (pid=15962) 0 Done Stopping statisticsDomain (pid=20363) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Done Stopping poolsDomain (pid=19126) 0 1 2 Done Stopping chimeraDomain (pid=15744) 0 Done Stopping gPlazma-dcachetogoDomain (pid=15646) 0 Done Stopping utilityDomain (pid=15560) 0 1 Done Stopping adminDoorDomain (pid=15379) 0 Done Stopping dirDomain (pid=15297) 0 Done Stopping dCacheDomain (pid=15224) 0 Done Stopping lmDomain (pid=15156) 0 Done [root@dcachetogo ~]# umount /pnfs [root@dcachetogo ~]# /etc/init.d/chimera stop Shutting down Chimera-NFSv3 interface
Migration.
Next, we update to dCache v1.9.12
[root@dcachetogo ~]# rpm -Uhv ~/install/dcache-server-1.9.12-r15295.noarch.rpm Preparing... ########################################### [100%] 1:dcache-server ########################################### [100%]
Apply the same patches to avoid bugs in the current migration script (these are already fixed in later release-candidates of 1.9.12).
[root@dcachetogo ~]# sed -ie 's/tail +4/tail -n +4/' /opt/d-cache/libexec/migrate-from-1.9.5.sh
[root@dcachetogo ~]# cp /opt/d-cache{_195,}/config/chimera-config.xml
Execute the migration script.
[root@dcachetogo ~]# /opt/d-cache/libexec/migrate-from-1.9.5.sh
Converting /opt/d-cache/config/dCacheSetup to
/opt/d-cache/etc/dcache.conf.
Renaming /opt/d-cache/config/dCacheSetup to
/opt/d-cache/config/dCacheSetup.pre-197
Converting /opt/d-cache/etc/node_config to
/opt/d-cache/etc/layouts/dcachetogo.conf.
Renaming /opt/d-cache/etc/node_config to
/opt/d-cache/etc/node_config.pre-197
DISCLAIMER: Please inspect the generated configuration. The import is
known to be incomplete; in particular if batch files have
been modified or the symbolic links to config/dCacheSetup
have been broken to make domain specific setups. This
includes the case where services have been moved between
domains or service specific configuration parameters are
used in a 1.9.6 style installation.
Checking the migrated configuration for any problems.
[WARNING] dcache.conf:22: Property serviceLocatorHost: use "broker.host"
instead; support for serviceLocatorHost will be
removed in the future
[ERROR] dcache.conf:23: Property java_options: may not be adjusted; See
dcache.java.options or dcache.java.options.extra
Found 1 error and 1 warning.
You probably remember these findings, as they already showed up in the previous migration example. In order to fix them, dit the file /opt/d-cache/etc/dcache.conf with your favourite editor and apply the following changes:
- change the line serviceLocatorHost=localhost to broker.host=localhost
- Remove the property line that starts java_options=
- Add the line dcache.java.memory.heap=64m
- Add the line dcache.java.memory.direct=64m
Now, the check-config command should give no errors.
[root@dcachetogo ~]# /opt/d-cache/bin/dcache check-config No problems found.
If you're sure that all conflicts are resolved, you may try to start dCache.
[root@dcachetogo d-cache]# /opt/d-cache/bin/dcache start Starting dCacheDomain done Starting dirDomain done Starting adminDoorDomain done Starting httpdDomain done Starting utilityDomain done Starting gPlazma-dcachetogoDomain done Starting namespaceDomain done Starting poolsDomain done Starting srm-dcachetogoDomain done Starting dcap-dcachetogoDomain done
But looking at the current status for the services, something went wrong with the httpd server.
[root@dcachetogo ~]# /opt/d-cache/bin/dcache status DOMAIN STATUS PID USER dCacheDomain running 12117 root dirDomain running 12154 root adminDoorDomain running 12195 root httpdDomain restarting [whoever runs "dcache start"] utilityDomain running 12278 root gPlazma-dcachetogoDomain running 12320 root namespaceDomain running 12371 root poolsDomain running 12425 root srm-dcachetogoDomain running 12480 root dcap-dcachetogoDomain running 12561 root
To understand what's happening, look back at the output when running /opt/d-cache/bin/dcache start with dCache v1.9.5. You see a line like:
httpdDomain might still be running
This is because the configuration attempts to start httpdDomain twice. In 1.9.5, the second attempt prints a warning message but doesn't affect the existing httpdDomain. It's working "by accident".
After migrating, the layout file will include two httpdDomain domain declarations with their corresponding collection of services. You should be able to see this in the file /opt/d-cache/etc/layouts/dcachetogo.conf. The effect is that the configuration attempts to instantiate each httpdDomain service twice. Some of these services cannot be instantiated twice with the default configuration (e.g., httpd service), which causes the httpdDomain to fail.
The fix is simple: delete one of the httpdDomain domain declarations along with it's corresponding set of services; then either restart the httpdDomain explicitly or simply wait for the automatic restart.
[root@dcachetogo ~]# /opt/d-cache/bin/dcache restart httpdDomain Stopping httpdDomain (pid=12231) done Starting httpdDomain done
Final step.
Let's tidy up the broken symbolic links.
[root@dcachetogo d-cache]# find -L /opt/d-cache/config -type l -exec rm {} ;
Hop-la seems that the http daemon on the old setup has his own settings
[root@dcachetogo d-cache]# find /opt/d-cache/config/ -name "*Setup"
/opt/d-cache/config/httpdSetup
[root@dcachetogo ~]# diff -u /opt/d-cache/config/{dCacheSetup.pre-197,httpdSetup}
--- /opt/d-cache/config/dCacheSetup.pre-197 2011-03-15 14:59:06.495324515 +0100
+++ /opt/d-cache/config/httpdSetup 2011-03-15 14:59:06.500325150 +0100
@@ -1,7 +1,7 @@
serviceLocatorHost=localhost
serviceLocatorPort=11111
java="/usr/bin/java"
-java_options="-server -Xmx64m -XX:MaxDirectMemorySize=64m
+java_options="-server -Xmx32m -XX:MaxDirectMemorySize=32m
-Dsun.net.inetaddr.ttl=1800
-Dorg.globus.tcp.port.range=20000,25000
-Djava.net.preferIPv4Stack=true
The httpdDomain has less memory allocated to it. We can configure this using the dcache.java.memory.heap and dcache.java.memory.direct properties inside the layout file.
[httpdDomain] dcache.java.memory.heap=32m dcache.java.memory.direct=32m [httpdDomain/httpd] [httpdDomain/billing] [httpdDomain/srm-loginbroker]
We need to restart of httpdDomain for it to use the reduced memory.
[root@dcachetogo ~]# /opt/d-cache/bin/dcache restart httpdDomain Stopping httpdDomain (pid=14844) 0 1 done Starting httpdDomain done
Here we double check that the httpdDomain has been started with the reduced memory
[root@dcachetogo ~]# ps -C java -o cmd|sed 's/-D.*start/[..]/' CMD /usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] dCacheDomain /usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] dirDomain /usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] adminDoorDomain /usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] utilityDomain /usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] gPlazma-dcachetogoDomain /usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] namespaceDomain /usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] poolsDomain /usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] srm-dcachetogoDomain /usr/bin/java -server -Xmx64m -XX:MaxDirectMemorySize=64m [..] dcap-dcachetogoDomain /usr/bin/java -server -Xmx32m -XX:MaxDirectMemorySize=32m [..] httpdDomain
Congratulations! You have completed the second migration successfully.
Now you may shuffle around services within domains, merge domains and enable new features like NFS4.1 and/or WebDAV.
