| Home | next >> |
- Installation of dCache server - Taipei2014
- Writing to / Reading from dCache
- WebDAV using certificate-based authentication
- Pool Management And Pool Selection
- Hardware Lifecycle (adding storage, migrating)
Installation of dCache server
(SERVER) (SERVER) (SERVER) (SERVER)
In this part we install dCache as a single node instance. Which means that all our services will run on the same host, it shows us the basics and necessary setup. At the end of this part we will have a running dCache that can be mounted locally as well as on your client.
Find the dCache server rpm on your server in the root's home direcrory /root/:
[root]# ls dcache-* dcache-2.6.21-1.noarch.rpm
If you are using your own virtual machine, please download the rpm from dCache Downloads
Install the rpm on your server machine:
[root]# rpm -ivh dcache-2.6.21-1.noarch.rpm Preparing... ########################################### [100%] 1:dcache ########################################### [100%] ...
Set the owner for the host certificates: We already placed a X.509 host certificate in /etc/grid-security/hostcert.pem and /etc/grid-security/hostkey.pem . During installation of the RPM a user and group "dcache" was created, and these two files need to get the right owners/permissions, so that dCache can use them:
[root]# chown dcache:dcache /etc/grid-security/host*.pem [root]# chmod u=rw,go=r /etc/grid-security/hostcert.pem [root]# chmod u=rw,go= /etc/grid-security/hostkey.pem
dCache server has dependencies: java-1.7.0-openjdk, postgresql-server (version >8.4), rpcbind and nfs-utils. You will find that openjdk, postgresql-server, rpcbind and nfs-utils are already installed on your machine. Run the following command to check for the java, postgresql, rpcbind and nfs-utils packages.
[root]# rpm -qa |grep -E "postgresql92-server|java-1.7.0-openjdk|nfs-utils|rpcbind" java-1.7.0-openjdk-1.7.0.51-2.4.4.1.el6_5.x86_64 postgresql92-server-9.2.7-1PGDG.rhel6.x86_64 nfs-utils-1.2.3-36.el6.x86_64 nfs-utils-lib-1.1.5-6.el6.x86_64 rpcbind-0.2.0-11.el6.x86_64
Look this up later: For more information on this please see dCache Book - Prerequisites. There are also several trivial steps that you have to do to setup postgresql, but they are not part of this tutorial and can be found in dCache Book - Readying the PostgreSQL server for the use with dCache We don't focus on detailed configuration of postgres server here, but have a look at the provided script for the commands needed if you are interested (configurepgsql.sh).
[root]# cat configurepgsql.sh
#/bin/bash
service postgresql-9.2 initdb
sed -ie 's/max_connections = 100/max_connections = 1000/' /var/lib/pgsql/9.2/data/postgresql.conf
service postgresql-9.2 start
chkconfig postgresql-9.2 on
mv /var/lib/pgsql/9.2/data/pg_hba.conf{,_ori}
cat >/var/lib/pgsql/9.2/data/pg_hba.conf <<EOF
# TYPE DATABASE USER CIDR-ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust
EOF
service postgresql-9.2 restart
createdb -U postgres chimera
createuser -U postgres --no-superuser --no-createrole --createdb chimera
createuser -U postgres --no-superuser --no-createrole --createdb srmdcache
createdb -U srmdcache dcache
createdb -O srmdcache -U postgres billing
dcache database update
Now execute the configurepgsql.sh script:
[root]# ./configurepgsql.sh Initializing database: [ OK ] Starting postgresql-9.2 service: [ OK ] Stopping postgresql-9.2 service: [ OK ] Starting postgresql-9.2 service: [ OK ] ... some strings here ... INFO - Successfully released change log lock Liquibase Update Successful [root]#
Now you have a running and fully configured postgresql server and installed dCache server.
Configuration files
We make use of flat files, to define and change the settings for the layout and the behavior of dCache on different levels (per host, per domain or per service). There are three main places for the configuration files:
- /usr/share/dcache/defaults
This directory is filled with files defining the default settings for all dCache services, as they are shipped by dCache.org. Do not modify these files, as they will be replaced by subsequent updates!
- /etc/dcache/dcache.conf
The central configuration file, that ideally should be nearly identical on all nodes of the dCache setup. Maybe it is possible to have only one difference among all nodes at all: the parameter pointing to the layout configuration files. To get to know what settings can be made in dcache.conf you can look through dcache.properties in the defaults directory.
- /etc/dcache/layouts
Layout files are the place to define the actual topology for the dCache services/domains on this node. So typically layout files are identical for nodes that have same services running on it (EX: gftp doors & pools).
dCache.org provides us with premade layout files that state a possible distribution of services over domains: head.conf, pool.conf and single.conf. Right now you could start dCache and it would use the empty dcache.conf file and the fallback layout file single.conf. With this the most important core services will be configured to run in one single domain with default settings. Alternatively, head.conf has predefined the mandatory services in a decent number of domains to be run on the headnode of your setup. Of course, you will need at least one other node using pool.conf to provide some disk space to dCache.
We would like to have our own layout file that we just use for this hands-on, therefor we create it from a template:
[root]# cp /etc/dcache/layouts/single.conf /etc/dcache/layouts/taipei2014_ws.conf
We need to tell dCache to use the layout file we just created and add this to /etc/dcache/dcache.conf:
[root]# vi /etc/dcache/dcache.conf
Do not make the mistake of entering the entire file name there, just add the following line:
dcache.layout=taipei2014_ws
Leave vi open for the moment and keep on reading.
Adjust the layout file
Firstly we need to tell dCache that there will be many domains communicating with each other. As we described in the introduction dCache services run inside so called domains. Since we will have many domains, we will need some mechanism for these domains to communicate. The cells framework is used for this communication, which is why we activate it by adding the following line to the beginning of /etc/dcache/dcache.conf:
broker.scheme=cells
NOTE: broker.scheme=cells is default dCache configuration so not really necessary to define.
As mentioned we want to be able to mount dCache locally using NFSv41. Therefor a service called nfsv41 needs to be started, which we will keep in a separate domain for convenience of restarting it separately from the rest of dCache. These lines need to be added at the end of /etc/dcache/layouts/taipei2014_ws.conf. They add a domain to dCache - [nfs-Domain] - that holds the nfsv41 service.
[nfs-Domain] [nfs-Domain/nfsv41]
Save and leave vi and have a look at the layout file:
[root]# cat /etc/dcache/layouts/taipei2014_ws.conf [dCacheDomain] [dCacheDomain/admin] [dCacheDomain/broadcast] [dCacheDomain/poolmanager] [dCacheDomain/loginbroker] [dCacheDomain/spacemanager] [dCacheDomain/pnfsmanager] [dCacheDomain/cleaner] [dCacheDomain/dir] [dCacheDomain/gplazma] [dCacheDomain/gsi-pam] [dCacheDomain/pinmanager] [dCacheDomain/billing] [dCacheDomain/srm-loginbroker] [dCacheDomain/httpd] [dCacheDomain/topo] [dCacheDomain/info] [nfs-Domain] [nfs-Domain/nfsv41] # [dCacheDomain/pool] # name=pool1 # path=/path/to/pool1 # [dCacheDomain/pool] # name=pool2 # path=/path/to/pool2 # [dCacheDomain/replica] # [dCacheDomain/dcap] # [dCacheDomain/gsidcap] # [dCacheDomain/gridftp] # [dCacheDomain/srm] # [dCacheDomain/transfermanagers] # [dCacheDomain/xrootd] # [dCacheDomain/webdav] # [dCacheDomain/webadmin] [root]#
Before we can start dCache we have to empty /etc/dcache/gplazma.conf as there is no security configured in a blank dCache:
[root]# echo "" > /etc/dcache/gplazma.conf
Then we want to be sure rpcbind service is up and running (in order to start nfs4 service correctly):
[root] # service rpcbind status rpcbind is stopped [root] # service rpcbind start Starting rpcbind: [ OK ] [root] # service rpcbind status rpcbind (pid 4178) is running...
Check that all is done properly before starting dcache:
[root]# dcache check-config No problems found.
Now start dCache by:
[root]# dcache start Starting dCacheDomain done Starting nfs-Domain done
Check if dCache has started up correctly for domains:
[root]# dcache status DOMAIN STATUS PID USER dCacheDomain running 18314 dcache nfs-Domain running 18359 dcache
and services:
[root]# dcache services DOMAIN SERVICE CELL LOG dCacheDomain admin alm /var/log/dcache/dCacheDomain.log dCacheDomain broadcast broadcast /var/log/dcache/dCacheDomain.log dCacheDomain poolmanager PoolManager /var/log/dcache/dCacheDomain.log dCacheDomain loginbroker LoginBroker /var/log/dcache/dCacheDomain.log dCacheDomain spacemanager SrmSpaceManager /var/log/dcache/dCacheDomain.log dCacheDomain pnfsmanager PnfsManager /var/log/dcache/dCacheDomain.log dCacheDomain cleaner cleaner /var/log/dcache/dCacheDomain.log dCacheDomain dir dirLookupPool /var/log/dcache/dCacheDomain.log dCacheDomain gplazma gPlazma /var/log/dcache/dCacheDomain.log dCacheDomain pinmanager PinManager /var/log/dcache/dCacheDomain.log dCacheDomain billing billing /var/log/dcache/dCacheDomain.log dCacheDomain srm-loginbroker srm-LoginBroker /var/log/dcache/dCacheDomain.log dCacheDomain httpd httpd /var/log/dcache/dCacheDomain.log dCacheDomain topo topo /var/log/dcache/dCacheDomain.log dCacheDomain info info /var/log/dcache/dCacheDomain.log nfs-Domain nfsv41 NFSv41-ws-server-001 /var/log/dcache/nfs-Domain.log
Then also check the log files:
[root]# tail -F /var/log/dcache/* ==> /var/log/dcache/dCacheDomain.log <== 2014-03-06 11:00:53 Launching /usr/bin/java -server -Xmx512m -XX:MaxDirectMemorySize=512m -Dsun.net.inetaddr.ttl=1800 -Dorg.globus.tcp.port.range=20000,25000 -Djava.net.preferIPv4Stack=true -Dorg.dcache.dcap.port=0 -Dorg.dcache.net.tcp.portrange=33115:33145 -Dorg.globus.jglobus.delegation.cache.lifetime=30000 -Dorg.globus.jglobus.crl.cache.lifetime=60000 -Djava.security.krb5.realm=EXAMPLE.ORG -Djava.security.krb5.kdc=localhost -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.auth.login.config=/etc/dcache/jgss.conf -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dcache/dCacheDomain-oom.hprof -javaagent:/usr/share/dcache/classes/spring-instrument-3.2.2.RELEASE.jar -Djava.awt.headless=true -DwantLog4jSetup=n -Ddcache.home=/usr/share/dcache -Ddcache.paths.defaults=/usr/share/dcache/defaults org.dcache.boot.BootLoader start dCacheDomain 06 Mar 2014 11:01:32 (PinManager) [] [AspectJ] javax.* types are not being woven because the weaver option '-Xset:weaveJavaxPackages=true' has not been specified ==> /var/log/dcache/nfs-Domain.log <== 2014-03-06 11:00:54 Launching /usr/bin/java -server -Xmx512m -XX:MaxDirectMemorySize=512m -Dsun.net.inetaddr.ttl=1800 -Dorg.globus.tcp.port.range=20000,25000 -Djava.net.preferIPv4Stack=true -Dorg.dcache.dcap.port=0 -Dorg.dcache.net.tcp.portrange=33115:33145 -Dorg.globus.jglobus.delegation.cache.lifetime=30000 -Dorg.globus.jglobus.crl.cache.lifetime=60000 -Djava.security.krb5.realm=EXAMPLE.ORG -Djava.security.krb5.kdc=localhost -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.auth.login.config=/etc/dcache/jgss.conf -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dcache/nfs-Domain-oom.hprof -javaagent:/usr/share/dcache/classes/spring-instrument-3.2.2.RELEASE.jar -Djava.awt.headless=true -DwantLog4jSetup=n -Ddcache.home=/usr/share/dcache -Ddcache.paths.defaults=/usr/share/dcache/defaults org.dcache.boot.BootLoader start nfs-Domain
Now exit the tail command by pressing control+c.
This should show you that domains were created and no error messages should be flying around, so the output that tail shows should stop at some point after starting dCache although some output is normal to inform about successful domain creation. Sometimes a domain might report that it could not connect to the dCacheDomain, because it might have started up in advance of dCacheDomain. If however "dcache services" reports that the domain and service have been started, it was correctly connected after the dCacheDomain came up.
First Contact
Now that we have a running dCache that does nothing we would like to make a first contact. This will be achieved by mounting dCache locally via NFSv41.
dCache needs several things to come together to be able to store data in it. It needs to authenticate and authorize people wanting to store data (gPlazma takes care of this), it needs a logical structure where file's names can be found in a directory structure, the namespace, which keeps file's meta data. The actual files are stored in a service that is called a pool that allows writing data to block devices and allows for a hierarchical storage management to be employed (More about this in the tertiary storage part of this hands-on).
dCache needs something that holds that actual data, the pools. So we first create a pool that is stored in a certain path in the local file system and set its maximum size. Execute the following command in your server console:
[root]# dcache pool create --size=419430400 --meta=db --lfs=precious /pools/nfsPool1 nfsPool1 poolDomain Created a pool in /pools/nfsPool1. The pool was added to poolDomain in file:/etc/dcache/layouts/taipei2014_ws.conf.
[root]# cat /etc/dcache/layouts/taipei2014_ws.conf
[dCacheDomain]
[dCacheDomain/admin]
...
...
[poolDomain]
[poolDomain/pool]
name=nfsPool1
path=/pools/nfsPool1
maxDiskSpace=419430400
metaDataRepository=org.dcache.pool.repository.meta.db.BerkeleyDBMetaDataRepository
waitForFiles=${path}/data:${path}/meta
lfs=precious
After we created the pool we need to start the domain that was created by executing:
[root]# dcache status DOMAIN STATUS PID USER dCacheDomain running (for 118 seconds) 4509 dcache nfs-Domain running (for 118 seconds) 4564 dcache poolDomain stopped dcache
[root]# dcache start poolDomain Starting poolDomain done
The pool service will always make sure that there is enough space on the pool to allow for some data to be moved around. The amount of space that is kept free is configurable and by default it is 4 GiB. For this tutorial we need to set the gap to a very low value (10 MB) because the disk space on the hands-on machines is so limited. In order to be able to administrate dCache we need to login to the admin interface. We will use key-based login with ssh2. Please create yourself a DSA key and store the public key in /etc/dcache/admin/authorized_keys2. Do the following steps to change the gap size (when generating dsa key use default file name and NO password):
[root]# ssh-keygen -t dsa ... [root]# cat .ssh/id_dsa.pub > /etc/dcache/admin/authorized_keys2 [root]# ssh -l admin -p 22224 localhost ... (local) admin > cd nfsPool1 ... (nfsPool) admin > set gap 10m Gap set to 10485760 ... (nfsPool) admin > save
Then exit the admin interface by pressing control+d.
Try to create yourself second pool with name "nfsPool2" assigned to the same domain it will be needed for the next exercises (don't forget about gap size).
This now enables us to actually store files in dCache, but how? ... for example via a mounted NFSv41.
We need to set the NFS domain to make sure nfs server and client are inside the same namespace. This is done by adding the following line to the end of /etc/dcache/dcache.conf:
nfs.domain = taipei-domain
As mentioned dCache also needs a namespace where file's meta data is stored. We need to create a directory in the dCache namespace by executing:
[root]# chimera-cli mkdir /data [root]# chimera-cli mkdir /data/world-writable
We also need to tell the nfs server that machines are allowed to mount certain directories that the dCache server provides access to. This is not really a dCache specific thing as you would also use the /etc/exports file for a non-dCache nfs servers. Please add the following line to /etc/exports:
/data localhost(rw,no_root_squash)
This states that the directory /data can be accessed by localhost with read/write permissions.
We do not have authentication setup yet, which is why we need to set the directory world writable to be able to write:
[root]# chimera-cli chmod /data/world-writable 777
Now we will be able to see the benefit of having the nfsv41 service in a separate domain as we need to restart the domain for the change in /etc/exports to take effect.
[root]# dcache status DOMAIN STATUS PID USER dCacheDomain running (for 27 minutes) 4509 dcache nfs-Domain running (for 27 minutes) 4564 dcache poolDomain running (for 25 minutes) 5229 dcache [root]# dcache restart nfs-Domain Stopping nfs-Domain 0 1 2 done Starting nfs-Domain done
If we not had the nfsv41 service running in a separate domain we would have had to restart entire dCache, which can be very bad when you want to maintain a production dCache in service.
After this we can mount dCache locally by doing the following:
[root]# mkdir /data [root]# mount -o intr,minorversion=1 localhost:/data /data
Now you can copy some file e.g.:
[root]# cp /etc/hosts /data/world-writable/myFirstNfsFile00001
just make an ls to be sure the file is actually copied
[root]# ls -la /data/world-writable/myFirstNfsFile00001
Congratulations you wrote your first file into dCache. We will now mount nfsv41 on the client. Therefor we need to grant it permission on the server side. Therefore edit /etc/exports:
[root]# vi /etc/exports
and change the one line from
/data localhost(rw,no_root_squash)
to
/data localhost(rw,no_root_squash) <your client IP address>(rw,no_root_squash)
or to
/data localhost(rw,no_root_squash) <your client HOSTNAME>(rw,no_root_squash)
then exit INSERT mode and save. This line now means that directory /data can be mounted with read-write permissions from localhost as well as <your client address>. After addition of this permission restart the nfs-Domain once more:
[root]# dcache restart nfs-Domain
Now go to your client:
(CLIENT) (CLIENT) (CLIENT) (CLIENT)
There you can have a look at:
[root]# less /etc/idmapd.conf
At the top of this file you will find:
[General] #Verbosity = 0 # The following should be set to the local NFSv4 domain name # The default is the host's DNS domain name. Domain = taipei-domain
Remember, earlier we set the nfs.domain = taipei-domain in the dcache.conf on the server. This is now telling server and client to be in the same domain. If you had to change this value, which we luckily already did for you, you would have to restart the /etc/rpcidmapd service. But that is not necessary, now.
Check if the machine has a /data directory (just type "ls /data"), in case not just create with:
[root]# mkdir /data
then mount and have a look at the mounted dCache and unmount:
[root]# mount -o intr,minorversion=1 <your server address>:/data /data [root]# ls -R /data ... [root]# umount /data
Please note: For the nfsv41 mount to work on our client we need a nfs-utils installed, an SL6 kernel > 2.6.32-220.
What you have now is a running dCache server that has a mounted NFSv41 server locally that you have access data without any authentication. In this section we also created started a pool service with one pool, we created a directory /data that holds our data and a subdirectory /data/world-writable that can be used without authentication.
| Home | next >> |
