wiki:Puppet Installation of dCache server - Karlsruhe2014
Last modified 6 years ago Last modified on 09/03/14 09:58:56
Home next >>

Puppet Installation of dCache server

(SERVER) (SERVER) (SERVER) (SERVER)

In this part we use Puppet to install dCache as a single node instance, which means that all our services will run on the same host. This part shows us the basics and necessary setup both for Puppet and dCache. At the end of this part we will have a running dCache that can be mounted locally as well as on your client.

Download the predefined manifests and modules

For this tutorial Puppet has been configured to use several environments. Each environment belongs to only one group of students and is defined for only one machine. The students will write their puppet code inside their own environment, will upload the code via git into the puppet master and will apply the changes on their own machine.

To start follow these simple steps:

  1. Get the code. Download the predefined code in your home.
    [dcache@dCacheServer ~]$ cd $HOME
    [dcache@dCacheServer ~]$ git clone ssh://gitolite@gks-121.scc.kit.edu:24/environments/students
    

Now you can see a directory called students. This directory contains all the code you need to change during the tutorial.

  1. Initialize your own repository. This step is needed to deploy your own code inside the puppet server.
    [dcache@dCacheServer ~]$ cd $HOME
    [dcache@dCacheServer ~]$ git clone ssh://gitolite@gks-121.scc.kit.edu:24/environments/gksYourMachineNumber/modules
    [dcache@dCacheServer ~]$ git clone ssh://gitolite@gks-121.scc.kit.edu:24/environments/gksYourMachineNumber/manifests
    

change the gksYourMachineNumber with the number of your machine. E.g. dcache@gks-126 --> gksYourMachineNumber=gks126

You should be in a situation similar to this:

[dcache@dCacheServer ~]$ ls
manifests  modules  students
  1. Copy the predefined code.

To start to write your own code copy the code from the students directory into yours, pay attention to the directory structure.

[dcache@dCacheServer ~]$ cd $HOME
[dcache@dCacheServer ~]$ cp -r students/manifests/* manifests/
[dcache@dCacheServer ~]$ cp -r students/modules/* modules/
[dcache@dCacheServer ~]$ rm -fr students

The following points 4. through 7. are to be done after EVERY time you have changed code in the directories. You do not need to do these steps just now. Read them now and after you have changed the code for the first time you can then execute steps 4. to 7.

  1. Upload your code. Once you are satisfied about your code upload it into the puppet master.

To do it we use again git. Go inside to the root directory you want to upload (e.g. manifests)

[dcache@dCacheServer ~]$ cd $HOME/manifests
[dcache@dCacheServer manifests]$ git add --all; git commit -m "write a comment"; git push origin master

At this point the puppet master has your code. You can explicitly write a comment to help you to trace your changes. Do not worry about warnings, regarding providing name or email, like these (these are git specific things):

Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly:

    git config --global user.name "Your Name"
    git config --global user.email you@example.com

If the identity used for this commit is wrong, you can fix it with:

    git commit --amend --author='Your Name <you@example.com>'
  1. Update the environment. We are using puppet in a not usual way for this reason after you uploaded your code, puppet needs to retrieve it and update your environment. The command to run is the same as you will use for the next step simulate the changes
[dcache@dCacheServer ~]$ sudo puppet agent --test --noop

In the output you will see a git update but the changes are still the old. Don't worry it's OK! Nothing has been done on your machine.

  1. Simulate the changes.

Once you uploaded your code AND updated the environment you can run a puppet agent sessions to simulate your changes on the node. This last task allows you to test your code and see what is going to happen on your node. Nothing will be applied to your machine.

The command is the same of the previous change.

[dcache@dCacheServer ~]$ sudo puppet agent --test --noop

If the answer is green and blue you can be happy!

  1. Apply the changes. If there are no errors coming from the simulation your are now ready to apply your changes.

Cross your fingers and type:

[dcache@dCacheServer ~]$ sudo puppet agent --test

If the answer is STILL green and blue you can be VERY happy!

ATTENTION: EVERY TIME you change your code, you need to repeat ALL the steps starting from point 4.

  • Brief comment:

we are using puppet in a not conventional way: puppet is not created to allow different users to code together on the same master. For this and for security reasons we use gitolite as code deploy-er, once you run puppet agent --test puppet synchronizes its code as first task.

Installation of dCache server

First of all, go into $HOME/manifests and fix the machine number inside the site.pp at the first line

EG: replace

node "gks-<machine number>.scc.kit.edu" {

with

node "gks-<your server number>.scc.kit.edu" 

Replace <your server number> with your actual server number, e.g. 161 to result in gks-161.scc.kit.edu Note: This time with a dash between gks and the number.

and commit the changes

[dcache@dCacheServer ~]$ cd $HOME/manifests
[dcache@dCacheServer manifest]$ git add --all; git commit -m "host name fixed"; git push origin master
[dcache@dCacheServer ~]$ cd $HOME/modules
[dcache@dCacheServer modules]$ git add --all; git commit -m "host name fixed"; git push origin master

With this you have copied the modules that we provided into your own git repository. However this leaves you with no understanding of what the actual puppet modules look like and what they do. So lets have a look to them:

ATTENTION: Please do not edit any of the files we are about to look at. Just take a look, understand what they do and ask if you do not understand them.

  • The dCache server has few packages dependencies: it needs java-1.7.0-openjdk, postgresql-server (version >8.4), rpcbind and nfs-utils. The install.pp ensures that the additional packages are present before installing dCache,. CREATION OF THIS FILE IS NOT REQUIRED. IT IS ALREADY PRESENT ON THE PUPPET MASTER
class dcache::install(){

	notify{"Hello dCache users: now we check installed packages":}

	Package { ensure => "installed" }
	$enhancers = [ "nfs-utils", "rpcbind" ]
	package { $enhancers: }

	notify{"Installing java-1.7.0-openjdk":}
	package{"java-1.7.0-openjdk":
		provider => "yum",
		ensure   => "installed",
	}

	notify{"Installing postgresql92":}
	package{"pgdg-redhat92-9.2-7.noarch.rpm":
		provider => "rpm",
		ensure   => "installed",
		source => 'http://yum.postgresql.org/9.2/redhat/rhel-6-x86_64/pgdg-sl92-9.2-7.noarch.rpm',
	}
	
        $install = "dcache-2.10.0-1.noarch.rpm"

	notify{"Installing dcache  $install":}
	package{$install:
		provider => "rpm",
		ensure   => "installed",
		source   => "http://www.dcache.org/downloads/1.9/repo/2.10/dcache-2.10.0-1.noarch.rpm",
	}

	exec { "check_install":
		command => "yum list installed java-1.7.0-openjdk nfs-utils rpcbind > /tmp/check_install.out ; rpm -qa | grep -i dcache >> /tmp/check_install.out; rpm -qa | grep -i postgresql >> /tmp/check_install.out",
		path    => [ "/usr/local/bin/", "/bin/", "/usr/bin/" ],  
	}

}

Puppet gives you the possibility to install packages via local repository present on your Puppet master, or via the network using standard Linux distribution repository or directly downloading a package via http protocol.
Here we use network installation both ways, standard Linux repository for nfs-utils, java and rpcbind and http links for postgres and dCache. The check_install method in install.pp produces a log file reporting which software has been installed, and we will check it later right after the first puppet agent run.

  • Then we now dig into config.pp puppet manifest responsible for the configuration of dCache and related services. CREATION OF THIS FILE IS NOT REQUIRED. IT IS ALREADY PRESENT ON THE PUPPET MASTER
class dcache::config(){

        notify{"Hello dCache users: now we configure all necessary services to run a dCache instance":}

        $grid_security_dir = "/etc/grid-security"

        file { $grid_security_dir:
                ensure => "directory",
                owner  => "root",
                group  => "root",
                mode   => "644";
                "$grid_security_dir/hostcert.pem":
                ensure => "present",
                owner  => "dcache",
                group  => "dcache",
                mode   => "u=rw,go=r";
                "$grid_security_dir/hostkey.pem":
                ensure => "present",
                owner  => "dcache",
                group  => "dcache",
                mode   => "u=rw,go=";
        }

        file { "/home/dcache/configurepgsql.sh":
        ensure   => "present",
        owner    => "root",
        group    => "root",
        mode     => "u=rwx,go=rwx",
        source => "puppet:///modules/psql/configurepgsql.sh";
        }

}

It sets the ownership for the host certificates and grid-security folder. We already placed a X.509 host certificate in /etc/grid-security/hostcert.pem and /etc/grid-security/hostkey.pem . During the installation of the dCache RPM a user and group "dcache" is created, and these two files need to get the right owners/permissions, so that dCache can use them. However, as you noticed before we logged into the machines as user dcache, which is why the user already existed. We nonetheless need to be sure that the files are present and set the right permissions via Puppet.

After the certificate Puppet will copy configurepgsql.sh necessary to configure the dCache needed postgresql databases.

  • The modules folder of the student environment (that you cloned under manifests/site.pp) is responsible to call two dcache class methods, the first one for installation of the main components, the second for the configuration and the latter to add custom configuration file for the GRIDKA school. CREATION OF THIS FILE IS NOT REQUIRED. IT IS ALREADY PRESENT ON THE PUPPET MASTER
node "gks-<machine number>.scc.kit.edu" {
        
        class{'dcache::install':
            before=>Class['dcache::config']
        }
        class{'dcache::config':
            before=>Class['dcache-exercise']
        }
        class{'dcache-exercise':
        }


}

Now is time to run the puppet agent (--noop to test it before):

[dcache@dCacheServer ~]$ sudo puppet agent --test --noop
[dcache@dCacheServer ~]$ sudo puppet agent --test

Please cat the log file to check for installed java, postgresql, rpcbind, dCache and nfs-utils packages. It should look like the following if everything is ok.

[dcache@dCacheServer ~]$ cat /tmp/check_install.out
Loaded plugins: security
Installed Packages
java-1.7.0-openjdk.x86_64         1:1.7.0.65-2.5.1.2.el6_5          @sl-security
nfs-utils.x86_64                  1:1.2.3-39.el6                    @sl         
rpcbind.x86_64                    0.2.0-11.el6                      @sl         
dcache-2.10.0-1.noarch
postgresql92-libs-9.2.9-1PGDG.rhel6.x86_64
postgresql92-9.2.9-1PGDG.rhel6.x86_64
postgresql92-server-9.2.9-1PGDG.rhel6.x86_64
  • Check the content of the configurepgsql.sh script

If you want have a look at the /home/dcache/configurepgsql.sh file and try to understand what it does, but if you don't. Don't worry.

Just look this up later: For more information on postgresql setup for dCache please see dCache Book - Prerequisites. There are also several trivial steps that you have to do to setup postgresql, but they are not part of this tutorial and can be found in dCache Book - Readying the PostgreSQL server for the use with dCache We don't focus on detailed configuration of postgres server here, but have a look at the provided script for the commands needed if you are interested (configurepgsql.sh).

[dcache@dCacheServer ~]$ cat /home/dcache/configurepgsql.sh
#/bin/bash
service postgresql-9.2 initdb
sed -ie 's/max_connections = 100/max_connections = 1000/' /var/lib/pgsql/9.2/data/postgresql.conf
service postgresql-9.2 start
chkconfig postgresql-9.2 on

mv /var/lib/pgsql/9.2/data/pg_hba.conf{,_ori}
cat >/var/lib/pgsql/9.2/data/pg_hba.conf <<EOF
# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

# "local" is for Unix domain socket connections only
local   all         all                               trust
# IPv4 local connections:
host    all         all         127.0.0.1/32          trust
# IPv6 local connections:
host    all         all         ::1/128               trust
EOF
service postgresql-9.2 restart
createdb -U postgres chimera
createuser -U postgres --no-superuser --no-createrole --createdb chimera
createuser -U postgres --no-superuser --no-createrole --createdb srmdcache
createdb -U srmdcache dcache
createdb -O srmdcache -U postgres billing
dcache database update

Note: the last command in this script updates the database schemas. This is very convenient as this allows you to easily keep up with the database changes from one release to another.

  • Now execute the configurepgsql.sh script. The script also needs to run as root as the puppet agent before, which is why we also use sudo to execute it:
[dcache@dCacheServer ~]$ sudo ./configurepgsql.sh
Initializing database:                                     [  OK  ]
Starting postgresql-9.2 service:                           [  OK  ]
Stopping postgresql-9.2 service:                           [  OK  ]
Starting postgresql-9.2 service:                           [  OK  ]
...
some strings here
...
INFO  - Successfully released change log lock
Liquibase Update Successful

Now you have a running and fully configured postgresql server. We also installed a dCache server but before to start the dCache services we need to be sure rpcbind is up and running:

[dcache@dCacheServer ~]$ service rpcbind status
rpcbind is stopped
[dcache@dCacheServer ~]$ sudo service rpcbind start
[sudo] password for dcache: 
Starting rpcbind:                                          [  OK  ]
[dcache@dCacheServer ~]$ service rpcbind status
rpcbind (pid  26887) is running...

Check also the status of postgresql:

[dcache@dCacheServer ~]$ service postgresql-9.2 status
postgresql-9.2 (pid  26257) is running...

Configuration files

dCache makes use of two files, to define and change the settings for the layout and the behavior of dCache on different levels (per host, per domain or per service). There are three main places for the configuration files:

  • /usr/share/dcache/defaults

This directory is filled with files defining the default settings for all dCache services, as they are shipped by dCache.org. Do not modify these files, as they will be replaced by subsequent updates!

  • /etc/dcache/dcache.conf

The central configuration file that ideally should be nearly identical on all nodes of the dCache setup. Maybe it is possible to have only one difference among all nodes at all: the parameter pointing to the layout configuration files. To get to know what settings can be made in dcache.conf you can look through dcache.properties in the defaults directory.

  • /etc/dcache/layouts

Layout files are the place to define the actual topology for the dCache services/domains on this node. So typically layout files are identical for nodes that have same services running on it (e.g.: doors & pools).

The dCache install does no longer provide us with pre made layout files. Right now you could start dCache and it would use the empty dcache.conf file and the fallback layout file <hostname>.conf. This would not exist after installing dCache, which is why we need puppet to copy to copy a file that we provide for this hands-on:

With ws_gridka_2014.conf layout file the most important core services will be configured to run one single domain with default settings.

NOTE: once you upload a modified version of dcache.conf or ws_gridka_2014.conf via git into the puppet server, the latter will push them into the right folder of the dcache server (the place where dcache server expect to find them).

We need to tell dCache to use the layout file we just created and add this to dcache.conf:

[dcache@dCacheServer ~]$ vi modules/dcache-exercise/files/dcache.conf 
# This is the main configuration file of dCache.
#
# This file used to be called config/dCacheSetup, but is now stored in
# etc/dcache.conf. Documentation may still refer to it as
# config/dCacheSetup. Most of the parameters are the same and the
# documentation also applies to this file.
#
# By default the file is empty. Default values are picked up from the
# properties files in share/defaults/. Do not modify the defaults
# files: any changes will be lost when upgrading. Instead override the
# defaults by redefining the parameters in this file.

dcache.layout=ws_gridka_2014

Do not make the mistake of entering the entire file name there, just add the following line:

dcache.layout=ws_gridka_2014

Leave vi open for now and keep on reading.

Adjust the layout file

Firstly we need to tell dCache that there will be many domains communicating with each other. As we described in the introduction dCache services run inside so called domains. Since we will have many domains, we will need some mechanism for these domains to communicate. The cells framework is used for this communication, which is why we activate it by adding the following line to the beginning of /etc/dcache/dcache.conf:

dcache.broker.scheme=cells

NOTE: broker.scheme=cells is default dCache configuration so not really necessary to define.

As mentioned we want to be able to mount dCache locally using NFSv41. Therefor a service called nfsv41 needs to be started, which we will keep in a separate domain for convenience of restarting it separately from the rest of dCache.

These lines need to be added at the end of ws_gridka_2014.conf in modules/dcache-exercise/files/ws_gridka_2014.conf. They add a domain to dCache - [nfs-Domain] - that holds the nfsv41 service.

[nfs-Domain]
[nfs-Domain/nfs]
nfs.version = 4.1

Save the file and quit vi. Then upload via a git commit to the Puppet server:

[dcache@dCacheServer ~]$ cd modules/
[dcache@dCacheServer modules]$ git add --all; git commit -m "added nfs support"; git push origin master

Have a look at the layout file that we just changed and pushed to the puppet master:

[dcache@gks-<your server number> modules]# cd
[dcache@dCacheServer ~]$ cat modules/dcache-exercise/files/ws_gridka_2014.conf
[dCacheDomain]
[dCacheDomain/admin]
[dCacheDomain/broadcast]
[dCacheDomain/poolmanager]
[dCacheDomain/loginbroker]
[dCacheDomain/spacemanager]
[dCacheDomain/pnfsmanager]
[dCacheDomain/cleaner]
[dCacheDomain/dir]
[dCacheDomain/gplazma]
[dCacheDomain/gsi-pam]
[dCacheDomain/pinmanager]
[dCacheDomain/billing]
[dCacheDomain/srm-loginbroker]
[dCacheDomain/httpd]
[dCacheDomain/topo]
[dCacheDomain/info]

[nfs-Domain]
[nfs-Domain/nfs]
nfs.version = 4.1


# [dCacheDomain/pool]
# name=pool1
# path=/path/to/pool1

# [dCacheDomain/pool]
# name=pool2
# path=/path/to/pool2

# [dCacheDomain/replica]
# [dCacheDomain/dcap]
# [dCacheDomain/gsidcap]
# [dCacheDomain/gridftp]
# [dCacheDomain/srm]
# [dCacheDomain/transfermanagers]
# [dCacheDomain/xrootd]
# [dCacheDomain/webdav]
# [dCacheDomain/webadmin]

Then compare the content of /etc/dcache/layouts/ws_gridka_2014.conf is the same of modules/dcache-exercise/files/ws_gridka_2014.conf.

[dcache@dCacheServer ~]$ vimdiff /home/dcache/modules/dcache-exercise/files/ws_gridka_2014.conf /etc/dcache/layouts/ws_gridka_2014.conf

They are different! We need to run a puppet agent in order to configure dCache server properly.

[dcache@dCacheServer ~]$ sudo puppet agent --test

and crosscheck the content of /etc/dcache/ws_gridka_2014.conf is the same of modules/dcache-exercise/files/ws_gridka_2014.conf.

[dcache@dCacheServer ~]$ vimdiff /home/dcache/modules/dcache-exercise/files/ws_gridka_2014.conf /etc/dcache/layouts/ws_gridka_2014.conf

NOTE: once you upload a modified version of dcache.conf or ws_gridka_2014.conf via git into the puppet server, the latter will push them into the right folder of the dcache server (the place where dcache server expect to find them).

Before we can start dCache we have to empty /etc/dcache/gplazma.conf as there is no security configured in a blank dCache:

[dcache@dCacheServer ~]$ sudo echo "" > /etc/dcache/gplazma.conf

Then we want to be sure rpcbind service is up and running (in order to start nfs4 service correctly):

[dcache@dCacheServer ~]$ service rpcbind status
rpcbind is stopped
[dcache@dCacheServer ~]$ sudo service rpcbind start
Starting rpcbind:                                          [  OK  ]
[dcache@dCacheServer ~]$ service rpcbind status
rpcbind (pid  4178) is running...

Check that all is configured properly before starting dcache by running "dcache check-config":

[dcache@gks-<your server number> ~]$ dcache check-config
No problems found.

Now we are ready to start dCache by:

[dcache@dCacheServer modules]$ sudo dcache start
Starting dCacheDomain done
Starting nfs-Domain done

Check if dCache has started up correctly for domains:

[dcache@dCacheServer modules]$ dcache status
DOMAIN       STATUS  PID   USER
dCacheDomain running 18314 dcache
nfs-Domain   running 18359 dcache

and services:

[dcache@dCacheServer modules]$ dcache services
DOMAIN       SERVICE         CELL            LOG
dCacheDomain admin           alm             /var/log/dcache/dCacheDomain.log
dCacheDomain broadcast       broadcast       /var/log/dcache/dCacheDomain.log
dCacheDomain poolmanager     PoolManager     /var/log/dcache/dCacheDomain.log
dCacheDomain loginbroker     LoginBroker     /var/log/dcache/dCacheDomain.log
dCacheDomain spacemanager    SrmSpaceManager /var/log/dcache/dCacheDomain.log
dCacheDomain pnfsmanager     PnfsManager     /var/log/dcache/dCacheDomain.log
dCacheDomain cleaner         cleaner         /var/log/dcache/dCacheDomain.log
dCacheDomain dir             dirLookupPool   /var/log/dcache/dCacheDomain.log
dCacheDomain gplazma         gPlazma         /var/log/dcache/dCacheDomain.log
dCacheDomain pinmanager      PinManager      /var/log/dcache/dCacheDomain.log
dCacheDomain billing         billing         /var/log/dcache/dCacheDomain.log
dCacheDomain srm-loginbroker srm-LoginBroker /var/log/dcache/dCacheDomain.log
dCacheDomain httpd           httpd           /var/log/dcache/dCacheDomain.log
dCacheDomain topo            topo            /var/log/dcache/dCacheDomain.log
dCacheDomain info            info            /var/log/dcache/dCacheDomain.log
nfs-Domain   nfsv41          NFSv41-ws-server-001   /var/log/dcache/nfs-Domain.log

Then also check the log files:

[dcache@dCacheServer ~]$ tail -F /var/log/dcache/*

==> /var/log/dcache/dCacheDomain.log <==

2014-03-06 11:00:53 Launching /usr/bin/java -server -Xmx512m -XX:MaxDirectMemorySize=512m -Dsun.net.inetaddr.ttl=1800 -Dorg.globus.tcp.port.range=20000,25000 -Djava.net.preferIPv4Stack=true -Dorg.dcache.dcap.port=0 -Dorg.dcache.net.tcp.portrange=33115:33145 -Dorg.globus.jglobus.delegation.cache.lifetime=30000 -Dorg.globus.jglobus.crl.cache.lifetime=60000 -Djava.security.krb5.realm=EXAMPLE.ORG -Djava.security.krb5.kdc=localhost -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.auth.login.config=/etc/dcache/jgss.conf -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dcache/dCacheDomain-oom.hprof -javaagent:/usr/share/dcache/classes/spring-instrument-3.2.2.RELEASE.jar -Djava.awt.headless=true -DwantLog4jSetup=n -Ddcache.home=/usr/share/dcache -Ddcache.paths.defaults=/usr/share/dcache/defaults org.dcache.boot.BootLoader start dCacheDomain
06 Mar 2014 11:01:32 (PinManager) [] [AspectJ] javax.* types are not being woven because the weaver option '-Xset:weaveJavaxPackages=true' has not been specified

==> /var/log/dcache/nfs-Domain.log <==

2014-03-06 11:00:54 Launching /usr/bin/java -server -Xmx512m -XX:MaxDirectMemorySize=512m -Dsun.net.inetaddr.ttl=1800 -Dorg.globus.tcp.port.range=20000,25000 -Djava.net.preferIPv4Stack=true -Dorg.dcache.dcap.port=0 -Dorg.dcache.net.tcp.portrange=33115:33145 -Dorg.globus.jglobus.delegation.cache.lifetime=30000 -Dorg.globus.jglobus.crl.cache.lifetime=60000 -Djava.security.krb5.realm=EXAMPLE.ORG -Djava.security.krb5.kdc=localhost -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.auth.login.config=/etc/dcache/jgss.conf -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dcache/nfs-Domain-oom.hprof -javaagent:/usr/share/dcache/classes/spring-instrument-3.2.2.RELEASE.jar -Djava.awt.headless=true -DwantLog4jSetup=n -Ddcache.home=/usr/share/dcache -Ddcache.paths.defaults=/usr/share/dcache/defaults org.dcache.boot.BootLoader start nfs-Domain

Now exit the tail command by pressing control+c.

This should show you that domains were created and no error messages should be flying around, so the output that tail shows should stop at some point after starting dCache although some output is normal to inform about successful domain creation. Sometimes a domain might report that it could not connect to the dCacheDomain, because it might have started up in advance of dCacheDomain. If however "dcache services" reports that the domain and service have been started, it was correctly connected after the dCacheDomain came up.

First Contact

Now that we have a running dCache that does nothing, we would like to make a first contact. This will be achieved by mounting dCache locally via NFSv41.

dCache needs several things to come together to be able to store data in it. It needs to authenticate and authorize people wanting to store data (gPlazma takes care of this), it needs a logical structure where file's names can be found in a directory structure, the namespace, which keeps file's meta data. The actual files are stored in a service that is called a pool that allows writing data to block devices and allows for a hierarchical storage management to be employed (More about this in the bonus part about tertiary storage of this hands-on).

dCache needs something that holds that actual data, the pools. So we first create a pool that is stored in a certain path in the local file system and set its maximum size. Execute the following command in your server console:

[dcache@dCacheServer modules]$ sudo dcache pool create --size=419430400  --meta=db --lfs=precious   /pools/nfsPool1 nfsPool1 poolDomain
Created a pool in /pools/nfsPool1. The pool was added to poolDomain in
file:/etc/dcache/layouts/ws_gridka_2014.conf.

here we need to make /etc/dcache/layouts/ws_gridka_2014.conf consistent with the "puppet one":

[dcache@dCacheServer modules]$ sudo vimdiff /home/dcache/modules/dcache-exercise/files/ws_gridka_2014.conf /etc/dcache/layouts/ws_gridka_2014.conf

then synk the repo and the puppet:

[dcache@dCacheServer modules]$ git add --all; git commit -m "pool 1 created"; git push origin master
[dcache@dCacheServer modules]$ sudo puppet agent --test

After we created the pool we need to start the domain that was created by executing:

[dcache@dCacheServer ~]$ dcache status
DOMAIN       STATUS  PID   USER
dCacheDomain running (for 118 seconds) 4509 dcache 
nfs-Domain   running (for 118 seconds) 4564 dcache 
poolDomain   stopped                   dcache
[dcache@dCacheServer modules]$ sudo dcache start poolDomain
Starting poolDomain done

The pool service will always make sure that there is enough space on the pool to allow for some data to be moved around. The amount of space that is kept free is configurable and by default it is 4 GiB. For this tutorial we need to set the gap to a very low value (10 MB) because the disk space on the hands-on machines is so limited. In order to be able to administrate dCache we need to login to the admin interface. We will use key-based login with ssh2. Please create yourself a DSA key and store the public key in /etc/dcache/admin/authorized_keys2. Do the following steps to change the gap size (when generating dsa key use default file name and NO password):

[dcache@dCacheServer modules]$ ssh-keygen -t dsa
...
[dcache@dCacheServer modules]$ echo  'cat /home/dcache/.ssh/id_dsa.pub >> /etc/dcache/admin/authorized_keys2' | sudo -s
[dcache@dCacheServer modules]$ ssh -l admin -p 22224 localhost
... (local) admin > cd nfsPool1
... (nfsPool) admin > set gap 10m
Gap set to 10485760
... (nfsPool) admin > save

Then exit the admin interface by pressing control+d.

Try to create yourself second pool with name "nfsPool2" assigned to the same domain it will be needed for the next exercises (don't forget about gap size). Also make sure you restart the poolDomain.

[dcache@dCacheServer modules]$ sudo restart poolDomain

This now enables us to actually store files in dCache, but how? ... for example via a mounted NFSv41.

We need to set the NFS domain to make sure nfs server and client are inside the same namespace. This is done by adding the following line to the end of dcache.conf:

nfs.domain = karlsruhe-domain
[dcache@dCacheServer modules]$ sudo vimdiff /home/dcache/modules/dcache-exercise/files/dcache.conf  /etc/dcache/dcache.conf 
[dcache@dCacheServer modules]$ git add --all; git commit -m "added nfs domain"; git push origin master
[dcache@dCacheServer modules]$ sudo puppet agent --test

As mentioned dCache also needs a namespace where file's meta data is stored. We need to create a directory in the dCache namespace by executing in the chimera front-end:

[dcache@dCacheServer modules]$ chimera
chimera:/# mkdir /data
chimera:/# mkdir /data/world-writable

We also need to tell the nfs server that machines are allowed to mount certain directories that the dCache server provides access to. This is not really a dCache specific thing as you would also use the /etc/exports file for a non-dCache nfs servers. Please add the following line to /etc/exports:

/data localhost(rw,no_root_squash)

This states that the directory /data can be accessed by localhost with read/write permissions. The no_root_squash option allows root to access/create files on the nfs server as root.

We do not have authentication setup yet, which is why we need to set the directory world writable to be able to write:

[dcache@dCacheServer modules]$ chimera
chimera:/# chmod 777 /data/world-writable

Leave the chimera front-end by pressing ctrl+d. Now we will be able to see the benefit of having the nfsv41 service in a separate domain as we need to restart the domain for the change in /etc/exports to take effect.

[dcache@dCacheServer modules]$ dcache status
DOMAIN       STATUS                   PID  USER   
dCacheDomain running (for 27 minutes) 4509 dcache 
nfs-Domain   running (for 27 minutes) 4564 dcache 
poolDomain   running (for 25 minutes) 5229 dcache 

[dcache@dCacheServer modules]$ sudo dcache restart nfs-Domain
Stopping nfs-Domain 0 1 2 done
Starting nfs-Domain done

Had we not put the nfs service in a separate domain, we would have had to restart entire dCache, all services. Thereby we would disrupt all of them just to realise a minor fix to the config of the nfs service.

After this we can mount dCache locally by doing the following:

[dcache@dCacheServer modules]$ sudo mkdir /data
[dcache@dCacheServer modules]$ sudo mount -o intr,minorversion=1 localhost:/data /data

Now you can copy some file e.g.:

[dcache@dCacheServer ~]$ cp /etc/hosts /data/world-writable/myFirstNfsFile00001

just make an ls to be sure the file is actually copied

[dcache@dCacheServer ~]$ ls -la /data/world-writable/myFirstNfsFile00001

Congratulations you wrote your first file into dCache. We will now mount nfsv41 on the client. Therefor we need to grant it permission on the server side. Therefore edit /etc/exports:

[dcache@dCacheServer ~]$ vi /etc/exports

and change the one line from

/data localhost(rw,no_root_squash)

to

/data localhost(rw,no_root_squash) <your client IP address>(rw,no_root_squash)

or to

/data localhost(rw,no_root_squash) <your client HOSTNAME>(rw,no_root_squash)

then exit INSERT mode and save. This line now means that directory /data can be mounted with read-write permissions from localhost as well as <your client address>. After addition of this permission restart the nfs-Domain once more:

[dcache@dCacheServer ~]$ sudo dcache restart nfs-Domain

Now go to your client:

(CLIENT) (CLIENT) (CLIENT) (CLIENT)

There you can have a look at:

[dcache@dCacheClient ~]$ less /etc/idmapd.conf

At the top of this file you will find:

[General]
#Verbosity = 0
# The following should be set to the local NFSv4 domain name
# The default is the host's DNS domain name.
Domain = karlsruhe-domain

Remember, earlier we set the nfs.domain = karlsruhe-domain in the dcache.conf on the server. This is now telling server and client to be in the same domain. If you had to change this value, which we luckily already did for you, you would have to restart the /etc/rpcidmapd service. But that is not necessary, now.

Check if the machine has a /data directory (just type "ls /data"), in case not just create with:

[dcache@dCacheClient ~]$ sudo mkdir /data

then mount and have a look at the mounted dCache and unmount:

[dcache@dCacheClient ~]$ sudo mount -o intr,minorversion=1 <your server address>:/data /data
[dcache@dCacheClient ~]$ ls -R /data
...
[dcache@dCacheClient ~]$ sudo umount /data

Please note: For the nfsv41 mount to work on our client we need a nfs-utils installed, an SL6 kernel > 2.6.32-220.

What you have now is a running dCache server that has a mounted NFSv41 server locally that you have access data without any authentication. In this section we also created started a pool service with one pool, we created a directory /data that holds our data and a subdirectory /data/world-writable that can be used without authentication.


Home next >>