Welcome to CernVM-FS’s documentation!¶

Linux¶

To install, proceed according to the following steps:

Step 1

Install the CernVM-FS packages. With yum, run

yum install cvmfs cvmfs-config-default

If yum does not show the latest packages, clean the yum cache by yum clean all. Packages can be also installed with rpm instead with the command rpm -vi. On Ubuntu, use dpkg -i on the cvmfs and cvmfs-config-default .deb packages.

Step 2

For the base setup, run cvmfs_config setup. Alternatively, you can do the base setup by hand: ensure that user_allow_other is set in /etc/fuse.conf, ensure that /cvmfs /etc/auto.cvmfs is set in /etc/auto.master and that the autofs service is running. If you migrate from a previous version of CernVM-FS, check the release notes if there is anything special to do for migration.

Step 3

Create /etc/cvmfs/default.local and open the file for editing.

Step 4

Select the desired repositories by setting CVMFS_REPOSITORIES=repo1,repo2,.... For ATLAS, for instance, set

CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch

Specify the HTTP proxy servers on your site with

CVMFS_HTTP_PROXY="http://myproxy1:port|http://myproxy2:port"

For the syntax of more complex HTTP proxy settings, see Network Settings. Make sure your local proxy servers allow access to all the Stratum 1 web servers (more on proxy configuration here). For Cern repositories, the Stratum 1 web servers are listed in /etc/cvmfs/domain.d/cern.ch.conf.

Step 5

Check if CernVM-FS mounts the specified repositories by cvmfs_config probe.

Mac OS X¶

On Mac OS X, CernVM-FS is based on OSXFuse. It is not integrated with autofs. In order to install, proceed according to the following steps:

Step 1

Install the CernVM-FS package by opening the .pkg file.

Step 2

Create /etc/cvmfs/default.local and open the file for editing.

Step 3

Select the desired repositories by setting CVMFS_REPOSITORIES=repo1,repo2,.... For CMS, for instance, set

CVMFS_REPOSITORIES=cms.cern.ch

Specify the HTTP proxy servers on your site with

CVMFS_HTTP_PROXY="http://myproxy1:port|http://myproxy2:port"

If you’re unsure about the proxy names, set CVMFS_HTTP_PROXY=DIRECT.

Step 4

Mount your repositories like

sudo mkdir -p /cvmfs/cms.cern.ch
sudo mount -t cvmfs cms.cern.ch /cvmfs/cms.cern.ch

The “Config Repository”¶

In addition to the local system configuration, a client can configure a dedicated “config repository”. A config repository is a standard mountable CernVM-FS repository that resembles the directory structure of /etc/cvmfs. It can be used to centrally maintain the public keys and configuration of repositories that should not be distributed with rather static packages. Configuration from the config repository is overwritten by the local configuration in case of conflicts. The config repository is set by the CVMFS_CONFIG_REPOSITORY parameter. The default configuration sets this parameter to cvmfs-config.cern.ch.

Private Mount Points¶

In contrast to the system’s mount command which requires root privileges, CernVM-FS can also be mounted like other Fuse file systems by normal users. In this case, CernVM-FS uses parameters from one or several user-provided config files instead of using the files under /etc/cvmfs. CernVM-FS private mount points do not appear as cvmfs2 file systems but as fuse file systems. The cvmfs_config and cvmfs_talk commands ignore privately mounted CernVM-FS repositories. On an interactive machine, private mount points are for instance unaffected by an administrator unmounting all system’s CernVM-FS mount points by cvmfs_config umount.

In order to mount CernVM-FS privately, use the cvmfs2 command like

cvmfs2 -o config=myparams.conf atlas.cern.ch /home/user/myatlas

A minimal sample myparams.conf file could look like this:

CVMFS_CACHE_BASE=/home/user/mycache
CVMFS_CLAIM_OWNERSHIP=yes
CVMFS_RELOAD_SOCKETS=/home/user/mycache
CVMFS_SERVER_URL=http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch
CVMFS_HTTP_PROXY=DIRECT

Make sure to use absolute path names for the mount point and for the cache directory. Use fusermount -u in order to unmount a privately mounted CernVM-FS repository.

The private mount points can also be used to use the CernVM-FS Fuse module in case it has not been installed under /usr and /etc. If the public keys are not installed under /etc/cvmfs/keys, the directory of the keys needs to be specified in the config file by CVMFS_KEYS_DIR=<directory>. If the libcvmfs_fuse.so library is not installed in one of the standard search paths, the LD_LIBRARY_PATH variable has to be set accordingly for the cvmfs2 command.

Docker Containers¶

There are two options to mount CernVM-FS in docker containers. The first option is to bind mount a mounted repository as a volume into the container. This has the advantage that the CernVM-FS cache is shared among multiple containers. The second option is to mount a repository inside a container, which requires a privileged container.

In both cases, autofs should not be used. Docker has often issues with autofs.

service autofs stop  # systemd: systemctl stop autofs
chkconfig autofs off  # systemd: systemctl disable autofs
mkdir -p /cvmfs/sft.cern.ch
mount -t cvmfs sft.cern.ch /cvmfs/sft.cern.ch

docker run -i -t -v /cvmfs/sft.cern.ch:/cvmfs/sft.cern.ch centos /bin/bash

docker run --privileged -i -t centos /bin/bash

Stratum 1 List¶

To specify the Stratum 1 servers, set CVMFS_SERVER_URL to a semicolon-separated list of known replica servers (enclose in quotes). The so defined URLs are organized as a ring buffer. Whenever download of files fails from a server, CernVM-FS automatically switches to the next mirror server. For repositories under the cern.ch domain, the Stratum 1 servers are specified in /etc/cvmfs/domain.d/cern.ch.conf.

It is recommended to adjust the order of Stratum 1 servers so that the closest servers are used with priority. This can be done automatically by using geographic ordering. Alternatively, for roaming clients (clients not using a proxy server), the Stratum 1 servers can be automatically sorted according to round trip time by cvmfs_talk host probe (see Auxiliary Tools). Otherwise, the proxy server would invalidate round trip time measurement.

The special sequence @fqrn@ in the CVMFS_SERVER_URL string is replaced by fully qualified repository name (atlas.cern.cn, cms.cern.ch, ...). That allows to use the same parameter for many repositories hosted under the same domain. For instance, http://cvmfs-stratum-one.cern.ch/cvmfs/@fqrn@ can resolve to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch, http://cvmfs-stratum-one.cern.ch/cvmfs/cms.cern.ch, and so on depending on the repository that is being mounted. The same works for the sequence @org@ which is replaced by the unqualified repository name (atlas, cms, ...).

Proxy Lists¶

CernVM-FS uses a dedicated HTTP proxy configuration, independent from system-wide settings. Instead of a single proxy, CernVM-FS uses a chain of load-balanced proxy groups. The CernVM-FS proxies are set by the CVMFS_HTTP_PROXY parameter.

Proxies within the same proxy group are considered as a load-balance group and a proxy is selected randomly. If a proxy fails, CernVM-FS automatically switches to another proxy from the current group. If all proxies from a group have failed, CernVM-FS switches to the next proxy group. After probing the last proxy group in the chain, the first proxy is probed again. To avoid endless loops, for each file download the number of switches is restricted by the total number of proxies.

The chain of proxy groups is specified by a string of semicolon separated entries, each group is a list of pipe separated hostnames [1]. Multiple IP addresses behind a single proxy host name (DNS round-robin entry) are automatically transformed into a load-balanced group. The DIRECT keyword for a hostname avoids using proxies. Note that the CVMFS_HTTP_PROXY parameter is necessary in order to mount. If you don’t use proxies, set the parameter to DIRECT.

Multiple proxy groups are often organized as a primary proxy group at the local site and backup proxy groups at remote sites. In order to avoid CernVM-FS being stuck with proxies at a remote site after a fail-over, CernVM-FS will automatically retry to use proxies from the primary group after some time. The delay for re-trying a proxies from the primary group is set in seconds by CVMFS_PROXY_RESET_AFTER. The distinction of primary and backup proxy groups can be turned off by setting this parameter to 0.

Fallback Proxy List¶

In addition to the regular proxy list set by CVMFS_HTTP_PROXY, a fallback proxy list is supported in CVMFS_FALLBACK_PROXY. The syntax of both lists is the same. The fallback proxy list is appended to the regular proxy list, and if the fallback proxy list is set, any DIRECT is removed from both lists. The automatic proxy configuration of the previous section only sets the regular proxy list, not the fallback proxy list. Also the fallback proxy list can be automatically reordered; see the next section.

Ordering of Servers according to Geographic Proximity¶

CernVM-FS Stratum 1 servers provide a RESTful service for geographic ordering. Clients can request http://<HOST>/cvmfs/<FQRN>/api/v1.0/geo/<proxy_address>/<server_list> The proxy address can be replaced by a UUID if no proxies are used, and the CernVM-FS client does that if there are no regular proxies. The server list is comma-separated. The result is an ordered list of indexes of the input host names. Use of this API can be enabled in a CernVM-FS client with CVMFS_USE_GEOAPI=yes. That will geographically sort both the servers set by CVMFS_SERVER_URL and the fallback proxies set by CVMFS_FALLBACK_PROXY.

Timeouts¶

CernVM-FS tries to gracefully recover from broken network links and temporarily overloaded paths. The timeout for connection attempts and for very slow downloads can be set by CVMFS_TIMEOUT and CVMFS_TIMEOUT_DIRECT. The two timeout parameters apply to a connection with a proxy server and to a direct connection to a Stratum 1 server, respectively. A download is considered to be “very slow” if the transfer rate is below for more than the timeout interval. The threshold can be adjusted with the CVMFS_LOW_SPEED_LIMIT parameter. A very slow download is treated like a broken connection.

On timeout errors and on connection failures (but not on name resolving failures), CernVM-FS will retry the path using an exponential backoff. This introduces a jitter in case there are many concurrent requests by a cluster of nodes, allowing a proxy server or web server to serve all the nodes consecutively. CVMFS_MAX_RETRIES sets the number of retries on a given path before CernVM-FS tries to switch to another proxy or host. The overall number of requests with a given proxy/host combination is $CVMFS_MAX_RETRIES+1. CVMFS_BACKOFF_INIT sets the maximum initial backoff in seconds. The actual initial backoff is picked with milliseconds precision randomly in the interval \([1, \text{\$CVMFS\_BACKOFF\_INIT}\cdot 1000]\). With every retry, the backoff is then doubled.

Alien Cache¶

An “alien cache” provides the possibility to use a data cache outside the control of CernVM-FS. This can be necessary, for instance, in HPC environments where local disk space is not available or scarce but powerful cluster file systems are available. The alien cache directory is a directory in addition to the ordinary cache directory. The ordinary cache directory is still used to store control files.

The alien cache directory is set by the CVMFS_ALIEN_CACHE option. It can be located anywhere including cluster and network file systems. If configured, all data chunks are stored there. CernVM-FS ensures atomic access to the cache directory. It is safe to have the alien directory shared by multiple CernVM-FS processes and it is safe to unlink files from the alien cache directory anytime. The contents of files, however, must not be touched by third-party programs.

In contrast to normal cache mode where files are store in mode 0600, in the alien cache files are stored in mode 0660. So all users being part of the alien cache directory’s owner group can use it.

The skeleton of the alien cache directory should be created upfront. Otherwise, the first CernVM-FS process accessing the alien cache determines the ownership. The cvmfs2 binary can create such a skeleton using

cvmfs2 __MK_ALIEN_CACHE__ $alien_cachedir $owner_uid $owner_gid

Since the alien cache is unmanaged, there is no automatic quota management provided by CernVM-FS; the alien cache directory is ever-growing. The CVMFS_ALIEN_CACHE requires CVMFS_QUOTA_LIMIT=-1 and CVMFS_SHARED_CACHE=no.

The alien cache might be used in combination with a special repository replication mode that preloads a cache directory (Section Setting up a Replica Server (Stratum 1)). This allows to propagate an entire repository into the cache of a cluster file system for HPC setups that do not allow outgoing connectivity.

Tuning¶

The default settings in CernVM-FS are tailored to the normal, non-NFS use case. For decent performance in the NFS deployment, the amount of memory given to the meta-data cache should be increased. By default, this is 16M. It can be increased, for instance, to 256M by setting CVMFS_MEMCACHE_SIZE to 256. Furthermore, the maximum number of download retries should be increased to at least 2.

The number of NFS daemons should be increased as well. A value of 128 NFS daemons has shown perform well. In Scientific Linux, the number of NFS daemons is set by the RPCNFSDCOUNT parameter in /etc/sysconfig/nfs.

The performance will benefit from large RAM on the NFS server (\(\geq\) 16GB) and CernVM-FS caches hosted on an SSD hard drive.

Shared NFS Maps (HA-NFS)¶

As an alternative to the existing, leveldb managed NFS maps, the NFS maps can optionally be managed out of the CernVM-FS cache directory by SQLite. This allows the NFS maps to be placed on shared storage and accessed by multiple CernVM-FS NFS export nodes simultaneously for clustering and active high-availablity setups. In order to enable shared NFS maps, set CVMFS_NFS_SHARED to the path that should be used to host the SQLite database. If the path is on shared storage, the shared storage has to support POSIX file locks. The drawback of the SQLite managed NFS maps is a significant performance penalty which in practice can be covered by the memory caches.

Example¶

An example entry /etc/exports (note: the fsid needs to be different for every exported CernVM-FS repository)

/cvmfs/atlas.cern.ch 172.16.192.0/24(ro,sync,no_root_squash,\
  no_subtree_check,fsid=101)

A sample entry /etc/fstab entry on a client:

172.16.192.210:/cvmfs/atlas.cern.ch /cvmfs/atlas.cern.ch nfs4 \
  ro,ac,actimeo=60,lookupcache=all,nolock,rsize=1048576,wsize=1048576 0 0

cvmfs_fsck¶

CernVM-FS assumes that the local cache directory is trustworthy. However, it might happen that files get corrupted in the cache directory caused by errors outside the scope of CernVM-FS. CernVM-FS stores files in the local disk cache with their cryptographic content hash key as name, which makes it easy to verify file integrity. CernVM-FS contains the cvmfs_fsck utility to do so for a specific cache directory. Its return value is comparable to the system’s fsck. For example,

cvmfs_fsck -j 8 /var/lib/cvmfs/shared

checks all the data files and catalogs in /var/lib/cvmfs/shared using 8 concurrent threads. Supported options are:

cvmfs_config¶

The cvmfs_config utility provides commands in order to setup the system for use with CernVM-FS.

setup

The setup command takes care of basic setup tasks, such as creating the cvmfs user and allowing access to CernVM-FS mount points by all users.

chksetup

The chksetup command inspects the system and the CernVM-FS configuration in /etc/cvmfs for common problems.

showconfig

The showconfig command prints the CernVM-FS parameters for all repositories or for the specific repository given as argument.

stat

The stat command prints file system and network statistics for currently mounted repositories.

status

The status command shows all currently mounted repositories and the process id (PID) of the CernVM-FS processes managing a mount point.

probe

The probe command tries to access /cvmfs/$repository for all repositories specified in CVMFS_REPOSITORIES.

reload

The reload command is used to reload or hotpatch CernVM-FS instances.

umount

The umount command unmounts all currently mounted CernVM-FS repositories, which will only succeed if there are no open file handles on the repositories.

wipecache

The wipecache command is an alias for reload -c.

bugreport

The bugreport command creates a tarball with collected system information which helps to debug a problem.

cvmfs_talk¶

The cvmfs_talk command provides a way to control a currently running CernVM-FS process and to extract information about the status of the corresponding mount point. Most of the commands are for special purposes only or covered by more convenient commands, such as cvmfs_config showconfig or cvmfs_config stat. Two commands might be of particular interest though.

cvmfs_talk cleanup 0

will, without interruption of service, immediately cleanup the cache from all files that are not currently pinned in the cache.

cvmfs_talk internal affairs

prints the internal status information and performance counters. It can be helpful for performance engineering.

Other¶

Information about the current cache usage can be gathered using the df utility. For repositories created with the CernVM-FS 2.1 toolchain, information about the overall number of file system entries in the repository as well as the number of entries covered by currently loaded meta-data can be gathered by df -i.

For the Nagios monitoring system [Schubert08], a checker plugin is available on our website.

System Requirements¶

• Apache HTTP server OR S3 compatible storage service
• aufs union file system in the kernel (see Installing the AUFS-enabled Kernel on Scientific Linux 6)
• Officially supported platforms
  • Scientific Linux 5 (64 bit)
  • Scientific Linux 6 (64 bit - with custom AUFS enabled kernel - Appendix “Available RPMs”)
  • Ubuntu 13.10 and above (64 bit - with installed AUFS kernel module)

Installation¶

1. Install cvmfs and cvmfs-server packages
2. Ensure enough disk space in /var/spool/cvmfs (>50GiB)
3. For local storage: Ensure enough disk space in /srv/cvmfs
4. Create a repository with cvmfs_server mkfs (See Repository Creation)

Content Publishing¶

1. cvmfs_server transaction <repository name>
2. Install content into /cvmfs/<repository name>
3. Create nested catalogs at proper locations
   • Create .cvmfscatalog files (See Managing Nested Catalogs) or
   • Consider using a .cvmfsdirtab file (See Managing Nested Catalogs with .cvmfsdirtab)
4. cvmfs_server publish <repository name>

Backup Policy¶

• Create backups of signing key files in /etc/cvmfs/keys
• Entire repository content
  • For local storage: /srv/cvmfs
  • Stratum 1s can serve as last-ressort backup of repository content

Repository Creation¶

A new repository is created by cvmfs_server mkfs:

cvmfs_server mkfs my.repo.name

The utility will ask for a user that should act as the owner of the repository and afterwards create all the infrastructure for the new CernVM-FS repository. Additionally it will create a reasonable default configuration and generate a new release manager certificate and software signing key. The public key in /etc/cvmfs/keys/my.repo.name.pub needs to be distributed to all client machines.

The cvmfs_server utility will use /srv/cvmfs as storage location by default. In case a separate hard disk should be used, a partition can be mounted on /src/cvmfs or /srv/cvmfs can be symlinked to another location (see Notable CernVM-FS Server Locations and Files). Besides local storage it is possible to use an S3 compatible storage service as data backend.

Once created, the repository is mounted under /cvmfs/my.repo.name containing only a single file called new_repository. The next steps describe how to change the repository content.

cvmfs_server mkfs -v my.repo.name

cvmfs_server mkfs -s /etc/cvmfs/.../mys3.conf \
  -w http://s3.amazonaws.com/mybucket-1-1 my.repo.name

http_access allow all
http_port 127.0.0.1:3128 intercept
cache_peer swift.cern.ch parent 80 0 no-query originserver
url_rewrite_program /usr/bin/s3_squid_rewrite.py
cache deny all

in: http://localhost:3128/data/.cvmfswhitelist
out: http://swift.cern.ch/cernbucket-9-91/data/.cvmfswhitelist

Repository Update¶

Typically a repository publisher does the following steps in order to create a new revision of a repository:

1. Run cvmfs_server transaction to switch to a copy-on-write enabled CernVM-FS volume
2. Make the necessary changes to the repository, add new directories, patch certain binaries, ...
3. Test the software installation
4. Do one of the following:
   • Run cvmfs_server publish to finalize the new repository revision or
   • Run cvmfs_server abort to clear all changes and start over again

CernVM-FS supports having more than one repository on a single server machine. In case of a multi-repository host, the target repository of a command needs to be given as a parameter when running the cvmfs_server utility. The cvmfs_server resign command should run every 30 days to update the signatures of the repository. Most cvmfs_server commands allow for wildcards to do manipulations on more than one repository at once, cvmfs_server migrate *.cern.ch would migrate all present repositories ending with .cern.ch.

Repository Import¶

The CernVM-FS 2.1 server tools support the import of a CernVM-FS file storage together with its corresponding signing keychain. With cvmfs_server import both CernVM-FS 2.0 and 2.1 compliant repository file storages can be imported.

cvmfs_server import works similar to cvmfs_server mkfs (described in Repository Creation) except it uses the provided data storage instead of creating a fresh (and empty) storage. In case of a CernVM-FS 2.0 file storage cvmfs_server import also takes care of the file catalog migration into the CernVM-FS 2.1 schema.

Customizable Actions Using Server Hooks¶

The cvmfs_server utility allows release managers to trigger custom actions before and after crucial repository manipulation steps. This can be useful for example for logging purposes, establishing backend storage connections automatically or other workflow triggers, depending on the application.

There are six designated server hooks that are potentially invoked during the repository update procedure:

• When running cvmfs_server transaction:
  • before the given repository is transitioned into transaction mode
  • after the transition was successful
• When running cvmfs_server publish:
  • before the publish procedure for the given repository is started
  • after it was published and remounted successfully
• When running cvmfs_server abort:
  • before the unpublished changes will be erased for the given repository
  • after the repository was successfully reverted to the last published state

All server hooks must be defined in a single shell script file called:

/etc/cvmfs/cvmfs_server_hooks.sh

The cvmfs_server utility will check the existence of this script and source it. To subscribe to the described hooks one needs to define one or more of the following shell script functions:

• transaction_before_hook()
• transaction_after_hook()
• publish_before_hook()
• publish_after_hook()
• abort_before_hook()
• abort_after_hook()

The defined functions get called at the specified positions in the repository update process and are provided with the fully qualified repository name as their only parameter ($1). Undefined functions automatically default to a NO-OP. An example script is located at cvmfs/cvmfs_server_hooks.sh.demo in the CernVM-FS sources.

Integrity Check¶

CernVM-FS provides an integrity checker for repositories. It is invoked by

cvmfs_server check

The integrity checker verifies the sanity of file catalogs and verifies that referenced data chunks are present. Ideally, the integrity checker is used after every publish operation. Where this is not affordable due to the size of the repositories, the integrity checker should run regularly.

Optionally cvmfs_server check can also verify the data integrity (command line flag -i) of each data object in the repository. This is a time consuming process and we recommend it only for diagnostic purposes.

Named Snapshots¶

Named snapshots or tags are an easy way to organise checkpoints in the file system history. CernVM-FS clients can explicitly mount a repository at a specific named snapshot to expose the file system content published with this tag. It also allows for rollbacks to previously created and tagged file system revisions. Tag names need to be unique for each repository and are not allowed to contain spaces or spacial characters. Besides the actual tag’s name they can also contain a free descriptive text and store a creation timestamp.

Named snapshots are best to use for larger modifications to the repository, for instance when a new major software release is installed. Named snapshots provide the ability to easily undo modifications and to preserve the state of the file system for the future. Nevertheless, named snapshots should not be used excessively. Less than 50 named snapshots are a good number of named snapshots in many cases.

By default, new repositories will automatically create a generic tag if no explicit tag is given during publish. The automatic tagging can be turned off using the -g option during repository creation or by setting CVMFS_AUTO_TAG=false in the /etc/cvmfs/repositories.d/$repository/server.conf file.

cvmfs_server transaction
# Changes
cvmfs_server publish -a release-1.0 -m "first stable release"

Managing Nested Catalogs¶

CernVM-FS stores meta-data (path names, file sizes, ...) in file catalogs. When a client accesses a repository, it has to download the file catalog first and then it downloads the files as they are opened. A single file catalog for an entire repository can quickly become large and impractical. Also, clients typically do not need all of the repository’s meta-data at the same time. For instance, clients using software release 1.0 do not need to know about the contents of software release 2.0.

With nested catalogs, CernVM-FS has a mechanism to partition the directory tree of a repository into many catalogs. Repository maintainers are responsible for sensible cutting of the directory trees into nested catalogs. They can do so by creating and removing magic files named .cvmfscatalog.

For example, in order to create a nested catalog for software release 1.0 in the hypothetical repository experiment.cern.ch, one would invoke

cvmfs_server transaction
touch /cvmfs/experiment.cern.ch/software/1.0/.cvmfscatalog
cvmfs_server publish

In order to merge a nested catalog with its parent catalog, the corresponding .cvmfscatalog file needs to be removed. Nested catalogs can be nested on arbitrary many levels.

Recommendations for Nested Catalogs¶

Nested catalogs should be created having in mind which files and directories are accessed together. This is typically the case for software releases, but can be also on the directory level that separates platforms. For instance, for a directory layout like

/cvmfs/experiment.cern.ch
  |- /software
  |    |- /i686
  |    |    |- 1.0
  |    |    |- 2.0
  |    `    |- common
  |    |- /x86_64
  |    |    |- 1.0
  |    `    |- common
  |- /grid-certificates
  |- /scripts

it makes sense to have nested catalogs at

/cvmfs/experiment.cern.ch/software/i686
/cvmfs/experiment.cern.ch/software/x86_64
/cvmfs/experiment.cern.ch/software/i686/1.0
/cvmfs/experiment.cern.ch/software/i686/2.0
/cvmfs/experiment.cern.ch/software/x86_64/1.0

A nested catalog at the top level of each software package release is generally the best approach because once package releases are installed they tend to never change, which reduces churn and garbage generated in the repository from old catalogs that have changed. In addition, each run only tends to access one version of any package so having a separate catalog per version avoids loading catalog information that will not be used. A nested catalog at the top level of each platform may make sense if there is a significant number of platform-specific files that aren’t included in other catalogs.

It could also make sense to have a nested catalog under grid-certificates, if the certificates are updated much more frequently than the other directories. It would not make sense to create a nested catalog under /cvmfs/experiment.cern.ch/software/i686/common, because this directory needs to be accessed anyway whenever its parent directory is needed. As a rule of thumb, a single file catalog should contain more than 1000 files and directories but not contain more than \(\approx\)200000 files. See Inspecting Nested Catalog Structure how to find catalogs that do not satisfy this recommendation.

Restructuring the repository’s directory tree is an expensive operation in CernVM-FS. Moreover, it can easily break client applications when they switch to a restructured file system snapshot. Therefore, the software directory tree layout should be relatively stable before filling the CernVM-FS repository.

Managing Nested Catalogs with .cvmfsdirtab¶

Rather than managing .cvmfscatalog files by hand, a repository administrator may create a file called .cvmfsdirtab, in the top directory of the repository, which contains a list of paths relative to the top of the repository where .cvmfscatalog files will be created. Those paths may contain shell wildcards such as asterisk (*) and question mark (?). This is useful for specifying patterns for creating nested catalogs as new files are installed. A very good use of the patterns is to identify directories where software releases will be installed.

In addition, lines in .cvmfsdirtab that begin with an exclamation point (!) are shell patterns that will be excluded from those matched by lines without an exclamation point. For example a .cvmfsdirtab might contain these lines for the repository of the previous subsection:

/software/*
/software/*/*
! */common
/grid-certificates

This will create nested catalogs at

/cvmfs/experiment.cern.ch/software/i686
/cvmfs/experiment.cern.ch/software/i686/1.0
/cvmfs/experiment.cern.ch/software/i686/2.0
/cvmfs/experiment.cern.ch/software/x86_64
/cvmfs/experiment.cern.ch/software/x86_64/1.0
/cvmfs/experiment.cern.ch/grid-certificates

Note that unlike the regular lines that add catalogs, asterisks in the exclamation point exclusion lines can span the slashes separating directory levels.

Inspecting Nested Catalog Structure¶

The following command visualizes the current nested file catalog layout of a repository.

cvmfs_server list-catalogs

Additionally this command allows to spot degenerated nested catalogs. As stated here the recommended maximal file entry count of a single catalog should not exceed \(\approx\)200000. One can use the switch list-catalogs -e to inspect the current nested catalog entry counts in the repository. Furthermore list-catalgos -s will print the file sizes of the catalogs in bytes.

Migrate File Catalogs¶

In rare cases the further development of CernVM-FS makes it necessary to change the internal structure of file catalogs. Updating the CernVM-FS installation on a Stratum 0 machine might require a migration of the file catalogs.

It is recommended that cvmfs_server list is issued after any CernVM-FS update to review if any of the maintained repositories need a migration. Outdated repositories will be marked as “INCOMPATIBLE” and cvmfs_server refuses all actions on these repositories until the file catalogs have been updated.

In order to run a file catalog migration use cvmfs_server migrate for each of the outdated repositories. This will essentially create a new repository revision that contains the exact same file structure as the current revision. However, all file catalogs will be recreated from scratch using the updated internal structure. Note that historic file catalogs of all previous repository revisions stay untouched and are not migrated.

After cvmfs_server migrate has successfully updated all file catalogs repository maintenance can continue as usual.

Garbage Sweeping Policy¶

The garbage collector of CernVM-FS is using a mark-and-sweep algorithm to detect unused files in the internal catalog graph. Revisions that are referenced by named snapshots (cf. Named Snapshots) or that are recent enough are preserved while all other revisions are condemned to be removed. By default this time-based threshold is three days but can be changed using the configuration variable CVMFS_AUTO_GC_TIMESPAN both on Stratum 0 and Stratum 1. The value of this variable is expected to be parseable by the date command, for example 3 days ago or 1 week ago.

Enabling Garbage Collection¶

Data files¶

First and foremost, CernVM-FS is designed to distribute executable code that is shared between a large number of jobs that run together at grid sites, clouds, or clusters. Worker node cache sizes and web proxy bandwidth are generally engineered to accommodate that application. The total amount read per job is expected to be roughly limited by the amount of RAM per job slot. The same files are also expected to be read from the worker node cache multiple times for the same type of job, and read from a caching web proxy by multiple worker nodes.

If there are data files distributed by CernVM-FS that follow similar access patterns and size limits as executable code, it will probably work fine. In addition, if there are files that are larger but read slowly throughout long jobs, as opposed to all at once at the beginning, that can also work well if the same files are read by many jobs. That is because web proxies have to be engineered for handling bursts at the beginning of jobs and so they tend to be lightly loaded a majority of the time.

In general, a good rule of thumb is to calculate the maximum rate at which jobs typically start and limit the amount of data that might be read from a web proxy to per thousand jobs, assuming a reasonable amount of overlap of jobs onto the same worker nodes. Also, limit the amount of data that will be put into any one worker node cache to . Of course, if you have a special arrangement with particular sites to have large caches and bandwidths available, these limits can be made higher at those sites. Web proxies may also need to be engineered with faster disks if the data causes their cache hit ratios to be reduced.

Also, keep in mind that the total amount of data distributed is not unlimited. The files are stored and distributed compressed, and files with the same content stored in multiple places in the same repository are collapsed to the same file in storage, but the storage space is used not only on the original repository server, it is also replicated onto multiple Stratum 1 servers. Generally if only executable code is distributed, there is no problem with the space taken on Stratum 1s, but if many large data files are distributed they may exceed the Stratum 1 storage capacity. Data files also tend to not compress as well, and that is especially the case of course if they are already compressed before installation.

Tarballs, zip files, and other archive files¶

If the contents of a tarball, zip file, or some other type of archive file is desired to be distributed by CernVM-FS, it is usually better to first unpack it into its separate pieces first. This is because it allows better sharing of content between multiple releases of the file; some pieces inside the archive file might change and other pieces might not in the next release, and pieces that don’t change will be stored as the same file in the repository. CernVM-FS will compress the content of the individual pieces, so even if there’s no sharing between releases it shouldn’t take much more space.

File permissions¶

Care should be taken to make all the files in a repository readable by “other”. This is because permissions on files in the original repository are generally the same as those seen by end clients, except the files are owned by the “cvmfs” user and group. The write permissions are ignored by the client since it is a read-only filesystem. However, unless the client has set

CVMFS_CHECK_PERMISSIONS=no

(and most do not), unprivileged users will not be able to read files unless they are readable by “other” and all their parent directories have at least “execute” permissions. It makes little sense to publish files in CernVM-FS if they won’t be able to be read by anyone.

Hardlinks¶

By default CernVM-FS does not allow hardlinks of a file to be in different directories. If there might be any such hardlinks in a repository, set the option

CVMFS_IGNORE_XDIR_HARDLINKS=true

in the repository’s server.conf. The file will not appear to be hardlinked to the client, but it will still be stored as only one file in the repository just like any other files that have identical content. Note that if, in a subsequent publish operation, only one of these cross-directory hardlinks gets changed, the other hardlinks remain unchanged (the hardlink got “broken”).

Content Hashes¶

CernVM-FS can use SHA-1 [Jones01], RIPEMD-160 [Dobbertin96] and SHAKE-128 [Bertoni09] as cryptographic hash function. The hash function can be changed on the Stratum 0 during the lifetime of repositories. On a change, new and updated files will use the new cryptographic hash while existing files remain unchanged. This is transparent to the clients since the hash function is stored in the flags field of file catalogs for each and every file. The default hash function is SHA-1. New software versions might introduce support for further cryptographic hash functions.

Nested Catalogs¶

In order to keep catalog sizes reasonable [1], repository subtrees may be cut and stored as separate nested catalogs. There is no limit on the level of nesting. A reasonable approach is to store separate software versions as separate nested catalogs. The figure below shows the simplified directory structure which we use for the ATLAS repository.

Directory structure useds for the ATLAS repository (simplified).

When a subtree is moved into a nested catalog, its entry directory serves as transition point for nested catalogs. This directory appears as empty directory in the parent catalog with flags set to 2. The same path appears as root-directory in the nested catalog with flags set to 33. Because the MD5 hash values refer to full absolute paths, nested catalogs store the root path prefix. This prefix is prepended transparently by CernVM-FS. The cryptographic hash of nested catalogs is stored in the parent catalog. Therefore, the root catalog fully defines an entire repository.

Loading of nested catalogs happens on demand by CernVM-FS on the first attempt to access of anything inside, a user won’t see the difference between a single large catalog and several nested catalogs. While this usually avoids unnecessary catalogs to be loaded, recursive operations like find can easily bypass this optimization.

Catalog Statistics¶

A CernVM-FS file catalog maintains several counters about its contents and the contents of all of its nested catalogs. The idea is that the catalogs know how many entries there are in their sub catalogs even without opening them. This way, one can immediately tell how many entries, for instance, the entire ATLAS repository has. Some of the numbers are shown using the number of inodes in statvfs. So df -i shows the overall number of entries in the repository and (as number of used inodes) the number of entries of currently loaded catalogs. Nested catalogs create an additional entry (the transition directory is stored in both the parent and the child catalog). File hardlinks are still individual entries (inodes) in the cvmfs catalogs. The following counters are maintained for both a catalog itself and for the subtree this catalog is root of:

• Number of regular files
• Number of symbolic links
• Number of directories
• Number of nested catalogs
• Number of chunked files
• Number of individual file chunks
• Overall file content size
• File content size stored in chunked files

Internal Manifest Structure¶

Below is an example of a typical manifest file. Each line starts with a capital letter specifying the meta data field, followed by the actual data string. The list of meta information is ended by a separator line (--) followed by signature information further described here.

C64551dccfbe0a48de7618dd7deb290200b474759
B1442336
Rd41d8cd98f00b204e9800998ecf8427e
D900
S42
Nexample.cern.ch
X731cca9476eb882f5a3f24aaa38001105a0e35eb
T1390301299
--
edde5308e502dd5e8fe405c56f5700f7477dc319
[...]

Please refer to table below for detailed information about each of the meta data fields.

Repository Signature¶

In order to provide authoritative information about a repository publisher, the repository manifest is signed by an X.509 certificate together with its private key.

DoS Protection¶

A subtle denial of service attack (DoS) can occur when CernVM-FS is successfully able to download a file but fails to store it in the local cache. This situation escalates into a DoS when the application using CernVM-FS remains in an endless loop and tries to open a file over and over again. Such a situation is prevented by CernVM-FS by re-trying with an exponential backoff. The backoff is triggered by consequtive filaures to cache a downloaded file within 10 seconds.

Keep-Alive¶

Although the HTTP protocol overhead is small in terms of data volume, in high latency networks we suffer from the bare number of requests: Each request-response cycle has a penalty of at least the network round trip time. Using plain HTTP/1.0, this results in at least \(3\cdot\text{round trip time}\) additional running time per file download for TCP handshake, HTTP GET, and TCP connection finalisation. By including the Connection: Keep-Alive header into HTTP requests, we advise the HTTP server end to keep the underlying TCP connection opened. This way, overhead ideally drops to just round trip time for a single HTTP GET. The impact of the keep-alive feature is shown in here.

This feature, of course, somewhat sabotages a server-side load-balancing. However, exploiting the HTTP keep-alive feature does not affect scalability per se. The servers and proxies may safely close idle connections anytime, in particular if they run out of resources.

Cache Control¶

In a limited way, CernVM-FS advises intermediate web caches how to handle its requests. Therefor it uses the Pragma: no-cache and the Cache-Control: no-cache headers in certain cases. These cache control headers apply to both, forward proxies as well as reverse proxies. This is not a guarantee that intermediate proxies fetch a fresh original copy (though they should).

By including these headers, CernVM-FS tries to not fetch outdated cache copies. Only in case CernVM-FS downloads a corrupted file from a proxy server, it retries having the HTTP no-cache header set. This way, the corrupted file gets replaced in the proxy server by a fresh copy from the backend.

Identification Header¶

CernVM-FS sends a custom header (X-CVMFS2) to be identified by the web server. If you have set the CernVM GUID, this GUID is also transmitted.

Redirects¶

Normally, the Stratum-1 servers directly respond to HTTP requests so CernVM-FS has no need to support HTTP redirect response codes. However, there are some high-bandwidth applications where HTTP redirects are used to transfer requests to multiple data servers. To enable support for redirects in the CernVM-FS client, set CVMFS_FOLLOW_REDIRECTS=yes.

mount¶

On mount, the file catalog has to be loaded. First, the file catalog manifest .cvmfspublished is loaded. The manifest is only accepted on successful validation of the signature. In order to validate the signature, the certificate and the white-list are downloaded in addition if not found in cache. If the download fails for whatever reason, CernVM-FS tries to load a local file catalog copy. As long as all requested files are in the disk cache as well, CernVM-FS continues to operate even without network access (offline mode). If there is no local copy of the manifest or the downloaded manifest and the cache copy differ, CernVM-FS downloads a fresh copy of the file catalog.

getattr and lookup¶

Requests for file attributes are entirely served from the mounted catalogs, there is no network traffic involved. This function is called as pre-requisite to other file system operations and therefore the most frequently called Fuse callback. In order to minimize relatively expensive SQLite queries, CernVM-FS uses a hash table to store negative and positive query results. The default size of for this memory cache is determined according to benchmarks with LHC experiment software.

Additionally, the callback takes care of the catalog TTL. If the TTL is expired, the catalog is re-mounted on the fly. Note that a re-mount might possibly break running programs. We rely on careful repository publishers that produce more or less immutable directory trees, new repository versions just add files.

If a directory with a nested catalog is accessed for the first time, the respective catalog is mounted in addition to the already mounted catalogs. Loading nested catalogs is transparent to the user.

readlink¶

A symbolic link is served from the file catalog. As a special extension, CernVM-FS detects environment variables in symlink strings written as $(VARIABLE) or $(VARIABLE:-/default/path). These variables are expanded by CernVM-FS dynamically on access (in the context of the cvmfs2 process). This way, a single symlink can point to different locations depending on the environment. This is helpful, for instance, to dynamically select software package versions residing in different directories.

readdir¶

A directory listing is served by a query on the file catalog. Although the “parent”-column is indexed (see Catalog table schema), this is a relatively slow function. We expect directory listing to happen rather seldom.

open / read¶

The open() call has to provide a file descriptor for a given path name. In CernVM-FS file requests are always served from the disk cache. The Fuse file handle is a file descriptor valid in the context of the CernVM-FS process. It points into the disk cache directory. Read requests are translated into the pread() system call.

getxattr¶

CernVM-FS uses extended attributes to display additional repository information. There are two supported attributes:

expires

Shows the remaining life time of the mounted root file catalog in minutes.

fqrn

Shows the fully qualified repository name of the mounted repository.

inode_max

Shows the highest possible inode with the current set of loaded catalogs.

hash

Shows the cryptographic hash of a regular file as listed in the file catalog.

host

Shows the currently active HTTP server.

host_list

Shows the ordered list of HTTP servers.

lhash

Shows the cryptographic hash of a regular file as stored in the local cache, if available.

maxfd

Shows the maximum number of file descriptors available to file system clients.

nclg

Shows the number of currently loaded nested catalogs.

ndiropen

Shows the overall number of opened directories.

ndownload

Shows the overall number of downloaded files since mounting.

nioerr

Shows the total number of I/O errors encoutered since mounting.

nopen

Shows the overall number of open() calls since mounting.

pid

Shows the process id of the CernVM-FS Fuse process.

proxy

Shows the currently active HTTP proxy.

rawlink

Shows unresolved variant symbolic links; only accessible as root.

revision

Shows the file catalog revision of the mounted root catalog, an auto-increment counter increased on every repository publish.

root_hash

Shows the cryptographic hash of the root file catalog.

rx

Shows the overall amount of downloaded kilobytes.

speed

Shows the average download speed.

timeout

Shows the timeout for proxied connections in seconds.

timeout_direct

Shows the timeout for direct connections in seconds.

rawlink

Shows the unresolved variant symlink.

uptime

Shows the time passed since mounting in minutes.

usedfd

Shows the number of file descriptors currently issued to file system clients.

version

Shows the version of the loaded CernVM-FS binary.

Extended attributes can be queried using the attr command. For instance, attr -g hash /cvmfs/atlas.cern.ch/ChangeLog returns the cryptographic hash of the file at hand. The extended attributes are used by the cvmfs_config stat command in order to show a current overview of health and performance numbers.

Read-write Interface using a Union File System¶

Union file systems combine several directories into one virtual file system that provides the view of merging these directories. These underlying directories are often called branches. Branches are ordered; in the case of operations on paths that exist in multiple branches, the branch selection is well-defined. By stacking a read-write branch on top of a read-only branch, union file systems can provide the illusion of a read-write file system for a read-only file system. All changes are in fact written to the read-write branch.

Preserving POSIX semantics in union file systems is non-trivial; the first fully functional implementation has been presented by Wright et al. [Wright04]. By now, union file systems are well established for “Live CD” builders, which use a RAM disk overlay on top of the read- only system partition in order to provide the illusion of a fully read-writable system. CernVM-FS uses the AUFS union file system. Another union file system with similar semantics can be plugged in if necessary. OverlayFS is supported as an experimental alternative.

Union file systems can be used to track changes on CernVM-FS repositories (Figure below). In this case, the read-only file system interface of CernVM-FS is used in conjunction with a writable scratch area for changes.

A union file system combines a CernVM-FS read-only mount point and a writable scratch area. It provides the illusion of a writable CernVM-FS mount point, tracking changes on the scratch area.

Based on the read-write interface to CernVM-FS, we create a feed-back loop that represents the addition of new software releases to a CernVM-FS repository. A repository in base revision \(r\) is mounted in read-write mode on the publisher’s end. Changes are written to the scratch area and, once published, are re-mounted as repository revision \(r+1\). In this way, CernVM-FS provides snapshots. In case of errors, one can safely resume from a previously committed revision.

Footnotes