Installing software using apt

Notes on migration if you have previously installed your software area with aptinstaller.sh

Contents:

Installing software using apt

GOAL

This page describes a simple method for installing the CMS software rpms as a non-root user using apt. A script developed by CMS, bootstrap.sh, provides the configuration necessary to use apt as a non-root user for software installation.

Note that you should not run the installation commands described on this page as root, but instead as some normal user.

Please report any bug / problem / feature request for what is described in this page to the following bug tracker:

https://savannah.cern.ch/projects/cmspackaging/

One time initialization of the software area

Commands in this section should only be performed once per software area.

Installing deprecated releases (locally)

A software area can be the one used for official installations or a "local" one which may be used to keep releases which are no longer supported in the official scheme (and are therefore deprecated and removed from the official area). For these "local" installations it is sufficient to use a different installation area (a different /x/y/z in the following examples) to install the software.

Prerequisites

cmssw depends upon a handful of system packages: which, wget, tk, tcsh, zsh, freetype, fontconfig, compat-libstdc++-33, libidn, libXmu, libXcursor, libXrandr, libXft, mesa-libGLU, libXi, libXinerama, libXpm (these are the SLC5 package names. In your distribution, these might differ ...).

For SL5 machines it is recommended to install the SL5DependencyRPM.

Setting up the software area environment

In order to use apt with a standard user we need to prepare some directory with the necessary configuration files and then set the environment so that the system apt lookups its configuration there. First you must choose an installation directory (/x/y/z/ in the following examples) and then choose one of two possibilities:

LCG style (also sufficient for laptop users)

setenv VO_CMS_SW_DIR /x/y/z
# Make sure that directory /x/y/z exists

x/y/z is the general installation folder, i.e. does NOT include the name of the actual release one wants to install

OSG style

# Software will be installed in $OSG_APP/cmssoft/cms
setenv OSG_APP /x/y/z
# Make sure that the directory /x/y/z/cmssoft/cms exists

Using the LCG-style is recommended for laptop or workstation users. It does not imply anything about the use of LCG Grid software, just that you are using the same "entry" environment variable to the software area that is used on LCG sites.

Once you have set one of these two environment variables, you should be able to use all of the commands which follow on this page simply by cutting and pasting, without needing to adapt them for your site.

Get the script

You now need to download the bootstarp.sh script which is used to setup and initialize a CMS software area.

setenv LANG "C"
wget -O $VO_CMS_SW_DIR/bootstrap.sh http://cmsrep.cern.ch/cmssw/cms/bootstrap.sh

Initialize the software area

This set depends on a directory /tmp/ being already present. If not

mkdir /tmp/$USER

In order to initialize a CMS software area for a given architecture you need to set the environment variable SCRAM_ARCH accordingly:

Architecture	SCRAM_ARCH
SL4 32bit	slc4_ia32_gcc345
SL5 32bit	slc5_ia32_gcc434
SL5 64bit	slc5_amd64_gcc434

export SCRAM_ARCH=<Value from table>
# csh/tcsh style is like so:
# setenv SCRAM_ARCH <Value from table>

Then you should execute

sh -x $VO_CMS_SW_DIR/bootstrap.sh setup -path $VO_CMS_SW_DIR -arch $SCRAM_ARCH >& $VO_CMS_SW_DIR/bootstrap_$SCRAM_ARCH.log

Once bootstrap.sh is successful, apt and rpm binaries from the installation area should be used by sourcing a setup file:

source $VO_CMS_SW_DIR/$SCRAM_ARCH/external/apt/<apt-version>/etc/profile.d/init.sh

# OR source $VO_CMS_SW_DIR/$SCRAM_ARCH/external/apt/<apt-version>/etc/profile.d/init.csh

After successful boot bootstrap of an slc5_{ia32|amd64}_gcc434 area you need to install some fake packages:

Find them via

 source $VO_CMS_SW_DIR/$SCRAM_ARCH/external/apt/<apt-version>/etc/profile.d/init.sh
 # OR source $VO_CMS_SW_DIR/$SCRAM_ARCH/external/apt/<apt-version>/etc/profile.d/init.csh 
 apt-cache pkgnames | grep fake

and install them with

apt-get install <fake-package>

Installing and updating software

Commands in the section can be repeated as often as needed.

Preparation for installation/updates

Each time the software manager wants to install or upgrade software, two commands should be executed:

source $VO_CMS_SW_DIR/$SCRAM_ARCH/external/apt/<apt-version>/etc/profile.d/init.sh
# OR source $VO_CMS_SW_DIR/$SCRAM_ARCH/external/apt/<apt-version>/etc/profile.d/init.csh
apt-get update

where apt-version is determined by what was installed by the bootstrap. This is most easily determined by looking in the $SCRAM_ARCH/external/apt area. Currently the apt-version is 0.5.15lorg3.2-CMS19c, however this may change or may be different for different platforms.

The second of these two commands will output something like:

apt-get:1 http://cmsdoc.cern.ch cms/cpt/Software/download/cms/apt/slc3_ia32_gcc323 release [1425B]
Fetched 1425B in 0s (23.9kB/s)
Hit http://cmsdoc.cern.ch cms/cpt/Software/download/cms/apt/slc3_ia32_gcc323/lcg pkglist
 Hit http://cmsdoc.cern.ch cms/cpt/Software/download/cms/apt/slc3_ia32_gcc323/lcg release
 Hit http://cmsdoc.cern.ch cms/cpt/Software/download/cms/apt/slc3_ia32_gcc323/cms pkglist
 Hit http://cmsdoc.cern.ch cms/cpt/Software/download/cms/apt/slc3_ia32_gcc323/cms release
 Hit http://cmsdoc.cern.ch cms/cpt/Software/download/cms/apt/slc3_ia32_gcc323/external pkglist
 Hit http://cmsdoc.cern.ch cms/cpt/Software/download/cms/apt/slc3_ia32_gcc323/external release
 Hit http://cmsdoc.cern.ch cms/cpt/Software/download/cms/apt/slc3_ia32_gcc323/lcg srclist
 Hit http://cmsdoc.cern.ch cms/cpt/Software/download/cms/apt/slc3_ia32_gcc323/cms srclist
 Hit http://cmsdoc.cern.ch cms/cpt/Software/download/cms/apt/slc3_ia32_gcc323/external srclist
 Reading Package Lists... Done
 Building Dependency Tree... Done

After this, all the subsequent apt-get / apt-cache commands will refer to the installation area in $VO_CMS_SW_DIR or $OSG_APP/cmssoft/cms.

Packages available for installation

The full set of rpms available for installation can be seen with:

source $VO_CMS_SW_DIR/$SCRAM_ARCH/external/apt/<apt-version>/etc/profile.d/init.sh
# OR source $VO_CMS_SW_DIR/$SCRAM_ARCH/external/apt/<apt-version>/etc/profile.d/init.csh
apt-cache pkgnames | egrep '^(cms|lcg|external)'

and in particular the available CMSSW releases can be seen with:

apt-cache search cmssw

The latter will output something like:

cms+cmssw+CMSSW_0_6_1 - CMS Software Package cmssw version CMSSW_0_6_1
cms+cmssw+CMSSW_0_7_0 - CMS Software Package cmssw version CMSSW_0_7_0
cms+cmssw+CMSSW_0_7_1 - CMS Software Package cmssw version CMSSW_0_7_1
cms+cmssw+CMSSW_0_8_0 - CMS Software Package cmssw version CMSSW_0_8_0
cms+cmssw+CMSSW_0_8_1 - CMS Software Package cmssw version CMSSW_0_8_1
cms+cmssw+CMSSW_0_9_0 - CMS Software Package cmssw version CMSSW_0_9_0
[snip]

Installing CMSSW with apt-get

Please be sure to read the warnings in the next sections before installing releases.

If not done already, the following step has to be done only once after the bootstrap.

Find them via

 source $VO_CMS_SW_DIR/$SCRAM_ARCH/external/apt/<apt-version>/etc/profile.d/init.csh
 # OR source $VO_CMS_SW_DIR/$SCRAM_ARCH/external/apt/<apt-version>/etc/profile.d/init.csh
 apt-cache pkgnames | grep fake

and install them with

apt-get install <fake-package>

To install a CMS release (e.g. CMSSW_0_9_0), do:

apt-get install cms+cmssw+CMSSW_0_9_0

which will give something like:

The following extra packages will be installed:
  cms+cms-env+1.0 cms+cmssw-tool-conf+CMS_137-withlcg46
  cms+ignominy+IGNOMINY_3_11_0-forlcg46
  cms+ignominy-tool-conf+CMS_137-withlcg46 cms+iguana+IGUANA_6_12_0-withlcg46
  cms+iguana-tool-conf+CMS_137-withlcg46 cms+rulechecker+2.6
  external+boost+1.33.1 external+boost-build+2.0-m10 external+bz2lib+1.0.2
  external+castor+2.1.0-0 external+clhep+1.9.2.2 external+cmake+2.4.2
  external+coin+2.4.4 external+cppunit+1.10.2 external+curl+7.15.3
  external+db4+4.4.20 external+dcap+1.2.35 external+doxygen+1.4.1
  external+elementtree+1.1 external+expat+2.0.0
  [snip]
The following NEW packages will be installed:
  cms+cms-env+1.0 cms+cmssw+CMSSW_0_9_0 cms+cmssw-tool-conf+CMS_137-withlcg46
  cms+ignominy+IGNOMINY_3_11_0-forlcg46
  cms+ignominy-tool-conf+CMS_137-withlcg46 cms+iguana+IGUANA_6_12_0-withlcg46
  cms+iguana-tool-conf+CMS_137-withlcg46 cms+rulechecker+2.6
  [snip]
0 upgraded, 82 newly installed, 0 removed and 0 not upgraded.
Need to get 1433MB of archives.
After unpacking 4143MB of additional disk space will be used.
Do you want to continue? [Y/n]

Hit 'Y', then watch:

Get:1 http://cmsdoc.cern.ch cms/cpt/Software/download/cms/apt/slc3_ia32_gcc323/external external+gcc+3.2.3 1-1006 [20.9MB]
Get:2 http://cmsdoc.cern.ch cms/cpt/Software/download/cms/apt/slc3_ia32_gcc323/cms cms+cms-env+1.0 1-1014 [5994B]
Get:3 http://cmsdoc.cern.ch cms/cpt/Software/download/cms/apt/slc3_ia32_gcc323/cms cms+rulechecker+2.6 1-1005 [2093kB]
[snip]
Fetched 1433MB in 2h7m53s (187kB/s)                                            
Executing RPM (-Uvh)...
Preparing...                ########################################### [100%]
   1:external+gcc+3.2.3     ########################################### [  1%]
Relocating to /mnt/cms1/elmer/cmspath
   2:cms+cms-env+1.0        ########################################### [  2%]
/mnt/cms1/elmer/cmspath
   3:cms+rulechecker+2.6    ########################################### [  4%]
Relocating to /mnt/cms1/elmer/cmspath
   4:external+expat+2.0.0   ########################################### [  5%]
[snip]
Done.

Note that apt takes care all of the installation of underlying dependencies, not just the CMSSW software release itself. The actual list of packages installed will depend on whether necessary rpms are already installed (e.g. due to a previous release installation).

Installing releases newer than 200pre1 into an existing software area

There is a small change required to install releases newer than 200pre1 into software areas bootstrapped using an old bootstrap recipe. [if you followed the recipe above, there is no need to follow this recipe].

Before installing 200pre2 or newer, do:

apt-get install cms+cms-common+1.0

Installing releases older than 200pre2

Conversely, if you have followed the bootstrap instructions above, an extra step is needed to install software releases older than 200pre2. This is required only for the first "old" release that you install.

Begin by installing the software release you require. For example

apt-get install cms+cmssw+CMSSW_1_8_0

After doing this, reinstall the cms-common rpm:

apt-get --reinstall install cms+cms-common+1.0-cms2

More precisely, this step is required anytime a version of SCRAMV1 is installed that is older than V1_2_0.

User environment configuration

Default environment

In order for a user to use the software you have just installed the user needs to set the relevant environment variable (LCG-style or OSG-style) and source the approriate cmsset_default.sh/csh. This typically means including one of the two following things in the users login scripts:

setenv VO_CMS_SW_DIR /x/y/z
source $VO_CMS_SW_DIR/cmsset_default.csh         # For LCG-style, or

setenv OSG_APP /x/y/z
source $OSG_APP/cmssoft/cms/cmsset_default.csh  # For OSG-style

With this configuration, the user can then do the usual scram list / scram project commands to create a new working area:

scramv1 list CMSSW
scramv1 project CMSSW CMSSW_0_9_0
etc.

Architecture specific environment

cmsset_default.csh script tries to guess the architecture of the current host machine, and sets SCRAM_ARCH environment variable to default value for this architecture, or to unsupported on the systems which it can not recognize.

To change the default user should set SCRAM_ARCH variable before sourcing the cmsset_default.csh script.

For example, if the user wants to run a cmsRun version available only for slc3_ia32_gcc323, like CMSSW_1_2_0, on slc4_amd64_gcc345 host, user can do:

setenv SCRAM_ARCH slc3_ia32_gcc323
source $VO_CMS_SW_DIR/cmsset_default.csh

In general changing the default does not guarantee that software will be available or compatible with the current system, or may work with limited functionality.

Recipes for succesful installation on different Linux distro's

Gentoo

https://twiki.cern.ch/twiki/bin/view/CMS/SWGuideCMSSWOnGentoo

Ubuntu

https://twiki.cern.ch/twiki/bin/view/CMS/SWGuideCMSSWOnUbuntu

F.A.Q.

Q: On my initial install on slc5 the log ends with "There was an error generating the platform seed" exit 1. What happened?

A: It could mean that one or more of the required packages is missing from your SLC5 installation. Refer to the above Prerequisites. Check which ones you are missing and install them.

Example: In the log file, above the "There was an error..." message you see:

-----------------------
+ echo 'Required package zsh is missing.'
-----------------------

so you perform:

sudo yum install zsh

ref: HyperNews forum reference

Q: Can I install SL4 builds on SL5 machines?

A: Yes. to perform the bootstrap you actually need to add the "-unsupported_distribution_hack" flag to the call.

sh -x $VO_CMS_SW_DIR/bootstrap.sh setup -path $VO_CMS_SW_DIR -arch $SCRAM_ARCH -unsupported_distribution_hack

Note that you need to install 2_x_x SL4 releases on SL4 machines if you want to use them on SL4 as well as SL5.

Q: apt-get dies with a message about "E: Dynamic MMap ran out of room" when trying to install a new release

A: On July 9, 2008, we exceeded the default cache size for apt. There is an easy fix to increase it. Download this script, change its name to fixApt.py, set up apt as you normally would for installing a release, and run the script. E.g.:

source $VO_CMS_SW_DIR/$SCRAM_ARCH/external/apt/0.5.15lorg3.2-CMS3/etc/profile.d/init.csh
python ./fixApt.py

Q: When I try to access files using castor I get "undefined symbol: Cns_lstat":

A: This is a known bug of some revisions of external+castor+2.1.0 package. Please do the following.

source $VO_CMS_SW_DIR/<arch>/external/apt/<apt-version>/etc/profile.d/init.csh
apt-get update 
apt-get dist-upgrade

Q: I found a bug and I reported it on the hn-cms-sw-develtools mailing list but I was told it's already fixed and that I should upgrade my installation. What do I need to do?

source $VO_CMS_SW_DIR/<arch>/external/apt/<apt-version>/etc/profile.d/init.csh
apt-get update
apt-get dist-upgrade

Q: apt-get is stuck since hours. What should I do?

First of all you need to kill all the stale rpm processes. Look with:

ps -ef | egrep '(rpm|apt)'

and 'kill -9 (pid)' for each of the rpm and/or apt processes that you see.

After that is done, do one of the following on SLC3:

rm -f $VO_CMS_SW_DIR/var/lib/rpm/__db.*                                # For LCG-style
rpmdb -vv --rebuilddb --dbpath $VO_CMS_SW_DIR/var/lib/rpm

rm -f $OSG_APP/cmssoft/cms/var/lib/rpm/__db.*                         # For OSG-style      
rpmdb --rebuilddb --dbpath $OSG_APP/cmssoft/cms/var/lib/rpm

On SLC4 rpm requires an addition command line option:

rm -f $VO_CMS_SW_DIR/var/lib/rpm/__db.*                                # For LCG-style
rpmdb -vv --define "_rpmlock_path $VO_CMS_SW_DIR/var/lib/rpm/lock" --rebuilddb --dbpath $VO_CMS_SW_DIR/var/lib/rpm

rm -f $OSG_APP/cmssoft/cms/var/lib/rpm/__db.*                         # For OSG-style      
rpmdb -vv --define "_rpmlock_path $OSG_APP/cmssoft/cms/var/lib/rpm/lock" --rebuilddb --dbpath $OSG_APP/cmssoft/cms/var/lib/rpm

If you are doing this at CERN, you may have to replace $VO_CMS_SW_DIR with "$CMS_PATH/sw". In addition you will probably find that rebuild is much quicker if you change the path(s) to point directly at the copy on local disk rather than have it 'stat' its way through afs, just to find a soft link to a local disk. (This is possible for the moment only for SLC3. You must be on lxbuild024 and the local disk path is /build/cmsbuild/rpmDatabase/rpm.)

Q: Something went wrong to my installation and now apt says something about multiple versions of a given package in the system. What should I do?:

There seems to be a bug in apt if the post installation script aborts or it is interrupted. On the subsequent call to apt-get you get a message like the following:

W: There are multiple versions of "xxx+yyy+zzz" in your system.

This package won't be cleanly updated, unless you leave
only one version. To leave multiple versions installed,
you may remove that warning by setting the following
option in your configuration file:

RPM::Allow-Duplicated { "^xxx+yyy+zzz$"; };

To disable these warnings completely set:

RPM::Allow-Duplicated-Warning "false

To solve this you should list the packages that are installaed with the same name:

rpm --dbpath $VO_CMS_SW_DIR/var/lib/rpm -qa | grep xxx+yyy+zzz

and then remove the one you have unsuccessfully installed.

rpm --dbpath $VO_CMS_SW_DIR/var/lib/rpm -e xxx+yyy+zzz-1-jjjj

Q: I cannot find cmscvsroot and project, what happened to them?

In order to have cmscvsroot and project installed you need to install the package cms-cvs-utils

source $VO_CMS_SW_DIR/<arch>/external/apt/<apt-version>/etc/profile.d/init.csh
apt-get install cms+cms-cvs-utils+1.0-cms

Q: Apt cannot install software because it say there are missing dependencies, what should I do?

If you get a message similar to:

Running installation of CMSSW_0_8_1
Reading Package Lists...
Building Dependency Tree...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.

Since you only requested a single operation it is extremely likely that
the package is simply not installable and a bug report against
that package should be filed.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
   cms+cmssw+CMSSW_0_8_1: Depends: xxxxxxx but it is not installable
                          Depends: yyyyyyy but it is not installable
Install failed

First of all make sure that rpmbuild is installed on the system.

which rpmbuild

If this is not the case, first of all ask your sysadmin to install it, and then repeat the setup proceedure. If this is still not enought report to hn-cms-sw-develtools@cern.ch specifying:

the distribution you are using
the result of running
```
     rpm -qa
     
```
on the machine you are working on.

the result of running

     rpm -q --whatprovides xxxxxx
     rpm -q --whatprovides yyyyyy

on the machine you are working on.

Notice that the message usually depends on the fact that you don't have one of the following packages installed in your system:

Q: apt-get fails with error message "E: Sub-process /build/eulisse/100g48/bin/rpm-wrapper returned an error code (some number)"

If apt-get fails with something like the following:

Preparing...                ^@########################################### [100%]
        installing package external+p5-dbi+1.50-1-1009 needs 2MB on the /build filesystem
        installing package external+p5-template-toolkit+2.14-1-1009 needs 12MB on the /build filesystem
E: Sub-process /build/eulisse/100g48/bin/rpm-wrapper returned an error code (_some number_)

It means you ran out of disk space and should free some before continuing. For more information on RPM return codes see here.

Q: I've updated my system installation with some packages and I want to reseed the local rpm database. How do I do it?

$VO_CMS_SW_DIR/bootstrap.sh reseed -path $VO_CMS_SW_DIR -csh

Q: What is APT?

Apt is an open source package manager. It was initially developed by debian and then extended to support rpm by connectiva. More information about it can be found on http://www.apt-rpm.org.

Q: I did apt-cache search cms+cmssw+CMSSW_X_Y_Z and it returned no results, even if I know it is available.

A: apt-cache needs to have the pluses double escaped in order to parse them correctly. The correct command is:

apt-cache search cms\\+cmssw\\+CMSSW_X_Y_Z

Q: I did apt-get install cms+cmssw+CMSSW_X_Y_Z and it wants to install an older version and/or cannot resolve dependencies.

A: Run "apt-cache pkgnames | grep akesystem" and install these packages first.

Q: I've installed a new version of CMSSW / I've done "apt-get dist-upgrade" and now I don't have scramv1 in my path anymore. What should I do?

Due to a problem in the older SCRAMV1 RPMs which would have prevented the usage of scram V1_0_2 in parallel with a new version V1_0_3, soon to be released, the following migration procedure has to be executed before installing any new release of CMSSW in an area where there are old ones as well:

source $VO_CMS_SW_DIR/<arch>/external/apt/<apt-version>/etc/profile.d/init.csh
apt-get update
apt-get dist-upgrade
apt-get reinstall lcg+SCRAMV1+V1_0_2

Please check at this point that you still see the 'scramv1' command in $CMS_PATH/bin. If you do not, rerun the command:

apt-get reinstall lcg+SCRAMV1+V1_0_2

and then verify again that it is in $CMS_PATH/bin.

If 'scramv1' is present, you can then proceed to install CMSSW as usual:

apt-get install cms+cmssw+CMSSW_X_Y_Z

This extra "dist-upgrade" and "reinstall" steps are NOT required if you are installing from scratch.

If you have any problems please report them to the "Software Distribution Tools Discussions" HN forum (hn-cms-softwareDistrib@cern.ch).

Q: My software area is NFS mounted, and rpm complains "cannot get shared/exclusive lock".

You need to have the nfslock service running on both nfs client and server. On SL systems this can be achieved with service nfslock start.

Q: How do I remove a software release?

The easiest is to use "apt-get remove --purge cms+cmssw+CMSSW_X_Y_Z". This will remove the CMSSW package, and with the --purge switch removes all dependencies which are not owned by another package.

You can also check out the tool cmsCleanSWArea from http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/PKGTOOLS/ and run it. Warning This command removed the "rpm" tool in my case, and I had to reinstall, be careful when using the script!

./cmsCleanSWArea -a \$SCRAM_ARCH -e cms+cmssw+CMSSW_CMSSW_X_Y_Z

Q: How do I make cmsRun run "standalone" without network connection?

Usually production jobs access databases via the network. But you can make your laptop provide the subset of conditions from Frontier needed for the cmsRun job(s) you would like to run, for the moment see https://hypernews.cern.ch/HyperNews/CMS/get/recoDevelopment/806/1/1/2.html

. See also the section "Personal Squid on a Desktop/Laptop" on SquidForCMS.

Q: I get an error like "cpio: rename failed - Inappropriate ioctl for device"

The apt-get install command reports an error like this:

Executing RPM (-Uvh)...
Preparing...                ##################################################
cms+cmssw+CMSSW_2_2_10      ##################################################
error: unpacking of archive failed on file /opt/storage/exp_software/cms/slc4_ia
32_gcc345/cms/cmssw/CMSSW_2_2_10/config/toolbox/slc4_ia32_gcc345/tools/selected/
ccompiler.xml: cpio: rename failed - Inappropriate ioctl for device

This might have two reasons: The RPM was corrupted during download or there is a filesystem corruption.

First try to flush the cached local of the RPM package, which will trigger a reload on the next apt-get install: