Difference between revisions of "OtherPackages"

From CASA Guides
Jump to: navigation, search
(Packages Already Distributed With CASA)
(Packages Already Distributed With CASA)
Line 179: Line 179:
 
Probably the best way to upgrade these is to install your own version of CASA so that you can write to the site-packages directory. Even doing so, you may encounter issues where different versions of the same package persist (based on anecdotal evidence, this so far seems like a bigger problem on Linux than Mac). To remedy this, you may want to take the dangerous but helpful step of deleting the old versions of the packages from CASA's site packages directory before installing your new ones.
 
Probably the best way to upgrade these is to install your own version of CASA so that you can write to the site-packages directory. Even doing so, you may encounter issues where different versions of the same package persist (based on anecdotal evidence, this so far seems like a bigger problem on Linux than Mac). To remedy this, you may want to take the dangerous but helpful step of deleting the old versions of the packages from CASA's site packages directory before installing your new ones.
  
On NRAO Charlottesville's network this was the sequence for a normal user to get an upgraded set of NumPy, SciPy, and matplotlib in CASA:
+
On NRAO Charlottesville's network this was the sequence for a normal user to get an upgraded set of NumPy in CASA:
  
 
* Download the compiled CASA from [http://casa.nrao.edu/casa_obtaining.shtml here.]
 
* Download the compiled CASA from [http://casa.nrao.edu/casa_obtaining.shtml here.]
*
+
* Set it up so that this is your version of CASA (mostly make sure it is in your path).
 +
* Figure out the site-packages directory for this version. For the 4.2 stable we checked it was casapy/lib64/python2.6/site-packages/
 +
* Go into that directory and remove numpy. (This is dangerous!)
 +
* Get the new version of numpy and untar it.
 +
* Identify CASA's python executable and use it to run Numpy's setup.py install.
 +
 
 +
This works - it wipes the old version of Numpy, which otherwise hangs around - and replaces it.
 +
 
 +
(Note that none of this appears to be necessary on a Mac.)
 +
 
 +
Scipy coming soon... still crashing.

Revision as of 19:04, 14 September 2013

CASA ships with a few third party packages (NumPy, SciPy, Matplotlib). It can be very useful to upgrade these or add additional packages. The method to do this can vary from package to package and OS to OS, but many of these packages use the same "distutils" approach and we can summarize a general path.

Disclaimer: This is a writeup documenting the experience at NRAO Charlottesville as of mid-2013. Your mileage may vary, but we hope this orients you and at least gets you started. Also BE CAREFUL when overwriting matplotlib especially. CASA does use these libraries and may not be entirely forward compatible. Be ready to delete the packages and roll back if needed. Better integration with these packages is a possible future goal.

Note: The instructions in this guide were used to successfully install NumPy, SciPy, and AstroPy on a Mac (self-administered) and Linux (administered by someone else). Let us know (temporarily: aleroy@nrao.edu eventually: via the NRAO helpdesk) if you have additional suggestions or refinements.

Danke: Josh Marvil, Remy Indebetouw, Tom Robitaille (x8 for the script)

The Basic Idea

The process of installing a third party package usually has two steps:

  • Building the package using the appropriate version of python for CASA.
  • Making sure the package is in a path visible inside CASA so that it can be imported.

The biggest complication is that CASA ships with its own version of python. In order to use a package inside CASA you need to be sure to build that package using this version of python, which may differ from other versions on your system. Another complication, which is usually more minor, is that (depending on permissions) you may need to add the locale of the built package to your path inside CASA.

Calling setup.py

Generally, once you identify the correct python version to use, you will

  • Download the buildable version of package in question (usually a ".tar.gz" file).
  • Untar and unzip it into a working directory and change directory to that directory.

In that directory you will see a script called "setup.py". You will build the package by calling a variation on

# In the shell
python setup.py install

from the shell.

Generally, an option to install will let you direct where the built package goes. By default it will go to python's "site-package" directory, which is the default home for third party packages. CASA has its own version of this (where it lives exactly depends on your installation), but you may or may not actually have access to this. Try

# In the shell
python setup.py install --help

To see your options. We'll discuss the specific cases of Mac and Linux next.

Your init.py File

Once you have a built version of a package, you need to be sure that CASA can see it. The easiest way to this is to modify your "init.py" file. This is a file that CASA executes every time it starts. It lives in your home directory in a subdirectory called .casa/ so that on a typical system you can see it by

# In the shell
cat ~/.casa/init.py

Now you may not necessarily have an init.py. If you don't, that's no problem you can just create one. If you haven't done so before you just make a text file and include

# python
print "This is my initialization file in ~/.casa/init.py"

Every time CASA starts you should see this line.

Key to installing third party packages, inside this init.py you can add lines like this:

# python
sys.path.append("/users/somebody/somedirectory/")

Then that directory will be in the path that casapy searches when trying to import something. If you don't build to a directory that is automatically in the path, you will need to add a line like this that points to where the built package lives.

How to Tell It Worked

Once you have installed the package, you can tell that it worked if you can import it successfully. For example, if you are installing the AstroPy package try

# python
import astropy

If it works without an error, odds are that you are in good shape.

On the Macintosh

You are in huge luck. Most astronomers can administrate their own Mac and the location of CASA is standardized from Mac to Mac. This allowed Tom Robitaille to write a simple script that makes installing third party packages extremely simple. The script lives here:

https://github.com/astrofrog/casa-python

along with clear, simple instructions! In short, grab this script, make it executable, and use it (casa-python) in place of python when calling setup.py.

# In the shell
casa-python setup.py install

As noted in the docs, this script assumes that CASA is still using Python 2.6. That will need to be upgraded if and when casapy steps forward in version.

Complications

You knew it wasn't always going to be that easy. Some of these packages, specifically the ultra-important SciPy, require a fortran compiler and Mac OS doesn't (at least at the moment) ship with one. In this case you will need to go grab a fortran compiler and install it on your Mac. One good option here is gfortran and you can find instructions to install it by googling gfortran and Mac.

(We'll try to expand this section with more specific information as time goes along.)

On Linux

Linux - especially Linux on a system administered by someone other than you - is more complicated. You will run into two obstacles:

  • paths - paths for CASA are less standard, so you will need to do a little detective work to identify the CASA python.
  • permissions - you may not have permission to write to the site-packages directory on a system with a shared installation of CASA. In this case you will need to install to somewhere else.

How to Identify Your CASA Python

Your first step on Linux is to figure out CASA's python executable. There's a very easy trick for this:

  • Start CASA by running casapy.
  • From the CASA prompt execute the shell command "!which python" to see the executable version of python being used by CASA:
# In casapy
!which python

Recall that the exclamation point executes a shell command from inside the casapy shell.

In fact, we can take this one step further. From inside the casapy shell in the directory where you untarred the package to install you can run

# In casapy
!python setup.py install

To build the package (you will usually have the option to append "--dry-run" to test this out). On systems not administered by you you will often encounter some variety of permission error. For example at NRAO we get "[Errno 30] Read-only file system", indicating that the site-packages directory for our CASA install is read-only.

Installing Packages to the Location of Your Choice

We need to build the package in question somewhere else. Easy enough. "setup.py" has an option to specify a destination other than the default site-packages directory. You can specify the target for your build via

# In casapy
!python setup.py install --root some_other_directory

Where "some_other_directory" refers to an alternate root directory (pick one where you have write permissions!). You'll get a whole crazy subdirectory structure that you own coming out of that root directory.

Once this runs successfully you have one more step to go. Go edit your ~/.casa/init.py file and add the relevant directory to your path. For example:

# In your init.py
sys.path.append("/users/aleroy/akl_python/third-party/usr/lib64/casapy/lib/python2.6/site-packages/")

Packages Already Distributed With CASA

The approach above works well for installing packages that aren't already included with CASA. Things are more complicated when you want to install a new version of one of the included packages:

  • NumPy
  • SciPy
  • matplotlib (pylab)

Upgrading these packages can be complicated and almost intractable if you don't have permission to edit the site-packages directory. Even when permission are not an issue, upgrading these packages can create errors. Matplotlib, in particular, is used heavily by CASA and checked when CASA starts. Always, always, always be ready to roll back to clean install of CASA in case your upgrade doesn't work.

Probably the best way to upgrade these is to install your own version of CASA so that you can write to the site-packages directory. Even doing so, you may encounter issues where different versions of the same package persist (based on anecdotal evidence, this so far seems like a bigger problem on Linux than Mac). To remedy this, you may want to take the dangerous but helpful step of deleting the old versions of the packages from CASA's site packages directory before installing your new ones.

On NRAO Charlottesville's network this was the sequence for a normal user to get an upgraded set of NumPy in CASA:

  • Download the compiled CASA from here.
  • Set it up so that this is your version of CASA (mostly make sure it is in your path).
  • Figure out the site-packages directory for this version. For the 4.2 stable we checked it was casapy/lib64/python2.6/site-packages/
  • Go into that directory and remove numpy. (This is dangerous!)
  • Get the new version of numpy and untar it.
  • Identify CASA's python executable and use it to run Numpy's setup.py install.

This works - it wipes the old version of Numpy, which otherwise hangs around - and replaces it.

(Note that none of this appears to be necessary on a Mac.)

Scipy coming soon... still crashing.