PythonBasics

Back to the PythonOverview.

Preface

This guide is intended as a practical introduction to python for astronomers using CASA. Along with the accompanying pages, we hope that this introduction gets you comfortable enough to take advantage of CASA's python shell to script your data reduction or integrate your analysis, observation planning, and reduction. Despite the CASA focus, we do aim to make this guide useful to any astronomer looking for a first dive into python. Given that similar material exists across the web we will not attempt a comprehensive introduction, just a tour of most of the basic astronomy-relevant python functionality.

After completing this guide, you can find more detailed discussion of other topics and some links to external material here: PythonOverview

Environment

We could write a lot on setting up python, a nice shell, installing key packages, etc. Because this is a CASA guide, we can short circuit some of this. CASA installs with it's own nice iPython shell and a core set of third-party packages (numpy, scipy, pylab, matplotlib). We will assume that you have downloaded and installed CASA from here and return to the issue of additional third-party software down the road.

Once you have CASA installed you can start a new session from your shell prompt by just typing

casapy

Depending on your computing environment you likely also have python and perhaps iPython installed, but the versions of these (especially the pre-installed version on the Mac) can vary widely. If you are interested in a non-CASA distribution that folds in a similar (in fact more extensive) suite of packages you might look at the free academic version here (no promises). We focus our discussion on the version of python that comes with CASA (2.6) and the examples will assume that you are working inside of a casapy shell.

Pasting Code

We will fairly rapidly reach the point where we want to cut and paste somewhat complex blocks of code. Python's use of indentation to implement code structure can make for awkward interactions with the shell at times. Even when it goes well after cutting and pasting a 'for' loop you often have to press return twice to execute. In very complex cases, you may end up needing to paste a line at a time.

Fortunately, CASA's iPython shell offers a very nice way around these challenges. You can type

cpaste

to initiate a 'code paste'. You will then be able to paste code directly into the shell and the code will appear exactly as you copied it. When you have pasted all of your code got to a new line and type "--". This ends the code paste and the pasted code should then execute. Use this (really!), it will save you a large amount of pasting-frustration.

Getting Help, Exploring Objects, the Shell

CASA's shell (and iPython shells in general) has a lot of useful functionality. For example, CASA knows some basic operating system commands like "ls", "cat", etc. CASA only knows a few of these but you can issue any commands to your Unix/Linux shell directly by prefacing those commands with an exclamation point. So this works:

ls

and so does this:

!ls -lh

this does not (because CASA does not know "df"):

df -h

but this does:

!df -h

The shell has many other nice features. For example, if you type the name of a variable, it will simply print the variable value.

You can readily get help on most objects via "help thisorthat," for example

help list

Along with "help," the other major exploration capability in the shell is tab-completion. This will let you explore what variables are defined in your shell. Even better, you can take any object, append a ".", hit tab, and the shell will show you list of associated methods.

Try this now by typing "list." and hitting tab. The shell will print all the things that a list can do. In addition to a bunch of system-defined names that begin and end with "__" (e.g., see the equality operator "__eq__") you can see that list has a set of methods "append", "count", "extend", "index", ... "sort".

If you saw that list.append exists and wanted to learn a bit about it you could type

help list.append

We're getting a bit ahead of ourselves but try to bear these exploratory capabilities as you look through the guide. They are a great way to poke around and find new functionality.

One very important thing to bear in mind as you begin more advanced programming is that the shell specific capabilities are not available outside the shell. In practice this means that you cannot, for example, print the name of a variable by just listing the variable inside a program.

Simple Variables

Let's begin by creating an manipulating some basic variables. At the python shell prompt create your first variable by typing

my_first_var = 1

That's it, you've created your first variable. Note that you didn't have to specify the data type, python works this out from context. To see the value of your variable use the print function.

print (my_first_var)

or (if and only if you are at the shell) you can see the value by just typing the variable name

my_first_var

(read: use 'print()' when programming).

You can see the type of the the variable by typing

print( type(my_first_var) )

Now try the same thing with some other data types.

# Make and print a string
a_string = "A String"
print( a_string, type(a_string) ) # a string

Note that this first line does nothing. Python's comment string is "#" and it ignores everything in a line that comes after this character.

Some similar examples for floating point numbers:

type(3.14159) # a float
a_float = 1e-10
print( type(a_float) )  # scientific notation

Notice the in-line comment and the on-the-fly creation of a variable in the first line.

Finally, python has a 'boolean' type that is either 'True' or 'False' (shorthand 'T' or 'F' in CASA, though not iPython in general).

type(True) # a boolean
boo = False
print( boo, type(boo) )

You can chain variable creation:

x = y = 1
print( x, y )
x = y = v = u = t = 0
print( u, t )

Basic Math

We have seen how to create variables. Now we will manipulate them with basic math operations.

Begin by creating a new variable with value 1.

x=1

and print the value after we add 2.

print ( x+2 )

We can assign the output of the operation to a new variable, which now exists in addition to x

y = x + 2
print (y)

z = x - 3
print (z)

Implicit operations work, e.g., "x += 3" will add three to the value of x

x = 1
print (x)
x += 3
print (x)
x *= 10
print (x)
x /= 20
print (x)

Exponents are done with "**", so for example

x = 2
y = x**2
print (x, y)

or to take the square root

x = 4
print (x**0.5)

Ints and Floats

We saw that python assigns a new variable the appropriate type based on context. Overall this is a nice feature, it lets us quickly develop scripts and programs without a lot of traditional programming overhead. However, there are a few rules that we need to keep in mind.

The first point to see is that dividing two integers returns an integer that is the "floor" (value rounded to the nearest integer closer to zero) of the result. For example, declare a new integer x and divide it by 20:

x = 1
print( x / 20 )

or try these

print( 19 / 20 )
print( -1 / 20 )
print( -19 / 20 )

That's the division of two integers. If we divide a float and an integer, on the other hand, we get a float:

(In Python 3 this changes, and the new default is to return a float. If you really want to return a truncated integer, use the int function explicitly, so your code will be forward compatible.)

x = 1
print ( type(x) )
print ( x / 1.5 )
print ( type(x/1.5) )

the operation implicitly casts the result as a float.

Explicit casts are okay and a reasonable way around this. Again, int() floors. For example:

x = 0.6
int_version = int(x)
int_version # remember this prints in the shell

Similarly, float() casts as a float:

float_version = float(int_version)
float_version

The order does matter. Notice the difference between these two cases:

print ( float(19)/20 )
print ( float(19/20) )

The takeaway for the average CASA user here is: be careful tossing integers around in math because things may unexpectedly round on you. When in doubt, you should probably make things floats.

Booleans

Python includes booleans as a basic data type. These are variables that have two possible values: True or False (for CASA the shorthands "T" and "F" work as well). You can declare booleans efficiently:

b = True  # note this is case sensitive TRUE or true does not work.
print(b)
b = False
print(b)

Booleans are particularly interesting as the output of many operators. For example:

b = 1 == 1
print(b)

or

c = 1 != 1
print(c)

Deleting Variables

Python variables at the shell level will hang around forever until you declare a variable with the same name or actively get rid of them (by contrast variables declared inside of a procedure will go away when the procedure ends). You can get rid of a variable (module, object, whatever) like so:

x = 1
del x
print (x)

and just like that x is gone. Be careful with this.

Checking Whether a Variable Exists

We just gave a practical example of using "print" to check variable existence. But this sort of catastrophic approach isn't ideal. A more robust way to check if a variable exists is:

try:
    the_answer
except NameError:
    print( "Does not exist!" )

The try statement accepts an "else":

the_answer = 42.
try:
    the_answer
except NameError:
    print( "Does not exist!" )
else:
    print( "We're cool." )

Now try the same thing after deleting your variable:

del the_answer
try:
    the_answer
except NameError:
    print( "Does not exist!" )
else:
    print( "We're cool." )

This kind of approach gives you the ability to write non-explosive scripts. There are other ways to approach this. If you are interested you may want to read up further on the locals and globals python dictionaries.

Data Collections: Lists

Python has several built in types of data collections. Here we'll look at "lists", which are loose collections of potentially mixed types of data. You can declare a list by placing a comma-delimited list of variables inside a set of square brackets. For example, let's create a list called "bands":

bands = ['L', 'S', 'C', 'X', 12]

One potentially surprising feature of python lists is that they freely mix and match data types. Notice that "bands" mixes strings and integers. You can print the type and number of elements of the list via

print( type(bands) )
print( len(bands) )

Simple Slicing

You "slice" a list, accessing parts of the data using square brackets:

bands = ['L', 'S', 'C', 'X', 12]
print( bands[0] )
print( bands[4] )

Notice that python is "zero-indexed" which means that for a 5-element list like "bands" the indices run from 0 (first) to 4 (last). As a result this will not work

print( bands[5] )

because there is no element 5.

Also notice that python allows you to use negative indices to index "backwards" from the last element of the array

print( bands[-1] )
print( bands[-2] )

You can pick out a range of indices using a ":" and the following syntax

print( bands[1:3] )

Notice that it prints elements 1 and 2 but not element 3 of the list. So "n:m" retrieves elements n through m-1.

Similarly "0:3" will retrieve elements 0, 1, and 2. For example

print( bands[1:3] )

A free ":" retrieves everything along the axis:

print( bands[:] )

while ":n" implicitly takes "0" as the initial index and "n:" runs n through the last index. For example:

print( bands[2:] )

Assignment

These same tricks can be used for assignment. We'll recreate bands and then replace the last element:

bands = ['L', 'S', 'C', 'X', 12]
bands
bands[4] = "Ku"
print( bands )

or use the more complex slicing to replace the first three elements:

bands[0:3] = [1.4,3.0,6.0]

Notice that the sizes of the left and right side of the assignment need to match. So this will not work

bands[0:3] = 3.0

In addition to replacing elements you can add to the list like so:

bands.append("Ka")

Exploring List Functionality (Back to the Shell)

Lists have several neat abilities in addition to "append." If you don't remember them offhand you can easily find them using the shell functionality discussed earlier. This is great opportunity to apply these techniques in practice. Do so now, making sure that "bands" is defined as above.

Type in "bands." and hit <tab>. the shell will offer you the the available auto-completions. Leave aside all those system defined __ items and look at the rest. We see a list from "append," "count," through "sort." These are methods of list and so will be called as "append" above.

Let's say that you want to find out more about the "count" function. Use the "help" function like so:

help(bands.count)

so bands.count() checks the number of occurrences of a value. Try it out:

bands = ['L', 'S', 'C', 'X', 'Ka']
bands.count("Ka")

Let's play a bit with the other functions. "Pop" returns the most recent addition (last element) in the list and simultaneously removes it:

bands = ['L', 'S', 'C', 'X', 'Ka']
print( bands )
print( bands.pop() )
print( bands )

Another function, "index" returns the index at which the specified element occurs for the first time in the list:

bands = ['L', 'S', 'C', 'X', 'Ka']
bands.index("Ka")

though it gets unhappy when asked to find an item not in the list

bands.index("Q")

"insert" places an item before the specified location. So we can for example combine "insert" and "index" to place "K" in the list before "Ka":

bands = ['L', 'S', 'C', 'X', 'Ka']
bands.insert(bands.index("Ka"),"K")
print(bands)

Now is a good time to explore the other functionality found here: removal, sorting, reversing, and so on.

A First Look at Iteration

We'll see much more on iteration in a moment, but one of the most important aspects of a list is that you can easily iterate over it. This simple syntax will get each element in bands, in order of their syntax, print its value, and tell you what type it is.

for element in bands:
    print( element, type(element) )

one of python's quirks is that looping occurs via iteration across collections like lists.

Unpacking

Lists and other collections like tuples or sets have the ability to "unpack." This means that they can be compactly assigned to a series of variables like so:

bands = ["X","Y","Z"]
a, b, c = bands
print(a)
print(b)
print(c)

Data Collections: Dictionaries

Dictionaries are another basic python data collection. Dictionaries store data in pairs, with a set of unique keys each mapped to some value (the values do not have to be unique). Dictionaries are extremely useful in scripting. For example, imagine that we are reducing a complex set of observations and want to be able to look up a the calibrator associated with a given source.

You can define a dictionary using the curly braces {} and "key"-"value" pairs separated by ":". For example

my_dict = {"a key":10,
           "b key":20}

You can then extract the value for a given key in a way similar to how we accessed lists. Index the dictionary with the key in square braces [].

print( my_dict["a key"] )

Dictionary keys can be numbers, strings, or tuples of either (this is a place where tuples are useful compared to lists or sets - you cannot have a list as a key for a dictionary). So you could, for example, define a dictionary holding source coordinates and fluxes using tuples (defined in parentheses, e.g., as "(1,2)").

catalog = {(1,2):10,
           (4,5):50,
           (10,10):10}
print( catalog[(1,2)] )

You can mix and match key types and use any data type for the value. For example:

my_dict = {1:5,
           "b":[1,2,3],
           (1,2):"string"}
my_dict[1]
my_dict["b"]
my_dict[(1,2)]

For scripting inteferometry reductions one useful application might be to pair sources and calibrators like so:

phasecal = {"source_1":"phasecal_1",
            "source_2":"phasecal_2",
            "source_3":"phasecal_1"}

Then during the reduction, one can simply call:

source_name = "source_1"
phasecal[source_name]

Manipulating Dictionaries

As with lists, dictionaries have significant associated functionality. Use tab completion and "help" to explore the dictionary functionality now.

Among other tricks, you can append new elements to a dictionary like so:

my_dict = {"a":1,
           "b":2,
           "c":3}
my_dict["d"] = 4

You can check if it has a given key like so (very useful for making sure you don't index with an invalid key):

print( my_dict.has_key("a"), my_dict.has_key("e") )

Or list the keys like so:

print( my_dict.keys() )

You can iterate over dictionaries like so:

for key in my_dict.keys():
    print( key )

Or iterate over pairs of keys and values using the "iteritems" functionality:

for key, value in my_dict.iteritems():
    print( key, value )

Dictionaries of Dictionaries

Dictionaries can also be values in dictionaries, making 2 dimensional dictionaries. For example, we could store the white pieces on a chess board in a 2D dictionary of files and ranks as:

files = ['a','b','c','d','e','f','g','h'] 
back_row = {'a':'Rook','b':'Knight','c':'Bishop','d':'Queen', \
    'e':'King','f':'Bishop','g':'Knight','h':'Rook'}
white = {}  #  sets to a 1D dictionary
for file in files:
    white[file] = {}   #  sets to a dictionary of dictionaries
    white[file][2] = 'Pawn'
    white[file][1] = back_row[file]

Now you can see what white piece is on a particular square simply by:

print( white['d'][1] )

There are more aspects to explore, but this gives a good introduction to the large functionality of dictionaries.

Other Types of Data Collections

You may also want to read up on sets (collections of unique elements) and tuples (immutable collections).

Sets

We won't discuss these at length, but "sets" are another useful python data collection. They contain unique collections of data:

my_set = set([1,2,3,3,3,3,4,4,4,4])
print( my_set )

sets have a large set of interesting functionality. Explore and read away.

Tuples

Tuples are collections of data defined using parentheses (). They have the key characteristic of being immutable, which means that they can be used in many places that other data collections don't work (e.g., to index numpy arrays).

test = (1,2)
print( test[0] )
print( test[1] )

but notice that this doesn't work

test[0] = 1

Read more here.

Control Flow

Control flow via if, for, and while-type statements is key to any programming. Python allows these approaches and offers some slick syntax to iterate in clever ways. It also allows very ready access to this functionality inside the interpreter, which is fantastic for rapid data analysis. Here we'll just look at the basics.

If

If is our most basic control statement. You can fork on a truth statement. Maybe the biggest thing to notice here is python's unconventional lack of curly braces, begin/ends, or other delimiters. Everything is done through the : and indentation. And it works on the command line, too. For example,

x = 55
if x < 100:
    print( "Not enough beer bottles." )
elif x > 100:
    print( "Too many beer bottles." )
else:
    print( "Just enough beer bottles." )

Try setting x to different values. If you aren't using "cpaste" notice that you may need an extra <enter> at the end of the paste to the shell.

While

While checks the truth statement and runs the commands inside the loop as long as it is true. This means that it can be pretty easy to get runaway loops, so get your CTRL+C fingers ready.

x = 55
while x < 100:
    print( "Too few bottles of beer. Filling one up." )
    x += 1
    print(x)

print(x)

Now let's write a loop that could get us in trouble. This would normally run away and we would need to hit CTRL+C to break out.

x = 55
while x < 100:
    print("Too few bottles of beer. Pouring out a bottle of beer.")
    x -= 1
    if x <= 0:
        print("Out of beer. Stopping.")
        break
    print("Still have beer left")

But we've built in a safeguard. The break statement breaks us out of our current loop. It's less destructive twin continue moves us to the next iteration of the innermost current loop.

The other key thing to notice above is the nested indentation. See how the if statement is set off by the : and indentation. But the print function below is back in line with the while and so will run every time.

For

For in python differs from its implementation in other languages in that its always iterating over some other sequence of data. This allows for some slick tricks but also takes a bit of getting used to.

Your basic for statement involves creating a list and looping over it (later you may want to replace this with a numpy array).

You can replicate the basic looping over sequential values using the "range" function. The following syntax will loop from 0 to 9 stepping by 1:

loop_over = range(10)
for i in loop_over:
    print(i)

The "range" function takes several parameters. For instance, try:

print( range(2,5) )
print( range(100,0,-30) )

There's no need to focus on integers when looping. Python can loop over any collection:

beatles = ['paul','john','george','ringo']
for musician in beatles:
    print musician
    if musician == 'ringo':
        print( "... oooh "+musician+" was the best." )

We could have obtained similar functionality by using "range" and the length of the list:

beatles = ['paul','john','george','ringo']
for i in range(len(beatles)):
    print( beatles[i] )

It's also possible to iterate in-line, for example to create lists. For example:

a = [x**2 for x in range(10)]
print(a)

or

b = [[x,x**2,x**3] for x in range(10)]
print(b)

Don't get too hung up on this syntax.

There is not a proper case statement in python though there are plenty of workarounds.

Continue, Break, and Pass

There are several additional bits of functionality that are useful for steering control flow in a program. Calling continue during an iteration will end the current iteration and move to the next one. For example

for i in range(11):
    if (i == 5):
        continue
    print(i)

break completely ends the innermost ongoing iteration. For example breaking the same loop yields

for i in range(11):
    if (i == 5):
        break
    print(i)

Finally, pass puts a do-nothing line of python in place, which for example lets you build an empty loop.

for i in range(10):
    pass

More Complex Programs

Executing Scripts

The most basic way to execute a set of python commands (aside from just copying and pasting to the shell) is to the execfile command. Calling execfile('myscript.py') from inside a python shell will execute 'myscript.py' one line at a time. You can use this to run a series of reduction commands or other simple scripts. In fact, calling execfile on one or more scripts will almost certainly be sufficient to script most basic CASA data reductions.

Try this now, download this script

• Download script: File:Intro script.py

and then in the same directory where you put the script type

execfile('intro_script.py')

The script should execute as through you had typed each line at the shell. Alternatively, you can get exactly the same results in the Ipython shell with:

run -i intro_script.py

The execfile function is nice if you want to run scripts from scripts. On the other hand, run gives you control of the namespace, which is helpful when working interactively.

You can combine the control flow that we learned above (if, for, while) with execfile to refine your scripts. For example, you might have a sophisticated reduction path that requires a few user inputs, which could be collected at the top of the script as variables. The reduction might have several discrete parts, which you could turn on or off using booleans and if statements.

As an example, download and examine the following file:

• Download script: File:Intro script 2.py

As you edit the variables and booleans inside the script then rerun it with execfile, various parts of the script will run tuned by the variables you set.

This simple but powerful approach can (if you desire) form the infrastructure for a lot of your CASA reduction scripting. Try playing around with these scripts now.

Functions

Python allows you to define functions either from the command line (or an execfile call) or as part of modules.

To define a function use def with the following syntax.

def first_function():
    """Prints Hello World and quits"""
    print "Hello World."

Now it's defined. You can call it via:

first_function()

And you can find out about it:

help(first_function)

or

first_function?

A function can return values like so:

def second_function():
    """Return the string 'Hello World'"""
    return "Hello World."

value = second_function()
value

It can also accept inputs:

def third_function(inp):
    """Just like the print function"""
    print(inp)

third_function("Hello World")

Your definition can set defaults for these inputs:

def fourth_function(optional="Shark Week!",optional2="Exciting!"):
    """Prints Shark Week Exciting unless told otherwise"""
    print optional
    print optional2

If you supply input this will override the default

fourth_function("Hello World")

If not then the default takes hold

fourth_function()

You had yet another option here too. What if we only wanted to specify the second parameter? We can do that with a keyword call. Specify the name of the input parameter in the call to the function.

fourth_function(optional2="Is just shameful.")

Note, that your keyword calls must come after your unlabeled input. These are legal:

fourth_function("Zebra week.", optional2="Is just shameful.")
fourth_function(optional="Giraffe week.", optional2="Is just shameful.")

This is not:

fourth_function(optional="Giraffe week.", "Is just awesome.")

Functions of Functions

You can also pass functions to functions. For example, imagine you define a cubic function as follows:

def cubic(x, A=1.0, B=0.0, C=0.0, D=0.0):
    """Returns A*x**3 + B*x**2 + C*x + D."""
    return A*x**3 + B*x**2 + C*x + D

And the derivative function as:

def slope(f, x, h=0.00001):
    """Takes the derivative at a point."""
    return (f(x+h) - f(x-h))/(2*h)

You can find the derivative of [math]\displaystyle{ x^3 }[/math] at [math]\displaystyle{ x=2 }[/math] by calling:

slope(cubic,2.0)

There is also a one-line function assignment called the lambda function. For example, one can simply define a function on one line as:

square = lambda x: x**2 
print(square(5))

This is handy in that it can be passed to another function. For example, what is the derivative of [math]\displaystyle{ 5x^3 + 3x^2 + 2x + 3 }[/math] evaluated at [math]\displaystyle{ x=7 }[/math]?

slope(lambda x: cubic(x,A=5.0,B=3.0,C=2.0,D=3.0), 7.0)

Unpacking again

Finally, you might want to pass out several values, for example a set of coordinates. One easy way to do this is:

def location():
    """Returns a location."""
    x_loc = 10
    y_loc = 20
    z_loc = 30
    return (x_loc,y_loc,z_loc)

and then the function can be called like so:

x, y, z = location()
print(x)
print(y)
print(z)

This is another application of the "unpacking" mentioned as we introduced lists.

Modules

Modules are collections of functions, variables, and objects that can be imported into a python session to add functionality. Python has a number of native modules, for example "os", "math", and "time". Third parties supply many more, including "numpy" for robust array handling, "scipy" for scientific analysis, and "matplotlib" for plotting. You can also easily write your own new modules in python.

To access a module in a python session or program you need to "import" it. For example, to import the math module type

import math

Then type "math." and try tab completion to see the math functionality. For example you can take 5 to the 3rd power via:

print( math.pow(5,3) )

When you import you can change the name of the module like so

import math as m
m.sqrt(4)

or you can import only specific functionality. For example if you wanted only the pow function you could import it like so:

from math import pow
print( pow(10,2) )

you can import all functionality from a module via

from math import *

There are a few important subtleties to know here:

• First, one reason that modules natively maintain their own namespace is to avoid variable and function names stepping on each other. For example, both math and numpy have a "sin" function. Having these as "numpy.sin" and "math.sin" keeps them distinct and clear.

• A CASA-specific issue: many of the tools have two-letter names so do be careful importing things as two-letter names - you can easily lose the CASA tools.

Writing Your Own Modules

Modules are easy. To define them you just need to create a file in your path with a .py extension. Here we create "intro_module.py". We'll define a few variables and functions.

First, you can define variables in your module:

a = 10.0
b = 15.0
c = 20.0

These definitions or any other executable code placed in the main body of the module will be executed the first time that the module is imported.

And you can define functions. You can use the variables you define in the module, if you specify them with the global statement. For example:

def square(x):
    """
    This takes a quadratic in form a*x**2 + b*x + c, with a,b,c
    defined in the module.
    """
    global a, b, c
    return a*x**2 + b*x + c

And finally, you import your modules into your programs using the standard import commands, such as:

import intro_module as intro
print(intro.a, intro.b, intro.c)
print( intro.square(2) )

If you change your module, you cannot import it again. If a module is imported that it already knows, it will ignore the command. Rather use the reload function:

reload( intro )

and note that you cannot reload material that you have imported via a "from X import Y" statement. Though a reasonable workaround is to delete and re-import. This means that as you develop your own modules and test them in the shell you almost certainly want to use the "import X" syntax.

The main use of modules will be to store reusable code or organize complex projects into discrete files. Read more on modules.

Future sections of the python basics touch on specific modules, either basic python or third party, useful for analysis and scripting.