PythonDataAccess

Back to the PythonOverview.

Preface

In addition to manipulating your data, you need some way to save and access it. Here we look at saving and loading files. We'll have a look at how to save and load ASCII data from disk, then how to quickly save and load more complex collections using pickle, and access to astronomical FITS data (or at least CASA images) via CASA. We'll begin by looking at how to get input from the user.

Input

Input can be accepted from the command line (or a script paused) using the raw_input command. Use this to query the user or to put a pause inside a script. "raw_input" returns the user input as its output:

verb = raw_input("Give me a verb: ")
noun = raw_input("Give me a noun: ")

and you can then use these as you would any other string variable:

mad_lib = "More fun than "+verb+"ing a "+noun
print mad_lib

Then inserting a pause in a script is as easy as:

dummy = raw_input("Hit <Enter> to continue.")

ASCII Files in Basic Python

Python provides easy basic file access. Grab our File:Example file.txt for the following example.

Python lets you open a file with the open command, like so:

a_file = open("example_file.txt", "r")

The second parameter determines how you will access the file. r means read, w means write, and a means append. You can both read and write at once if you want. Read up here for more.

Now that it's open we can read the lines in the file into a list like so:

lines = a_file.readlines()
print lines

We could also have read a single line with readline() or only a fixed set of bytes with read().

After we've written or read our data, we will want to close the file. Do this with the .close() method like so.

a_file.close()

We write using similar syntax. Here we open a new file for writing, write out the list of lines, and then close the new file.

a_new_file = open("new_file.txt", "w")
a_new_file.writelines(lines)
a_new_file.close()

The file now exists. Pull in the native os module and use it to list the contents of the file:

import os
os.system('cat new_file.txt')

(Note that readlines and writelines want ASCII strings from you. You need to convert floats and integers to strings before writing.)

Pickling

It's possible to directly save and load variables from a file without making them into strings or worrying about parsing lines into discrete variables. To do this, we use python's built-in serializer pickle. First import the pickle module:

import pickle

Now make a dictionary, which we will shortly save:

a_dict = {"field1":100,
          50:[1,2,3,5],
          3.14:"hello"}

To save the dictionary using pickle you open a file for output, initialize a "pickler" pointing at the file, and dump the dictionary to the pickler.

f = open("pickle_jar.pkl","w")
p = pickle.Pickler(f)
p.dump(a_dict)
f.close()

Go ahead and have a look at what it's doing by just listing the file.

import os
os.system("cat pickle_jar.pkl")

the file is ASCII, but it's not English.

To get back the things that you just pickled, we reverse the previous sequence. We use an Unpickler in place of the original Pickler and open the file for reading instead of writing. The result looks like this:

f = open("pickle_jar.pkl","r")
u = pickle.Unpickler(f)
read_back = u.load()
f.close()

So read_back is a variable containing the output of the unpickler. Have a look, it should be identical to the input dictionary:

print a_dict
print read_back

(There's also a more compact syntax to just load and dump directly from a file. Options allow binary instead of ASCII writing. And there's a faster version called cPickle.)

What if we wanted to store many variables? Pickle gives you back the variables in the order that you put them in, so that if you have three simple variables:

a = 1
b = 2
c = 3

and save them using pickle:

f = open("another_pickle_jar.pkl","w")
p = pickle.Pickler(f)
p.dump(a)
p.dump(b)
p.dump(c)
f.close()

then you can pull them back out like so:

f = open("another_pickle_jar.pkl","r")
u = pickle.Unpickler(f)
var1 = u.load()
var2 = u.load()
var3 = u.load()
print a, b, c
print var1, var2, var3

But you can only pull out as many variables as you put in. If we go a variable too far:

var4 = u.load()

Then we run into trouble.

f.close()

Of course the disadvantage of pickling is that you need to unpickle it. This is not a generic format to save data and share with other people.

Other Approaches

You don't need to waste a lot of effort duplicating previous work on reading and writing text files. Adam Ginsburg's "readcol.py" (loosely patterned after the IDL version, linked from the page) will save you a lot of effort. The package astroasciidata also looks promising but I have not yet gotten a chance to experiment with it.