PythonDataAccess

Back to the PythonOverview.

Preface

In addition to manipulating your data, you need some way to save and access it. Here we look at saving and loading files. We'll have a look at how to save and load ASCII data from disk, then how to quickly save and load more complex collections using pickle, and access to astronomical FITS data (or at least CASA images) via CASA. We'll begin by looking at how to get input from the user.

Input

Input can be accepted from the command line (or a script paused) using the raw_input command. Use this to query the user or to put a pause inside a script. "raw_input" returns the user input as its output:

verb = raw_input("Give me a verb: ")
noun = raw_input("Give me a noun: ")

and you can then use these as you would any other string variable:

mad_lib = "More fun than "+verb+"ing a "+noun
print mad_lib

ASCII Files in Basic Python

Python provides easy basic file access. Grab our File:Example file.txt for the following example.

Python lets you open a file with the open command, like so:

a_file = open("example_file.txt", "r")

The second parameter determines how you will access the file. r means read, w means write, and a means append. You can both read and write at once if you want. Read up here for more.

Now that it's open we can read the lines in the file into a list like so:

lines = a_file.readlines()
print lines

We could also have read a single line with readline() or only a fixed set of bytes with read().

After we've written or read our data, we will want to close the file. Do this with the .close() method like so.

a_file.close()

We write using similar syntax. Here we open a new file for writing, write out the list of lines, and then close the new file.

a_new_file = open("new_file.txt", "w")
a_new_file.writelines(lines)
a_new_file.close()

The file now exists. Pull in the native os module and use it to list the contents of the file:

import os
os.system('cat new_file.txt')

(Note that readlines and writelines want ASCII strings from you. You need to convert floats and integers to strings before writing.)

Pickling

It's possible to directly save and load variables from a file (without making them into strings and worrying about parsing).

import pickle

Make a dictionary

a_dict = {"field1":100,
          50:[1,2,3,5],
          3.14:"hello"}

Save the dictionary

f = open("pickle.jar","w")
p = pickle.Pickler(f)
p.dump(a_dict)
f.close()

Go ahead and have a look at what it's doing.

import os
os.system("cat pickle.jar")

ascii but not english.

Get the stuff back

f = open("pickle.jar","r")
u = pickle.Unpickler(f)
read_back = u.load()
f.close()

print a_dict
print read_back

There's also a more compact syntax to just load and dump directly from a file. Options allow binary instead of ascii writing. And there's a faster version called cPickle.

Pickle is stack-based by the way, so:

a = 1
b = 2
c = 3

Save the dictionary

f = open("another_pickle.jar","w")
p = pickle.Pickler(f)
p.dump(a)
p.dump(b)
p.dump(c)
f.close()

Get the stuff back

f = open("another_pickle.jar","r")
u = pickle.Unpickler(f)
var1 = u.load()
var2 = u.load()
var3 = u.load()

... a variable too far:

var4 = u.load()

uhoh!

f.close()

print var1, var2, var3

Of course the disadvantage of pickling is that you need to unpickle it. This is not a generic format to save data and share with other people.

Other Approaches

You don't need to waste a lot of effort duplicating previous work on reading and writing text files. Adam Ginsburg's "readcol.py" (loosely patterned after the IDL version, linked from the page) will save you a lot of effort. The package astroasciidata also looks promising but I have not yet gotten a chance to experiment with it.