Download historical financial data from Yahoo! using Python

Update! I’ve extended and polished the library and published it as pyyahoofinance at github. Feel free to critizise my coding style and laugh at me, and of course to use it and/or contribute.

I’m working on a project where I need to analyze the ups and downs of different stock values along the history and find correlations between the contraction and expansion of the market, the evolution of the indexes and simulate certain trading strategies to become rich and famous. ;)

Yahoo! Finance lets you download historical financial data in CSV, but only provides data for one stock at a time. Clicking, saving, cutting and pasting is always tedious, error-prone and a waste of time in general, so I’ve taken advantage of my good friend Python to write a script to plunder (yeah, I’m a pirate, you know ;) Yahoo!’s Standard & Poor’s 500 index in a couple of minutes and save it in a space-separated-values file named “results.txt”.

You can then generate beautiful charts like this one:

Code follows:


import urllib2

CLOSE_COLUMN = 4 # the index of the column containing the close value
TICKER_COLUMN = 0 # the index of the column containing the ticker name

# get the Standard & Poor stock tickers
tickers = []
for n in range(0, 500, 50):
    url = urllib2.urlopen("".replace('PAGE', str(n)))
    data =
    stocks = data.split('rn')
    for stock in stocks:
            ticker = stock.split(',')[TICKER_COLUMN]
            ticker = ticker.replace('"', '') # remove surrounding quotes
            if ticker: # not empty ticker
        except IndexError: # empty row

global_closes = {}

for ticker in tickers:
    print "getting data from ticker: %s" % ticker
    url = urllib2.urlopen("" % ticker)

    history =

    measures = history.split('n')
    measures = measures[1:-1] # the last row is empty and the first
                              # one contains the labels

    closes = [measure.split(',')[CLOSE_COLUMN] for measure in measures]

    global_closes[ticker] = closes

columns = [[ticker] + global_closes[ticker] for ticker in global_closes.keys()]

rows = zip(*columns)
out = open('results.txt', 'w')
for row in rows:
    out.write(' '.join(row))

Installing Plone 4.0a1 on Debian

I just read on a tweet that the first alpha release of Plone 4 is already available for testing and I wasn’t able to resist the temptation. :P  As some people have already pointed out, Python 2.6 for Debian is only available from the experimental repository, and most of us prefer to stay testing or unstable at most. :)

Thanks god, there’s a buildout recipe (we should create a Linux distro based on buildout someday, alecm ;) to build Python from source in an isolated environment. Steps:

  1. svn co buildout.python
  2. cd buildout.python; python
  3. Edit buildout.cfg to fit your needs. You might want to comment out the references to the Python versions you don’t want to install.
  4. bin/buildout
  5. cd to-another-directory; paster create -t plone3_buildout plone4.
  6. Enter “4.0a1″ (without quotes) when asked about “Which Plone version to install”. Make sure you have the last version of ZopeSkel (2.14.1 while I’m writing this – easy_install -U ZopeSkel) or the generated buildout.cfg won’t be valid for Plone 4 otherwise. Thanks to MatthewWilkes for the pointer and of course for the ZopeSkel release. :)
  7. path-to-buildout.python/python-2.6/bin/python
  8. bin/buildout. If you get tons of lines in your console output about fetching distribute, see this post by Reinout van Rees.
  9. You’ll need to install PIL to run Plone, so do either:
    1. python-2.6/bin/easy_install-2.6, or
    2. Add PIL or PILwoTk to any of the eggs sections of your Plone 4 buildout and re-run bin/buildout.

If you stumble upon an error message similar to:

Processing PIL-1.1.6.tar.gz
Running PIL-1.1.6/ -q bdist_egg --dist-dir /tmp/easy_install-Fev48C/PIL-1.1.6/egg-dist-tmp-SNtCRu
In file included from decode.c:608:
libImaging/Zip.h:11:18: error: zlib.h: No such file or directory
In file included from decode.c:608:
libImaging/Zip.h:37: error: expected specifier-qualifier-list before ‘z_stream’
error: Setup script exited with error: command 'gcc' failed with exit status 1

try “aptitude install zlib1g-dev”. Thanks davisagli!

The previous procedure should work… But it didn’t in my system. :(  I get an error in the buildout.python bin/buildout:

SystemError: ('Failed', 'patch -p0 < /somepath/buildout.python/parts/readline-patch/readline.patch')

I’ve already tweeted fschulze (who is presumably the author of the collective.buildout.python stuff, according to comments in the #plone IRC channel) about this issue. I hope it will get solved soon! :)

In the meanwhile, removing or commenting out the stuff about readline.patch in the collective.buildout.python buildout.cfg before running bin/buildout appears to be a valid workaround, at least for me.

The Charter for Innovation, Creativity and Access to Knowledge has just been published

(I’m republishing here the press announcement from the Culture Forum, please help to spread it!)

A huge international coalition has come together to campaign for respect for the civil rights of citizens and artists in the digital era. Today they are launching internationally the Charter for Innovation, Creativity and Access to Knowledge. This initiative is a response to the pressure of the lobbies of the culture industry on the European Parliament and national parliaments. The Charter is an outcome of the Culture Forum held last week in Barcelona.

The press recently announced the accord reached by the European Commission on the “protection” of Internet access, the so-called Telecommunications Package. The agreeement is ambiguous but on the positive side, it is not granting the full demands of the culture industry lobbies which included the drastic measure of cutting off anyone who interchanged files on the Internet.

The changes added in defense of fundamental rights and in favor of the jurisdictions of each country were the results of the efforts of hundreds of organizations and citizens who pressured the European Parliament members and served to counteract the pressures of the culture industry lobbies.

This case, like many others, shows the need for civil society to organize and make firm demands so that their rights are respected in adapting laws to the new structural advances of the digital era.

More than 100 renowned specialists from 20 different countries participated last week in the Culture Forum of Barcelona and created a huge international coalition to urge respect for the civil rights of citizens and artists in the digital era.

This historic gathering, whose activities included coordinating the response to the final meeting of the commission on the Telecommunications Package, constitutes the beginning of an unprecedented offensive of civil society in defense of the fundamental rights in the digital era. These rights range from the right to freedom of expression to the right of access to culture and knowledge; the defense of a just division of authors’ rights, the inviolability of communications and of privacy and the neutrality of access to the Internet. The Charter sees all these rights as great levers for the transformation of economic, political and social relations.

After three days of intense work, the Charter of the Culture Forum of Barcelona for Innovation, Creativity and Access to Knowledge was produced.

This Charter, which invites citizens to take it as theirs and use it in their requests and demands, will be disseminated worldwide through formal presentations to governments and also through a variety of actions by individuals and organisations.

The Charter will be presented to more than 1000 political institutions and governments, including WIPO, the Obama administration, the European Commission and many national governments. Some of these organizations have already shown an interest in listening to the demands, and two representatives of the European Commission and official observers from the Brazilian Ministry of Culture, among others, were present during the approval of the Charter.

The campaign will make a particular appeal to the Spanish government, which has made the regulation of the digital environment one of the flagship items in its upcoming presidency of the European Union. In 10 days the Charter will be delivered personally to “la infanta Cristina”, the daughter of the king of Spain.

Back from the Free Culture Forum

Back to Stockholm! I’ve spent this whole weekend in Barcelona in the Free Culture Forum, working together with more than a hundred people from all over the world to debate about how to face the new (and old) challenges with copyright, net neutrality and education, among others, and decide the next actions to take.

First of all, I’d like to congratulate and thank everyone who contributed to organize and manage this. Despite the lack of a solid financial and human support, the forum was an undeniable success in terms of organization – I felt like at home and had a real good time.

Our main task was to elaborate a document reflecting our vision, goals and demands. We worked in five groups: Education, Economy, Politics, Digital Infrastructure Rights and Legal. I worked on Education since I was invited to the event in representation of Alqua.

Since the document has been composed with the input of so many people, I’m a bit concerned about its coherence and homogeneity, so I’ll advocate for several different proof-reads before publishing anything, including mine. Also, the idea is that the different individuals and organizations who attended the forum will sign it as a whole, so we have to be very careful with the inclusion of any “too” radical demand in order to reach a reasonably broad acceptance.

One of the atendees suggested that we should speak about the next steps we, as individuals, either representing a project/group or not, will take to achieve the goals stated in our chart. I think this is rather important since otherwise (and even if we do so) we’re in risk of this awesome forum to become no more than a declaration of intentions, an utopy, without any real commitment of work and thus failed. Don’t get me wrong – I’m the first who starts slackering and procrastrinating if I don’t state my own commitments clearly. So here are my agenda of next actions, which I will post to our (hopefully upcoming) mailing list:

- Concact the Spanish universities and ask them to upload their already existing educational material to OAI-enabled repositories. OAI is a metadata standard for easy dissemination of content. Contact Ignasi Labastida (Creative Common Spain), who showed us some of the different tools available, to help me to understand how these repositories work.

- Place the already existing Alqua documents in OAI-enabled repositories, or create a new one, to increase their visibility.

- Contact David Gómez-Ullate, who is currently leading a (publicly funded!) pedagogical innovation project to use free software in some courses and research work in Universidad Complutense de Madrid.

- Investigate the current use of Free Software in the Spanish universities to find out use cases and good practices.

This is all for now, friends. If you’re (even slightly) interested in helping or have any suggestions, don’t hesitate to contact me! :)