Linguistics analysis: State of the Union addresses

Always looking for an idea that will generate a much needed income stream , combined with my unabated fascination by technology, and my insatiable curiosity about various problem domains, I decided to look at linguistics as a tool to help generate content.

So I downloaded and installed the nltk package from nltk.org . I did install it on Google AppEngine account together with DJango. I had a few minor technical difficulties to overcome, but it all ended working well.

I decided to test the nltk package with an analysis of the State of the Union addresses over the years. I wanted to explore what were the dominant words used by the various presidents during their various addresses. The objective was to do a word frequency count and to compare the top words use in each address.

This was easily achieved with the nltk package. I was able to calculate the word frequency, exclude stopwords from the results sets, and exclude a few other words that were devoid meaning. If anyone is interested, I will post the list of words I used and the code used to provide the results described here.

I had to add a handful of missing state of the Union addresses to the downloaded corpus from nltk.org and I went on developing the following very simple quantitative linguistics analysis.

Here is a list of the State of Union addresses followed by a list of twelve words for each. These words are listed in the order of the most frequently occuring first, then the next, etc. in that address.

Some quick salient observations:
1. the focus by Clinto on children.
2. the focus of Eisenhower, a military man, on military and defence
3. Reagan the second to focus on the term billion, as in dollars
4. Bush and the looming terror threat
..

etc

Feel free to add any other noteworthy observation via a comment. Thanks for that. It will encourage me do more of these, perhaps.

Anyway, here are the results:

1945-Truman
peace world nations hope freedom faith wars man victory justice defense war
1946-Truman
dollars war million fiscal expenditures billion Federal national world business production economic
1947-Truman
labor national management war world public housing Commission legislation veterans peace number
1948-Truman
world economic tax nations living resources war peace system prices inflation wage
1949-Truman
world prosperity production housing economy security power democratic national business public social
1950-Truman
world economic life system peace democratic benefits national freedom growth power past
1951-Truman
world free nations Soviet defense military strength freedom aggression Europe together fighting
1953-Eisenhower
Federal security economic policy labor freedom strength legislation world defense military administration
1954-Eisenhower
Federal world military security tax freedom economic system assistance defense present peace
1955-Eisenhower
Federal military economic free strength world past peace forces citizens nations economy
1956-Eisenhower
Federal legislation system Administration national world economic military effort peace progress free
1957-Eisenhower
world free security nations economy peace human vital military rights principles economic
1958-Eisenhower
military world peace strength security economic effort power action development defense free
1959-Eisenhower
world freedom Federal economy free progress growth public development economic military law
1960-Eisenhower
nations world peace freedom World progress Free help economic system free past
1961-Kennedy
freedom defense dollars space million world nations President efforts civil nuclear necessary
1962-Kennedy
free world strength help nations growth efforts economy economic increased freedom aid
1963-Johnson
action strong dream act Kennedy strength forward national John bill peace rights
1963-Kennedy
world free nations tax defense billion help nuclear freedom peace Communist alliance
1964-Johnson
session public million help poverty budget faith hopes history Federal tax war
1965-Johnson-1
help world freedom man live interest life men effort open nations begin
1965-Johnson-2
rights bill man President men vote Selma cause law poverty Constitution history
1966-Johnson
Vietnam world help South peace work billion men life recommend war principle
1967-Johnson
Vietnam war help nations President billion men think progress home security together
1968-Johnson
billion million budget cities help peace hope act tax Act consumer increase
1969-Johnson
think believe world President peace hope already commitments budget administration Vietnam public
1970-Nixon
world peace nations war growth land Federal responsibility life crime air live
1971-Nixon
Federal open way history goal power chance resources achieve budget President localities
1972-Nixon
world help believe Federal goal President proposals war peace look action policy
1973-Nixon
Union resources together spending Federal messages taxes without world goals peace help
1974-Nixon
peace world energy system President ago needs Union past Federal health progress
1975-Ford
energy world oil Federal tax President economic legislation billion taxes major nations
1976-Ford
Federal world budget economy help tax local much cost jobs President growth
1977-Ford
President energy peace Federal world defense Union strategic free economic national nations
1978-Carter
world inflation tax policy economic economy major foreign together work jobs energy
1979-Carter
world peace inflation foundation nuclear rights SALT strong build control economy nations
1980-Carter
Soviet world peace Union military oil security nations energy preserve power East
1981-Reagan
tax billion Federal jobs inflation effective economic rates plan increase interest cost
1982-Reagan
Federal tax billion help economic spending growth local world social peace taxes
1983-Reagan
economic spending world work problems recovery peace growth strength together trade freedom
1984-Reagan
peace freedom tax free help spending hope space work God develop children
1985-Reagan
freedom tax economy growth world help lives free defense human Federal jobs
1986-Reagan
work free world family budget freedom Federal tax speak peace dream families
1987-Reagan
world freedom Soviet trade work free budget century welfare Constitution national children
1988-Reagan
freedom world family budget work federal agreement free democratic economic peace revolution
1989-Bush
budget work help hope drugs plan President increase world growth families billion
1990-Bush
world free freedom idea ago kids capital change Europe Union hope budget
1991-Bush-1
world freedom power struggle Gulf home leadership war work peace growth aggression
1991-Bush-2
world war peace home work Gulf Middle victory Iraq Desert East Storm
1992-Bush
world plan help health tax Federal economy pay together changes work thing
1993-Clinton
plan care jobs work health tax deficit investment world interest children believe
1994-Clinton
health care work world welfare jobs system crime children insurance President reform
1995-Clinton
work way children welfare citizens together families working world responsibility help economy
1996-Clinton
challenge children work working families together crime way challenges hard community parents
1997-Clinton
world children work budget century challenge school welfare schools help national education
1998-Clinton
children help care world century child work college tax support families way
1999-Clinton
century work support Security Social children help budget school world schools care
2000-Clinton
children help work care thank gun tax child support world community century
2001-GWBush-1
budget tax Social Security care money work energy health Medicare help children
2001-GWBush-2
world terrorists freedom war terror citizens together Afghanistan Qaeda fight forget leaders
2002-GWBush
world security terror Afghanistan weapons war jobs freedom citizens terrorist terrorists allies
2003-GWBush
weapons world Saddam Hussein Iraq dollars million nuclear AIDS war work inspectors
2004-GWBush
health help Iraq law economy children work free war world schools care
2005-GWBush
freedom Security Social world terror workers Iraq system benefits retirement economy help
2006-GWBush
world freedom economy life support members lead citizens fight hopeful military competitive
2007-GWBush
Iraq help health insurance terrorists work Qaida war world children enemy Iraqi
2008-GWBush
Iraq help trust world terrorists forces Iraqi free past empower Qaida hope
2009-Obama
economy plan health care jobs energy education budget recovery cost crisis world
2010-Obama
jobs work families businesses economy energy took small bill tax values business

Copyright 2010 - Michel de Greef