Monthly Archives: April 2012

Hurdy Gurdy (Daniel Seideneder and Daniel Pfeiffer, 2011)

Posted by Ezra Glenn on April 29, 2012
Film / 2 Comments

In World on a Wire (reviewed previously), Rainer Werner Fassbinder explored the possibility of creating a miniature world through the use of a computer. In Hurdy Gurdy, a wonderful new short film from a German and Estonian collaboration, we get to enjoy the ways that the camera itself can render our real-world in apparent miniature (although I suspect a computer played a part as well…), giving us an entirely new and delightfully playful perspective on everyday scenes of urban life.

The film — all of four minutes long — uses stop-motion photography along with a technique that either is, or perhaps simulates, what is known as “tilt-shift” photography. The images below give a rough sense of the effect, which is to change the depth of focus and the level of detail; when combined with the increased speed and mechanical jerkiness (due to the stop-motion animation), the film transforms footage of a typical sea-side town into a magical micropolis of urban interaction: a true sidewalk ballet which unfolds as tourists arrive, streetcars come and go, crowds surge and flow, and daily life weaves and cycles in an endless state of humming activity. (The title itself refers to the mechanical music box, where one could just wind it up again and have the whole scene-and-song play over again and again.)

A number of other short videos using the tilt-shift technique can be found on-line, and quite a few choose city scenes or the movement of crowds to show off the magic; for example, see this popular short depicting a day in the life at Disney or this one showing streetlife in New York City. But it would be wrong to regard Hurdy Gurdy as nothing more than a cool demonstration of a visual trick: rather than letting the technique be the whole story, Seideneder and Pfeiffer use the effect to focus us on the beauty, color, and harmony of our ever-changing world. In a surprising way, even as we watch the film and smile and wonder and are entertained and entranced by this non-stop motion, we discover the time and space to meditate on the smallness of our individual existence and the majesty of the patterns we collectively create.

The film was screened in Somerville, MA, as part of the 2012 Independent Film Festival of Boston, and it seems to be making the rounds of similar festivals here and abroad, including Cleveland, Woodstock, Florida, Lisbon, Rotterdam, and Cannes. Look for it wherever independent films are found.

Tags: , , , ,

Pitfalls of Working with Time-Series Data

Posted by Ezra Glenn on April 24, 2012
Data, Missions, Shape Your Neighborhood, Simulation / No Comments

In addition to the general caution against using past data for projecting future conditions (and the need for equally spaced time intervals mentioned above), the particulars of time series data require additional attention to some special issues.

Inflation and Constant Dollars

Any time series that deals with dollars (or yen, pounds sterling, wampum, or other forms of currency) must confront the fact that the value of money changes over time. If you are simply making a time series showing the shrinking value of the dollar, that’s fine — it’s what you want to show — but if you want to show something else (say, changes in wages or home prices), then you will need to correct your data to some common base. Usually this is done by starting with a base year (often the start or end of the series, or the “current” year) and adjusting values based on changes to some official inflation statistic (e.g., the consumer price index).1

Growth and Change to the Underlying Population

Over time — especially over long periods — the population of a place can change quite a lot, both in terms of overall numbers and the demographic components. As with inflation, this may be precisely the change that you are interested in observing and predicting (as in the first examples in this chapter), but at times it can introduce a spurious or intervening variable into your analysis.

Continue reading…

Tags: , , , , ,

Examining Historical Growth III: The forecast() package

Posted by Ezra Glenn on April 21, 2012
Data, Missions, Shape Your Neighborhood, Simulation / No Comments

In our last mission we used R to plot a trend-line for population growth in Houston, based on historical data from the past century. Depending on which of two different methods we used, we arrived at an estimate for the city’s 2010 population of 2,144,531 (based on the 100-year growth trend for the city) or 2,225,125 (based on the steeper growth trend of the past fifty years). Looking now at the official Census count for 2010, it turns out that our guesses are close, but both of too high: the actual reported figure for 2010 is 2,099,451.

It would have been surprising to have guessed perfectly based on nothing other than a linear trend — and the fact that we came as close as we did speaks well of this sort of “back of the envelope” projection technique (at least for the case of steady-growth). But there was a lot of information contained in those data points that we essentially ignored: our two trendlines were really based on nothing more than a start and an end point.

A more sophisticated set of tools for making projections — which may be able to extract some extra meaning from the variation contained in the data — is provided in R by the excellent forecast package, developed by Rob Hyndman of the Monash University in Australia. To access these added functions, you’ll need to install it:

> install.packages(forecast)
> library(forecast)

Time-series in R: an object with class

Although R is perfectly happy to help you analyze and plot time series data organized in vectors and dataframes, it actually has a specialized object class for this sort of thing, created with the ts() function. Remember: R is an “object-oriented” language. Every object (a variable, a dataframe, a function, a time series) is associated with a certain class, which helps the language figure out how to manage and interact with them. To find the class of an object, use the class() functions:

> a=c(1,2)
> class(a)
[1] "numeric"
> a=TRUE
> class(a)
[1] "logical"
> class(plot)
[1] "function"
> a=ts(1)
> class(a)
[1] "ts"

Continue reading…

Tags: , , , , ,

Examining Historical Growth II: Using the past to predict the future

Posted by Ezra Glenn on April 12, 2012
Data, Missions, Shape Your Neighborhood, Simulation / No Comments

In our previous mission we plotted population numbers in Houston for 1900–2000, to start to understand the growth trend for that city. Now, what if we didn’t have access to the latest Census figures, and we wanted to try to guess Houston’s population for 2010, using nothing but the data from 1900–2000?

One place to start would be with the 2000 population (1,953,631) and adjust it a bit based on historical trends. With 100 year’s worth of data, we can do this in R with a simple call to some vector math.1

> attach(houston.pop) # optional, see footnote
> population[11]      # don't forget: 11, not 10, data points
[1] 1953631
> annual.increase=(population[11]-population[1])/100   # watch the parentheses!
> population[11]+10*annual.increase
[1] 2144531

Remember that we actually have eleven data points, since we have both 1900 and 2000, so we need to specify population[11] as our endpoint. But since there are only ten decade intervals, we divide by 100 to get the annual increase. Adding ten times this increase to the 2000 population, we get an estimate for 2010 of 2,144,531. (Bonus question: based on this estimated annual increase, in what year would Houston have passed the two-million mark?2)

Continue reading…

Tags: , , , , ,

Examining Historical Growth I: Basic trends

Posted by Ezra Glenn on April 11, 2012
Data, Missions, Shape Your Neighborhood, Simulation / No Comments

The nature of predictions

To paraphrase John Allen Paulos, author of A Mathematician Reads the Newspaper, all expert predictions can be essentially restated in one of two ways: “Things will continue roughly as they have been until something changes”; and its corollary, “Things will change after an indeterminate period of stability.” Although these statements are both true and absurd, they contain a kernel of wisdom: simply assuming a relative degree of stability and painting a picture of the future based on current trends is the first step of scenario planning. The trick, of course, is to never completely forget the “other shoe” of Paulos’s statement: as the disclaimer states on all investment offerings, “Past performance is not a guarantee of future results”; at some point in the future our present trends will no longer accurately describe where we are headed. (We will deal with this as well, with a few “safety valves.”)

From the second stage of the Rational Planning Paradigm (covered in the background sections of the book) we should have gathered information on both past and present circumstances related to our planning effort. If we are looking at housing production, we might have data on annual numbers of building permits and new subdivision approvals, mortgage rates, and housing prices; if we are looking at public transportation we might need monthly ridership numbers, information of fare changes, population and employment figures, and even data on past weather patterns or changes in vehicle ownership and gas prices. The first step of projection, therefore, is to gather relevant information and get it into a form that you can use.

Since we will be thinking about changes over time in order to project a trend into the future, we’ll need to make sure that our data has time as an element: a series of data points with one observation for each point or period of time is known as a time series. The exact units of time are not important—they could be days, months, years, decades, or something different—but it is customary (and important) to obtain data where points are regularly spaced at even intervals.1 Essentially, time series data is a special case of multivariate data in which we treat time itself as an additional variable and look for relationships as it changes. Luckily, R has some excellent functions and packages for dealing with time-series data, which we will cover below in passing. For starters, however, let’s consider a simple example, to start to think about what goes into projections. Continue reading…

Tags: , , , , ,

acs Package at Upcoming Conference: UseR! 2012

Posted by Ezra Glenn on April 09, 2012
Census, Code, Self-promotion / No Comments

I’m happy to report that I’ll be giving a paper on my acs package at the 8th annual useR! conference, Coming June 12-15th to Vanderbilt University in Nashville, TN. The paper is titled “Estimates with Errors and Errors with Estimates: Using the R acs Package for Analysis of American Community Survey Data.” Here’s the abstract:

"Estimates with Errors and Errors with Estimates: Using the R acs
Package for Analysis of American Community Survey Data"
Ezra Haber Glenn

Over the past decade, the U.S. Census Bureau has implemented the
American Community Survey (ACS) as a replacement for its traditional
decennial ``long-form'' survey.  Last year—for the first time
ever—ACS data was made available at the census tract and block group
level for the entire nation, representing geographies small enough to
be useful to local planners; in the future these estimates will be
updated on a yearly basis, providing much more current data than was
ever available in the past.  Although the ACS represents a bold
strategy with great promise for government planners, policy-makers,
and other advocates working at the neighborhood scale, it will require
them to become comfortable with statistical techniques and concerns
that they have traditionally been able to avoid.

To help with this challenge the author has been working with
local-level planners to determine the most common problems associated
with using ACS data, and has implemented these functions as a package
in R.  The package—currently hosted on CRAN in version 0.8—defines
a new ``acs'' class object (containing estimates, standard errors, and
metadata for tables from the ACS), with methods to deal appropriately
with common tasks (e.g., combining subgroups or geographies,
mathematical operations on estimates, tests of significance, plots of
confidence intervals, etc.).

This paper will present both the use and the internal structure of the
package, with discussion of additional lines of development.

Hope to see you all there!

Tags: , , , , ,

A Richer Neighborhood Profile, Part I: Getting tract-level data

Posted by Ezra Glenn on April 08, 2012
Census, Missions, Reconnaissance, Shape Your Neighborhood / No Comments

In a previous mission (see Finding Obama in the smallest Census geography) we delved down to the see what data was available at the level of individual blocks. Unfortunately, as we noted there, the Census doesn’t provide a whole lot of useful data at the block-level, since the results will exclude sample data from the SF3 “long form” (or, post-2000, the American Community Survey). If we want to know more about a neighborhood we will need to think in slightly larger geographies, and seek data at the tract-level or higher.

For this mission, we’ll be zooming into to Park Slope neighborhood on Brooklyn, and gathering data on income, race, education, and the breakdown of owners and renters for a single census tract. Since its often helpful to be able to view data like this in the context of the surrounding neighborhood, subsequent missions will explore ways to make comparisons with this sort of data, either to other tracts or to larger geographies.

But for starters, our target: although defining the exact edges of a neighborhood is never easy – especially ones in dense, diverse areas, where even residents disagree over terminology and the continual processes of gentrification, urban decline, migration, and other demographic shifts continually redefine the categories – most observers would agree that the neighborhood extends roughly north and west from Bartel Pritchard Square, at the lower corner of Prospect Park, with both 15th Street and Prospect Park itself providing something of an “edge.” Since edges are often exciting places to observe change, we will select an address along 15th Street, near the corner of 5th Avenue. Continue reading…

Tags: , , , ,

Master Plan (Robert Todd, 2011)

Posted by Ezra Glenn on April 03, 2012
Film / No Comments

Yesterday we screened Robert Todd’s Master Plan at MIT’s Department of Urban Studies & Planning, and I’m still thinking about it. It’s a beautiful documentary of the best kind: one that presents stirring images and thought-provoking juxtapositions, but once stirred and provoked the viewer’s thoughts are allowed to marinate a while. The film shies away from any pat conclusions, seeming much more comfortable presenting a landscape of places, ideas, and lines of inquiry for us to wander and ponder along with Todd, rather than a single “punch line” he wants us to “get”; I was reminded of the line from Zen and the Art of Motorcycle Maintenance, where Pirsig talks about the importance of thinking about “what things are,” and not just “what things mean.”

Indeed, the film had a certain Zen-like quality, both in its attention to small details and quietly “just being” in the places it explores, as well as its non-attachment to a single-purpose narrative. Although described as “a feature length film about housing,” its scope extends far beyond simply looking at physical housing: its subject is homes, habitats, communities, neighborhoods, buildings, landscapes, and the ways people interact in, around, and with them; the bulk of the footage presents a wonderfully rich portrait — or perhaps nonstop pan — of the ways humans live in places. Beyond all this — and the luxuriously decompressed pace takes plenty of time meandering before arriving at this point — the focal point of the film finally settles on a prolonged meditation on the homes and communities of incarcerated individuals, which is apparently a longer-term project for Todd. (An earlier film, In Loving Memory, explored the experiences of prisoners on death row; his next major project will examine ways that former prisoners are re-integrated into their home communities.)

Continue reading…

Tags: , , , ,

Building Blocks: Finding Obama in the smallest Census geography

Posted by Ezra Glenn on April 02, 2012
Missions, Reconnaissance, Shape Your Neighborhood / 1 Comment

The most basic unit of the U.S. Census is the individual household — that’s who fills out the surveys – but the Census won’t report data at the household level: in order to deliver on its promise of privacy and confidentiality (and thereby ensure our willingness to be enumerated), the Census always aggregates data before releasing it. This is important, and should become something of a mantra for would-be data analysts: all Census data is summary data. That said, we can still learn quite a lot at these micro-geographies, especially when we know what we are looking for.

Finding Barack

As an example of how to work with the building blocks of Census summary data – the individual “blocks” – let’s go back a bit in time and look at a very particular neighborhood in Chicago. At the time of the 2000 Census, President Obama was serving as a Senator from Illinois, living at 5429 S. Harper Avenue in Chicago. Starting with just an address, you can easily find how it fits into the census geography on the “American FactFinder” site: just visit the main Census site, click the menu-bar for Data, and select the link for American FactFinder.

Continue reading…

Tags: , , , , , ,