CityState

We’re pleased to announce the creation of a new mailing list for the acs.R package. The “acs” package allows users to download, manipulate, analyze, and visualize data from the American Community Survey in R; the “acs-r” e-mail list allows members to keep in touch and share information about the package, including updates from the development team concerning improvements, user questions and help requests, worked examples, and more. To register, visit http://mailman.mit.edu/mailman/listinfo/acs-r.

A very nice user wrote the following in an email to me about the latest version of the acs.R package:

 
> Thanks for providing such a wonderful package in R. I'm having
> difficulty defining a geo at the block group level. Would you mind
> sharing an example with me?

I responded via email, but thought that my answer — which took the form of a short worked-example — might be helpful to others, so I am posting it here as well. Here’s what I said:

To showcase how the package can create new census geographies based on stuff like blockgroups, let’s look in my home state of Massachusetts, in Middlesex County. If I wanted to get info on all the block groups for tract 387201,¹ I could create a new geo like this:

> my.tract=geo.make(state="MA", county="Middlesex", 
  tract=387201, block.group="*", check=T)
Testing geography item 1: Tract 387201, Blockgroup *, 
  Middlesex County, Massachusetts .... OK.
> 

(This might be a useful first step, especially if I didn’t know how many block groups there were in the tract, or what they were called. Also, note that check=T is not required, but can often help ensure you are dealing with valid geos.)

If I then wanted to get very basic info on these block groups – say, table number B01003 (Total Population), I could type:

> total.pop=acs.fetch(geo=my.tract, table.number="B01003")
> total.pop
ACS DATA: 
 2007 -- 2011 ;
  Estimates w/90% confidence intervals;
  for different intervals, see confint()
              B01003_001  
Block Group 1 2681 +/- 319
Block Group 2 952 +/- 213 
Block Group 3 1010 +/- 156
Block Group 4 938 +/- 214 
> 

Here we can see that the block.group=”*” has yielded the actual four block groups for the tract.

Now, if instead of wanting all of them, we only wanted the first two, we could just type:

> my.bgs=geo.make(state="MA", county="Middlesex", 
  tract=387201, block.group=1:2, check=T)
Testing geography item 1: Tract 387201, Blockgroup 1, 
  Middlesex County, Massachusetts .... OK.
Testing geography item 2: Tract 387201, Blockgroup 2, 
  Middlesex County, Massachusetts .... OK.
> 

And then:

> bg.total.pop=acs.fetch(geo=my.bgs, table.number="B01003")
> bg.total.pop
ACS DATA: 
 2007 -- 2011 ;
  Estimates w/90% confidence intervals;
  for different intervals, see confint()
              B01003_001  
Block Group 1 2681 +/- 319
Block Group 2 952 +/- 213 
> 

Now, if we wanted to add in some blockgroups from tract 387100 (a.k.a. “tract 3871″ — but remember: we need those trailing zeroes) – say, blockgroups 2 and 3 – we could enter:

> my.bgs=my.bgs+geo.make(state="MA", county="Middlesex", 
  tract=387100, block.group=2:3, check=T)
Testing geography item 1: Tract 387100, Blockgroup 2, 
  Middlesex County, Massachusetts .... OK.
Testing geography item 2: Tract 387100, Blockgroup 3, 
  Middlesex County, Massachusetts .... OK.

And then:

> new.total.pop=acs.fetch(geo=my.bgs, table.number="B01003")
> new.total.pop
ACS DATA: 
 2007 -- 2011 ;
  Estimates w/90% confidence intervals;
  for different intervals, see confint()
              B01003_001  
Block Group 1 2681 +/- 319
Block Group 2 952 +/- 213 
Block Group 2 827 +/- 171 
Block Group 3 1821 +/- 236
> 

Note that the short rownames can be confusing – as in this example — but if you type:

> geography(new.total.pop)
           NAME state county  tract blockgroup
1 Block Group 1    25     17 387201          1
2 Block Group 2    25     17 387201          2
3 Block Group 2    25     17 387100          2
4 Block Group 3    25     17 387100          3
> 

you can see that the two entries for “Block Group 2″ are actually in different tracts. (Also note: you can combine block groups and other levels of geography, all in a single geo objects…)

And now, to show off the coolest part! Let’s say I don’t just want to get data on the four blockgroups, but I want to combine them into a single new geographic entity. Before downloading, I could simply say:

> combine(my.bgs)=T
> combine.term(my.bgs)="Select Blockgroups"
> new.total.pop=acs.fetch(geo=my.bgs, table.number="B01003")
> new.total.pop
ACS DATA: 
 2007 -- 2011 ;
  Estimates w/90% confidence intervals;
  for different intervals, see confint()
                   B01003_001               
Select Blockgroups 6281 +/- 481.733328720362
>

And see – voila! – it sums the estimates and deals with the margins of error, so you don’t need to get your hands dirty with square roots and standard errors and all that messy stuff.

You can even create interesting nested geo.sets, where some of the lower levels are combined, like this:

> combine.term(my.bgs)="Select Blockgroups, 
  Tracts 387100 and 387201"
> more.bgs=c(my.bgs, geo.make(state="MA", 
  county="Middlesex", tract=370300, block.group=1:2, check=T), 
  geo.make(state="MA", county="Middlesex", tract=370400, 
  block.group=1:3, combine=T, combine.term="Select Blockgroups, 
  Tract 3703", check=T)) 
Testing geography item 1: Tract 370300, Blockgroup 1, 
  Middlesex County, Massachusetts .... OK.
Testing geography item 2: Tract 370300, Blockgroup 2, 
  Middlesex County, Massachusetts .... OK.
Testing geography item 1: Tract 370400, Blockgroup 1, 
  Middlesex County, Massachusetts .... OK.
Testing geography item 2: Tract 370400, Blockgroup 2, 
  Middlesex County, Massachusetts .... OK.
Testing geography item 3: Tract 370400, Blockgroup 3, 
 Middlesex County, Massachusetts .... OK.
> more.total.pop=acs.fetch(geo=more.bgs, table.number="B01003")
> more.total.pop
ACS DATA: 
 2007 -- 2011 ;
  Estimates w/90% confidence intervals;
  for different intervals, see confint()
                                             B01003_001               
Select Blockgroups, Tracts 387100 and 387201 6281 +/- 481.733328720362
Block Group 1                                315 +/- 132              
Block Group 2                                1460 +/- 358             
Select Blockgroups, Tract 3703               2594 +/- 487.719181496894
> 

In closing: I hope this helps, and be sure to contact me if you have other questions/problems about using the package.

It’s been a while since I last updated the acs.R package, but as noted here, I’ll be using CityState to provide updates and test-versions of the package prior to uploading to CRAN. I’m happy to report that we now have a near-final package of version 1.0.

The most significant improvements to the package (beyond those mentioned previously) are the following;

• The package is now capable of downloading data directly from the new Census American Community Survey API and importing into R (with proper statistical treatment of estimates and error, variable and geographic relabeling, and more), all through a single “acs.fetch()” function;
• The package includes a new “geo.make()” function to allow users to create their own custom geographies for organize and download data; and
• The package provides two special “lookup” tools to help filter through all the existing Census geographies (with the “geo.lookup()” function) and tables (with the “acs.lookup()” function) to find exactly what they want. These functions return new R “lookup” objects which can be saved, manipulated, and passed to acs.fetch() for downloading data.

I want to thank the very kind folks at the Puget Sound Regional Council, who have been supporting the development of this package (in exchange for some special attention to scripts and functions they really want to include for themselves and their member communities). They continue to provide excellent help and advice “from the trenches” as we refine the package.

If you’re interested in trying out the new version, you can download it below, along with a brief set of “Introductory Notes” written for the team at PSRC. (Users may also want to check out the manual for the previous version of the package and this article from 2011 on the package.)

• acs_1.0.tar.gz

• working_with_acs.R.pdf

Previous missions have demonstrated a whole lot of things you can do with census data. Here are a few of the problems you can get yourself into.

In our last mission we used R to plot a trend-line for population growth in Houston, based on historical data from the past century. Depending on which of two different methods we used, we arrived at an estimate for the city’s 2010 population of 2,144,531 (based on the 100-year growth trend for the city) or 2,225,125 (based on the steeper growth trend of the past fifty years). Looking now at the official Census count for 2010, it turns out that our guesses are close, but both of too high: the actual reported figure for 2010 is 2,099,451.

It would have been surprising to have guessed perfectly based on nothing other than a linear trend — and the fact that we came as close as we did speaks well of this sort of “back of the envelope” projection technique (at least for the case of steady-growth). But there was a lot of information contained in those data points that we essentially ignored: our two trendlines were really based on nothing more than a start and an end point.

A more sophisticated set of tools for making projections — which may be able to extract some extra meaning from the variation contained in the data — is provided in R by the excellent forecast package, developed by Rob Hyndman of the Monash University in Australia. To access these added functions, you’ll need to install it:

> install.packages(forecast)
> library(forecast)

> a=c(1,2)
> class(a)
[1] "numeric"
> a=TRUE
> class(a)
[1] "logical"
> class(plot)
[1] "function"
> a=ts(1)
> class(a)
[1] "ts"
> 

In our previous mission we plotted population numbers in Houston for 1900–2000, to start to understand the growth trend for that city. Now, what if we didn’t have access to the latest Census figures, and we wanted to try to guess Houston’s population for 2010, using nothing but the data from 1900–2000?

One place to start would be with the 2000 population (1,953,631) and adjust it a bit based on historical trends. With 100 year’s worth of data, we can do this in R with a simple call to some vector math.¹

> attach(houston.pop) # optional, see footnote
> population[11]      # don't forget: 11, not 10, data points
[1] 1953631
> annual.increase=(population[11]-population[1])/100   # watch the parentheses!
> population[11]+10*annual.increase
[1] 2144531
> 

Remember that we actually have eleven data points, since we have both 1900 and 2000, so we need to specify population[11] as our endpoint. But since there are only ten decade intervals, we divide by 100 to get the annual increase. Adding ten times this increase to the 2000 population, we get an estimate for 2010 of 2,144,531. (Bonus question: based on this estimated annual increase, in what year would Houston have passed the two-million mark?²)

Continue reading…

I’m happy to report that I’ll be giving a paper on my acs package at the 8th annual useR! conference, Coming June 12-15^th to Vanderbilt University in Nashville, TN. The paper is titled “Estimates with Errors and Errors with Estimates: Using the R acs Package for Analysis of American Community Survey Data.” Here’s the abstract:

"Estimates with Errors and Errors with Estimates: Using the R acs
Package for Analysis of American Community Survey Data"
Ezra Haber Glenn

Over the past decade, the U.S. Census Bureau has implemented the
American Community Survey (ACS) as a replacement for its traditional
decennial ``long-form'' survey.  Last year—for the first time
ever—ACS data was made available at the census tract and block group
level for the entire nation, representing geographies small enough to
be useful to local planners; in the future these estimates will be
updated on a yearly basis, providing much more current data than was
ever available in the past.  Although the ACS represents a bold
strategy with great promise for government planners, policy-makers,
and other advocates working at the neighborhood scale, it will require
them to become comfortable with statistical techniques and concerns
that they have traditionally been able to avoid.

To help with this challenge the author has been working with
local-level planners to determine the most common problems associated
with using ACS data, and has implemented these functions as a package
in R.  The package—currently hosted on CRAN in version 0.8—defines
a new ``acs'' class object (containing estimates, standard errors, and
metadata for tables from the ACS), with methods to deal appropriately
with common tasks (e.g., combining subgroups or geographies,
mathematical operations on estimates, tests of significance, plots of
confidence intervals, etc.).

This paper will present both the use and the internal structure of the
package, with discussion of additional lines of development.

Hope to see you all there!

In a previous mission (see Finding Obama in the smallest Census geography) we delved down to the see what data was available at the level of individual blocks. Unfortunately, as we noted there, the Census doesn’t provide a whole lot of useful data at the block-level, since the results will exclude sample data from the SF3 “long form” (or, post-2000, the American Community Survey). If we want to know more about a neighborhood we will need to think in slightly larger geographies, and seek data at the tract-level or higher.

For this mission, we’ll be zooming into to Park Slope neighborhood on Brooklyn, and gathering data on income, race, education, and the breakdown of owners and renters for a single census tract. Since its often helpful to be able to view data like this in the context of the surrounding neighborhood, subsequent missions will explore ways to make comparisons with this sort of data, either to other tracts or to larger geographies.

But for starters, our target: although defining the exact edges of a neighborhood is never easy – especially ones in dense, diverse areas, where even residents disagree over terminology and the continual processes of gentrification, urban decline, migration, and other demographic shifts continually redefine the categories – most observers would agree that the neighborhood extends roughly north and west from Bartel Pritchard Square, at the lower corner of Prospect Park, with both 15th Street and Prospect Park itself providing something of an “edge.” Since edges are often exciting places to observe change, we will select an address along 15th Street, near the corner of 5th Avenue. Continue reading…

The most basic unit of the U.S. Census is the individual household — that’s who fills out the surveys – but the Census won’t report data at the household level: in order to deliver on its promise of privacy and confidentiality (and thereby ensure our willingness to be enumerated), the Census always aggregates data before releasing it. This is important, and should become something of a mantra for would-be data analysts: all Census data is summary data. That said, we can still learn quite a lot at these micro-geographies, especially when we know what we are looking for.