Climbing the Ladder: Investigating the “American Dream” with Python

Climbing the Ladder: Investigating the “American Dream” with Python

College was everything to me.

Until the day I got my first acceptance letter, every grade, project and passion was pursued with college in mind. I knew since I was 5 that I needed a scholarship—that my future rested on money that I didn’t have.

My parents, teachers and counselors worked hard to make it happen, and due in no small part to the combination of their efforts and sheer good luck, I was able to graduate from an amazing school. But I was lucky. I have known many people throughout my life who shared my drive but not my fortune.

As a lifelong observer, explorer, and beneficiary of the American higher education system, I want to understand it. I want to know its history, its benefits, and the role it plays in intergenerational mobility. I want to know whether there is more than just luck that separates attainment and success from rejection or failure. This article represents my best attempt to do so.

A brief history of American Education

American schools, as it turns out, have always been very different.

In the late 18th century, American colonies varied drastically in their education systems. Geography, immigration, religious sectarianism and the rise of enlightenment thinking combined to create a patchwork of different educational philosophies. The privatized, economically stratified elementary schools in the South formed a different education ecosystem to the governmental and religious institutions in the North.

Prior to 1747, Harvard, Yale and William and Mary were the only colleges within the United States. But the religious fervor inspired by the Great Awakening caused various Christian sects to open schools, tripling the number of colonial colleges, and setting the groundwork for higher education in the country.


The original Harvard College building (1638–1670)

Americans became far more educated throughout the 19th century, as the embrace of public funding and a decentralized education system saw the United States become one of the best-educated countries in the world. American colleges, initially considered inferior to their European counterparts, would benefit massively from public commitment and investment.

The 20th century greatly accelerated the transition to higher education, particularly so after World War II. The GI Bill covered college tuition and expenses for World War II and Korean veterans, encouraging many to pursue a degree after returning to the states. The SAT attempted to democratize the admissions process. Low-cost community colleges and two-year schools began popping up throughout the country. Most importantly, the percentage of high school graduates went from 25% in 1940 to nearly 80% by 2000.

Likewise, college graduation rates have exploded, with the U.S. Census Bureau’s data showing that the percentage of college graduates in the general population has doubled in the past 40 years:


Via U.S Census Bureau.

However, this growth has not been even. The geographic, fiscal, and, above all, racial composition of different localities has led to a wide range of different outcomes for different people.

The combination of exploding demand for higher education and variable local factors has resulted in wide diversity among colleges and their student bodies. These differences are key to understanding the role higher education plays in intergenerational mobility. By analyzing how wealth, location and collegiate selectivity affect student outcomes, we can better understand the role higher education plays in achieving the “American Dream.”

College and outcome

As the American workforce has adapted to an increasingly automated world, demand for a college degree has drastically risen. Per the Georgetown Public Policy Institute, we know that degree holders have seen the highest job growth for the past 20 years:


Via Georgetown Public Policy Institute.

We also know that degree holders have higher wages:


Via Bureau of Labor Statistics.

Lower rates of unemployment:


Via Statista.

And have benefitted massively from increased educational requirements over the past 45 years:


Via Georgetown Public Policy Institute.

While the U.S. population has made huge strides in college participation, the distribution of these gains has been uneven at the local level. One example of this is the disconnect between increasing high school and college graduation rates.

The Hamilton Project, an economic research group within the Brookings Institution, found that counties with the lowest percentage of high school graduates in 1980 had the highest gains in high school graduation rates by 2016:


Via The Hamilton Project.

The opposite is true for college graduates. Counties with lower percentages of college graduates in 1980 saw the smallest increases, while counties with higher percentages of college graduates in 1980 saw the largest gains:


Via The Hamilton Project.

To understand the ramifications of this trend, we can use the Hamilton Project’s Vitality Index. This index assigns each county in the United States a composite score according to factors like household income, poverty rate, life expectancy and other measurements of the county’s health.

Comparing county vitality rate, we can see that this lack of collegiate participation has had dire consequences for many counties. Counties with low college attainment had significantly worse vitality scores in 1980 and 2016.


Via The Hamilton Project.

So, what does this tell us?

  • An increasing number of jobs require a college education
  • College graduates tend to be significantly better off than non-graduates
  • An increasing percentage of the population are getting a college education
  • Collegiate educational gains have been smallest in counties with fewer graduates
  • The presence of college graduates has significant ramifications for community well-being.

Altogether, this gives us a detailed picture of outcomesw, but economic mobility is a different beast. To understand how colleges impact mobility, we need to understand the opportunities colleges provide for those on the bottom end of the national income distribution. We can’t just know that colleges are associated with better circumstances—we need to know how they affect those who come from worse circumstances.

Determining Mobility

So which colleges provide such opportunities for low-income students? Where are these colleges located?

To answer this question, we’ll need some data. Thankfully, a group of researchers at Opportunity Insights have done the legwork for us. Using IRS tax records and Department of Education data, this team has released various data sets concerning student outcomes for children who were born between 1980 and 1982. With Pandas, we can visualize this data, and in the process paint a clearer picture of economic mobility in the United States.

Let’s start with a head count.



There are two main takeaways from this data:

  • Most children attended some type of college, but a sizable minority (averaging a bit under 1.4 million per year) didn’t.
  • College graduation is primarily driven by Selective and Two-year colleges

Remember this when considering the data below. While a non-trivial number of students attend Elite, Highly Selective, and Non-Selective colleges, the schools most representative of the student experience are Selective and Two-year institutions. Other schools types, while enrolling millions of students, represent only a fraction of national student body.

Median Parental Household Income

As mobility describes upward economic movement, we need to know these students’ “starting position”. To do this, we can compare the range of parental income by college type:


Unsurprisingly, median parental household income moves upward by tier. Not a single Elite college had a median parental income below a hundred thousand, while the opposite was true for for-profit colleges. The income gap between college tiers increases the more rightward you go in the graph, ranging from median incomes between 20 and 30k to median incomes above 220k.

Individual Income

The pattern observed with parental income persists with student income:


Again, we see a clear rightward shift in the income distribution by college tier. This tells us that:

  • Individual and parental income generally increases according to college tier.
  • There is a wide gap between the incomes for the lowest tier (under two years/for profits) and the highest tier(most selective schools)

Bottom to Top

We can see that there is a significant wealthy disparity according to college type. But how does the distribution of wealth look if we focus on lower income students? Are the colleges with the wealthiest students on average seeing the greatest gains for students from poor families?

To determine this, we can use what I will henceforth call “Bottom to top mobility,” the percentage of students from families in the bottom 20% of the national income distribution who rise to the top quintile of the individual national income distribution.


The answer is a resounding yes. Like the income distributions, we see a clear upward trajectory by college tier. Bottom quintile students show more upward progression at higher tier colleges, with a majority at most elite schools reaching the top individual quintile. In general, poorer students seem to do considerably better after graduating from top schools.

Mobility Rates

So if more selective schools lead to higher income for lower income students, and they admit fewer low income students, how can we assess these schools’ overall impact on mobility? We can take those “mobile students”, the students from low income families who made it into the top quintile, and divide the number of them at a given school by the total number of students at that school. Doing so gives us these colleges’ “Mobility Rates”:

(Note: The following graph is “zoomed in” for clarity. 7 colleges with Mobility Rates above 8.5 are not shown.)


Mobility Rates don’t diverge noticeably by selectivity. While there is variation in the distributions, there isn’t a clear “upward path” between college tier and Mobility Rates. Selective schools do better for poor students, yet this seems to be “balanced out” by more selective colleges having fewer poor students in the first place.


So while poorer students tend to do better at more selective schools, the lower percentage of poor students at these schools means that these institutions have a smaller footprint on economic mobility.

We can graph this footprint geographically. If we multiply the mobility rate by the number of students, then group these students together by their colleges counties, we can see the locations in the United States that play the biggest role in driving economic mobility.


County Count

Naturally, our largest urban centers had the most representation. Los Angeles; San Diego; Miami-Dade; Cook County, Illinois; Harris County, Texas; Prince George’s County, Maryland; and the New York City boroughs were responsible for a substantial portion of mobile students. This is in line with general trends in college enrollment:

“Across the nation, college graduates are overwhelmingly concentrated in urban areas. Almost 90 percent of college grads live in urban counties, with more than 60 percent of them in large metros with over one million people. Just a bit more than one in ten college graduates reside in rural communities.”
Richard Florida, Citylab

Finally, we can split these mobile students by college type.


The majority of “mobile” students come from selective and two year colleges. While more selective schools might be more conducive to economic mobility, they represent a smaller fraction of overall college attendance, and percentage-wise they admit far fewer lower income students.

Looking Ahead

James Tuslow Adams defined the American Dream as:

“…(not) a dream of motor cars and high wages merely, but a dream of social order in which each man and each woman shall be able to attain to the fullest stature of which they are innately capable, and be recognized by others for what they are, regardless of the fortuitous circumstances of birth or position.”

Economic mobility lies at the heart of the American Dream, and education has long served as it’s ladder. College, once available to only a select few, has become an essential component of the American workforce and economy.

These longstanding trends in mobility won’t reverse themselves, and a college degree will doubtlessly become even more essential to employment prospects in the future. However, I hope recognizing the colleges that drive mobility can help us determine the avenues by which we can increase the availability of opportunity and elevate those for whom the opportunities are slipping further from reach.

Time’s arrow always marches forward, but it is worth considering how we might aid those left in its trail.