7. Data and Data Frames

In this chapter we introduce Pandas.

Pandas is Python’s most popular toolset for manipulating data in tabular form (Excel sheets, data tables). This module has two main goals. The first is to introduce the two main pandas data types, DataFrame and Series.

A DataFrame is a table of data. Datasets at all levels of analysis of analysis can be represented as DataFrames.

You can think of a DataFrame as being organized in rows and column, like a numpy 2D array, but differing from it in two important ways:

1. A DataFrame uses keyword indexing instead of positional indexing. Columns and rows have names, so this makes them a little more like spread sheet tables.

2. The data types of the columns typically differ from one another (Pandas columns may contain strings, number types, and date time types, among others).

Despite these changes in how indexing and typing works, all the principles that apply to computing with numpy arrays will carry over with minor modifications to computing with Pandas DataFrames. This is especially true of Boolean indexing, which will be your fundamental tool for selecting and reshaping data in pandas. Where a DataFrame is like a 2D array, a Series is like a 1D array; both the rows and the columns of Pandas DataFrames are Series objects.

The second goal of the chaoter is to introduce you to some of pandas analytical tools, especially cross tabulation, and grouping. We will briefly examine a few basic uses of pivot tables.

Contents

• 7.1. Pandas Introduction
  • 7.1.1. Create Data
  • 7.1.2. Selecting Columns
  • 7.1.3. Selecting rows
  • 7.1.4. Boolean conditions
  • 7.1.5. Selecting Rows with Boolean Conditions
  • 7.1.6. Combining Conditions with Boolean operators
  • 7.1.7. Keyword indexing and Alignment
  • 7.1.8. Sorting and positional indexing
  • 7.1.9. Loading Data: A more realistic example
  • 7.1.10. Selection: Selecting parts of Pandas data frames
  • 7.1.11. Summary/Review: Selection & Indexing
  • 7.1.12. The .value_counts( ) method
  • 7.1.13. Cross-tabulation
  • 7.1.14. Solution 1
  • 7.1.15. Solution 2
  • 7.1.16. Complaints: a new dataset
  • 7.1.17. Using groupby
  • 7.1.18. Understanding cross-tabulation
  • 7.1.19. Exercises combining everything we’ve learned in Part One.
• 7.2. Pivot Tables
  • 7.2.1. Introducing the pivot table
• 7.3. Merging
  • 7.3.1. Another kind of merging: Join
  • 7.3.2. Adding Lat Longs to the data
  • 7.3.3. Doing the join