Data Manipulation with pandas By/With Kyesswa Steven

Data Manipulation with pandas with Kyesswa Steven( Kyessvar Stevenz )

Run the hidden code cell below to import the data used in this course.

Notes Taken By Kyesswa Steven On 11/5/2023

Inspecting a DataFrame Exercise

When you get a new DataFrame to work with, the first thing you need to do is explore it and see what it contains. There are several useful methods and attributes for this.

• .head() returns the first few rows (the “head” of the DataFrame).
• .info() shows information on each of the columns, such as the data type and number of missing values.
• .shape returns the number of rows and columns of the DataFrame.
• .describe() calculates a few summary statistics for each column.

homelessness is a DataFrame containing estimates of homelessness in each U.S. state in 2018. The individual column is the number of homeless individuals not part of a family with children. The family_members column is the number of homeless individuals part of a family with children. The state_pop column is the state's total population.

Instructions 1/4

1. Print the head of the homelessness DataFrame.
2. Print information about the column types and missing values in homelessness.
3. Print the number of rows and columns in homelessness.
4. Print some summary statistics that describe the homelessness DataFrame.

Parts of a DataFrame Exercise

To better understand DataFrame objects, it's useful to know that they consist of three components, stored as attributes:

.values: A two-dimensional NumPy array of values. .columns: An index of columns: the column names. .index: An index for the rows: either row numbers or row names.

You can usually think of indexes as a list of strings or numbers, though the pandas Index data type allows for more sophisticated options. (These will be covered later in the course.)

Instructions

• Import pandas using the alias pd.
• Print a 2D NumPy array of the values in homelessness.
• Print the column names of homelessness.
• Print the index of homelessness.

Sorting rows Exercise

Finding interesting bits of data in a DataFrame is often easier if you change the order of the rows. You can sort the rows by passing a column name to .sort_values().

In cases where rows have the same value (this is common if you sort on a categorical variable), you may wish to break the ties by sorting on another column. You can sort on multiple columns in this way by passing a list of column names.

Sort on … Syntax one column df.sort_values("breed") multiple columns df.sort_values(["breed", "weight_kg"]) By combining .sort_values() with .head(), you can answer questions in the form, "What are the top cases where…?".

Instructions 1/3

1. Sort homelessness by the number of homeless individuals, from smallest to largest, and save this as homelessness_ind.
2. Print the head of the sorted DataFrame.
3. Sort homelessness by the number of homeless family_members in descending order, and save this as homelessness_fam.
4. Print the head of the sorted DataFrame.
5. Sort homelessness first by region (ascending), and then by number of family members (descending). Save this as homelessness_reg_fam.
6. Print the head of the sorted DataFrame.

Code Below

Subsetting columns Exercise

When working with data, you may not need all of the variables in your dataset. Square brackets ([]) can be used to select only the columns that matter to you in an order that makes sense to you. To select only "col_a" of the DataFrame df, use

df["col_a"]

To select "col_a" and "col_b" of df, use

df[["col_a", "col_b"]]

homelessness is available and pandas is loaded as pd.

Instructions 1/3

1. Create a DataFrame called individuals that contains only the individuals column of homelessness. Print the head of the result.
2. Create a DataFrame called state_fam that contains only the state and family_members columns of homelessness, in that order. Print the head of the result.
3. Create a DataFrame called ind_state that contains the individuals and state columns of homelessness, in that order. Print the head of the result.

Code Below:

Subsetting rows Exercise

A large part of data science is about finding which bits of your dataset are interesting. One of the simplest techniques for this is to find a subset of rows that match some criteria. This is sometimes known as filtering rows or selecting rows.

There are many ways to subset a DataFrame, perhaps the most common is to use relational operators to return True or False for each row, then pass that inside square brackets.

dogs[dogs["height_cm"] > 60] dogs[dogs["color"] == "tan"]

You can filter for multiple conditions at once by using the "bitwise and" operator, &.

dogs[(dogs["height_cm"] > 60) & (dogs["color"] == "tan")]

homelessness is available and pandas is loaded as pd.

Instructions 1/3

Filter homelessness for cases where the number of individuals is greater than ten thousand, assigning to ind_gt_10k. View the printed result. Filter homelessness for cases where the USA Census region is "Mountain", assigning to mountain_reg. View the printed result. Filter homelessness for cases where the number of family_members is less than one thousand and the region is "Pacific", assigning to fam_lt_1k_pac. View the printed result.

Code Below:

Subsetting rows by categorical variables Exercise

Subsetting data based on a categorical variable often involves using the "or" operator (|) to select rows from multiple categories. This can get tedious when you want all states in one of three different regions, for example. Instead, use the .isin() method, which will allow you to tackle this problem by writing one condition instead of three separate ones.

colors = ["brown", "black", "tan"] condition = dogs["color"].isin(colors) dogs[condition]

homelessness is available and pandas is loaded as pd.

Instructions 1/2

1. Filter homelessness for cases where the USA census region is "South Atlantic" or it is "Mid-Atlantic", assigning to south_mid_atlantic. View the printed result.
2. Filter homelessness for cases where the USA census state is in the list of Mojave states, canu, assigning to mojave_homelessness. View the printed result.

Code Below:

Adding new columns Exercise

You aren't stuck with just the data you are given. Instead, you can add new columns to a DataFrame. This has many names, such as transforming, mutating, and feature engineering.

You can create new columns from scratch, but it is also common to derive them from other columns, for example, by adding columns together or by changing their units.

homelessness is available and pandas is loaded as pd.

Instructions

• Add a new column to homelessness, named total, containing the sum of the individuals and family_members columns.
• Add another column to homelessness, named p_individuals, containing the proportion of homeless people in each state who are individuals.

Code Below: