<p>A database models real-life entities like professors and universities by storing them in tables. Each table contains data from a single entity type. This reduces redundancy by storing entities only once. For example, there only needs to be one row of data containing a certain company&#39;s details. Lastly, a database can be used to model the relationship between entities.</p>
<p>SQL is Structured Query Language, which is a computer language for storing, manipulating, and retrieving data stored in a relational database.</p>
<p>SQL is the standard language for Relational Database System. All the Relational Database Management Systems (RDMS) like MySQL, MS Access, Oracle, Sybase, Informix, Postgres, and SQL Server use SQL as their standard database language.</p>
<h2 id="postgresql-database">PostgreSQL Database</h2>
<p><code>information_schema</code> is a meta-database that holds information about your current database. It&#39;s not PostgreSQL specific and also available in other database management systems like MySQL or SQL Server. <code>information_schema</code> has multiple tables you can query with the known <code>SELECT * FROM</code> syntax:</p>
<ul>
<li>tables: information about all tables in your current database</li>
</ul>
<ul>
<li>columns: information about all columns in all of the tables in your current database</li>
</ul>
<pre><code class="lang-SQL">  SELECT table_schema, table_name
  FROM information_schema.tables;
</code></pre>
<pre><code>      table_schema    |              table_name
   -------------------+------------------------------
   pg_catalog         | pg_statistic
   pg_catalog         | pg_type
   pg_catalog         | pg_policy
   pg_catalog         | pg_authid
   pg_catalog         | pg_shadow
   public             | university_professors
   pg_catalog         | pg_settings
  ...
</code></pre><h2 id="commands-in-sql">Commands in SQL</h2>
<h4 id="data-definition-language-ddl-">Data Definition Language (DDL)</h4>
<ul>
<li><strong>CREATE</strong>: It creates a new table, a view of a table.</li>
<li><strong>ALTER</strong>: It modifies the existing table.</li>
<li><strong>DROP</strong>: It deletes the entire table or other objects in the database.</li>
</ul>
<h4 id="data-manipulation-language-dml-">Data Manipulation Language (DML)</h4>
<ul>
<li><strong>SELECT</strong>: It extracts certain records from one or more tables.</li>
<li><strong>INSERT</strong>: It creates a record in the existing table.</li>
<li><strong>UPDATE</strong>: It modifies the existing record of the table.</li>
<li><strong>DELETE</strong>: It deletes the records in the table and even delete the complete table.</li>
</ul>
<h2 id="looking-at-the-columns-of-a-certain-table">Looking at the Columns of a Certain Table</h2>
<p><code>information_schema</code> also holds information about columns in the &quot;columns&quot; table. Once you know the name of a table, you can query its columns by accessing the &quot;columns&quot; table.</p>
<pre><code class="lang-SQL">SELECT table_name, column_name, data_type
FROM information_schema.columns
WHERE table_name = &#39;pg_config&#39;;
</code></pre>
<pre><code class="lang-out"> table_name | column_name | data_type
 -----------+-------------+-----------
 pg_config  | name        | text
 pg_config  | setting     | text
</code></pre>
<p>Here, you see that the system table <code>pg_config</code> has only two columns – supposedly for storing name-value pairs.</p>
<h2 id="interactive-example">Interactive Example</h2>
<p>In this exercise, you&#39;ll only need information from the <code>&#39;public&#39;</code> schema, which is specified as the column <code>table_schema</code> of the <code>tables</code> and <code>columns</code> tables. The <code>&#39;public&#39;</code> schema holds information about user-defined tables and databases. The other types of <code>table_schema</code> hold system information – here, we are only interested in user-defined stuff.</p>
<p>You will get information on all table names in the current database while limiting your query to the <code>&#39;public&#39;</code> <code>table_schema</code>.</p>
<pre><code class="lang-python">## Query the right table in information_schema
SELECT table_name
FROM information_schema.tables
## Specify the correct table_schema value
WHERE table_schema = &#39;public&#39;;
</code></pre>
<p>When we run above code, it produces the following result:</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1603468261/sql2_ki8l1w.png" /></center>

<p><a href="https://campus.datacamp.com/courses/introduction-to-relational-databases-in-sql/your-first-database?ex=3">Try it for yourself</a>.</p>
<p>To learn more about relational databases in SQL, please see this video from our course <a href="https://campus.datacamp.com/courses/introduction-to-relational-databases-in-sql/your-first-database?ex=1">Introduction to Relational Databases in SQL</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/dDoFZyUHMVo?start=98" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://www.datacamp.com/courses/introduction-to-python-for-finance">Introduction to Relational Databases in SQL</a> course by Timo Grossenbacher.</p>


https://www.datacamp.com/community/tutorials/sql-database

(Tutorial) SQL Database Overview

SQL Database Overview

<p>An object-oriented approach is most useful when your code involves complex interactions of many objects. In real production code, classes can have dozens of attributes and methods with complicated logic, but the underlying structure is the same as with the most simple class.</p>
<p>Classes are like a blueprint for objects outlining possible behaviors and states that every object of a certain type could have. For example, if you say, &quot;every customer will have a phone number and an email, and will be able to place and cancel orders&quot;, you just defined a class! This way, you can talk about customers in a unified way. Then a specific Customer object is just a realization of this class with a particular state value.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1603476086/class2_b3ufes.png" /></center>

<h2 id="finding-python-classes">Finding Python Classes</h2>
<p>In Python, everything is an object. Numbers, strings, DataFrames, even functions are objects. In particular, everything you deal with in Python has a class, a blueprint associated with it under the hood. The existence of these unified interfaces is why you can use, for example, any DataFrame in the same way.</p>
<p>You can call <code>type()</code> on any Python object to find out its class. For example, the class of a <code>numpy</code> array is actually called <code>ndarray</code> (for n-dimensional array).</p>
<pre><code class="lang-py">import numpy as np
a = np.array([1,2,3,4])
print(type(a))
</code></pre>
<pre><code>numpy.ndarray
</code></pre><p>Classes incorporate information about state and behavior. State information in Python is contained in attributes and behavior information in methods.</p>
<h2 id="attributes-and-methods">Attributes and Methods</h2>
<p>Take a <code>numpy</code> array: you have already been using some of its methods and attributes!</p>
<p>For example, every <code>numpy</code> array has an attribute &quot;shape&quot; that you can access by specifying the array&#39;s name followed by a dot and shape.</p>
<h3 id="state-attributes">State &lt;--&gt; Attributes</h3>
<pre><code class="lang-py">import numpy as np
a = np.array([1,2,3,4])
</code></pre>
<pre><code class="lang-py"># shape attribute
a.shape
</code></pre>
<pre><code>(4,)
</code></pre><p>It also has methods like <code>max</code> and <code>reshape</code> which are also accessible via dot.</p>
<h3 id="behavior-methods">Behavior &lt;--&gt; Methods</h3>
<pre><code class="lang-py">import numpy as np
a = np.array([1,2,3,4])
</code></pre>
<pre><code class="lang-py"># reshape method
a.reshape(2,2)
</code></pre>
<pre><code>array([[1, 2],
       [3, 4]])
</code></pre><h2 id="creating-your-first-class">Creating Your First Class</h2>
<p>In this example, you will create an empty class <code>Employee</code>. Then you will create an object <code>emp</code> of the class <code>Employee</code> by calling <code>Employee()</code>.</p>
<p>Try printing the <code>.name</code> attribute of <code>emp</code> object in the console. What happens?</p>
<pre><code class="lang-python"># Create an empty class Employee
class Employee:
    pass
# Create an object emp of class Employee
emp = Employee()
</code></pre>
<p><a href="https://campus.datacamp.com/courses/object-oriented-programming-in-python/oop-fundamentals?ex=6">Try it for yourself</a>.</p>
<p>To learn more about object-oriented programming in python, please see this video from our course, <a href="https://campus.datacamp.com/courses/object-oriented-programming-in-python/oop-fundamentals?ex=1">Object-Oriented Programming in Python</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/SV7Is9IPNk8?start=145" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://www.datacamp.com/courses/data-processing-in-shell">Object-Oriented Programming in Python</a> course by Aliaksandra Yarosh.</p>


https://www.datacamp.com/community/tutorials/python-classes

(Tutorial) Python Classes

Python Classes

<p>The <code>boxplot()</code> function shows how the distribution of a numerical variable <code>y</code> differs across the unique levels of a second variable, <code>x</code>. To be effective, this second variable should not have too many unique levels (e.g., 10 or fewer is good; many more than this makes the plot difficult to interpret).</p>
<p>The <code>boxplot()</code> function also has a number of optional parameters, and this exercise asks you to use three of them to obtain a more informative plot:</p>
<ul>
<li><code>varwidth</code> allows for variable-width Box Plot that shows the different sizes of the data subsets.</li>
<li><code>log</code> allows for log-transformed y-values.</li>
<li><code>las</code> allows for more readable axis labels.</li>
</ul>
<h2 id="when-you-should-use-a-box-plot-">When you should use a Box Plot?</h2>
<ul>
<li><p>When you have a continuous variable, split by a categorical variable.</p>
</li>
<li><p>When you want to compare the distributions of the continuous variable for each category.</p>
</li>
</ul>
<h2 id="histogram-vs-box-plot">Histogram vs. Box Plot</h2>
<ul>
<li>Below is the comparison of a Histogram vs. a Box Plot. The line in the middle shows the median of the distribution. That is, half the monarchs started ruling before this age, and half after this age.</li>
</ul>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/c_scale,f_auto,q_auto:best/v1601058164/boxplot3_aej9he.png" /></center>


<ul>
<li>The box in the Box Plot extends from the lower quartile to the upper quartile. The lower quartile is the point where one-quarter of the values are below it. That is, one-quarter of the monarchs started ruling before this age, and three-quarters after it. Likewise, the upper quartile is the age where three quarters of the monarchs started ruling below this age. The difference between the upper quartile and the lower quartile is called the inter-quartile range.</li>
</ul>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1601058164/boxplot4_sjwpbm.png" /></center>

<p>The horizontal lines, known as &quot;whiskers&quot;, have a more complicated definition. Each bar extends to one and a half times the interquartile range, but then they are limited to reaching actual data points.</p>
<p>The technical definition is shown in the image below, but in practice, you can think of the whiskers as extending far enough that anything outside of them is an extreme value.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1601058084/Screen_Shot_2020-09-25_at_2.20.50_PM_xizuqo.png" /></center>

<p>As mentioned before, the power of Box Plots is that you can compare many distributions at once. Here, the royal houses are ordered from oldest at the top to newest at the bottom.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best,w_400/v1601058084/Screen_Shot_2020-09-25_at_2.20.59_PM_ksobqy.png" /></center>

<p>A trend is visible: since the Plantagenets in the fourteenth century, the boxes gradually move right, showing that the ages when new monarchs ascend to the throne have been increasing.</p>
<p>Godwin and Blois appear as a single line because there was only one king from each house. The Anjou house only had three kings, and forms a box with one whisker, not two.</p>
<p>Notice that the Box Plots for the houses of Denmark and Windsor show some points. These are extreme values, that is, values that are outside the range of the whiskers. Windsor&#39;s left-most outlier is Elizabeth the second, who ascended at age 26.</p>
<h2 id="-boxplot-function"><code>boxplot()</code> Function</h2>
<p>In the following example, using the formula interface, you will create a Box Plot showing the distribution of numerical <code>crim</code> values over the different distinct <code>rad</code> values from the <a href="https://www.rdocumentation.org/packages/MASS/topics/Boston">Boston</a> data frame. Then, use the <code>varwidth</code> parameter to obtain variable-width Box Plots, specify a log-transformed y-axis, and set the <code>las</code> parameter equal to 1 to obtain horizontal labels for both the x and y-axes.</p>
<p>Finally, use the <code>title()</code> function to add the title <code>&quot;Crime rate vs. radial highway index&quot;</code>.</p>
<pre><code class="lang-r"># Create a variable-width Box Plot with log y-axis &amp; horizontal labels
boxplot(crim ~ rad, data = Boston,
        varwidth = TRUE, log = &quot;y&quot;, las = 1)

# Add a title
title(&quot;Crime rate vs. radial highway index&quot;)
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best,w_400/v1601058164/boxplot2_rjcbyw.png" /></center>

<p><a href="https://campus.datacamp.com/courses/data-visualization-in-r/different-plot-types?ex=7">Try it for yourself</a>.</p>
<p>To learn more about Box Plots, please see this video from our course <a href="https://campus.datacamp.com/courses/data-visualization-for-everyone/visualizing-distributions?ex=7">Data Visualization for Everyone</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/rs0P8wLLDEQ?start=90" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://www.datacamp.com/courses/data-visualization-for-everyone">Data Visualization for Everyone</a> course by Richie Cotton and our <a href="https://learn.datacamp.com/courses/data-processing-in-shell">Data Visualization in R</a> course by Ronald Pearson.</p>


https://www.datacamp.com/community/tutorials/boxplot-in-r

(Tutorial) Box Plot in R

Box Plot in R

<p>Removing duplicates is an essential skill to get accurate counts because you often don&#39;t want to count the same thing multiple times. In Python, this could be accomplished by using the Pandas module, which has a method known as <code>drop_duplicates</code>.</p>
<p>Let&#39;s understand how to use it with the help of a few examples.</p>
<h2 id="dropping-duplicate-names">Dropping Duplicate Names</h2>
<p>Let&#39;s say you have a dataframe that contains vet visits, and the vet&#39;s office wants to know how many dogs of each breed have visited their office. However, there are dogs like Max and Stella, who have visited the vet more than once in your dataset. Hence, you cannot just count the number of each breed in the breed column.</p>
<pre><code class="lang-python">print(vet_visits)
</code></pre>
<pre><code>          date     name        breed  weight_kg
0   2018-09-02    Bella     Labrador      24.87
1   2019-06-07      Max     Labrador      28.35
2   2018-01-17   Stella    Chihuahua       1.51
3   2019-10-19     Lucy    Chow Chow      24.07
..         ...      ...          ...        ...
71  2018-01-20   Stella    Chihuahua       2.83
72  2019-06-07      Max    Chow Chow      24.01
73  2018-08-20     Lucy    Chow Chow      24.40
74  2019-04-22      Max     Labrador      28.54
</code></pre><p>To achieve this, you would remove rows that contain a dog name already listed earlier, or in other words, you will extract a dog with each name from the dataset once.</p>
<p>You would do this using the <code>drop_duplicates</code> method. It takes an argument <code>subset</code>, which is the column we want to find or duplicates based on - in this case, we want all the unique names.</p>
<pre><code class="lang-python">vet_visits.drop_duplicates(subset=&quot;name&quot;)
</code></pre>
<pre><code>          date     name        breed  weight_kg
0   2018-09-02    Bella     Labrador      24.87
1   2019-06-07      Max    Chow Chow      24.01
2   2019-03-19  Charlie       Poodle      24.95
3   2018-01-17   Stella    Chihuahua       1.51
4   2019-10-19     Lucy    Chow Chow      24.07
7   2019-03-30   Cooper    Schnauzer      16.91
10  2019-01-04   Bernie  St. Bernard      74.98
</code></pre><pre><code>(6  2019-06-07      Max     Labrador      28.35)
</code></pre><p>But, what if we have dogs with the same name?</p>
<h2 id="dropping-duplicate-pairs">Dropping Duplicate Pairs</h2>
<p>In that case, we need to consider more than just <code>name</code> when dropping duplicates. Since <code>Max</code> and <code>Max</code> are different breeds, we can drop the rows with pairs of names and breeds listed earlier in the dataset.</p>
<pre><code class="lang-python">unique_dogs = vet_visits.drop_duplicates(subset=[&quot;name&quot;, &quot;breed&quot;])
print(unique_dogs)
</code></pre>
<pre><code>          date     name        breed  weight_kg
0   2018-09-02    Bella     Labrador      24.87
1   2019-03-13      Max    Chow Chow      24.13
2   2019-03-19  Charlie       Poodle      24.95
3   2018-01-17   Stella    Chihuahua       1.51
4   2019-10-19     Lucy    Chow Chow      24.07
6   2019-06-07      Max     Labrador      28.35
7   2019-03-30   Cooper    Schnauzer      16.91
10  2019-01-04   Bernie  St. Bernard      74.98
</code></pre><p>To base our duplicate dropping on multiple columns, we can pass a list of column names to the subset argument, in this case, name and breed.</p>
<p>Now both Max&#39;s have been included.</p>
<h2 id="interactive-example">Interactive Example</h2>
<p>In this exercise, you&#39;ll create some new DataFrames using unique values from <code>sales</code>. <code>sales</code> is available, and <code>pandas</code> is imported as <code>pd</code>.</p>
<p>You will perform the following steps:</p>
<ul>
<li>First, you will remove rows of <code>sales</code> with duplicate pairs of <code>store</code> and <code>type</code> and save as <code>store_types</code> and print the head.</li>
<li>Then, you will remove rows of <code>sales</code> with duplicate pairs of <code>store</code> and <code>department</code> and save as <code>store_depts</code> and print the head.</li>
<li>Subset the rows that are holiday weeks, and drop the duplicate <code>dates</code>, saving as <code>holiday_dates</code>.</li>
<li>Finally, select the <code>date</code> column of <code>holiday_dates</code>, and print the <code>holiday_dates</code> dataframe.</li>
</ul>
<pre><code class="lang-python"># Drop duplicate store/type combinations
store_types = sales.drop_duplicates(subset=[&quot;store&quot;, &quot;type&quot;])
print(store_types.head())

# Drop duplicate store/department combinations
store_depts = sales.drop_duplicates(subset=[&quot;store&quot;, &quot;department&quot;])
print(store_depts.head())

# Subset the rows that are holiday weeks and drop duplicate dates
holiday_dates = sales[sales[&quot;is_holiday&quot;]].drop_duplicates(subset=&quot;date&quot;)

# Print date col of holiday_dates
print(holiday_dates[&quot;date&quot;])
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>      store type  department       date  weekly_sales  is_holiday  temperature_c  fuel_price_usd_per_l  unemployment
0         1    A           1 2010-02-05      24924.50       False          5.728                 0.679         8.106
901       2    A           1 2010-02-05      35034.06       False          4.550                 0.679         8.324
1798      4    A           1 2010-02-05      38724.42       False          6.533                 0.686         8.623
2699      6    A           1 2010-02-05      25619.00       False          4.683                 0.679         7.259
3593     10    B           1 2010-02-05      40212.84       False         12.411                 0.782         9.765
    store type  department       date  weekly_sales  is_holiday  temperature_c  fuel_price_usd_per_l  unemployment
0       1    A           1 2010-02-05      24924.50       False          5.728                 0.679         8.106
12      1    A           2 2010-02-05      50605.27       False          5.728                 0.679         8.106
24      1    A           3 2010-02-05      13740.12       False          5.728                 0.679         8.106
36      1    A           4 2010-02-05      39954.04       False          5.728                 0.679         8.106
48      1    A           5 2010-02-05      32229.38       False          5.728                 0.679         8.106
498    2010-09-10
691    2011-11-25
2315   2010-02-12
6735   2012-09-07
6810   2010-12-31
6815   2012-02-10
6820   2011-09-09
Name: date, dtype: datetime64[ns]
</code></pre><p><a href="https://campus.datacamp.com/courses/data-manipulation-with-pandas/aggregating-data-848344db-cbd9-4b34-a95f-949ce8af06c2?ex=7">Try it for yourself</a>.</p>
<p>To learn more about counting and aggregating data, please see this video from our course <a href="https://campus.datacamp.com/courses/data-manipulation-with-pandas/aggregating-data-848344db-cbd9-4b34-a95f-949ce8af06c2?ex=6">Data Manipulation with pandas</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/IbRokvcf6Ow?start=19" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://learn.datacamp.com/courses/data-manipulation-with-pandas">Data Manipulation with pandas</a> course by Maggie Matsui and Richie Cotton.</p>


https://www.datacamp.com/community/tutorials/pandas-drop-duplicates

(Tutorial) Pandas Drop Duplicates

Pandas Drop Duplicates

<p>Finding interesting bits of data in a DataFrame is often easier if you change the rows&#39; order. You can sort the rows by passing a column name to <code>.sort_values()</code>.</p>
<p>In cases where rows have the same value (this is common if you sort on a categorical variable), you may wish to break the ties by sorting on another column. You can sort on multiple columns in this way by passing a list of column names.</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1601054651/37_fa9fh6.png" /></center>


<h2 id="modifying-the-order-of-columns">Modifying the Order of Columns</h2>
<p>You can change the rows&#39; order by sorting them so that the most interesting data is at the top of the dataframe.</p>
<p>For example, when we apply <code>sort_values()</code> on the <code>weight_kg</code> column of the dogs dataframe, we get the lightest dog at the top, Stella the Chihuahua, and the heaviest dog at the bottom, Bernie the Saint Bernard.</p>
<pre><code class="lang-python">dogs.sort_values(&quot;weight_kg&quot;)
</code></pre>
<pre><code>      name        breed  color  height_cm  weight_kg date_of_birth
5   Stella    Chihuahua    Tan         18          2    2015-04-20
3   Cooper    Schnauzer   Gray         49         17    2011-12-11
0    Bella     Labrador  Brown         56         24    2013-07-01
1  Charlie       Poodle  Black         43         24    2016-09-16
2     Lucy    Chow Chow  Brown         46         24    2014-08-25
4      Max     Labrador  Black         59         29    2017-01-20
6   Bernie  St. Bernard  White         77         74    2018-02-27
</code></pre><p>Setting the <code>ascending</code> argument to False will sort the data the other way round, from heaviest to lightest dog.</p>
<pre><code class="lang-python">dogs.sort_values(&quot;weight_kg&quot;, ascending=False)
</code></pre>
<pre><code>      name        breed  color  height_cm  weight_kg date_of_birth
6   Bernie  St. Bernard  White         77         74    2018-02-27
4      Max     Labrador  Black         59         29    2017-01-20
0    Bella     Labrador  Brown         56         24    2013-07-01
1  Charlie       Poodle  Black         43         24    2016-09-16
2     Lucy    Chow Chow  Brown         46         24    2014-08-25
3   Cooper    Schnauzer   Gray         49         17    2011-12-11
5   Stella    Chihuahua    Tan         18          2    2015-04-20
</code></pre><h2 id="sorting-by-multiple-variables">Sorting by Multiple Variables</h2>
<p>We can sort by multiple variables by passing a list of column names to <code>sort_values</code>. Here, we sort first by weight, then by height. Now, Charlie, Lucy, and Bella are ordered from shortest to tallest, even though they all weigh the same.</p>
<pre><code class="lang-python">dogs.sort_values([&quot;weight_kg&quot;, &quot;height_cm&quot;])
</code></pre>
<pre><code>      name        breed  color  height_cm  weight_kg date_of_birth
5   Stella    Chihuahua    Tan         18          2    2015-04-20
3   Cooper    Schnauzer   Gray         49         17    2011-12-11
1  Charlie       Poodle  Black         43         24    2016-09-16
2     Lucy    Chow Chow  Brown         46         24    2014-08-25
0    Bella     Labrador  Brown         56         24    2013-07-01
4      Max     Labrador  Black         59         29    2017-01-20
6   Bernie  St. Bernard  White         77         74    2018-02-27
</code></pre><p>To change the direction values are sorted in, pass a list to the ascending argument to specify which direction sorting should be done for each variable. Now, Charlie, Lucy, and Bella are ordered from tallest to shortest.</p>
<pre><code class="lang-python">dogs.sort_values([&quot;weight_kg&quot;, &quot;height_cm&quot;], ascending=[True, False])
</code></pre>
<pre><code>      name        breed  color  height_cm  weight_kg date_of_birth
5   Stella    Chihuahua    Tan         18          2    2015-04-20
3   Cooper    Schnauzer   Gray         49         17    2011-12-11
0    Bella     Labrador  Brown         56         24    2013-07-01
2     Lucy    Chow Chow  Brown         46         24    2014-08-25
1  Charlie       Poodle  Black         43         24    2016-09-16
4      Max     Labrador  Black         59         29    2017-01-20
6   Bernie  St. Bernard  White         77         74    2018-02-27
</code></pre><h2 id="interactive-example">Interactive Example</h2>
<p>In the following example, you will sort <code>homelessness</code> by the number of homeless individuals, from smallest to largest, and save this as <code>homelessness_ind</code>. Finally, you will print the head of the sorted DataFrame.</p>
<pre><code class="lang-python"># Sort homelessness by individuals
homelessness_ind = homelessness.sort_values(&quot;individuals&quot;)

# Print the top few rows
print(homelessness_ind.head())
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>                region         state  individuals  family_members  state_pop
50            Mountain       Wyoming        434.0           205.0     577601
34  West North Central  North Dakota        467.0            75.0     758080
7       South Atlantic      Delaware        708.0           374.0     965479
39         New England  Rhode Island        747.0           354.0    1058287
45         New England       Vermont        780.0           511.0     624358
</code></pre><p><a href="https://campus.datacamp.com/courses/data-manipulation-with-pandas/transforming-data?ex=5">Try it for yourself</a>.</p>
<p>To learn more about sorting and subsetting the data, please see this video from our course <a href="https://campus.datacamp.com/courses/data-manipulation-with-pandas/transforming-data?ex=4">Data Manipulation with pandas</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/F0tQ1YG6BWQ" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://www.datacamp.com/courses/data-manipulation-with-pandas">Data Manipulation with pandas</a> course by Maggie Matsui and Richie Cotton.</p>


https://www.datacamp.com/community/tutorials/pandas-sort-values

(Tutorial) Pandas Sort Values

Pandas Sort Values

<p>Data visualization is an essential skill for data scientists. It combines statistics and design in meaningful and appropriate ways. On the one hand, data visualization is a form of graphical data analysis, emphasizing accurate representation, and data interpretation. On the other hand, data visualization relies on good design choices to make our plots attractive and aid both the understanding and communication of results. On top of that, there is an element of creativity, since data visualization is a form of visual communication at its heart.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/c_scale,f_auto,q_auto:best,w_400/v1601047315/Screen_Shot_2020-09-25_at_11.19.09_AM_mufbol.png" /></center>

<p>It&#39;s important to understand the distinction between exploratory and explanatory visualizations. Exploratory visualizations are easily-generated, data-heavy, and intended for a small specialist audience, such as yourself and your colleagues - their primary purpose is graphical data analysis. Explanatory visualizations are labor-intensive, data-specific, and intended for a broader audience, e.g., in publications or presentations - they are part of the communications process. As a data scientist, it&#39;s essential that you can quickly explore data, but you&#39;ll also be tasked with explaining your results to stake-holders. Good design begins with thinking about the audience - and sometimes that just means ourselves.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1601047315/Screen_Shot_2020-09-25_at_11.19.25_AM_xcki8p.png" /></center>

<h2 id="scatter-plot">Scatter Plot</h2>
<p>Below, we have a dataset that contains the average brain and body weights of 62 mammals.</p>
<pre><code>MASS::mammals
</code></pre><pre><code>                              body   brain
Arctic fox                   3.385   44.50
Owl monkey                   0.480   15.50
Mountain beaver              1.350    8.10
Cow                        465.000  423.00
Grey wolf                   36.330  119.50
Goat                        27.660  115.00
Roe deer                    14.830   98.20
...
Pig                        192.000  180.00
Echidna                      3.000   25.00
Brazilian tapir            160.000  169.00
Tenrec                       0.900    2.60
Phalanger                    1.620   11.40
Tree shrew                   0.104    2.50
Red fox                      4.235   50.40
</code></pre><p>To understand the relationship here, the most obvious first step is to make a scatter plot, like the one shown below:</p>
<pre><code class="lang-r">ggplot(mammals, aes(x = body, y = brain)) +
  geom_point()
</code></pre>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/c_scale,f_auto,q_auto:best,w_400/v1601047315/Screen_Shot_2020-09-25_at_11.19.41_AM_psxeu9.png" /></center>

<p>Two mammals, the African and the Asian Elephants have both very large brain and body weights, leading to a positive skew on both axes.</p>
<h2 id="linear-model">Linear Model</h2>
<p>Now, if we were to apply a linear model, it would be a poor choice since a few extreme values have a large influence.</p>
<pre><code class="lang-r">ggplot(mammals, aes(x = body, y = brain)) +
  geom_point(alpha = 0.6) +
  stat_smooth(
    method = &quot;lm&quot;,
    color = &quot;red&quot;,
    se = FALSE
  )
</code></pre>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/c_scale,f_auto,q_auto:best,w_400/v1601047315/Screen_Shot_2020-09-25_at_11.19.52_AM_sduhm8.png" /></center>

<p>Applying a log transformation of both variables allows for a better fit.</p>
<pre><code>ggplot(mammals, aes(x = body, y = brain)) +
  geom_point(alpha = 0.6) +
  coord_fixed() +
  scale_x_log10() +
  scale_y_log10() +
  stat_smooth(
    method = &quot;lm&quot;,
    color = &quot;#C42126&quot;,
    se = FALSE,
    size = 1
  )
</code></pre><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/c_scale,f_auto,q_auto:best,w_400/v1601047315/Screen_Shot_2020-09-25_at_11.20.10_AM_mkqjpw.png" /></center>


<p>So, although we began with a rough exploratory plot, it informed us about our data and led us to a meaningful result.</p>
<h3 id="anscombe-s-plots">Anscombe&#39;s Plots</h3>
<p>When we imagine a linear model, as presented on this anonymous plot, we imagine that we are describing data that looks something like this.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/c_scale,f_auto,q_auto:best,w_400/v1601047784/Screen_Shot_2020-09-25_at_11.29.23_AM_n4rynf.png" /></center>

<p>But this same model could be describing a very different set of data, such as a parabolic relationship, which calls for a different model.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best,w_400/v1601047791/Screen_Shot_2020-09-25_at_11.29.30_AM_yondtv.png" /></center>

<p>Or data in which an extreme value has a large effect. which becomes clear when the outlier is removed.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best,w_400/v1601048085/Screen_Shot_2020-09-25_at_11.34.30_AM_y0cqcq.png" /></center>

<p>And sometimes, the model may be describing a relationship where, in fact, there is none at all because some extreme values may be incorrect.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best,w_400/v1601048272/Screen_Shot_2020-09-25_at_11.37.23_AM_fiuxww.png" /></center>

<p>If we relied solely on the numerical output without plotting our data, we&#39;d have missed distinct and interesting underlying trends.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1601048272/Screen_Shot_2020-09-25_at_11.37.38_AM_ohqkck.png" /></center>

<p>We can see that data visualization is rooted in statistics and graphical data analysis, but it&#39;s also a creative process that involves some amount of trial and error.</p>
<h2 id="interactive-example">Interactive Example</h2>
<p>In the following example, you will first Load the <code>ggplot2</code> package using <a href="http://www.rdocumentation.org/packages/base/functions/library">library()</a>. Then, you will use <a href="http://www.rdocumentation.org/packages/utils/functions/str">str()</a> to explore the structure of the <code>mtcars</code> dataset.</p>
<p>Finally, you will visualize the ggplot and try to understand what ggplot does with the data.</p>
<p>You will use the <code>mtcars</code> dataset contains information on 32 cars from a 1973 issue of Motor Trend magazine. This dataset is small, intuitive, and contains a variety of continuous and categorical variables.</p>
<pre><code class="lang-python"># Load the ggplot2 package
library(ggplot2)

# Explore the mtcars data frame with str()
str(mtcars)

# Execute the following command
p &lt;- ggplot(mtcars, aes(cyl, mpg)) +
  geom_point()
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>data.frame&#39;:    32 obs. of  11 variables:
 $ mpg : num  21 21 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 ...
 $ cyl : num  6 6 4 6 8 6 8 4 4 6 ...
 $ disp: num  160 160 108 258 360 ...
 $ hp  : num  110 110 93 110 175 105 245 62 95 123 ...
 $ drat: num  3.9 3.9 3.85 3.08 3.15 2.76 3.21 3.69 3.92 3.92 ...
 $ wt  : num  2.62 2.88 2.32 3.21 3.44 ...
 $ qsec: num  16.5 17 18.6 19.4 17 ...
 $ vs  : num  0 0 1 1 0 1 0 1 1 1 ...
 $ am  : num  1 1 1 0 0 0 0 0 0 0 ...
 $ gear: num  4 4 4 3 3 3 3 4 4 4 ...
 $ carb: num  4 4 1 1 2 1 4 2 2 4 ...
</code></pre><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1601049048/ggplot2_au2hht.png" /></center>


<p><a href="https://campus.datacamp.com/courses/introduction-to-data-visualization-with-ggplot2/introduction-c90b4341-2fc9-4d0e-9c57-8d74ae7fae5f?ex=3">Try it for yourself</a>.</p>
<p>To learn more about data visualization with ggplot, please see this video from our course <a href="https://www.datacamp.com/courses/introduction-to-data-visualization-with-ggplot2/introduction-c90b4341-2fc9-4d0e-9c57-8d74ae7fae5f?ex=1">Introduction to Data Visualization with ggplot2</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/K8Al2CEd_6k?start=17" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://www.datacamp.com/courses/introduction-to-data-visualization-with-ggplot2">Introduction to Data Visualization with ggplot2</a> course by Rick Scavetta.</p>


https://www.datacamp.com/community/tutorials/graphics-with-ggplot2

(Tutorial) Graphics with ggplot2

Graphics with ggplot2

<p>If you are looking to find or replace items in a string, Python has several built-in methods that can help you search a target string for a specified substring.</p>
<h2 id="-find-method"><code>.find()</code> Method</h2>
<h3 id="syntax">Syntax</h3>
<pre><code class="lang-Python">string.find(substring, start, end)
</code></pre>
<p>Note: <code>start</code> and <code>end</code> are optional arguments.</p>
<p>From the above syntax, you can observe that the <code>.find()</code> method takes the desired substring as the mandatory argument. You can specify the other two arguments: an inclusive starting position and an exclusive ending position.</p>
<h3 id="substring-search">Substring Search</h3>
<p>In the example code, you search for <code>Waldo</code> in the string <code>Where&#39;s Waldo?</code>. The <code>.find()</code> method returns the lowest index in the string where it can find the substring, in this case, eight.</p>
<pre><code class="lang-python">my_string = &quot;Where&#39;s Waldo?&quot;
my_string.find(&quot;Waldo&quot;)
</code></pre>
<pre><code>8
</code></pre><p>If you search for <code>Wenda</code>, it returns <code>-1</code> since the substring is not found.</p>
<pre><code class="lang-python">my_string.find(&quot;Wenda&quot;)
</code></pre>
<pre><code>-1
</code></pre><p>Let&#39;s see if you can find <code>Waldo</code> between characters zero and five. In the code, you specify the starting position zero and ending position as six, since this position is not inclusive.</p>
<pre><code class="lang-python">my_string = &quot;Where&#39;s Waldo?&quot;
</code></pre>
<pre><code class="lang-python">my_string.find(&quot;Waldo&quot;, 0, 6)
</code></pre>
<pre><code class="lang-out">-1
</code></pre>
<p>The <code>.find()</code> method does not find the substring and returns <code>-1</code>, as shown above.</p>
<h2 id="-index-method"><code>.index()</code> Method</h2>
<h3 id="syntax">Syntax</h3>
<pre><code class="lang-Python">string.index(substring, start, end)
</code></pre>
<p>Note: <code>start</code> and <code>end</code> are optional arguments.</p>
<p>From the above syntax, you can observe that the <code>.index()</code> method takes the desired substring as a mandatory argument. It can take optional starting and ending positions as well.</p>
<h3 id="substring-search">Substring Search</h3>
<p>In the example, we search again for Waldo using <code>.index()</code>.</p>
<pre><code class="lang-Python">my_string = &quot;Where&#39;s Waldo?&quot;
my_string.index(&quot;Waldo&quot;)
</code></pre>
<pre><code>8
</code></pre><p>We get eight again. When we look for a substring that is not there, we have a difference.</p>
<pre><code class="lang-Python">my_string.index(&quot;Wenda&quot;)
</code></pre>
<pre><code>File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
ValueError: substring not found
</code></pre><p>The <code>.index()</code> method raises an exception, as we can see in the output. We can handle this using the try except block.</p>
<pre><code class="lang-python">my_string = &quot;Where&#39;s Waldo?&quot;
</code></pre>
<pre><code class="lang-python">try:
    my_string.index(&quot;Wenda&quot;)
except ValueError:
      print(&quot;Not found&quot;)
</code></pre>
<p>Above, you can observe the syntax. The try part will test the given code. If any error appears, the except part will be executed, obtaining the following output as a result.</p>
<pre><code>&quot;Not found&quot;
</code></pre><h2 id="-count-method"><code>.count()</code> Method</h2>
<p>The <code>.count()</code> method searches for a specified substring in the target string. It returns the number of non-overlapping occurrences. In simple words, how many times the substring is present in the string.</p>
<h3 id="syntax">Syntax</h3>
<p>The syntax of <code>.count()</code> is very similar to the other methods, as we can observe.</p>
<pre><code class="lang-Python">string.count(substring, start, end)
</code></pre>
<p>Note: <code>start</code> and <code>end</code> are optional arguments.</p>
<h3 id="substring-count">Substring Count</h3>
<p>In the example, we use the <code>.count()</code> method to get how many times fruit appears.</p>
<pre><code class="lang-python">my_string = &quot;How many fruits do you have in your fruit basket?&quot;
my_string.count(&quot;fruit&quot;)
</code></pre>
<pre><code>2
</code></pre><p>In the output, we see that is is two.</p>
<p>We can then limit the occurrences of fruit between character zero and fifteen of the string, as we can observe in the code below.</p>
<pre><code class="lang-python">my_string.count(&quot;fruit&quot;, 0, 16)
</code></pre>
<pre><code>1
</code></pre><p>The method will return <code>1</code>. Remember that the starting position is inclusive, but the ending is not.</p>
<h2 id="-replace-method"><code>.replace</code> Method</h2>
<p>Sometimes you will want to replace occurrences of a substring with a new substring. In this case, Python provides us with the <code>.replace</code> method.</p>
<h3 id="syntax">Syntax</h3>
<pre><code class="lang-Python">string.replace(old, new, count)
</code></pre>
<p>Note: <code>count</code> is an optional argument.</p>
<p>As we see in the syntax above, it takes three arguments: the substring being replaced, the new string to replace it, and an optional number that indicates how many occurrences to replace.</p>
<h3 id="replacing-a-substring">Replacing a Substring</h3>
<p>In the example code, we replace the substring house with car.</p>
<pre><code class="lang-python">my_string = &quot;The red house is between the blue house and the old house&quot;
print(my_string.replace(&quot;house&quot;, &quot;car&quot;))
</code></pre>
<pre><code>The red car is between the blue car and the old car
</code></pre><p>The method will return a copy with all house substrings replaced.</p>
<h3 id="replacing-a-specific-number-of-occurrences">Replacing a Specific Number of Occurrences</h3>
<p>In this example, we specified that we only want 2 of the occurrences to be replaced.</p>
<pre><code class="lang-python">print(my_string.replace(&quot;house&quot;, &quot;car&quot;, 2))
</code></pre>
<pre><code>The red car is between the blue car and the old house
</code></pre><p>In the output, we see that the method returns a copy of the string with the first two occurrences of house replaced by car.</p>
<h2 id="interactive-example">Interactive Example</h2>
<p>In the below example, you will:</p>
<ul>
<li>Find if the substring <code>actor</code> occurs between the characters with index <code>37</code> and <code>41</code> inclusive. If it is not detected, print the statement <code>Word not found</code>.</li>
<li>Replace <code>actor actor</code> with the substring <code>actor</code> if <code>actor</code> occurs only two repeated times.</li>
<li>Replace <code>actor actor actor</code> with the substring actor if <code>actor</code> appears three repeated times.</li>
</ul>
<pre><code class="lang-Python">for movie in movies:
      # Find if actor occurrs between 37 and 41 inclusive
    if movie.find(&quot;actor&quot;, 37, 42) == -1:
        print(&quot;Word not found&quot;)
    # Count occurrences and replace two by one
    elif movie.count(&quot;actor&quot;) == 2:  
        print(movie.replace(&quot;actor actor&quot;, &quot;actor&quot;))
    else:
        # Replace three occurrences by one
        print(movie.replace(&quot;actor actor actor&quot;, &quot;actor&quot;))
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>Word not found
I believe you I always said that the actor is amazing in every movie he has played
it&#39;s astonishing how frightening the actor norton looks with a shaved head and a swastika on his chest.
</code></pre><p><a href="https://campus.datacamp.com/courses/regular-expressions-in-python/basic-concepts-of-string-manipulation?ex=10">Try it for yourself</a>.</p>
<p>To learn more about finding and replacing strings, please see this video from our course, <a href="https://campus.datacamp.com/courses/regular-expressions-in-python/basic-concepts-of-string-manipulation?ex=9">Regular Expressions in Python</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/bJu-lc5coZw?start=16" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://learn.datacamp.com/courses/regular-expressions-in-python">Regular Expressions in Python</a> course by Maria Eugenia Inzaugarat.</p>


https://www.datacamp.com/community/tutorials/python-string-contains

Does your data contain what you need? Learn about Python strings and it's methods such as .find(), .index(), & .count() in our tutorial with code examples now!

Python String Contains Tutorial with .find(), .index() & .count()

Python String Contains

<p>You are never stuck with just the data you are given. Instead, you can add new columns to a DataFrame. This has many names, such as <em>transforming</em>, <em>mutating</em>, and <em>feature engineering</em>.</p>
<p>You can create new columns from scratch, but it is also common to derive them from other columns, for example, by adding columns together or by changing their units.</p>
<h2 id="deriving-a-column">Deriving a Column</h2>
<p>Using a dog dataset, let&#39;s say you want to add a new column to your DataFrame that has each dog&#39;s height in meters instead of centimeters.</p>
<p>On the left-hand side of the equals, you use square brackets with the name of the new column you want to create, in this case, <code>height_m</code>. On the right-hand side, you have the calculation.</p>
<pre><code class="lang-python">dogs[&quot;height_m&quot;] = dogs[&quot;height_cm&quot;] / 100
print(dogs)
</code></pre>
<pre><code>        name        breed   color   height_cm   weight_kg   date_of_birth   height_m
0      Bella     Labrador   Brown          56          24      2013-07-01       0.56
1    Charlie       Poodle   Black          43          24      2016-09-16       0.43
2       Lucy    Chow Chow   Brown          46          24      2014-08-25       0.46
3     Cooper    Schnauzer    Gray          49          17      2016-09-16       0.49
4        Max     Labrador   Black          59          29      2016-09-16       0.59
5     Stella    Chihuahua     Tan          18           2      2016-09-16       0.18
6     Bernie  St. Bernard   White          77          74      2016-09-16       0.77
</code></pre><p>Notice that both the existing and the derived column are in the dataframe you modified.</p>
<h2 id="adding-a-column">Adding a Column</h2>
<p>In this example, you will calculate doggy mass index and add it as a column to your dataframe. BMI stands for body mass index, which is calculated by weight in kilograms divided by their height in meters, squared.</p>
<pre><code class="lang-python">dogs[&quot;height_m&quot;] = dogs[&quot;weight_kg&quot;] / dogs[&quot;height_m&quot;] ** 2
print(dogs.head())
</code></pre>
<pre><code>        name        breed   color   height_cm   weight_kg   date_of_birth   height_m          bmi
0      Bella     Labrador   Brown          56          24      2013-07-01       0.56    76.530612
1    Charlie       Poodle   Black          43          24      2016-09-16       0.43   129.799892
2       Lucy    Chow Chow   Brown          46          24      2014-08-25       0.46   113.421550
3     Cooper    Schnauzer    Gray          49          17      2016-09-16       0.49    70.803832
4        Max     Labrador   Black          59          29      2016-09-16       0.59    83.309394
</code></pre><p>Again, the new column is on the left-hand side of the equals, but this time, our calculation involves two columns.</p>
<h2 id="adding-a-column-with-multiple-manipulations">Adding a Column With Multiple Manipulations</h2>
<p>The real power of pandas comes in when you combine all the skills that you have learned so far. Let&#39;s figure out the names of skinny, tall dogs.</p>
<p>First, to define the skinny dogs, you take the subset of dogs that have a BMI of less than 100. Next, you sort the height in descending order of height to get the tallest skinny dogs at the top.</p>
<p>Finally, this time you will only keep the columns you are interested in.</p>
<pre><code class="lang-python">bmi_lt_100 = dogs[dogs[&quot;bmi&quot;] &lt; 100]
bmi_lt_100_height = bmi_lt_100.short_values(&quot;height_cm&quot;, ascending=False)
bmi_lt_100_height[[&quot;name&quot;, &quot;height_cm&quot;, &quot;bmi&quot;]]
</code></pre>
<pre><code>        name      height_cm           bmi
4        Max             59     83.309394
0      Bella             56     76.530612
3     Cooper             49     70.803832
5     Stella             18     61.728395
</code></pre><p>Here, you can see that <code>Max</code> is the tallest dog with a BMI of under 100.</p>
<h2 id="interactive-example">Interactive Example</h2>
<p>In the below example, you add a new column to DataFrame <code>homelessness</code>, named <code>total</code>, containing the sum of the <code>individuals</code> and <code>family_members</code> columns. Then, add another column to <code>homelessness</code>, named <code>p_individuals</code>, containing the proportion of homeless people in each state who are individuals. Finally, print the <code>homelessness</code> dataframe.</p>
<pre><code class="lang-python"># Add total col as sum of individuals and family_members
homelessness[&quot;total&quot;] = homelessness[&quot;individuals&quot;] + homelessness[&quot;family_members&quot;]

# Add p_individuals col as proportion of individuals
homelessness[&quot;p_individuals&quot;] = homelessness[&quot;individuals&quot;] / homelessness[&quot;total&quot;]

# See the result
print(homelessness)
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>                region                 state  individuals  family_members  state_pop     total  p_individuals
0   East South Central               Alabama       2570.0           864.0    4887681    3434.0          0.748
1              Pacific                Alaska       1434.0           582.0     735139    2016.0          0.711
2             Mountain               Arizona       7259.0          2606.0    7158024    9865.0          0.736
3   West South Central              Arkansas       2280.0           432.0    3009733    2712.0          0.841
4              Pacific            California     109008.0         20964.0   39461588  129972.0          0.8
...
48      South Atlantic         West Virginia       1021.0           222.0    1804291    1243.0          0.821
49  East North Central             Wisconsin       2740.0          2167.0    5807406    4907.0          0.558
50            Mountain               Wyoming        434.0           205.0     577601     639.0          0.679
</code></pre><p><a href="https://campus.datacamp.com/courses/data-manipulation-with-pandas/transforming-data?ex=10">Try it for yourself</a>.</p>
<p>To learn more about adding new columns in pandas, please see this video from our course <a href="https://campus.datacamp.com/courses/data-manipulation-with-pandas/transforming-data?ex=9">Data Manipulation with pandas
</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/w-pGkEvGYao?start=17" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://learn.datacamp.com/courses/data-types-for-data-science-in-python">Data Manipulation with pandas</a> course by Maggie Matsui and Richie Cotton.</p>


https://www.datacamp.com/community/tutorials/pandas-add-column

(Tutorial) Pandas Add Column

Pandas Add Column

<p>One alternative to using a loop to iterate over a DataFrame is to use the pandas <code>.apply()</code> method. This function acts as a <code>map()</code> function in Python. It takes a function as an input and applies this function to an entire DataFrame.</p>
<p>If you are working with tabular data, you must specify an axis you want your function to act on (<code>0</code> for columns; and <code>1</code> for rows).</p>
<p>Much like the <code>map()</code> function, the <code>apply()</code> method can also be used with anonymous functions or lambda functions. Let&#39;s look at some <code>apply()</code> examples using baseball data.</p>
<h2 id="calculating-run-differentials-with-apply-">Calculating Run Differentials With <code>.apply()</code></h2>
<p>First, you will call the <code>.apply()</code> method on the <code>basebal_df</code> dataframe. Then use the <code>lambda</code> function to iterate over the rows of the dataframe. For every row, we grab the <code>RS</code> and <code>RA</code> columns and pass them to the <code>calc_run_diff</code> function. Finally, you will specify the <code>axis=1</code> to tell the <code>.apply()</code> method that we want to apply it on the rows instead of columns.</p>
<pre><code class="lang-python">baseball_df.apply(
    lambda row: calc_run_diff(row[&#39;RS&#39;], row[&#39;RA&#39;]),
    axis=1
)
</code></pre>
<p>You will notice that we don&#39;t need to use a <code>for</code> loop. You can collect the run differentials directly into an object called <code>run_diffs_apply</code>. After creating a new column and printing the dataframe you will notice that the results are similar to what you will get with the <code>.iterrows()</code> method.</p>
<pre><code class="lang-python">run_diffs_apply = baseball_df.apply(
         lambda row: calc_run_diff(row[&#39;RS&#39;], row[&#39;RA&#39;]),
         axis=1)
baseball_df[&#39;RD&#39;] = run_diffs_apply
print(baseball_df)
</code></pre>
<pre><code>      Team    League    year   RS    RA    W    G   Playoffs    RD
0      ARI        NL    2012  734   688   81  162          0    46
1      ATL        NL    2012  700   600   94  162          1   100
2      BAL        AL    2012  712   705   93  162          1     7
</code></pre><h2 id="interactive-example-using-apply-">Interactive Example Using <code>.apply()</code></h2>
<p>The Tampa Bay Rays want you to analyze their data.</p>
<p>They&#39;d like the following metrics:</p>
<ul>
<li>The sum of each column in the data</li>
<li>The total amount of runs scored in a year (<code>&#39;RS&#39;</code> + <code>&#39;RA&#39;</code> for each year)</li>
<li>The <code>&#39;Playoffs&#39;</code> column in text format rather than using <code>1&#39;s</code> and <code>0&#39;s</code></li>
</ul>
<p>The below function can be used to convert the <code>&#39;Playoffs&#39;</code> column to text:</p>
<pre><code class="lang-python">def text_playoffs(num_playoffs):
    if num_playoffs == 1:
        return &#39;Yes&#39;
    else:
      return &#39;No&#39;
</code></pre>
<p>Use <code>.apply()</code> to get these metrics. A DataFrame (<code>rays_df</code>) has been printed below. This DataFrame is indexed on the <code>&#39;Year&#39;</code> column.</p>
<pre><code>       RS   RA   W  Playoffs
2012  697  577  90         0
2011  707  614  91         1
2010  802  649  96         1
2009  803  754  84         0
2008  774  671  97         1
</code></pre><ul>
<li>Apply <code>sum()</code> to each column of the <code>rays_df</code> to collect the sum of each column. Be sure to specify the correct <code>axis</code>.</li>
</ul>
<pre><code class="lang-Python"># Gather sum of all columns
stat_totals = rays_df.apply(sum, axis=0)
print(stat_totals)
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>RS          3783
RA          3265
W            458
Playoffs       3
dtype: int64
</code></pre><p><a href="https://campus.datacamp.com/courses/writing-efficient-python-code/basic-pandas-optimizations?ex=8">Try it for yourself</a>.</p>
<p>To learn more about pandas alternative to looping, please see this video from our course <a href="https://campus.datacamp.com/courses/writing-efficient-python-code/basic-pandas-optimizations?ex=7">Writing Efficient Python Code</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/v6yf43eOqjM?start=90" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://learn.datacamp.com/courses/intermediate-python-for-data-science">Writing Efficient Python Code</a> course by Logan Thomas.</p>


https://www.datacamp.com/community/tutorials/pandas-apply

 Learn what is and how to use Python Pandas .apply for DataFrames. Learn how to iterate over DataFrames using the .apply() function today!

Pandas Apply for Python DataFrames

Pandas Apply

<p>When working with data, you are often going to need to count items, create dictionaries values before you know keys to store them in, or maintain order in a dictionary.</p>
<p><code>Counter</code> is a powerful tool for counting, validating, and learning more about the elements within a dataset that is found in the <code>collections</code> module. You pass an iterable (list, set, tuple) or a dictionary to the <code>Counter</code>. You can also use the <code>Counter</code> object similarly to a dictionary with key/value assignment, for example, <code>counter[key] = value</code>.</p>
<p>A common usage for <code>Counter</code> is checking data for consistency prior to using it.</p>
<h2 id="counter-module">Counter Module</h2>
<p>The <code>counter</code> module is based on dictionary; you can use all of the normal dictionary features. In this example, we have the list named <code>nyc_eatery_types</code> that contains one column of data called type from a table about eateries in NYC parks. We create a new counter based on that list and print it.</p>
<pre><code class="lang-Python">from collections import Counter
nyc_eatery_count_by_types = Counter(nyc_eatery_types)
print(nyc_eatery_count_by_types)
</code></pre>
<pre><code>Counter({&#39;Mobile Food Truck&#39;: 114, &#39;Food Cart&#39;: 74, &#39;Snack Bar&#39;: 24,
&#39;Specialty Cart&#39;: 18, &#39;Restaurant&#39;: 15, &#39;Fruit &amp; Vegetable Cart&#39;: 4})
</code></pre><p>You can see each type from the list and the number of times it was found in the list. We can also see how many restaurants are in the counter by using <code>Restaurant</code> as the index and printing it.</p>
<pre><code class="lang-python">print(nyc_eatery_count_by_types[&#39;Restaurant&#39;])
</code></pre>
<pre><code>15
</code></pre><h2 id="using-counter-to-find-the-most-common">Using Counter to Find the Most Common</h2>
<p>Counters also provide a wonderful way to find the most common values they contain. The <code>most_common()</code> method on a Counter returns a list of tuples containing the items and their count in descending order.</p>
<p>Let&#39;s print the top 3 eatery types in the NYC park system with the <code>most_common()</code> method and pass it 3 as the number items to return.</p>
<pre><code class="lang-python">print(nyc_eatery_count_by_types.most_common(3))
</code></pre>
<pre><code>[(&#39;Mobile Food Truck&#39;: 114), (&#39;Food Cart&#39;: 74), (&#39;Snack Bar&#39;: 24)]
</code></pre><p><code>most_common()</code> is excellent for frequency analytics and finding out how often an item occurs.</p>
<h2 id="interactive-example">Interactive Example</h2>
<p>In the following example, you will:</p>
<ul>
<li>Use the data from the Chicago Transit Authority on ridership.</li>
<li>Import the <code>Counter</code> object from <code>collections</code>.</li>
<li>Print the first ten items from the <code>stations</code> list.</li>
<li>Create a <code>Counter</code> of the <code>stations</code> list called <code>station_count</code>.</li>
<li>Print the <code>station_count</code>.</li>
</ul>
<pre><code class="lang-Python"># Import the Counter object
from collections import Counter

# Print the first ten items from the stations list
print(stations[:10])

# Create a Counter of the stations list: station_count
station_count = Counter(stations)

# Print the station_count
print(station_count)
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>[&#39;stationname&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;]
Counter({&#39;California-Cermak&#39;: 700, &#39;Damen-Cermak&#39;: 700, &#39;Ashland-Orange&#39;: 700, &#39;Argyle&#39;: 700, &#39;Halsted-Orange&#39;: 700, &#39;Laramie&#39;: 700, &#39;Diversey&#39;: 700, &#39;79th&#39;: 700, &#39;Clinton-Lake&#39;: 700, &#39;Monroe/Dearborn&#39;: 700, &#39;Wellington&#39;: 700, &#39;Merchandise Mart&#39;: 700, &#39;Cicero-Cermak&#39;: 700, &#39;Kedzie-Lake&#39;: 700, &#39;Southport&#39;: 700, &#39;Washington/Wells&#39;: 700, &#39;Clark/Division&#39;: 700, &#39;stationname&#39;: 1})
</code></pre><p><a href="https://campus.datacamp.com/courses/data-types-for-data-science-in-python/meet-the-collections-module?ex=2">Try it for yourself</a>.</p>
<p>To learn more about the collections module for counting, please see this video from our course <a href="https://campus.datacamp.com/courses/data-types-for-data-science-in-python/meet-the-collections-module?ex=1">Data Types for Data Science in Python</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/dfgIoThVpkc" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://www.datacamp.com/courses/data-types-for-data-science-in-python">Data Types for Data Science in Python</a> course by Jason Myers.</p>


https://www.datacamp.com/community/tutorials/python-count

(Tutorial) Python Count

Python Count

<p>If you have a DataFrame and would like to access or select a specific few rows/columns from that DataFrame, you can use square brackets or other advanced methods such as <code>loc</code> and <code>iloc</code>.</p>
<h2 id="selecting-columns-using-square-brackets">Selecting Columns Using Square Brackets</h2>
<p>Now suppose that you want to select the country column from the <code>brics</code> DataFrame. To achieve this, you will type <code>brics</code> and then the column label inside the square brackets.</p>
<h3 id="selecting-a-column">Selecting a Column</h3>
<pre><code>          country     capital        area    population
BR         Brazil    Brasilia       8.516        200.40
RU         Russia      Moscow      17.100        143.50
IN         India    New Dehli       3.286       1252.00
CH         China      Beijing       9.597       1357.00
SA  South Africa     Pretoria       1.221         52.98
</code></pre><pre><code class="lang-Python">brics[&quot;country&quot;]
</code></pre>
<pre><code>BR         Brazil
RU         Russia
IN         India
CH         China
SA  South Africa
Name: country, dtype: object
</code></pre><h3 id="checking-the-type-of-the-object">Checking the Type of the Object</h3>
<p>Let&#39;s check the type of the object that gets returned with the type function.</p>
<pre><code class="lang-Python">type(brics[&quot;country&quot;])
</code></pre>
<pre><code>pandas.core.series.Series
</code></pre><p>As we can see from the above output, we are dealing with a <code>pandas</code> series here! Series could be thought of as a one-dimensional array that could be labeled just like a DataFrame.</p>
<p>If you want to select data and keep it in a DataFrame, you will need to use double square brackets:</p>
<pre><code class="lang-Python">brics[[&quot;country&quot;]]
</code></pre>
<pre><code>BR         Brazil
RU         Russia
IN         India
CH         China
SA  South Africa
</code></pre><p>If we check the type of this output, it&#39;s a DataFrame! With only one column, though.</p>
<pre><code class="lang-Python">type(brics[[&quot;country&quot;]])
</code></pre>
<pre><code>pandas.core.frame.DataFrame
</code></pre><h3 id="selecting-multiple-columns">Selecting Multiple Columns</h3>
<p>You can extend this call to select two columns. Let&#39;s try to select <code>country</code> and <code>capital</code>.</p>
<pre><code class="lang-Python">brics[[&quot;country&quot;, &quot;capital&quot;]]
</code></pre>
<pre><code>          country     capital
BR         Brazil    Brasilia
RU         Russia      Moscow
IN         India    New Dehli
CH         China      Beijing
SA  South Africa     Pretoria
</code></pre><p>If you look at this closely, you are actually putting a list with column labels inside another set of square brackets and end up with a sub DataFrame containing only the <code>country</code> and <code>capital</code> columns.</p>
<h2 id="selecting-rows-using-square-brackets">Selecting Rows Using Square Brackets</h2>
<p>Square brackets can do more than just selecting columns. You can also use them to get rows, or observations, from a DataFrame.</p>
<h3 id="example">Example</h3>
<p>You can only select rows using square brackets if you specify a slice, like <code>0:4</code>. Also, you&#39;re using the integer indexes of the rows here, not the row labels!</p>
<p>To get the second, third, and fourth rows of <code>brics</code> DataFrame, we use the slice 1 through 4. Remember that end the of the slice is exclusive, and the index starts at zero.</p>
<pre><code class="lang-Python">brics[1:4]
</code></pre>
<pre><code>          country     capital     area    population
RU         Russia      Moscow   17.100        143.50
IN         India    New Dehli    3.286       1252.00
CH         China      Beijing    9.597       1357.00
</code></pre><p>These square brackets work, but they only offer limited functionality. Ideally, we would want something similar to 2D <code>Numpy</code> arrays, where you also use square brackets. The index, or slice, before the comma refers to the rows, and the slice after the comma refers to the columns.</p>
<p>Example of 2D <code>Numpy</code> array:</p>
<pre><code>my_array[rows, columns]
</code></pre><p>If you want to do something similar with <code>pandas</code>, you need to look at using the <code>loc</code> and <code>iloc</code> functions.</p>
<ul>
<li><code>loc</code>: label-based</li>
<li><code>iloc</code>: integer position-based</li>
</ul>
<h2 id="-loc-function"><code>loc</code> Function</h2>
<p><code>loc</code> is a technique to select parts of your data based on labels. Let&#39;s look at the brics DataFrame and get the rows for Russia.</p>
<p>To achieve this, you will put the label of interest in square brackets after <code>loc</code>.</p>
<h3 id="selecting-rows">Selecting Rows</h3>
<pre><code class="lang-Python">brics.loc[&quot;RU&quot;]
</code></pre>
<pre><code>country     Russia
capital     Moscow
area          17.1
population   143.5
Name: RU, dtype: object
</code></pre><p>We get a <code>pandas</code> series containing all of the rows information; inconveniently, though, it is shown on different lines. To get a DataFrame, we have to put the <code>RU</code> sting in another pair of brackets. We can also select multiple rows at the same time. Suppose you want to also include India and China. Simply add those row labels to the list.</p>
<pre><code class="lang-Python">brics.loc[[&quot;RU&quot;, &quot;IN&quot;, &quot;CH&quot;]]
</code></pre>
<pre><code>          country     capital     area    population
RU         Russia      Moscow   17.100        143.50
IN         India    New Dehli    3.286       1252.00
CH         China      Beijing    9.597       1357.00
</code></pre><p>The difference between using a <code>loc</code> and basic square brackets is that you can extend the selection with a comma and a specification of the columns of interest.</p>
<h3 id="selecting-rows-and-columns">Selecting Rows and Columns</h3>
<p>Let&#39;s extend the previous call to only include the <code>country</code> and <code>capital</code> columns. We add a comma and list the column labels we want to keep. The intersection gets returned.</p>
<pre><code class="lang-Python">brics.loc[[&quot;RU&quot;, &quot;IN&quot;, &quot;CH&quot;], [&quot;country&quot;, &quot;capital&quot;]]
</code></pre>
<pre><code>          country     capital
RU         Russia      Moscow
IN         India    New Dehli
CH         China      Beijing
</code></pre><p>You can also use <code>loc</code> to select all rows but only a specific number of columns. Simply replace the first list that specifies the row labels with a colon. A slice going from beginning to end. This time, we get back all of the rows but only two columns.</p>
<h3 id="selecting-all-rows-and-specific-columns">Selecting All Rows and Specific Columns</h3>
<pre><code class="lang-Python">brics.loc[:, [&quot;country&quot;, &quot;capital&quot;]]
</code></pre>
<pre><code>          country     capital
BR         Brazil    Brasilia
RU         Russia      Moscow
IN         India    New Dehli
CH         China      Beijing
SA  South Africa     Pretoria
</code></pre><h2 id="-iloc-function"><code>iloc</code> Function</h2>
<p>The <code>iloc</code> function allows you to subset <code>pandas</code> DataFrames based on their position or index.</p>
<h3 id="selecting-rows">Selecting Rows</h3>
<p>Let&#39;s use the same data and similar examples as we did for <code>loc</code>. Let&#39;s start by getting the row for Russia.</p>
<pre><code class="lang-Python">brics.iloc[[1]]
</code></pre>
<pre><code>          country     capital     area    population
RU         Russia      Moscow   17.100        143.50
</code></pre><p>To get the rows for Russia, India, and China. You can now use a list of index <code>1, 2, 3</code>.</p>
<pre><code class="lang-Python">brics.iloc[[1, 2, 3]]
</code></pre>
<pre><code>          country     capital     area    population
RU         Russia      Moscow   17.100        143.50
IN         India    New Dehli    3.286       1252.00
CH         China      Beijing    9.597       1357.00
</code></pre><h3 id="selecting-rows-and-columns">Selecting Rows and Columns</h3>
<p>Similar to <code>loc</code>, we can also select both rows and columns using <code>iloc</code>. Here, we will select rows for Russia, India, and China and columns <code>country</code> and <code>capital</code>.</p>
<pre><code class="lang-Python">brics.iloc[[1, 2, 3], [0, 1]]
</code></pre>
<pre><code>          country     capital
RU         Russia      Moscow
IN         India    New Dehli
CH         China      Beijing
</code></pre><h3 id="selecting-all-rows-and-specific-columns">Selecting All Rows and Specific Columns</h3>
<p>Finally, if you wanted to select all rows but just keep the <code>country</code> and <code>capital</code> columns, you can:</p>
<pre><code class="lang-python">brics.loc[:, [0, 1]]
</code></pre>
<pre><code>          country     capital
BR         Brazil    Brasilia
RU         Russia      Moscow
IN         India    New Dehli
CH         China      Beijing
SA  South Africa     Pretoria
</code></pre><p><code>loc</code> and <code>iloc</code> functions are pretty similar. The only difference is how you refer to columns and rows.</p>
<h2 id="interactive-example-on-selecting-a-subset-of-data">Interactive Example on Selecting a Subset of Data</h2>
<p>In the following example, the cars data is imported from a CSV files as a Pandas DataFrame. To select only the <code>cars_per_cap column</code> from cars, you can use:</p>
<pre><code>cars[&#39;cars_per_cap&#39;]
cars[[&#39;cars_per_cap&#39;]]
</code></pre><p>The single bracket version gives a Pandas Series; the double bracket version gives a Pandas DataFrame.</p>
<ul>
<li>You will use single square brackets to print out the <code>country</code> column of <code>cars</code> as a Pandas Series.</li>
<li>Then use double square brackets to print out the <code>country</code> column of <code>cars</code> as a Pandas DataFrame.</li>
<li>Finally, use the double square brackets to print out a DataFrame with both the <code>country</code> and <code>drives_right</code> columns of <code>cars</code>, in this order.</li>
</ul>
<pre><code class="lang-Python"># Import cars data
import pandas as pd
cars = pd.read_csv(&#39;cars.csv&#39;, index_col = 0)

# Print out country column as Pandas Series
print(cars[&#39;country&#39;])

# Print out country column as Pandas DataFrame
print(cars[[&#39;country&#39;]])

# Print out DataFrame with country and drives_right columns
print(cars[[&#39;country&#39;, &#39;drives_right&#39;]])
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>US     United States
AUS        Australia
JPN            Japan
IN             India
RU            Russia
MOR          Morocco
EG             Egypt
Name: country, dtype: object
           country
US   United States
AUS      Australia
JPN          Japan
IN           India
RU          Russia
MOR        Morocco
EG           Egypt
           country  drives_right
US   United States          True
AUS      Australia         False
JPN          Japan         False
IN           India         False
RU          Russia          True
MOR        Morocco          True
EG           Egypt          True
</code></pre><p><a href="https://campus.datacamp.com/courses/intermediate-python/dictionaries-pandas?ex=15">Try it for yourself</a>.</p>
<p>To learn more about pandas, please see this video from our course <a href="https://campus.datacamp.com/courses/intermediate-python/dictionaries-pandas?ex=14">Intermediate Python</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/UQIUhGWYg6Q" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://learn.datacamp.com/courses/intermediate-python-for-data-science">Intermediate Python</a> course by Hugo Bowne-Anderson.</p>


https://www.datacamp.com/community/tutorials/python-select-columns

 Use Python Pandas and select columns from DataFrames. Follow our tutorial with code examples and learn different ways to select your data today! 

Python Pandas Select Columns from DataFrames

Python Select Columns

<center>
<a target=“_blank” href="https://www.datacamp.com/learn/python/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1582839819/python_4_t5pp5t.png" /></a>
</center>

<p>Are you tired of waiting in long queues for your call to be connected to the customer service executive? Does reading FAQ&#39;s make you feel lethargic? Then you are on the right page. Can you remember the last time you communicated to a customer service agent via chat for the wrong item being delivered to you? There is a high probability that you were being communicated to by a bot rather than a customer service representative. So what exactly are bots? How do we build one? What source of code does it require? These are some of the questions which will be answered in this blog post!</p>
<p>Artificial intelligence, which brings into play machine learning and Natural language Processing (NLP) for building bot or chatbot, is specifically designed to unravel the smooth interaction between humans and computers. Chatbots are everywhere, be it a banking website, pizza store, to e-commerce shopping stores, you will find chatbots left, right, and center. Chatbots provide real-time customer service assistance on a range of pre-defined questions related to the domain it is built on. It adapts natural human language and converses with humans in a human-like manner.</p>
<p>To simplify the chatbot&#39;s definition, we can say chatbots are the evolution of Question Answer systems employing natural language processing. As per sources by the year 2024, the global conversation market&#39;s size will grow to $15.7 billion, with 30.2% being the annual growth rate. For instance, amidst the CoronaVirus Pandemic, we have witnessed thousands of hoaxes circulating on WhatsApp, such as what can be used to treat COVID or what can be beneficial in increasing immunity, or whether the virus was developed in a lab. Putting an end to such hoaxes, Facebook launched a chatbot that works as a fact-checker.</p>
<p>In this tutorial, we’ll learn Chatbot building in detail, including:</p>
<ul>
<li>Introduction</li>
<li>What is Chatbot?</li>
<li>How Does Chatbot Work?</li>
<li>Comparison of Chatbot platforms</li>
<li>Chatterbot Library</li>
<li>Building  a Chatbot using Chatterbot</li>
<li>Chatbot Testing</li>
<li>Conclusion</li>
</ul>
<h2 id="what-is-a-chatbot-">What is a chatbot?</h2>
<p>The term “chatterbot” came in existence in 1994 when Michael Mauldin created his first chatbot named “Julia”. As per the Oxford Dictionary, a chatbot is defined as “A computer program designed to simulate conversation with human users, especially over the internet.” It can be looked upon as a virtual assistant that communicates with users via text messages and helps businesses in getting close to their customers. It is a program designed to imitate the way humans communicate with each other. It can be done through a chat interface or by voice call. Developers usually design chatbots so that it is difficult to tell for users whether they are communicating with a person or a robot.</p>
<p>Chatbots helps any business/organization in accomplishing the following goals:</p>
<ul>
<li>Increases operational efficiency.</li>
<li>Automating customer request fulfillment.</li>
<li>Handling basic queries, which in turn free employees to work for complex &amp; higher value inquiries.</li>
<li>Offers Multi-language support.</li>
<li>Saves time &amp; effort by automating customer support.</li>
<li>Improves the response rate as well as customer engagement.</li>
<li>Personalization of communication</li>
</ul>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1598043769/chatterbot1_uy1uar.png" /></center>

<center><a href="https://medium.com/appedus/5-chatbot-companies-that-helps-you-automate-your-business-1acba5fa14e4">Source</a></center>

<h2 id="how-does-a-chatbot-work-">How Does a Chatbot Work?</h2>
<p>Chatbots are nothing but software applications that have an application layer, a database, and APIs. To simplify the working of the chatbot, we can say it works on pattern matching to classify text and produce a suitable response for the questions/queries addressed by the user. The chatbot responds to the user as per the program that has been fed in it. Chatbots are of different types, depending on how they are used. Mainly there are three types of chatbots, and they are as follows:</p>
<ul>
<li><p><strong>Rule-Based Chatbot:</strong> This is the basic chatbot made, the user interacts with this kind of bot by using predefined options. To get answers from these bots, users need to click on certain options. These kinds of bots collect the user&#39;s request, analyze it, and then offer results in the form of buttons. These bots are commonly used to replace frequently asked questions when it comes to complex queries; they aren&#39;t always the best solution.</p>
</li>
<li><p><strong>Independent(Keyword) Chatbots:</strong> These are machine learning bots, unlike rule-based chatbots, they analyze what the user wants and respond appropriately. These chatbots use customizable keywords and machine learning to determine how to respond to users&#39; requests effectively and efficiently.</p>
</li>
<li><p><strong>NLP (Contextual) Chatbots:</strong> These are so far the most advanced chatbots. They are a combination of best from rule-based and keyword chatbots. These chatbots use NLP to understand the context and intent in users&#39; requests and thus act accordingly. These chatbots can handle multiple requests from the same user at ease.</p>
</li>
</ul>
<h2 id="comparisons-of-chatbot-platform">Comparisons of Chatbot Platform</h2>
<p>Many platforms offer customized chatbots with automation making seamless, always on-time, best in class support services available to customers whenever they need it without the box conversational capacities. Customers are also keen to purchase from a business that they can easily connect over messages.</p>
<p>Listing down the AI chatbot building platform in 2020:</p>
<ul>
<li><p><strong>Azure Bot Service:</strong> Azure bot service offers to build a chatbot from scratch, i.e., you can build, connect, test, and deploy. It allows developers to use the open-source SDK and tools. It also enables developers to create advanced bots such as virtual assistants to handle complex queries.</p>
</li>
<li><p><strong>Botsify:</strong> This tool&#39;s uniqueness is that it enables non-technical users to build a chatbot with its intuitive interface. Bots are represented here as stories, i.e., you can create multiple stories or multiple chatbots and deploy them as per the requirement. Another great feature is the ability to save users&#39; responses to a form that can be easily exported to a CSV.</p>
</li>
<li><p><strong>Amazon Lex:</strong> Amazon lex allows developers to build conversational interfaces using text and voice. It comes up with advanced deep learning functionalities and NLP for understanding the context of the text. It provides an easy to use console for building chatbot in minutes.</p>
</li>
<li><p><strong>Mobile Monkey:</strong> Mobile monkey offers developers to build chatbots specifically for marketing purposes. It allows developers to make Facebook ad bots, SMS bots, and native web chatbots all in a single platform. They also offer ready-made chatbot templates for every business possible, which can be directly embedded on a website.</p>
</li>
<li><p><strong>ChatterOn:</strong> It claims to handle different types of rich content responses from the Bot as it allows developers to connect  different APIs at each interaction with the user. ChatterOn offers more than 20 pre-built bolts that can be put to use in a single click. It doesn’t require much coding, which makes it easier for non-technical users to build chatbots.</p>
</li>
<li><p><strong>TARS:</strong> It offers you to build a conversational landing page, which enables you to create an automated chatbot to greet you, customers, give them relevant information about their queries regarding your products, and ask for their contact simultaneously. Tars offers many pre-defined chatbot templates, which are classified into two parts - by industry and by use-case.</p>
</li>
</ul>
<p>By now, you must be curious to build a chatbot of your own. And what&#39;s better than a customizable NLP chatbot? Let&#39;s get started for building our very own chatbot in Python using library chatterbot.</p>
<h2 id="chatterbot">Chatterbot</h2>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1598043769/chatterbot2_lejxhc.png" /></center>

<p>As the name suggests, chatterbot is a python library specifically designed to generate chatbots. This algorithm uses a selection of machine learning algorithms to fabricate varying responses to users as per their requests.</p>
<p>Chatterbot makes it easier to develop chatbots that can engage in conversations. It starts by creating an untrained chatterbot that has no prior experience or knowledge regarding how to communicate. As the users enter statements, the library saves the request made by the user as well as it also saves the responses that are sent back to the users. As the  number of instances increases in chatterbot, the accuracy of the responses made by chatterbot also increases.</p>
<p>Chatterbot is trained to search the closest analogous response by finding the closest analogous request made by users that is equivalent to the new request made. Then it selects a response from the already existing responses. The USP of chatterbot is that it enables developers to create their own dataset and structures at ease.</p>
<h2 id="building-a-chatbot-using-chatterbot">Building  a Chatbot using Chatterbot</h2>
<p>Let&#39;s begin by installing the chatterbot library. For creating chatbot also need to install chatterbot corpus. Corpus - literal meaning is a collection of words. This contains a corpus of data that is included in the chatterbot module. Each corpus is nothing but a prototype of different input statements and their responses. These corpus are used by bots to train themselves. The most recommended method for installing chatterbot and chatterbot_corpus is by using pip.</p>
<p>Installation commands for terminal:</p>
<pre><code>pip install chatterbot
</code></pre><pre><code>pip install chatterbot_corpus
</code></pre><p>Installation commmands for Jupyter Notebook:</p>
<pre><code>!pip install chatterbot
</code></pre><pre><code>!pip install chatterbot_corpus
</code></pre><p>Let&#39;s first import the Chatbot class of the chatterbot module.</p>
<pre><code class="lang-python"># Importing chatterbot
from chatterbot import ChatBot
</code></pre>
<h3 id="create-chatbot-instance">Create Chatbot Instance</h3>
<p>Now, it&#39;s time for the most interesting part i.e., naming your chatbot by creating a Chatbot object. You can choose any name you want. This single line of code generates our very own new bot named Buddy. We need to specify some more parameters before running our first program.</p>
<pre><code class="lang-python"># Create object of ChatBot class
bot = ChatBot(&#39;Buddy&#39;)
</code></pre>
<pre><code>[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.
[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.
[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data]   Unzipping corpora/wordnet.zip.
</code></pre><p>You can position the storage adapter with the chatbot object. Storage Adapters allows you to connect to a particular storage unit or network. For using a storage adapter, we need to specify it. We will position the storage adapter by assigning it to the import path of the storage we want to use. Here we are using SQL Storage Adapter, which permits chatbot to connect to databases in SQL. By using the database parameter, we will create a new SQLite Database. Please follow the code below, for creating a new database for chatbot.</p>
<pre><code class="lang-python"># Create object of ChatBot class with Storage Adapter
bot = ChatBot(
    &#39;Buddy&#39;,
    storage_adapter=&#39;chatterbot.storage.SQLStorageAdapter&#39;,
    database_uri=&#39;sqlite:///database.sqlite3&#39;
)
</code></pre>
<p>You can also position the logical adapter with a chatbot object. As the name implies, Logical Adapter regulates the logic behind the chatterbot, i.e., it picks responses for any input provided to it. This parameter contains a list of logical operators. Chatterbot allows us to use a number of logical Adapters. When more than one logical adapter is put to use, the chatbot will calculate the confidence level, and the response with the highest calculated confidence will be returned as output. Here we have used two logical adapters: <code>BestMatch</code> and <code>TimeLogicAdapter</code>.</p>
<pre><code class="lang-python"># Create object of ChatBot class with Logic Adapter
bot = ChatBot(
    &#39;Buddy&#39;,  
    logic_adapters=[
        &#39;chatterbot.logic.BestMatch&#39;,
        &#39;chatterbot.logic.TimeLogicAdapter&#39;],
)
</code></pre>
<h3 id="training-the-chatbot">Training the chatbot</h3>
<p>Now the final step in making a chatbot is to train the chatbot using the modules available in chatterbot. Training a chatbot using chatterbot is as simple as providing a conversation into the chatbot database. As soon as the chatbot is given a dataset, it produces the essential entries in the chatbot&#39;s knowledge graph to represent the input and output in the right manner. Firstly, let&#39;s import the <code>ListTrainer</code>, create its object by passing the <code>Chatbot</code> object, and call the <code>train()</code> method by passing a list of sentences.</p>
<pre><code class="lang-python"># Inport ListTrainer
from chatterbot.trainers import ListTrainer

trainer = ListTrainer(bot)

trainer.train([
&#39;Hi&#39;,
&#39;Hello&#39;,
&#39;I need your assistance regarding my order&#39;,
&#39;Please, Provide me with your order id&#39;,
&#39;I have a complaint.&#39;,
&#39;Please elaborate, your concern&#39;,
&#39;How long it will take to receive an order ?&#39;,
&#39;An order takes 3-5 Business days to get delivered.&#39;,
&#39;Okay Thanks&#39;,
&#39;No Problem! Have a Good Day!&#39;
])
</code></pre>
<pre><code>List Trainer: [####################] 100%
</code></pre><h2 id="chatbot-testing">Chatbot Testing</h2>
<p>The last step of this tutorial is to test the chatterbot’s conversational skills. For testing its responses, we will call the <code>get_responses()</code> method of <code>Chatbot</code> instance.</p>
<pre><code class="lang-python"># Get a response to the input text &#39;I would like to book a flight.&#39;
response = bot.get_response(&#39;I have a complaint.&#39;)

print(&quot;Bot Response:&quot;, response)
</code></pre>
<pre><code>Bot Response: Please elaborate, your concern
</code></pre><p>We will create a while loop for our chatbot to run in. When statements are passed in the loop, we will get an appropriate response for it, as we have already entered data into our database. If we get &quot;Bye&quot; or &quot;bye&quot; statement from the user, we can put an end to the loop and stop the program.</p>
<pre><code class="lang-python">name=input(&quot;Enter Your Name: &quot;)
print(&quot;Welcome to the Bot Service! Let me know how can I help you?&quot;)
while True:
    request=input(name+&#39;:&#39;)
    if request==&#39;Bye&#39; or request ==&#39;bye&#39;:
        print(&#39;Bot: Bye&#39;)
        break
    else:
        response=bot.get_response(request)
        print(&#39;Bot:&#39;,response)
</code></pre>
<pre><code>Enter Your Name: Avinash
Welcome to the Bot Service! Let me know how can I help you?
Avinash:I need your assistance regarding my order
Bot: Please, Provide me with your order id
Avinash:12345
Bot: No Problem! Have a Good Day!
Avinash:Bye
Bot: Bye
</code></pre><h2 id="conclusion">Conclusion</h2>
<p>Congratulations, you have made it to the end of this tutorial!</p>
<p>This article was based on learning how to make a chatbot in Python using the ChatterBot library. Building a chatbot with ChatterBot was not only simple, but also, the results were accurate. With Artificial Intelligence and Machine Learning,  in advancement, everything and anything is possible to achieve whether it is creating bots with conversational skills like humans or be it anything else.<br></p>
<center>
<a target=“_blank” href="https://www.datacamp.com/learn/python/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1583330891/python_1_b5jdxv.png" /></a>
</center>   


https://www.datacamp.com/community/tutorials/building-a-chatbot-using-chatterbot

In this tutorial, you'll learn how to build a chatbot using chatterbot in Python.

Leverage Python Chatterbot and easily make your very own chatbot. Follow our step-by-step tutorial and code examples to build a website chatbox today!

Python Chatterbot: How to Make a Chatbot using Python

Building a Chatbot using Chatterbot in Python

<p>R is the most popular language for Data Science. There are many packages and libraries provided for doing different tasks. For example, there is <code>dplyr</code> and <code>data.table</code> for data manipulation, whereas libraries like <code>ggplot2</code> for data visualization and data cleaning library like <code>tidyr</code>. Also, there is a library like &#39;Shiny&#39; to create a Web application and <code>knitr</code> for the Report generation where finally <code>mlr3</code>, <code>xgboost</code>, and <code>caret</code> are used in Machine Learning.</p>
<center>
<a target=“_blank” href="https://www.datacamp.com/learn/r/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1583330891/r_4_br67yx.png" /></a>
</center>

<h2>1. <code>ggplot2</code></h2>

<p><code>ggplot2</code> is based on the &#39;Grammar of Graphics&quot;, which is a popular data visualization library. Graphs with one variable, two variables, and three variables, along with both categorical and numerical data, can be built. Also, grouping can be done through symbol, size, color, etc.
The interactive graphics can be made with the help of <code>plot.ly</code>, where the 3D image should be made from <code>plot3D</code>.</p>
<p>You can easily install the package <code>ggplot2</code> in R&#39;s console as seen below:<br></p>
<pre><code>install.packages(&quot;ggplot2&quot;)
</code></pre><p>You can easily load the package <code>ggplot2</code> by using the following syntax:<br></p>
<pre><code>library(ggplot2)
</code></pre><p>The following tutorials on DataCamp provide much detailed knowledge about &#39;ggplot2&#39;.<br></p>
<ol>
<li><a href="https://learn.datacamp.com/courses/data-visualization-with-ggplot2-1">Data Visualization with ggplot2 (Part 1)</a><br></li>
<li><a href="https://learn.datacamp.com/courses/data-visualization-with-ggplot2-2">Data Visualization with ggplot2 (Part 2)</a><br></li>
<li><a href="https://learn.datacamp.com/courses/data-visualization-with-ggplot2-3">Data Visualization with ggplot2 (Part 3)</a><br></li>
</ol>
<h2>2. <code>data.table</code></h2>

<p><code>data.table</code> is the fastest package that can handle a vast amount of data during data manipulation. It is mostly used for health care domains for genomic data and fields like business for predictive analytics. Also, the data size ranges from more than 10 GB to 100GB.</p>
<p>You can easily install the package <code>data.table</code> in R&#39;s console as seen below:<br></p>
<pre><code>install.packages(&quot;data.table&quot;)
</code></pre><p>You can easily load the package <code>data.table</code> in R as seen below:<br></p>
<pre><code>library(data.table)
</code></pre><p>You can look up to following tutorial and course in the DataCamp:</p>
<ol>
<li><a href="https://www.datacamp.com/courses/data-table-data-manipulation-r-tutorial/">Data Analysis in R, the data.table Way</a>.</li>
<li><a href="https://www.datacamp.com/community/tutorials/data-table-r-tutorial">A data.table R Tutorial: Intro to DT[i, j, by]</a>.</li>
</ol>
<h2>3. <code>dplyr</code></h2>

<p><code>dplyr</code> is the package which is used for data manipulation by providing different sets of verbs like <code>select()</code>, <code>arrange()</code>, <code>filter()</code>, <code>summarise()</code>, and <code>mutate()</code>. It can also work with computational backends like <code>dplyr</code>, <code>sparklyr</code>, and <code>dtplyr</code>.</p>
<ol>
<li><p>You can install <code>dplyr</code> through using the <code>tidyverse</code> package, which will come with the package <code>dplyr</code>.<br></p>
<pre><code>install.packages(&quot;tidyverse&quot;)
</code></pre></li>
<li><p>Alternatively, you can install <code>dplyr</code> using the following command.<br></p>
<pre><code>install.packages(&quot;dplyr&quot;)
</code></pre></li>
<li><p>You can load the package by using the following command.<br></p>
<pre><code>library(dplyr)
</code></pre></li>
</ol>
<p>The following tutorial and course in DataCamp provide detailed knowledge of <code>dplyr</code>.</p>
<ol>
<li><a href="https://www.datacamp.com/courses/data-manipulation-with-dplyr-in-r">Data Manipulation with dplyr</a></li>
<li><a href="https://www.datacamp.com/courses/joining-data-with-dplyr-in-r/">Joining Data with dplyr</a></li>
<li><a href="https://www.datacamp.com/courses/introduction-to-the-tidyverse">Introduction to the Tidyverse</a></li>
</ol>
<h2>4. <code>tidyr</code></h2>

<p><code>tidyr</code> helps to create tidy data. The significant amount of work mostly goes on when cleaning and tidying the data. Basically, tidy data consists of those datasets where every cell acts as a single value, where every row is an observation, and every column is variable.</p>
<p>You can install <code>tidyr</code> using the following command.<br></p>
<pre><code>install.packages(&quot;tidyr&quot;)
</code></pre><p>You can load <code>tidyr</code> using the following command.<br></p>
<pre><code>library(tidyr)
</code></pre><p>The following tutorial in DataCamp provides detailed knowledge in <code>tidyr</code>.
<a href="https://www.datacamp.com/courses/cleaning-data-in-r">Cleaning Data in R</a></p>
<h2>5. <code>Shiny</code></h2>

<p><code>Shiny</code> can be used to build the web application without requiring JavaScript. It can be used together with htmlwidgets, JavaScript actions, and CSS themes to have extended features. Also, it can be used to build dashboards along with the standalone web applications.</p>
<p>You can install the <code>Shiny</code> package by the following command.<br></p>
<pre><code>install.packages(&quot;shiny&quot;)
</code></pre><p>You can load <code>Shiny</code> using the following command.<br></p>
<pre><code>library(shiny)
</code></pre><p>You can visit the link mentioned below to learn more about <code>Shiny</code>.<br>
<a href="https://www.datacamp.com/tracks/shiny-fundamentals-with-r">Shiny Fundamentals with R</a></p>
<h2>6. <code>plotly</code></h2>

<p><code>plotly</code> is the graphing library used to create graphs that are interactive and can also be used with JavaScript known as <code>plotly.js</code>.</p>
<p>You can install the <code>plotly</code> package by the following command.<br></p>
<pre><code>install.packages(&quot;plotly&quot;)
</code></pre><p>You can load <code>plotly</code> using the following command.<br></p>
<pre><code>library(plotly)
</code></pre><p>You can visit the link mentioned below to learn more about <code>plotly</code>.<br>
<a href="https://www.datacamp.com/courses/intermediate-interactive-data-visualization-with-plotly-in-r">Intermediate Interactive Data Visualization with plotly in R</a></p>
<h2>7. <code>knitr</code></h2>

<p><code>knitr</code> is the package mostly used for research. It is reproducible, used for report creation, and integrates with various types of code structures like LaTeX, HTML, Markdown, LyX, etc. It was inspired by Sweave and has extended the features by adding lots of packages like a weaver, animation, cacheSweave, etc.</p>
<p>You can install the <code>knitr</code> package by the following command.<br></p>
<pre><code>install.packages(&quot;knitr&quot;)
</code></pre><p>You can load <code>knitr</code> using the following command.<br></p>
<pre><code>library(knitr)
</code></pre><p>You can visit the link mentioned below to learn more about <code>knitr</code>.<br>
<a href="https://www.datacamp.com/courses/reporting-with-r-markdown/">Reporting with R Markdown
</a></p>
<h2>8. <code>mlr3</code></h2>

<p><code>mlr3</code> package is created for doing Machine Learning. It is also efficient, which supports Object-Oriented programming where &#39;R6&#39; objects are being provided along with machine learning workflow. It is also seen as one of the extensible frameworks for clustering, regression, classification, and survival analysis.</p>
<p>You can install the <code>mlr3</code> package by the following command.<br></p>
<pre><code>install.packages(&quot;mlr3&quot;)
</code></pre><p>You can load <code>knitr</code> using the following command.<br></p>
<pre><code>library(mlr3)
</code></pre><p>You can visit the link mentioned below to learn more about <code>mlr3</code>.<br>
<a href="https://mlr3book.mlr-org.com/">mlr3Book</a></p>
<h2>9. <code>XGBoost</code></h2>

<p><code>XGBoost</code> is an implementation of the gradient boosting framework. It also provides an interface for R where the model in R&#39;s caret package is also present. Its speed and performance are faster than the implementation in H20, Spark, and Python. This package&#39;s primary use case is for machine learning tasks like classification, ranking problems, and regression.</p>
<p>You can install the <code>XGBoost</code> package by the following command.<br></p>
<pre><code>install.packages(&#39;xgboost&#39;)
</code></pre><p>You can load <code>XGBoost</code> using the following command.<br></p>
<pre><code>library(xgboost)
</code></pre><p>You can visit the link mentioned below to learn more about <code>XGBoost</code>.<br>
<a href="https://www.datacamp.com/courses/extreme-gradient-boosting-with-xgboost">Extreme Gradient Boosting with XGBoost</a></p>
<h2>10. <code>Caret</code></h2>

<p>A <code>caret</code> package is a short form of Classification And Regression Training used for predictive modeling where it provides the tools for the following process.</p>
<ol>
<li>Pre-Processing: Where data is pre-processed and also the missing data is checked.preprocess() is provided by caret for doing such task.</li>
<li>Data splitting: Splitting the training data into two similar categorical data sets is done.</li>
<li>Feature selection: Techniques which is most suitable like Recursive Feature selection can be used.</li>
<li>Training Model: caret provides many packages for machine learning algorithms.</li>
<li>Resampling for model tuning: The model can be tuned using repeated k-fold, k-fold, etc. Also, the parameter can be tuned using &#39;tuneLength.&#39;</li>
<li>Variable importance estimation: <code>vlamp()</code> can be used for any model to access the variable importance estimation.</li>
</ol>
<p>You can install the <code>caret</code> package by the following command.<br></p>
<pre><code>install.packages(&#39;caret&#39;)
</code></pre><p>You can load <code>caret</code> using the following command.<br></p>
<pre><code>library(caret)
</code></pre><p>You can visit the link mentioned below to learn more about <code>caret</code> from the author &quot;Max Kuhn&quot;.<br>
<a href="https://www.datacamp.com/courses/machine-learning-toolbox">Machine Learning with caret in R</a></p>
<h2 id="congratulations">Congratulations</h2>
<p>Congratulations, you have made it to the end of this tutorial!</p>
<p>In this tutorial, you&#39;ve learned about different packages in R used for the Data Science process. This tutorial focused on installation, loading, and finally, getting the resources to DataCamp for learning about these packages.</p>
<p><br><center>
<a target=“_blank” href="https://www.datacamp.com/learn/r/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1583330891/r_2_t6hgbx.png" /></a>
</center>  </p>


https://www.datacamp.com/community/tutorials/top-ten-most-important-packages-in-r-for-data-science

Learn about different packages in R used for data science. Including how to load them and different resources you can use to advance your skills with them.

(Tutorial) The 10 Most Important Packages in R for Data Science

The 10 Most Important Packages in R for Data Science

https://www.datacamp.com/community/tutorials/streaming-platform-analysis

In this Python tutorial, explore movie data of popular streaming platforms and build a recommendation system.

(Tutorial) Recommendation System for Streaming Platforms

Recommendation System for Streaming Platforms

<p><b>Pyspark = Python + Apache Spark</b></p>
<p>Apache Spark is a new and open-source framework used in the big data industry for real-time processing and batch processing. It supports different languages, like Python, Scala, Java, and R.</p>
<p>Apache Spark is initially written in a Java Virtual Machine(JVM) language called Scala, whereas Pyspark is like a Python API which contains a library called Py4J. This allows dynamic interaction with JVM objects.</p>
<center>
<a target=“_blank” href="https://www.datacamp.com/learn/python/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1583330891/python_3_qgciu1.png" /></a>
</center>   

<h2 id="windows-installation">Windows Installation</h2>
<p>The installation which is going to be shown is for the Windows Operating System. It consists of the installation of Java with the environment variable and Apache Spark with the environment variable.</p>
<p>The recommended pre-requisite installation is Python, which is done from <a href="https://www.python.org/downloads/">here</a>.</p>
<h2 id="java-installation">Java installation</h2>
<ol>
<li>Go to <a href="https://www.oracle.com/java/technologies/javase-jdk13-downloads.html">Download Java JDK</a>.<br>
Visit Oracle&#39;s website for the download of the Java Development Kit(JDK).<br><br></li>
<li>Move to download section consisting of operating system Windows, and in my case, it&#39;s Windows Offline(64-bit). The installer file will be downloaded.
<img src="https://i.imgur.com/9SpClJs.png" width="500" height="500"><br><br></li>
<li>Open the installer file, and the download begins.
<img src="https://i.imgur.com/thvLwKx.png" width="500" height="500"><br><br></li>
<li><p>Go to &quot;Command Prompt&quot; and type &quot;java -version&quot; to know the version and know  whether it is installed or not.
<img src="https://i.imgur.com/UBKQIES.png" width="500" height="500"></p>
</li>
<li><p>Add the Java path
<img src="https://i.imgur.com/lU2y7xE.png" width="500" height="500"></p>
</li>
<li>Go to the search bar and &quot;EDIT THE ENVIRONMENT VARIABLES.
<img SRC="https://i.imgur.com/NU0LqoR.png" width="500" height="500"></li>
<li>Click into the &quot;Environment Variables&#39;
<img src="https://i.imgur.com/SNRZyyW.png" width="500" height="500"></li>
<li>Click into &quot;New&quot; to create your new Environment variable.
<img src="https://i.imgur.com/HAs9InC.png" width="500" height="500"></li>
<li>Use Variable Name as &quot;JAVA_HOME&#39; and your Variable Value as &#39;C:\Program Files (x86)\Java\jdk1.8.0_251&#39;. This is your location of the Java file. Click &#39;OK&#39; after you&#39;ve finished the process.
<img src="https://i.imgur.com/tkUM5Pu.png" width="500" height="500"></li>
<li>Let&#39;s add the User variable and select &#39;Path&#39; and click &#39;New&#39; to create it.
<img src="https://i.imgur.com/sJZlilZ.png" width="500" height="500"></li>
<li>Add the Variable name as &#39;PATH&#39; and path value as &#39;C:\Program Files (x86)\Java\jdk1.8.0_251\bin&#39;, which is your location of Java bin file. Click &#39;OK&#39; after you&#39;ve finished the process.
<img src="https://i.imgur.com/wopPx9j.png" width="500" height="500"></li>
</ol>
<p><b>Note:</b> You can locate your Java file by going to C drive,  which is C:\Program Files (x86)\Java\jdk1.8.0_251&#39; if you&#39;ve not changed location during the download.
<img src="https://i.imgur.com/SOWtRR6.png" width="500" height="500"></p>
<h3 id="installing-pyspark">Installing Pyspark</h3>
<ol>
<li><p>Head over to the <a href="https://spark.apache.org/downloads.html">Spark homepage</a>.</p>
</li>
<li><p>Select the Spark release and package type as following and download the .tgz file.</p>
</li>
</ol>
<p><img src="https://i.imgur.com/KlKb21g.png">
<img src="https://i.imgur.com/As7McNw.png"></p>
<p>You can make a new folder called &#39;spark&#39; in the C directory and extract the given file by using &#39;Winrar&#39;, which will be helpful afterward.</p>
<h3 id="download-and-setup-winutils-exe">Download and setup winutils.exe</h3>
<p>Go to <a href="https://github.com/steveloughran/winutils">Winutils</a> choose your previously downloaded Hadoop version, then download the winutils.exe file by going inside &#39;bin&#39;. The link to my Hadoop version is: <a href="https://github.com/steveloughran/winutils/blob/master/hadoop-2.7.1/bin/winutils.exe">https://github.com/steveloughran/winutils/blob/master/hadoop-2.7.1/bin/winutils.exe</a></p>
<p>Make a new folder called &#39;winutils&#39; and inside of it create again a new folder called &#39;bin&#39;.Then put the file recently download &#39;winutils&#39; inside it.</p>
<h3 id="environment-variables">Environment variables</h3>
<ol>
<li>Let&#39;s create a new environment where variable name as &quot;hadoop_home&quot; and variable value to be the location of winutils, which is &quot;C:\winutils&quot; and click &quot;OK&quot;.<br>
<img src="https://i.imgur.com/mwUJqro.png"><br></li>
<li>For spark, also let&#39;s create a new environment where the variable name is &quot;Spark_home&quot; and the variable value to be the location of spark, which is &quot;C:\spark&quot; and click &quot;OK&quot;.<br>
<img src="https://i.imgur.com/aE1lQqq.png"><br></li>
<li>Finally, double click the &#39;path&#39; and change the following as done below where a new path is created &quot;%Spark_Home%\bin&#39; is added and click &quot;OK&quot;.<br>
<img src="https://i.imgur.com/eKDy5iB.png"></li>
</ol>
<h3 id="finalizing-pyspark-installation">Finalizing Pyspark Installation</h3>
<ol>
<li>Open Command Prompt and type the following command.<br>
<img src="https://i.imgur.com/KAVautu.png"><br></li>
<li>Once everything is successfully done, the following message is obtained.<br>
<img src="https://i.imgur.com/1O9Hj9d.png" width="700" height="700"></li>
</ol>
<h2 id="linux-installation">Linux Installation</h2>
<p>The installation which is going to be shown is for the Linux Operating System. It consists of the installation of Java with the environment variable along with Apache Spark and the environment variable.</p>
<p>The recommended pre-requisite installation is Python, which is done from <a href="https://www.python.org/downloads/source/">here</a>.</p>
<h3 id="java-installation">Java Installation</h3>
<ol>
<li>Go to <a href="https://www.oracle.com/java/technologies/javase-jdk13-downloads.html">Download Java JDK</a>.<br>
Visit Oracle&#39;s website for the download of the Java Development Kit (JDK).<br><br></li>
<li>Move to the download section consisting of the operating system Linux and download it according to your system requirement.<br>
<img src="https://i.imgur.com/8W8zNLv.png" width = "700" height = "700"><br></li>
<li>Save the file and click &quot;Ok&quot; to save in your local machine.<br>
<img src="https://i.imgur.com/gQF2eXb.png" width = "700" height = "700"></li>
<li>Go to your terminal and check the recently downloaded file using &#39;ls&#39; command.<br>
<img src="https://i.imgur.com/OqaIa1m.png"><br></li>
<li>Install the package using the following command, which will install the debian package of java, which is recently downloaded.
<img src="https://i.imgur.com/AwZ3hk4.png"></li>
<li>Finally, you can check your java version using &#39;java --version&#39; command.<br>
<img src="https://i.imgur.com/m3sCj1C.png"><br></li>
<li>For configuring environment variables, let&#39;s open the &#39;gedit&#39; text editor using the following command.<br>
<img src="https://i.imgur.com/nU9yraN.png"><br></li>
<li>Let&#39;s make the change by providing the following information where the &#39;Java&#39; path is specified.<br>
<img src="https://i.imgur.com/N5RtFrI.png"><br></li>
<li>To make a final change, let&#39;s type the following command.
<img src="https://i.imgur.com/07T8aJd.png"></li>
</ol>
<h2 id="installing-spark">Installing Spark</h2>
<ol>
<li>Head over to the <a href="https://spark.apache.org/downloads.html">Spark homepage</a>.</li>
<li>Select the Spark release and package type as following and download the .tgz file.
<img src="https://i.imgur.com/KlKb21g.png"><br>
<img src="https://i.imgur.com/As7McNw.png"><br></li>
<li>Save the file to your local machine and click &#39;Ok&#39;.<br>
<img src="https://i.imgur.com/HeDHHGA.png"><br></li>
<li>Open your terminal and go to the recently downloaded file.<br>
<img src="https://i.imgur.com/RW4svhQ.png"></li>
<li>Let&#39;s extract the file using the following command.<br>
<img src="https://i.imgur.com/9GYmv5C.png"></li>
<li>After extracting the file, the new file is created and shown using the list(&#39;ls&#39;) command.<br>
<img src="https://i.imgur.com/XQPkqhp.png"><br></li>
</ol>
<h3 id="configuring-environment-variable-in-linux">Configuring Environment Variable in Linux</h3>
<ol>
<li>Let&#39;s open the &#39;bashrc&#39; file using &#39;vim editor&#39; by the command &#39;vim ~/.bashrc&#39;.<br>
<img src="https://i.imgur.com/yToxRFy.png"><br></li>
<li>Provide the following information according to your suitable path on your computer. In my case, the following were the required path to my Spark location, Python path, and Java path. Also, first press &#39;Esc&#39; and then type &quot;:wq&quot; to save and exit from vim.<br>
<img src="https://i.imgur.com/AuGyKm5.png"><br></li>
<li>To make a final change, save, and exit. This results in accessing the pyspark command everywhere in the directory.
<img src="https://i.imgur.com/xlEUe7d.png"><br></li>
<li>Open pyspark using &#39;pyspark&#39; command, and the final message will be shown as below.
<img src="https://i.imgur.com/NVzj0eL.png">
<img src="https://i.imgur.com/Zeq2AYQ.png"></li>
</ol>
<h2 id="mac-installation">Mac Installation</h2>
<p>The installation which is going to be shown is for the Mac Operating System. It consists of the installation of Java with the environment variable along with Apache Spark and the environment variable.</p>
<p>The recommended pre-requisite installation is Python, which is done from <a href="https://www.python.org/downloads/mac-osx/">here</a>.</p>
<h3 id="java-installation">Java Installation</h3>
<ol>
<li>Go to <a href="https://www.oracle.com/java/technologies/javase-jdk13-downloads.html">Download Java JDK</a>.<br>
Visit Oracle&#39;s website for the download of the Java Development Kit (JDK).<br><br></li>
<li>Move to download section consisting of the operating system Linux and download according to your system requirement.<br>
<img src="https://i.imgur.com/yZS58VJ.png" width = "700" height = "700"><br></li>
<li>The installation of Java can be confirmed by using <code>$java --showversion</code> in the Terminal.</li>
</ol>
<h3 id="installing-apache-spark">Installing Apache Spark</h3>
<ol>
<li>Head over to the <a href="https://spark.apache.org/downloads.html">Spark homepage</a>.</li>
<li>Select the Spark release and package type as following and download the .tgz file.
<img src="https://i.imgur.com/KlKb21g.png"><br>
<img src="https://i.imgur.com/As7McNw.png"><br></li>
<li>Save the file to your local machine and click &#39;Ok&#39;.<br></li>
<li>Let&#39;s extract the file using the following command.<br>
<code>$ tar -xzf spark-2.4.6-bin-hadoop2.7.tgz</code></li>
</ol>
<h3 id="configuring-environment-variable-for-apache-spark-and-python">Configuring Environment Variable for Apache Spark and Python</h3>
<p>You need to open the ~/.bashrc or  ~/.zshrc file depending upon your current Mac version.</p>
<pre><code>export SPARK_HOME=&quot;/Downloads/spark&quot;
export PATH=$SPARK_HOME/bin:$PATH
export PYSPARK_PYTHON=python3
</code></pre><p>Open pyspark using &#39;pyspark&#39; command, and the final message will be shown as below.
<img src="https://i.imgur.com/9Mw3906.png"></p>
<h2 id="congratulations">Congratulations</h2>
<p>Congratulations, you have made it to the end of this tutorial!</p>
<p>In this tutorial, you&#39;ve learned about the installation of Pyspark, starting the installation of Java along with Apache Spark and managing the environment variables in Windows, Linux, and Mac Operating System.</p>
<p>If you would like to learn more about Pyspark, take DataCamp&#39;s <a href="https://www.datacamp.com/courses/introduction-to-pyspark">Introduction to Pyspark</a>.</p>


https://www.datacamp.com/community/tutorials/installation-of-pyspark

This tutorial will demonstrate the installation of Pyspark and hot to manage the environment variables in Windows, Linux, and Mac Operating System.

community

datacamp

Data Science Tutorials