<p>You are never stuck with just the data you are given. Instead, you can add new columns to a DataFrame. This has many names, such as <em>transforming</em>, <em>mutating</em>, and <em>feature engineering</em>.</p>
<p>You can create new columns from scratch, but it is also common to derive them from other columns, for example, by adding columns together or by changing their units.</p>
<h2 id="deriving-a-column">Deriving a Column</h2>
<p>Using a dog dataset, let&#39;s say you want to add a new column to your DataFrame that has each dog&#39;s height in meters instead of centimeters.</p>
<p>On the left-hand side of the equals, you use square brackets with the name of the new column you want to create, in this case, <code>height_m</code>. On the right-hand side, you have the calculation.</p>
<pre><code class="lang-python">dogs[&quot;height_m&quot;] = dogs[&quot;height_cm&quot;] / 100
print(dogs)
</code></pre>
<pre><code>        name        breed   color   height_cm   weight_kg   date_of_birth   height_m
0      Bella     Labrador   Brown          56          24      2013-07-01       0.56
1    Charlie       Poodle   Black          43          24      2016-09-16       0.43
2       Lucy    Chow Chow   Brown          46          24      2014-08-25       0.46
3     Cooper    Schnauzer    Gray          49          17      2016-09-16       0.49
4        Max     Labrador   Black          59          29      2016-09-16       0.59
5     Stella    Chihuahua     Tan          18           2      2016-09-16       0.18
6     Bernie  St. Bernard   White          77          74      2016-09-16       0.77
</code></pre><p>Notice that both the existing and the derived column are in the dataframe you modified.</p>
<h2 id="adding-a-column">Adding a Column</h2>
<p>In this example, you will calculate doggy mass index and add it as a column to your dataframe. BMI stands for body mass index, which is calculated by weight in kilograms divided by their height in meters, squared.</p>
<pre><code class="lang-python">dogs[&quot;height_m&quot;] = dogs[&quot;weight_kg&quot;] / dogs[&quot;height_m&quot;] ** 2
print(dogs.head())
</code></pre>
<pre><code>        name        breed   color   height_cm   weight_kg   date_of_birth   height_m          bmi
0      Bella     Labrador   Brown          56          24      2013-07-01       0.56    76.530612
1    Charlie       Poodle   Black          43          24      2016-09-16       0.43   129.799892
2       Lucy    Chow Chow   Brown          46          24      2014-08-25       0.46   113.421550
3     Cooper    Schnauzer    Gray          49          17      2016-09-16       0.49    70.803832
4        Max     Labrador   Black          59          29      2016-09-16       0.59    83.309394
</code></pre><p>Again, the new column is on the left-hand side of the equals, but this time, our calculation involves two columns.</p>
<h2 id="adding-a-column-with-multiple-manipulations">Adding a Column With Multiple Manipulations</h2>
<p>The real power of pandas comes in when you combine all the skills that you have learned so far. Let&#39;s figure out the names of skinny, tall dogs.</p>
<p>First, to define the skinny dogs, you take the subset of dogs that have a BMI of less than 100. Next, you sort the height in descending order of height to get the tallest skinny dogs at the top.</p>
<p>Finally, this time you will only keep the columns you are interested in.</p>
<pre><code class="lang-python">bmi_lt_100 = dogs[dogs[&quot;bmi&quot;] &lt; 100]
bmi_lt_100_height = bmi_lt_100.short_values(&quot;height_cm&quot;, ascending=False)
bmi_lt_100_height[[&quot;name&quot;, &quot;height_cm&quot;, &quot;bmi&quot;]]
</code></pre>
<pre><code>        name      height_cm           bmi
4        Max             59     83.309394
0      Bella             56     76.530612
3     Cooper             49     70.803832
5     Stella             18     61.728395
</code></pre><p>Here, you can see that <code>Max</code> is the tallest dog with a BMI of under 100.</p>
<h2 id="interactive-example">Interactive Example</h2>
<p>In the below example, you add a new column to DataFrame <code>homelessness</code>, named <code>total</code>, containing the sum of the <code>individuals</code> and <code>family_members</code> columns. Then, add another column to <code>homelessness</code>, named <code>p_individuals</code>, containing the proportion of homeless people in each state who are individuals. Finally, print the <code>homelessness</code> dataframe.</p>
<pre><code class="lang-python"># Add total col as sum of individuals and family_members
homelessness[&quot;total&quot;] = homelessness[&quot;individuals&quot;] + homelessness[&quot;family_members&quot;]

# Add p_individuals col as proportion of individuals
homelessness[&quot;p_individuals&quot;] = homelessness[&quot;individuals&quot;] / homelessness[&quot;total&quot;]

# See the result
print(homelessness)
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>                region                 state  individuals  family_members  state_pop     total  p_individuals
0   East South Central               Alabama       2570.0           864.0    4887681    3434.0          0.748
1              Pacific                Alaska       1434.0           582.0     735139    2016.0          0.711
2             Mountain               Arizona       7259.0          2606.0    7158024    9865.0          0.736
3   West South Central              Arkansas       2280.0           432.0    3009733    2712.0          0.841
4              Pacific            California     109008.0         20964.0   39461588  129972.0          0.8
...
48      South Atlantic         West Virginia       1021.0           222.0    1804291    1243.0          0.821
49  East North Central             Wisconsin       2740.0          2167.0    5807406    4907.0          0.558
50            Mountain               Wyoming        434.0           205.0     577601     639.0          0.679
</code></pre><p><a href="https://campus.datacamp.com/courses/data-manipulation-with-pandas/transforming-data?ex=10">Try it for yourself</a>.</p>
<p>To learn more about adding new columns in pandas, please see this video from our course <a href="https://campus.datacamp.com/courses/data-manipulation-with-pandas/transforming-data?ex=9">Data Manipulation with pandas
</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/w-pGkEvGYao?start=17" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://learn.datacamp.com/courses/data-types-for-data-science-in-python">Data Manipulation with pandas</a> course by Maggie Matsui and Richie Cotton.</p>


https://www.datacamp.com/community/tutorials/pandas-add-column

(Tutorial) Pandas Add Column

Pandas Add Column

<p>If you are looking to find or replace items in a string, Python has several built-in methods that can help you search a target string for a specified substring.</p>
<h2 id="-find-method"><code>.find()</code> Method</h2>
<h3 id="syntax">Syntax</h3>
<pre><code class="lang-Python">string.find(substring, start, end)
</code></pre>
<p>Note: <code>start</code> and <code>end</code> are optional arguments.</p>
<p>From the above syntax, you can observe that the <code>.find()</code> method takes the desired substring as the mandatory argument. You can specify the other two arguments: an inclusive starting position and an exclusive ending position.</p>
<h3 id="substring-search">Substring Search</h3>
<p>In the example code, you search for <code>Waldo</code> in the string <code>Where&#39;s Waldo?</code>. The <code>.find()</code> method returns the lowest index in the string where it can find the substring, in this case, eight.</p>
<pre><code class="lang-python">my_string = &quot;Where&#39;s Waldo?&quot;
my_string.find(&quot;Waldo&quot;)
</code></pre>
<pre><code>8
</code></pre><p>If you search for <code>Wenda</code>, it returns <code>-1</code> since the substring is not found.</p>
<pre><code class="lang-python">my_string.find(&quot;Wenda&quot;)
</code></pre>
<pre><code>-1
</code></pre><p>Let&#39;s see if you can find <code>Waldo</code> between characters zero and five. In the code, you specify the starting position zero and ending position as six, since this position is not inclusive.</p>
<pre><code class="lang-python">my_string = &quot;Where&#39;s Waldo?&quot;
</code></pre>
<pre><code class="lang-python">my_string.find(&quot;Waldo&quot;, 0, 6)
</code></pre>
<pre><code class="lang-out">-1
</code></pre>
<p>The <code>.find()</code> method does not find the substring and returns <code>-1</code>, as shown above.</p>
<h2 id="-index-method"><code>.index()</code> Method</h2>
<h3 id="syntax">Syntax</h3>
<pre><code class="lang-Python">string.index(substring, start, end)
</code></pre>
<p>Note: <code>start</code> and <code>end</code> are optional arguments.</p>
<p>From the above syntax, you can observe that the <code>.index()</code> method takes the desired substring as a mandatory argument. It can take optional starting and ending positions as well.</p>
<h3 id="substring-search">Substring Search</h3>
<p>In the example, we search again for Waldo using <code>.index()</code>.</p>
<pre><code class="lang-Python">my_string = &quot;Where&#39;s Waldo?&quot;
my_string.index(&quot;Waldo&quot;)
</code></pre>
<pre><code>8
</code></pre><p>We get eight again. When we look for a substring that is not there, we have a difference.</p>
<pre><code class="lang-Python">my_string.index(&quot;Wenda&quot;)
</code></pre>
<pre><code>File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
ValueError: substring not found
</code></pre><p>The <code>.index()</code> method raises an exception, as we can see in the output. We can handle this using the try except block.</p>
<pre><code class="lang-python">my_string = &quot;Where&#39;s Waldo?&quot;
</code></pre>
<pre><code class="lang-python">try:
    my_string.index(&quot;Wenda&quot;)
except ValueError:
      print(&quot;Not found&quot;)
</code></pre>
<p>Above, you can observe the syntax. The try part will test the given code. If any error appears, the except part will be executed, obtaining the following output as a result.</p>
<pre><code>&quot;Not found&quot;
</code></pre><h2 id="-count-method"><code>.count()</code> Method</h2>
<p>The <code>.count()</code> method searches for a specified substring in the target string. It returns the number of non-overlapping occurrences. In simple words, how many times the substring is present in the string.</p>
<h3 id="syntax">Syntax</h3>
<p>The syntax of <code>.count()</code> is very similar to the other methods, as we can observe.</p>
<pre><code class="lang-Python">string.count(substring, start, end)
</code></pre>
<p>Note: <code>start</code> and <code>end</code> are optional arguments.</p>
<h3 id="substring-count">Substring Count</h3>
<p>In the example, we use the <code>.count()</code> method to get how many times fruit appears.</p>
<pre><code class="lang-python">my_string = &quot;How many fruits do you have in your fruit basket?&quot;
my_string.count(&quot;fruit&quot;)
</code></pre>
<pre><code>2
</code></pre><p>In the output, we see that is is two.</p>
<p>We can then limit the occurrences of fruit between character zero and fifteen of the string, as we can observe in the code below.</p>
<pre><code class="lang-python">my_string.count(&quot;fruit&quot;, 0, 16)
</code></pre>
<pre><code>1
</code></pre><p>The method will return <code>1</code>. Remember that the starting position is inclusive, but the ending is not.</p>
<h2 id="-replace-method"><code>.replace</code> Method</h2>
<p>Sometimes you will want to replace occurrences of a substring with a new substring. In this case, Python provides us with the <code>.replace</code> method.</p>
<h3 id="syntax">Syntax</h3>
<pre><code class="lang-Python">string.replace(old, new, count)
</code></pre>
<p>Note: <code>count</code> is an optional argument.</p>
<p>As we see in the syntax above, it takes three arguments: the substring being replaced, the new string to replace it, and an optional number that indicates how many occurrences to replace.</p>
<h3 id="replacing-a-substring">Replacing a Substring</h3>
<p>In the example code, we replace the substring house with car.</p>
<pre><code class="lang-python">my_string = &quot;The red house is between the blue house and the old house&quot;
print(my_string.replace(&quot;house&quot;, &quot;car&quot;))
</code></pre>
<pre><code>The red car is between the blue car and the old car
</code></pre><p>The method will return a copy with all house substrings replaced.</p>
<h3 id="replacing-a-specific-number-of-occurrences">Replacing a Specific Number of Occurrences</h3>
<p>In this example, we specified that we only want 2 of the occurrences to be replaced.</p>
<pre><code class="lang-python">print(my_string.replace(&quot;house&quot;, &quot;car&quot;, 2))
</code></pre>
<pre><code>The red car is between the blue car and the old house
</code></pre><p>In the output, we see that the method returns a copy of the string with the first two occurrences of house replaced by car.</p>
<h2 id="interactive-example">Interactive Example</h2>
<p>In the below example, you will:</p>
<ul>
<li>Find if the substring <code>actor</code> occurs between the characters with index <code>37</code> and <code>41</code> inclusive. If it is not detected, print the statement <code>Word not found</code>.</li>
<li>Replace <code>actor actor</code> with the substring <code>actor</code> if <code>actor</code> occurs only two repeated times.</li>
<li>Replace <code>actor actor actor</code> with the substring actor if <code>actor</code> appears three repeated times.</li>
</ul>
<pre><code class="lang-Python">for movie in movies:
      # Find if actor occurrs between 37 and 41 inclusive
    if movie.find(&quot;actor&quot;, 37, 42) == -1:
        print(&quot;Word not found&quot;)
    # Count occurrences and replace two by one
    elif movie.count(&quot;actor&quot;) == 2:  
        print(movie.replace(&quot;actor actor&quot;, &quot;actor&quot;))
    else:
        # Replace three occurrences by one
        print(movie.replace(&quot;actor actor actor&quot;, &quot;actor&quot;))
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>Word not found
I believe you I always said that the actor is amazing in every movie he has played
it&#39;s astonishing how frightening the actor norton looks with a shaved head and a swastika on his chest.
</code></pre><p><a href="https://campus.datacamp.com/courses/regular-expressions-in-python/basic-concepts-of-string-manipulation?ex=10">Try it for yourself</a>.</p>
<p>To learn more about finding and replacing strings, please see this video from our course, <a href="https://campus.datacamp.com/courses/regular-expressions-in-python/basic-concepts-of-string-manipulation?ex=9">Regular Expressions in Python</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/bJu-lc5coZw?start=16" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://learn.datacamp.com/courses/regular-expressions-in-python">Regular Expressions in Python</a> course by Maria Eugenia Inzaugarat.</p>


https://www.datacamp.com/community/tutorials/python-string-contains

(Tutorial) Python String Contains

Python String Contains

<p>When working with data, you are often going to need to count items, create dictionaries values before you know keys to store them in, or maintain order in a dictionary.</p>
<p><code>Counter</code> is a powerful tool for counting, validating, and learning more about the elements within a dataset that is found in the <code>collections</code> module. You pass an iterable (list, set, tuple) or a dictionary to the <code>Counter</code>. You can also use the <code>Counter</code> object similarly to a dictionary with key/value assignment, for example, <code>counter[key] = value</code>.</p>
<p>A common usage for <code>Counter</code> is checking data for consistency prior to using it.</p>
<h2 id="counter-module">Counter Module</h2>
<p>The <code>counter</code> module is based on dictionary; you can use all of the normal dictionary features. In this example, we have the list named <code>nyc_eatery_types</code> that contains one column of data called type from a table about eateries in NYC parks. We create a new counter based on that list and print it.</p>
<pre><code class="lang-Python">from collections import Counter
nyc_eatery_count_by_types = Counter(nyc_eatery_types)
print(nyc_eatery_count_by_types)
</code></pre>
<pre><code>Counter({&#39;Mobile Food Truck&#39;: 114, &#39;Food Cart&#39;: 74, &#39;Snack Bar&#39;: 24,
&#39;Specialty Cart&#39;: 18, &#39;Restaurant&#39;: 15, &#39;Fruit &amp; Vegetable Cart&#39;: 4})
</code></pre><p>You can see each type from the list and the number of times it was found in the list. We can also see how many restaurants are in the counter by using <code>Restaurant</code> as the index and printing it.</p>
<pre><code class="lang-python">print(nyc_eatery_count_by_types[&#39;Restaurant&#39;])
</code></pre>
<pre><code>15
</code></pre><h2 id="using-counter-to-find-the-most-common">Using Counter to Find the Most Common</h2>
<p>Counters also provide a wonderful way to find the most common values they contain. The <code>most_common()</code> method on a Counter returns a list of tuples containing the items and their count in descending order.</p>
<p>Let&#39;s print the top 3 eatery types in the NYC park system with the <code>most_common()</code> method and pass it 3 as the number items to return.</p>
<pre><code class="lang-python">print(nyc_eatery_count_by_types.most_common(3))
</code></pre>
<pre><code>[(&#39;Mobile Food Truck&#39;: 114), (&#39;Food Cart&#39;: 74), (&#39;Snack Bar&#39;: 24)]
</code></pre><p><code>most_common()</code> is excellent for frequency analytics and finding out how often an item occurs.</p>
<h2 id="interactive-example">Interactive Example</h2>
<p>In the following example, you will:</p>
<ul>
<li>Use the data from the Chicago Transit Authority on ridership.</li>
<li>Import the <code>Counter</code> object from <code>collections</code>.</li>
<li>Print the first ten items from the <code>stations</code> list.</li>
<li>Create a <code>Counter</code> of the <code>stations</code> list called <code>station_count</code>.</li>
<li>Print the <code>station_count</code>.</li>
</ul>
<pre><code class="lang-Python"># Import the Counter object
from collections import Counter

# Print the first ten items from the stations list
print(stations[:10])

# Create a Counter of the stations list: station_count
station_count = Counter(stations)

# Print the station_count
print(station_count)
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>[&#39;stationname&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;, &#39;Austin-Forest Park&#39;]
Counter({&#39;California-Cermak&#39;: 700, &#39;Damen-Cermak&#39;: 700, &#39;Ashland-Orange&#39;: 700, &#39;Argyle&#39;: 700, &#39;Halsted-Orange&#39;: 700, &#39;Laramie&#39;: 700, &#39;Diversey&#39;: 700, &#39;79th&#39;: 700, &#39;Clinton-Lake&#39;: 700, &#39;Monroe/Dearborn&#39;: 700, &#39;Wellington&#39;: 700, &#39;Merchandise Mart&#39;: 700, &#39;Cicero-Cermak&#39;: 700, &#39;Kedzie-Lake&#39;: 700, &#39;Southport&#39;: 700, &#39;Washington/Wells&#39;: 700, &#39;Clark/Division&#39;: 700, &#39;stationname&#39;: 1})
</code></pre><p><a href="https://campus.datacamp.com/courses/data-types-for-data-science-in-python/meet-the-collections-module?ex=2">Try it for yourself</a>.</p>
<p>To learn more about the collections module for counting, please see this video from our course <a href="https://campus.datacamp.com/courses/data-types-for-data-science-in-python/meet-the-collections-module?ex=1">Data Types for Data Science in Python</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/dfgIoThVpkc" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://www.datacamp.com/courses/data-types-for-data-science-in-python">Data Types for Data Science in Python</a> course by Jason Myers.</p>


https://www.datacamp.com/community/tutorials/python-count

(Tutorial) Python Count

Python Count

<p>One alternative to using a loop to iterate over a DataFrame is to use the pandas <code>.apply()</code> method. This function acts as a <code>map()</code> function in Python. It takes a function as an input and applies this function to an entire DataFrame.</p>
<p>If you are working with tabular data, you must specify an axis you want your function to act on (<code>0</code> for columns; and <code>1</code> for rows).</p>
<p>Much like the <code>map()</code> function, the <code>apply()</code> method can also be used with anonymous functions or lambda functions. Let&#39;s look at some <code>apply()</code> examples using baseball data.</p>
<h2 id="calculating-run-differentials-with-apply-">Calculating Run Differentials With <code>.apply()</code></h2>
<p>First, you will call the <code>.apply()</code> method on the <code>basebal_df</code> dataframe. Then use the <code>lambda</code> function to iterate over the rows of the dataframe. For every row, we grab the <code>RS</code> and <code>RA</code> columns and pass them to the <code>calc_run_diff</code> function. Finally, you will specify the <code>axis=1</code> to tell the <code>.apply()</code> method that we want to apply it on the rows instead of columns.</p>
<pre><code class="lang-python">baseball_df.apply(
    lambda row: calc_run_diff(row[&#39;RS&#39;], row[&#39;RA&#39;]),
    axis=1
)
</code></pre>
<p>You will notice that we don&#39;t need to use a <code>for</code> loop. You can collect the run differentials directly into an object called <code>run_diffs_apply</code>. After creating a new column and printing the dataframe you will notice that the results are similar to what you will get with the <code>.iterrows()</code> method.</p>
<pre><code class="lang-python">run_diffs_apply = baseball_df.apply(
         lambda row: calc_run_diff(row[&#39;RS&#39;], row[&#39;RA&#39;]),
         axis=1)
baseball_df[&#39;RD&#39;] = run_diffs_apply
print(baseball_df)
</code></pre>
<pre><code>      Team    League    year   RS    RA    W    G   Playoffs    RD
0      ARI        NL    2012  734   688   81  162          0    46
1      ATL        NL    2012  700   600   94  162          1   100
2      BAL        AL    2012  712   705   93  162          1     7
</code></pre><h2 id="interactive-example-using-apply-">Interactive Example Using <code>.apply()</code></h2>
<p>The Tampa Bay Rays want you to analyze their data.</p>
<p>They&#39;d like the following metrics:</p>
<ul>
<li>The sum of each column in the data</li>
<li>The total amount of runs scored in a year (<code>&#39;RS&#39;</code> + <code>&#39;RA&#39;</code> for each year)</li>
<li>The <code>&#39;Playoffs&#39;</code> column in text format rather than using <code>1&#39;s</code> and <code>0&#39;s</code></li>
</ul>
<p>The below function can be used to convert the <code>&#39;Playoffs&#39;</code> column to text:</p>
<pre><code class="lang-python">def text_playoffs(num_playoffs):
    if num_playoffs == 1:
        return &#39;Yes&#39;
    else:
      return &#39;No&#39;
</code></pre>
<p>Use <code>.apply()</code> to get these metrics. A DataFrame (<code>rays_df</code>) has been printed below. This DataFrame is indexed on the <code>&#39;Year&#39;</code> column.</p>
<pre><code>       RS   RA   W  Playoffs
2012  697  577  90         0
2011  707  614  91         1
2010  802  649  96         1
2009  803  754  84         0
2008  774  671  97         1
</code></pre><ul>
<li>Apply <code>sum()</code> to each column of the <code>rays_df</code> to collect the sum of each column. Be sure to specify the correct <code>axis</code>.</li>
</ul>
<pre><code class="lang-Python"># Gather sum of all columns
stat_totals = rays_df.apply(sum, axis=0)
print(stat_totals)
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>RS          3783
RA          3265
W            458
Playoffs       3
dtype: int64
</code></pre><p><a href="https://campus.datacamp.com/courses/writing-efficient-python-code/basic-pandas-optimizations?ex=8">Try it for yourself</a>.</p>
<p>To learn more about pandas alternative to looping, please see this video from our course <a href="https://campus.datacamp.com/courses/writing-efficient-python-code/basic-pandas-optimizations?ex=7">Writing Efficient Python Code</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/v6yf43eOqjM?start=90" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://learn.datacamp.com/courses/intermediate-python-for-data-science">Writing Efficient Python Code</a> course by Logan Thomas.</p>


https://www.datacamp.com/community/tutorials/pandas-apply

(Tutorial) Pandas Apply

Pandas Apply

<p>If you have a DataFrame and would like to access or select a specific few rows/columns from that DataFrame, you can use square brackets or other advanced methods such as <code>loc</code> and <code>iloc</code>.</p>
<h2 id="selecting-columns-using-square-brackets">Selecting Columns Using Square Brackets</h2>
<p>Now suppose that you want to select the country column from the <code>brics</code> DataFrame. To achieve this, you will type <code>brics</code> and then the column label inside the square brackets.</p>
<h3 id="selecting-a-column">Selecting a Column</h3>
<pre><code>          country     capital        area    population
BR         Brazil    Brasilia       8.516        200.40
RU         Russia      Moscow      17.100        143.50
IN         India    New Dehli       3.286       1252.00
CH         China      Beijing       9.597       1357.00
SA  South Africa     Pretoria       1.221         52.98
</code></pre><pre><code class="lang-Python">brics[&quot;country&quot;]
</code></pre>
<pre><code>BR         Brazil
RU         Russia
IN         India
CH         China
SA  South Africa
Name: country, dtype: object
</code></pre><h3 id="checking-the-type-of-the-object">Checking the Type of the Object</h3>
<p>Let&#39;s check the type of the object that gets returned with the type function.</p>
<pre><code class="lang-Python">type(brics[&quot;country&quot;])
</code></pre>
<pre><code>pandas.core.series.Series
</code></pre><p>As we can see from the above output, we are dealing with a <code>pandas</code> series here! Series could be thought of as a one-dimensional array that could be labeled just like a DataFrame.</p>
<p>If you want to select data and keep it in a DataFrame, you will need to use double square brackets:</p>
<pre><code class="lang-Python">brics[[&quot;country&quot;]]
</code></pre>
<pre><code>BR         Brazil
RU         Russia
IN         India
CH         China
SA  South Africa
</code></pre><p>If we check the type of this output, it&#39;s a DataFrame! With only one column, though.</p>
<pre><code class="lang-Python">type(brics[[&quot;country&quot;]])
</code></pre>
<pre><code>pandas.core.frame.DataFrame
</code></pre><h3 id="selecting-multiple-columns">Selecting Multiple Columns</h3>
<p>You can extend this call to select two columns. Let&#39;s try to select <code>country</code> and <code>capital</code>.</p>
<pre><code class="lang-Python">brics[[&quot;country&quot;, &quot;capital&quot;]]
</code></pre>
<pre><code>          country     capital
BR         Brazil    Brasilia
RU         Russia      Moscow
IN         India    New Dehli
CH         China      Beijing
SA  South Africa     Pretoria
</code></pre><p>If you look at this closely, you are actually putting a list with column labels inside another set of square brackets and end up with a sub DataFrame containing only the <code>country</code> and <code>capital</code> columns.</p>
<h2 id="selecting-rows-using-square-brackets">Selecting Rows Using Square Brackets</h2>
<p>Square brackets can do more than just selecting columns. You can also use them to get rows, or observations, from a DataFrame.</p>
<h3 id="example">Example</h3>
<p>You can only select rows using square brackets if you specify a slice, like <code>0:4</code>. Also, you&#39;re using the integer indexes of the rows here, not the row labels!</p>
<p>To get the second, third, and fourth rows of <code>brics</code> DataFrame, we use the slice 1 through 4. Remember that end the of the slice is exclusive, and the index starts at zero.</p>
<pre><code class="lang-Python">brics[1:4]
</code></pre>
<pre><code>          country     capital     area    population
RU         Russia      Moscow   17.100        143.50
IN         India    New Dehli    3.286       1252.00
CH         China      Beijing    9.597       1357.00
</code></pre><p>These square brackets work, but they only offer limited functionality. Ideally, we would want something similar to 2D <code>Numpy</code> arrays, where you also use square brackets. The index, or slice, before the comma refers to the rows, and the slice after the comma refers to the columns.</p>
<p>Example of 2D <code>Numpy</code> array:</p>
<pre><code>my_array[rows, columns]
</code></pre><p>If you want to do something similar with <code>pandas</code>, you need to look at using the <code>loc</code> and <code>iloc</code> functions.</p>
<ul>
<li><code>loc</code>: label-based</li>
<li><code>iloc</code>: integer position-based</li>
</ul>
<h2 id="-loc-function"><code>loc</code> Function</h2>
<p><code>loc</code> is a technique to select parts of your data based on labels. Let&#39;s look at the brics DataFrame and get the rows for Russia.</p>
<p>To achieve this, you will put the label of interest in square brackets after <code>loc</code>.</p>
<h3 id="selecting-rows">Selecting Rows</h3>
<pre><code class="lang-Python">brics.loc[&quot;RU&quot;]
</code></pre>
<pre><code>country     Russia
capital     Moscow
area          17.1
population   143.5
Name: RU, dtype: object
</code></pre><p>We get a <code>pandas</code> series containing all of the rows information; inconveniently, though, it is shown on different lines. To get a DataFrame, we have to put the <code>RU</code> sting in another pair of brackets. We can also select multiple rows at the same time. Suppose you want to also include India and China. Simply add those row labels to the list.</p>
<pre><code class="lang-Python">brics.loc[[&quot;RU&quot;, &quot;IN&quot;, &quot;CH&quot;]]
</code></pre>
<pre><code>          country     capital     area    population
RU         Russia      Moscow   17.100        143.50
IN         India    New Dehli    3.286       1252.00
CH         China      Beijing    9.597       1357.00
</code></pre><p>The difference between using a <code>loc</code> and basic square brackets is that you can extend the selection with a comma and a specification of the columns of interest.</p>
<h3 id="selecting-rows-and-columns">Selecting Rows and Columns</h3>
<p>Let&#39;s extend the previous call to only include the <code>country</code> and <code>capital</code> columns. We add a comma and list the column labels we want to keep. The intersection gets returned.</p>
<pre><code class="lang-Python">brics.loc[[&quot;RU&quot;, &quot;IN&quot;, &quot;CH&quot;], [&quot;country&quot;, &quot;capital&quot;]]
</code></pre>
<pre><code>          country     capital
RU         Russia      Moscow
IN         India    New Dehli
CH         China      Beijing
</code></pre><p>You can also use <code>loc</code> to select all rows but only a specific number of columns. Simply replace the first list that specifies the row labels with a colon. A slice going from beginning to end. This time, we get back all of the rows but only two columns.</p>
<h3 id="selecting-all-rows-and-specific-columns">Selecting All Rows and Specific Columns</h3>
<pre><code class="lang-Python">brics.loc[:, [&quot;country&quot;, &quot;capital&quot;]]
</code></pre>
<pre><code>          country     capital
BR         Brazil    Brasilia
RU         Russia      Moscow
IN         India    New Dehli
CH         China      Beijing
SA  South Africa     Pretoria
</code></pre><h2 id="-iloc-function"><code>iloc</code> Function</h2>
<p>The <code>iloc</code> function allows you to subset <code>pandas</code> DataFrames based on their position or index.</p>
<h3 id="selecting-rows">Selecting Rows</h3>
<p>Let&#39;s use the same data and similar examples as we did for <code>loc</code>. Let&#39;s start by getting the row for Russia.</p>
<pre><code class="lang-Python">brics.iloc[[1]]
</code></pre>
<pre><code>          country     capital     area    population
RU         Russia      Moscow   17.100        143.50
</code></pre><p>To get the rows for Russia, India, and China. You can now use a list of index <code>1, 2, 3</code>.</p>
<pre><code class="lang-Python">brics.iloc[[1, 2, 3]]
</code></pre>
<pre><code>          country     capital     area    population
RU         Russia      Moscow   17.100        143.50
IN         India    New Dehli    3.286       1252.00
CH         China      Beijing    9.597       1357.00
</code></pre><h3 id="selecting-rows-and-columns">Selecting Rows and Columns</h3>
<p>Similar to <code>loc</code>, we can also select both rows and columns using <code>iloc</code>. Here, we will select rows for Russia, India, and China and columns <code>country</code> and <code>capital</code>.</p>
<pre><code class="lang-Python">brics.iloc[[1, 2, 3], [0, 1]]
</code></pre>
<pre><code>          country     capital
RU         Russia      Moscow
IN         India    New Dehli
CH         China      Beijing
</code></pre><h3 id="selecting-all-rows-and-specific-columns">Selecting All Rows and Specific Columns</h3>
<p>Finally, if you wanted to select all rows but just keep the <code>country</code> and <code>capital</code> columns, you can:</p>
<pre><code class="lang-python">brics.loc[:, [0, 1]]
</code></pre>
<pre><code>          country     capital
BR         Brazil    Brasilia
RU         Russia      Moscow
IN         India    New Dehli
CH         China      Beijing
SA  South Africa     Pretoria
</code></pre><p><code>loc</code> and <code>iloc</code> functions are pretty similar. The only difference is how you refer to columns and rows.</p>
<h2 id="interactive-example-on-selecting-a-subset-of-data">Interactive Example on Selecting a Subset of Data</h2>
<p>In the following example, the cars data is imported from a CSV files as a Pandas DataFrame. To select only the <code>cars_per_cap column</code> from cars, you can use:</p>
<pre><code>cars[&#39;cars_per_cap&#39;]
cars[[&#39;cars_per_cap&#39;]]
</code></pre><p>The single bracket version gives a Pandas Series; the double bracket version gives a Pandas DataFrame.</p>
<ul>
<li>You will use single square brackets to print out the <code>country</code> column of <code>cars</code> as a Pandas Series.</li>
<li>Then use double square brackets to print out the <code>country</code> column of <code>cars</code> as a Pandas DataFrame.</li>
<li>Finally, use the double square brackets to print out a DataFrame with both the <code>country</code> and <code>drives_right</code> columns of <code>cars</code>, in this order.</li>
</ul>
<pre><code class="lang-Python"># Import cars data
import pandas as pd
cars = pd.read_csv(&#39;cars.csv&#39;, index_col = 0)

# Print out country column as Pandas Series
print(cars[&#39;country&#39;])

# Print out country column as Pandas DataFrame
print(cars[[&#39;country&#39;]])

# Print out DataFrame with country and drives_right columns
print(cars[[&#39;country&#39;, &#39;drives_right&#39;]])
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>US     United States
AUS        Australia
JPN            Japan
IN             India
RU            Russia
MOR          Morocco
EG             Egypt
Name: country, dtype: object
           country
US   United States
AUS      Australia
JPN          Japan
IN           India
RU          Russia
MOR        Morocco
EG           Egypt
           country  drives_right
US   United States          True
AUS      Australia         False
JPN          Japan         False
IN           India         False
RU          Russia          True
MOR        Morocco          True
EG           Egypt          True
</code></pre><p><a href="https://campus.datacamp.com/courses/intermediate-python/dictionaries-pandas?ex=15">Try it for yourself</a>.</p>
<p>To learn more about pandas, please see this video from our course <a href="https://campus.datacamp.com/courses/intermediate-python/dictionaries-pandas?ex=14">Intermediate Python</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/UQIUhGWYg6Q" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://learn.datacamp.com/courses/intermediate-python-for-data-science">Intermediate Python</a> course by Hugo Bowne-Anderson.</p>


https://www.datacamp.com/community/tutorials/python-select-columns

(Tutorial) Python Select Columns

Python Select Columns

<p>R is the most popular language for Data Science. There are many packages and libraries provided for doing different tasks. For example, there is <code>dplyr</code> and <code>data.table</code> for data manipulation, whereas libraries like <code>ggplot2</code> for data visualization and data cleaning library like <code>tidyr</code>. Also, there is a library like &#39;Shiny&#39; to create a Web application and <code>knitr</code> for the Report generation where finally <code>mlr3</code>, <code>xgboost</code>, and <code>caret</code> are used in Machine Learning.</p>
<center>
<a target=“_blank” href="https://www.datacamp.com/learn/r/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1583330891/r_4_br67yx.png" /></a>
</center>

<h2>1. <code>ggplot2</code></h2>

<p><code>ggplot2</code> is based on the &#39;Grammar of Graphics&quot;, which is a popular data visualization library. Graphs with one variable, two variables, and three variables, along with both categorical and numerical data, can be built. Also, grouping can be done through symbol, size, color, etc.
The interactive graphics can be made with the help of <code>plot.ly</code>, where the 3D image should be made from <code>plot3D</code>.</p>
<p>You can easily install the package <code>ggplot2</code> in R&#39;s console as seen below:<br></p>
<pre><code>install.packages(&quot;ggplot2&quot;)
</code></pre><p>You can easily load the package <code>ggplot2</code> by using the following syntax:<br></p>
<pre><code>library(ggplot2)
</code></pre><p>The following tutorials on DataCamp provide much detailed knowledge about &#39;ggplot2&#39;.<br></p>
<ol>
<li><a href="https://learn.datacamp.com/courses/data-visualization-with-ggplot2-1">Data Visualization with ggplot2 (Part 1)</a><br></li>
<li><a href="https://learn.datacamp.com/courses/data-visualization-with-ggplot2-2">Data Visualization with ggplot2 (Part 2)</a><br></li>
<li><a href="https://learn.datacamp.com/courses/data-visualization-with-ggplot2-3">Data Visualization with ggplot2 (Part 3)</a><br></li>
</ol>
<h2>2. <code>data.table</code></h2>

<p><code>data.table</code> is the fastest package that can handle a vast amount of data during data manipulation. It is mostly used for health care domains for genomic data and fields like business for predictive analytics. Also, the data size ranges from more than 10 GB to 100GB.</p>
<p>You can easily install the package <code>data.table</code> in R&#39;s console as seen below:<br></p>
<pre><code>install.packages(&quot;data.table&quot;)
</code></pre><p>You can easily load the package <code>data.table</code> in R as seen below:<br></p>
<pre><code>library(data.table)
</code></pre><p>You can look up to following tutorial and course in the DataCamp:</p>
<ol>
<li><a href="https://www.datacamp.com/courses/data-table-data-manipulation-r-tutorial/">Data Analysis in R, the data.table Way</a>.</li>
<li><a href="https://www.datacamp.com/community/tutorials/data-table-r-tutorial">A data.table R Tutorial: Intro to DT[i, j, by]</a>.</li>
</ol>
<h2>3. <code>dplyr</code></h2>

<p><code>dplyr</code> is the package which is used for data manipulation by providing different sets of verbs like <code>select()</code>, <code>arrange()</code>, <code>filter()</code>, <code>summarise()</code>, and <code>mutate()</code>. It can also work with computational backends like <code>dplyr</code>, <code>sparklyr</code>, and <code>dtplyr</code>.</p>
<ol>
<li><p>You can install <code>dplyr</code> through using the <code>tidyverse</code> package, which will come with the package <code>dplyr</code>.<br></p>
<pre><code>install.packages(&quot;tidyverse&quot;)
</code></pre></li>
<li><p>Alternatively, you can install <code>dplyr</code> using the following command.<br></p>
<pre><code>install.packages(&quot;dplyr&quot;)
</code></pre></li>
<li><p>You can load the package by using the following command.<br></p>
<pre><code>library(dplyr)
</code></pre></li>
</ol>
<p>The following tutorial and course in DataCamp provide detailed knowledge of <code>dplyr</code>.</p>
<ol>
<li><a href="https://www.datacamp.com/courses/data-manipulation-with-dplyr-in-r">Data Manipulation with dplyr</a></li>
<li><a href="https://www.datacamp.com/courses/joining-data-with-dplyr-in-r/">Joining Data with dplyr</a></li>
<li><a href="https://www.datacamp.com/courses/introduction-to-the-tidyverse">Introduction to the Tidyverse</a></li>
</ol>
<h2>4. <code>tidyr</code></h2>

<p><code>tidyr</code> helps to create tidy data. The significant amount of work mostly goes on when cleaning and tidying the data. Basically, tidy data consists of those datasets where every cell acts as a single value, where every row is an observation, and every column is variable.</p>
<p>You can install <code>tidyr</code> using the following command.<br></p>
<pre><code>install.packages(&quot;tidyr&quot;)
</code></pre><p>You can load <code>tidyr</code> using the following command.<br></p>
<pre><code>library(tidyr)
</code></pre><p>The following tutorial in DataCamp provides detailed knowledge in <code>tidyr</code>.
<a href="https://www.datacamp.com/courses/cleaning-data-in-r">Cleaning Data in R</a></p>
<h2>5. <code>Shiny</code></h2>

<p><code>Shiny</code> can be used to build the web application without requiring JavaScript. It can be used together with htmlwidgets, JavaScript actions, and CSS themes to have extended features. Also, it can be used to build dashboards along with the standalone web applications.</p>
<p>You can install the <code>Shiny</code> package by the following command.<br></p>
<pre><code>install.packages(&quot;shiny&quot;)
</code></pre><p>You can load <code>Shiny</code> using the following command.<br></p>
<pre><code>library(shiny)
</code></pre><p>You can visit the link mentioned below to learn more about <code>Shiny</code>.<br>
<a href="https://www.datacamp.com/tracks/shiny-fundamentals-with-r">Shiny Fundamentals with R</a></p>
<h2>6. <code>plotly</code></h2>

<p><code>plotly</code> is the graphing library used to create graphs that are interactive and can also be used with JavaScript known as <code>plotly.js</code>.</p>
<p>You can install the <code>plotly</code> package by the following command.<br></p>
<pre><code>install.packages(&quot;plotly&quot;)
</code></pre><p>You can load <code>plotly</code> using the following command.<br></p>
<pre><code>library(plotly)
</code></pre><p>You can visit the link mentioned below to learn more about <code>plotly</code>.<br>
<a href="https://www.datacamp.com/courses/intermediate-interactive-data-visualization-with-plotly-in-r">Intermediate Interactive Data Visualization with plotly in R</a></p>
<h2>7. <code>knitr</code></h2>

<p><code>knitr</code> is the package mostly used for research. It is reproducible, used for report creation, and integrates with various types of code structures like LaTeX, HTML, Markdown, LyX, etc. It was inspired by Sweave and has extended the features by adding lots of packages like a weaver, animation, cacheSweave, etc.</p>
<p>You can install the <code>knitr</code> package by the following command.<br></p>
<pre><code>install.packages(&quot;knitr&quot;)
</code></pre><p>You can load <code>knitr</code> using the following command.<br></p>
<pre><code>library(knitr)
</code></pre><p>You can visit the link mentioned below to learn more about <code>knitr</code>.<br>
<a href="https://www.datacamp.com/courses/reporting-with-r-markdown/">Reporting with R Markdown
</a></p>
<h2>8. <code>mlr3</code></h2>

<p><code>mlr3</code> package is created for doing Machine Learning. It is also efficient, which supports Object-Oriented programming where &#39;R6&#39; objects are being provided along with machine learning workflow. It is also seen as one of the extensible frameworks for clustering, regression, classification, and survival analysis.</p>
<p>You can install the <code>mlr3</code> package by the following command.<br></p>
<pre><code>install.packages(&quot;mlr3&quot;)
</code></pre><p>You can load <code>knitr</code> using the following command.<br></p>
<pre><code>library(mlr3)
</code></pre><p>You can visit the link mentioned below to learn more about <code>mlr3</code>.<br>
<a href="https://mlr3book.mlr-org.com/">mlr3Book</a></p>
<h2>9. <code>XGBoost</code></h2>

<p><code>XGBoost</code> is an implementation of the gradient boosting framework. It also provides an interface for R where the model in R&#39;s caret package is also present. Its speed and performance are faster than the implementation in H20, Spark, and Python. This package&#39;s primary use case is for machine learning tasks like classification, ranking problems, and regression.</p>
<p>You can install the <code>XGBoost</code> package by the following command.<br></p>
<pre><code>install.packages(&#39;xgboost&#39;)
</code></pre><p>You can load <code>XGBoost</code> using the following command.<br></p>
<pre><code>library(xgboost)
</code></pre><p>You can visit the link mentioned below to learn more about <code>XGBoost</code>.<br>
<a href="https://www.datacamp.com/courses/extreme-gradient-boosting-with-xgboost">Extreme Gradient Boosting with XGBoost</a></p>
<h2>10. <code>Caret</code></h2>

<p>A <code>caret</code> package is a short form of Classification And Regression Training used for predictive modeling where it provides the tools for the following process.</p>
<ol>
<li>Pre-Processing: Where data is pre-processed and also the missing data is checked.preprocess() is provided by caret for doing such task.</li>
<li>Data splitting: Splitting the training data into two similar categorical data sets is done.</li>
<li>Feature selection: Techniques which is most suitable like Recursive Feature selection can be used.</li>
<li>Training Model: caret provides many packages for machine learning algorithms.</li>
<li>Resampling for model tuning: The model can be tuned using repeated k-fold, k-fold, etc. Also, the parameter can be tuned using &#39;tuneLength.&#39;</li>
<li>Variable importance estimation: <code>vlamp()</code> can be used for any model to access the variable importance estimation.</li>
</ol>
<p>You can install the <code>caret</code> package by the following command.<br></p>
<pre><code>install.packages(&#39;caret&#39;)
</code></pre><p>You can load <code>caret</code> using the following command.<br></p>
<pre><code>library(caret)
</code></pre><p>You can visit the link mentioned below to learn more about <code>caret</code> from the author &quot;Max Kuhn&quot;.<br>
<a href="https://www.datacamp.com/courses/machine-learning-toolbox">Machine Learning with caret in R</a></p>
<h2 id="congratulations">Congratulations</h2>
<p>Congratulations, you have made it to the end of this tutorial!</p>
<p>In this tutorial, you&#39;ve learned about different packages in R used for the Data Science process. This tutorial focused on installation, loading, and finally, getting the resources to DataCamp for learning about these packages.</p>
<p><br><center>
<a target=“_blank” href="https://www.datacamp.com/learn/r/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1583330891/r_2_t6hgbx.png" /></a>
</center>  </p>


https://www.datacamp.com/community/tutorials/top-ten-most-important-packages-in-r-for-data-science

Learn about different packages in R used for data science. Including how to load them and different resources you can use to advance your skills with them.

(Tutorial) The 10 Most Important Packages in R for Data Science

The 10 Most Important Packages in R for Data Science

https://www.datacamp.com/community/tutorials/streaming-platform-analysis

In this Python tutorial, explore movie data of popular streaming platforms and build a recommendation system.

(Tutorial) Recommendation System for Streaming Platforms

Recommendation System for Streaming Platforms

<center>
<a target=“_blank” href="https://www.datacamp.com/learn/python/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1582839819/python_4_t5pp5t.png" /></a>
</center>

<p>Are you tired of waiting in long queues for your call to be connected to the customer service executive? Does reading FAQ&#39;s make you feel lethargic? Then you are on the right page. Can you remember the last time you communicated to a customer service agent via chat for the wrong item being delivered to you? There is a high probability that you were being communicated to by a bot rather than a customer service representative. So what exactly are bots? How do we build one? What source of code does it require? These are some of the questions which will be answered in this blog post!</p>
<p>Artificial intelligence, which brings into play machine learning and Natural language Processing (NLP) for building bot or chatbot, is specifically designed to unravel the smooth interaction between humans and computers. Chatbots are everywhere, be it a banking website, pizza store, to e-commerce shopping stores, you will find chatbots left, right, and center. Chatbots provide real-time customer service assistance on a range of pre-defined questions related to the domain it is built on. It adapts natural human language and converses with humans in a human-like manner.</p>
<p>To simplify the chatbot&#39;s definition, we can say chatbots are the evolution of Question Answer systems employing natural language processing. As per sources by the year 2024, the global conversation market&#39;s size will grow to $15.7 billion, with 30.2% being the annual growth rate. For instance, amidst the CoronaVirus Pandemic, we have witnessed thousands of hoaxes circulating on WhatsApp, such as what can be used to treat COVID or what can be beneficial in increasing immunity, or whether the virus was developed in a lab. Putting an end to such hoaxes, Facebook launched a chatbot that works as a fact-checker.</p>
<p>In this tutorial, we’ll learn Chatbot building in detail, including:</p>
<ul>
<li>Introduction</li>
<li>What is Chatbot?</li>
<li>How Does Chatbot Work?</li>
<li>Comparison of Chatbot platforms</li>
<li>Chatterbot Library</li>
<li>Building  a Chatbot using Chatterbot</li>
<li>Chatbot Testing</li>
<li>Conclusion</li>
</ul>
<h2 id="what-is-a-chatbot-">What is a chatbot?</h2>
<p>The term “chatterbot” came in existence in 1994 when Michael Mauldin created his first chatbot named “Julia”. As per the Oxford Dictionary, a chatbot is defined as “A computer program designed to simulate conversation with human users, especially over the internet.” It can be looked upon as a virtual assistant that communicates with users via text messages and helps businesses in getting close to their customers. It is a program designed to imitate the way humans communicate with each other. It can be done through a chat interface or by voice call. Developers usually design chatbots so that it is difficult to tell for users whether they are communicating with a person or a robot.</p>
<p>Chatbots helps any business/organization in accomplishing the following goals:</p>
<ul>
<li>Increases operational efficiency.</li>
<li>Automating customer request fulfillment.</li>
<li>Handling basic queries, which in turn free employees to work for complex &amp; higher value inquiries.</li>
<li>Offers Multi-language support.</li>
<li>Saves time &amp; effort by automating customer support.</li>
<li>Improves the response rate as well as customer engagement.</li>
<li>Personalization of communication</li>
</ul>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1598043769/chatterbot1_uy1uar.png" /></center>

<center><a href="https://medium.com/appedus/5-chatbot-companies-that-helps-you-automate-your-business-1acba5fa14e4">Source</a></center>

<h2 id="how-does-a-chatbot-work-">How Does a Chatbot Work?</h2>
<p>Chatbots are nothing but software applications that have an application layer, a database, and APIs. To simplify the working of the chatbot, we can say it works on pattern matching to classify text and produce a suitable response for the questions/queries addressed by the user. The chatbot responds to the user as per the program that has been fed in it. Chatbots are of different types, depending on how they are used. Mainly there are three types of chatbots, and they are as follows:</p>
<ul>
<li><p><strong>Rule-Based Chatbot:</strong> This is the basic chatbot made, the user interacts with this kind of bot by using predefined options. To get answers from these bots, users need to click on certain options. These kinds of bots collect the user&#39;s request, analyze it, and then offer results in the form of buttons. These bots are commonly used to replace frequently asked questions when it comes to complex queries; they aren&#39;t always the best solution.</p>
</li>
<li><p><strong>Independent(Keyword) Chatbots:</strong> These are machine learning bots, unlike rule-based chatbots, they analyze what the user wants and respond appropriately. These chatbots use customizable keywords and machine learning to determine how to respond to users&#39; requests effectively and efficiently.</p>
</li>
<li><p><strong>NLP (Contextual) Chatbots:</strong> These are so far the most advanced chatbots. They are a combination of best from rule-based and keyword chatbots. These chatbots use NLP to understand the context and intent in users&#39; requests and thus act accordingly. These chatbots can handle multiple requests from the same user at ease.</p>
</li>
</ul>
<h2 id="comparisons-of-chatbot-platform">Comparisons of Chatbot Platform</h2>
<p>Many platforms offer customized chatbots with automation making seamless, always on-time, best in class support services available to customers whenever they need it without the box conversational capacities. Customers are also keen to purchase from a business that they can easily connect over messages.</p>
<p>Listing down the AI chatbot building platform in 2020:</p>
<ul>
<li><p><strong>Azure Bot Service:</strong> Azure bot service offers to build a chatbot from scratch, i.e., you can build, connect, test, and deploy. It allows developers to use the open-source SDK and tools. It also enables developers to create advanced bots such as virtual assistants to handle complex queries.</p>
</li>
<li><p><strong>Botsify:</strong> This tool&#39;s uniqueness is that it enables non-technical users to build a chatbot with its intuitive interface. Bots are represented here as stories, i.e., you can create multiple stories or multiple chatbots and deploy them as per the requirement. Another great feature is the ability to save users&#39; responses to a form that can be easily exported to a CSV.</p>
</li>
<li><p><strong>Amazon Lex:</strong> Amazon lex allows developers to build conversational interfaces using text and voice. It comes up with advanced deep learning functionalities and NLP for understanding the context of the text. It provides an easy to use console for building chatbot in minutes.</p>
</li>
<li><p><strong>Mobile Monkey:</strong> Mobile monkey offers developers to build chatbots specifically for marketing purposes. It allows developers to make Facebook ad bots, SMS bots, and native web chatbots all in a single platform. They also offer ready-made chatbot templates for every business possible, which can be directly embedded on a website.</p>
</li>
<li><p><strong>ChatterOn:</strong> It claims to handle different types of rich content responses from the Bot as it allows developers to connect  different APIs at each interaction with the user. ChatterOn offers more than 20 pre-built bolts that can be put to use in a single click. It doesn’t require much coding, which makes it easier for non-technical users to build chatbots.</p>
</li>
<li><p><strong>TARS:</strong> It offers you to build a conversational landing page, which enables you to create an automated chatbot to greet you, customers, give them relevant information about their queries regarding your products, and ask for their contact simultaneously. Tars offers many pre-defined chatbot templates, which are classified into two parts - by industry and by use-case.</p>
</li>
</ul>
<p>By now, you must be curious to build a chatbot of your own. And what&#39;s better than a customizable NLP chatbot? Let&#39;s get started for building our very own chatbot in Python using library chatterbot.</p>
<h2 id="chatterbot">Chatterbot</h2>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1598043769/chatterbot2_lejxhc.png" /></center>

<p>As the name suggests, chatterbot is a python library specifically designed to generate chatbots. This algorithm uses a selection of machine learning algorithms to fabricate varying responses to users as per their requests.</p>
<p>Chatterbot makes it easier to develop chatbots that can engage in conversations. It starts by creating an untrained chatterbot that has no prior experience or knowledge regarding how to communicate. As the users enter statements, the library saves the request made by the user as well as it also saves the responses that are sent back to the users. As the  number of instances increases in chatterbot, the accuracy of the responses made by chatterbot also increases.</p>
<p>Chatterbot is trained to search the closest analogous response by finding the closest analogous request made by users that is equivalent to the new request made. Then it selects a response from the already existing responses. The USP of chatterbot is that it enables developers to create their own dataset and structures at ease.</p>
<h2 id="building-a-chatbot-using-chatterbot">Building  a Chatbot using Chatterbot</h2>
<p>Let&#39;s begin by installing the chatterbot library. For creating chatbot also need to install chatterbot corpus. Corpus - literal meaning is a collection of words. This contains a corpus of data that is included in the chatterbot module. Each corpus is nothing but a prototype of different input statements and their responses. These corpus are used by bots to train themselves. The most recommended method for installing chatterbot and chatterbot_corpus is by using pip.</p>
<p>Installation commands for terminal:</p>
<pre><code>pip install chatterbot
</code></pre><pre><code>pip install chatterbot_corpus
</code></pre><p>Installation commmands for Jupyter Notebook:</p>
<pre><code>!pip install chatterbot
</code></pre><pre><code>!pip install chatterbot_corpus
</code></pre><p>Let&#39;s first import the Chatbot class of the chatterbot module.</p>
<pre><code class="lang-python"># Importing chatterbot
from chatterbot import ChatBot
</code></pre>
<h3 id="create-chatbot-instance">Create Chatbot Instance</h3>
<p>Now, it&#39;s time for the most interesting part i.e., naming your chatbot by creating a Chatbot object. You can choose any name you want. This single line of code generates our very own new bot named Buddy. We need to specify some more parameters before running our first program.</p>
<pre><code class="lang-python"># Create object of ChatBot class
bot = ChatBot(&#39;Buddy&#39;)
</code></pre>
<pre><code>[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.
[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.
[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data]   Unzipping corpora/wordnet.zip.
</code></pre><p>You can position the storage adapter with the chatbot object. Storage Adapters allows you to connect to a particular storage unit or network. For using a storage adapter, we need to specify it. We will position the storage adapter by assigning it to the import path of the storage we want to use. Here we are using SQL Storage Adapter, which permits chatbot to connect to databases in SQL. By using the database parameter, we will create a new SQLite Database. Please follow the code below, for creating a new database for chatbot.</p>
<pre><code class="lang-python"># Create object of ChatBot class with Storage Adapter
bot = ChatBot(
    &#39;Buddy&#39;,
    storage_adapter=&#39;chatterbot.storage.SQLStorageAdapter&#39;,
    database_uri=&#39;sqlite:///database.sqlite3&#39;
)
</code></pre>
<p>You can also position the logical adapter with a chatbot object. As the name implies, Logical Adapter regulates the logic behind the chatterbot, i.e., it picks responses for any input provided to it. This parameter contains a list of logical operators. Chatterbot allows us to use a number of logical Adapters. When more than one logical adapter is put to use, the chatbot will calculate the confidence level, and the response with the highest calculated confidence will be returned as output. Here we have used two logical adapters: <code>BestMatch</code> and <code>TimeLogicAdapter</code>.</p>
<pre><code class="lang-python"># Create object of ChatBot class with Logic Adapter
bot = ChatBot(
    &#39;Buddy&#39;,  
    logic_adapters=[
        &#39;chatterbot.logic.BestMatch&#39;,
        &#39;chatterbot.logic.TimeLogicAdapter&#39;],
)
</code></pre>
<h3 id="training-the-chatbot">Training the chatbot</h3>
<p>Now the final step in making a chatbot is to train the chatbot using the modules available in chatterbot. Training a chatbot using chatterbot is as simple as providing a conversation into the chatbot database. As soon as the chatbot is given a dataset, it produces the essential entries in the chatbot&#39;s knowledge graph to represent the input and output in the right manner. Firstly, let&#39;s import the <code>ListTrainer</code>, create its object by passing the <code>Chatbot</code> object, and call the <code>train()</code> method by passing a list of sentences.</p>
<pre><code class="lang-python"># Inport ListTrainer
from chatterbot.trainers import ListTrainer

trainer = ListTrainer(bot)

trainer.train([
&#39;Hi&#39;,
&#39;Hello&#39;,
&#39;I need your assistance regarding my order&#39;,
&#39;Please, Provide me with your order id&#39;,
&#39;I have a complaint.&#39;,
&#39;Please elaborate, your concern&#39;,
&#39;How long it will take to receive an order ?&#39;,
&#39;An order takes 3-5 Business days to get delivered.&#39;,
&#39;Okay Thanks&#39;,
&#39;No Problem! Have a Good Day!&#39;
])
</code></pre>
<pre><code>List Trainer: [####################] 100%
</code></pre><h2 id="chatbot-testing">Chatbot Testing</h2>
<p>The last step of this tutorial is to test the chatterbot’s conversational skills. For testing its responses, we will call the <code>get_responses()</code> method of <code>Chatbot</code> instance.</p>
<pre><code class="lang-python"># Get a response to the input text &#39;I would like to book a flight.&#39;
response = bot.get_response(&#39;I have a complaint.&#39;)

print(&quot;Bot Response:&quot;, response)
</code></pre>
<pre><code>Bot Response: Please elaborate, your concern
</code></pre><p>We will create a while loop for our chatbot to run in. When statements are passed in the loop, we will get an appropriate response for it, as we have already entered data into our database. If we get &quot;Bye&quot; or &quot;bye&quot; statement from the user, we can put an end to the loop and stop the program.</p>
<pre><code class="lang-python">name=input(&quot;Enter Your Name: &quot;)
print(&quot;Welcome to the Bot Service! Let me know how can I help you?&quot;)
while True:
    request=input(name+&#39;:&#39;)
    if request==&#39;Bye&#39; or request ==&#39;bye&#39;:
        print(&#39;Bot: Bye&#39;)
        break
    else:
        response=bot.get_response(request)
        print(&#39;Bot:&#39;,response)
</code></pre>
<pre><code>Enter Your Name: Avinash
Welcome to the Bot Service! Let me know how can I help you?
Avinash:I need your assistance regarding my order
Bot: Please, Provide me with your order id
Avinash:12345
Bot: No Problem! Have a Good Day!
Avinash:Bye
Bot: Bye
</code></pre><h2 id="conclusion">Conclusion</h2>
<p>Congratulations, you have made it to the end of this tutorial!</p>
<p>This article was based on learning how to make a chatbot in Python using the ChatterBot library. Building a chatbot with ChatterBot was not only simple, but also, the results were accurate. With Artificial Intelligence and Machine Learning,  in advancement, everything and anything is possible to achieve whether it is creating bots with conversational skills like humans or be it anything else.<br></p>
<center>
<a target=“_blank” href="https://www.datacamp.com/learn/python/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1583330891/python_1_b5jdxv.png" /></a>
</center>   


https://www.datacamp.com/community/tutorials/building-a-chatbot-using-chatterbot

In this tutorial, you'll learn how to build a chatbot using chatterbot in Python.

(Tutorial) Building a Chatbot using Chatterbot in Python

Building a Chatbot using Chatterbot in Python

<center>
<a target=“_blank” href="https://www.datacamp.com/learn/r/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1583330891/r_3_ijnnwq.png" /></a>
</center>   

<p>R provides a different way to sort the data either in ascending or descending order; Data-analysts, and Data scientists use <code>order()</code>, <code>sort()</code> and packages like <code>dplyr</code> to sort data depending upon the structure of the obtained data.<br></p>
<p><code>order()</code> can sort vector, matrix, and also a dataframe can be sorted in ascending and descending order with its help, which is shown in the final section of this tutorial.</p>
<h2 id="syntax-of-order-">Syntax of <code>order()</code></h2>
<p>The syntax of <code>order()</code> is shown below:<br></p>
<pre><code>order(x, decreasing = TRUE or FALSE, na.last = TRUE or FLASE, method = c(&quot;auto&quot;, &quot;shell&quot;, &quot;quick&quot;, &quot;radix&quot;))
</code></pre><p>The argument above in <code>order()</code> states that:<br><br></p>
<ul>
<li><b>x:</b> data-frames, matrices, or vectors<br></li>
<li><b>decreasing:</b> boolean value; TRUE then sort in descending order or FALSE then sort in ascending order.<br></li>
<li><b>na.last:</b> boolean value; TRUE then NA indices are put at last or FLASE THEN NA indices are put first.<br></li>
<li><b>method:</b> sorting method to be used.</li>
</ul>


<h2 id="-order-in-r"><code>order()</code> in R</h2>
<p>Let&#39;s look at an example of <code>order()</code> in action.</p>
<p>Below the code contains variable <code>x</code>, which includes a vector with a list of numbers. The numbers are ordered  according to its index by using <code>order(x)</code>.</p>
<pre><code class="lang-python">y = c(4,12,6,7,2,9,5)
order(y)
</code></pre>
<p>The above code gives the following output:<br></p>
<pre><code>5 1 7 3 4 6 2
</code></pre><p>Here the <code>order()</code> will sort the given numbers according to its index in the ascending order. Since number 2 is the smallest, which has an index as five and number 4 is index 1, and similarly, the process moves forward in the same pattern.</p>
<pre><code class="lang-python">y = c(4,12,6,7,2,9,5)
y[order(y)]
</code></pre>
<p>The above code gives the following output:<br></p>
<pre><code>2 4 5 6 7 9 12
</code></pre><p>Here the indexing of order is done where the actual values are printed in the ascending order. The values are ordered according to the index using <code>order()</code> then after each value accessed using <code>y[some-value]</code>.</p>
<h2 id="sorting-vector-using-different-parameters-in-order-">Sorting vector using different parameters in <code>order()</code></h2>
<p>Let&#39;s look at an example where the datasets contain the value as symbol <b>NA</b>(Not available).</p>
<h4 id="order-x-na-last-true-">order(x,na.last=TRUE)</h4>
<pre><code class="lang-python">x &lt;- c(8,2,4,1,-4,NA,46,8,9,5,3)
order(x,na.last = TRUE)
</code></pre>
<p>The above code gives the following output:<br></p>
<pre><code>5 4 2 11 3 10 1 8 9 7 6
</code></pre><p>Here the <code>order()</code> will also sort the given list of numbers according to its index in the ascending order. Since <b>NA</b> is present, its index will be placed last, where 6 will be placed last because of <code>na.last=TRUE</code>.</p>
<h4 id="-order-x-na-last-false-"><code>order(x,na.last=FALSE)</code></h4>
<pre><code class="lang-python">order(x,na.last=FALSE)
</code></pre>
<p>The above code gives the following output:<br></p>
<pre><code>6 5 4 2 11 3 10 1 8 9 7
</code></pre><p>Here the <code>order()</code> will also sort the given list of numbers according to its index in the ascending order. Since <b>NA</b> is present, it&#39;s index, which is 6, will be placed first because of <code>na.last=FALSE</code>.</p>
<h4 id="order-x-decreasing-true-na-last-true-">order(x,decreasing=TRUE,na.last=TRUE)</h4>
<pre><code class="lang-python">order(x,decreasing=TRUE,na.last=TRUE)
</code></pre>
<p>The above code gives the following output:<br></p>
<pre><code>7 9 1 8 10 3 11 2 4 5 6
</code></pre><p>Here <code>order()</code> will sort a given list of numbers according to its index in the descending order because of <code>decreasing=TRUE</code>: 46. The largest is placed at index 7, and the other values are arranged in a decreasing manner. Since <b>NA</b> is present, index 6 will be placed last because of <code>na.last=TRUE</code>.</p>
<h4 id="order-x-decreasing-false-na-last-false-">order(x,decreasing=FALSE,na.last=FALSE)</h4>
<pre><code class="lang-python">order(x,decreasing=FALSE,na.last=FALSE)
</code></pre>
<p>The above code gives the following output:<br></p>
<pre><code>6 5 4 2 11 3 10 1 8 9 7
</code></pre><p>Here  <b>NA</b> is present which index is 6 will be placed at first because of <code>na.last=FALSE</code>. <code>order()</code> will sort a given list of numbers according to its index in the ascending order because of <code>decreasing=FALSE</code>: -4, which is smallest placed at index 5, and the other values are arranged increasingly.</p>
<h2 id="sorting-a-dataframe-by-using-order-">Sorting a dataframe by using <code>order()</code></h2>
<p>Let&#39;s create a dataframe where the population value is 10. The variable gender consists of vector values &#39;male&#39; and &#39;female&#39; where 10 sample values could be obtained with the help of <code>sample()</code>, whereas <code>replace = TRUE</code> will generate only the unique values.
Similarly, the age consists of value from 25 to 75, along with a degree of possible value as c(&quot;MA,&quot; &quot;ME,&quot; &quot;BE,&quot; &quot;BSCS&quot;), which again will generate unique values.<br></p>
<p><b>Task</b>: To sort the given data in the ascending order based on the given population&#39;s age.<br><br></p>
<p><b>Note:</b> The sample data shown may differ while you&#39;re trying to use it in your local machine because each time running a code will create a unique dataframe.</p>
<pre><code class="lang-python">population = 10
gender=sample(c(&quot;male&quot;,&quot;female&quot;),population,replace=TRUE)
age = sample(25:75, population, replace=TRUE)
degree = sample(c(&quot;MA&quot;,&quot;ME&quot;,&quot;BE&quot;,&quot;BSCS&quot;), population, replace=TRUE)
(final.data = data.frame(gender=gender, age=age, degree=degree))
</code></pre>
<div class="output_wrapper"><table>
<thead><tr><th scope=col>gender</th><th scope=col>age</th><th scope=col>degree</th></tr></thead>
<tbody>
    <tr><td>male  </td><td>40    </td><td>MA    </td></tr>
    <tr><td>female</td><td>57    </td><td>BSCS  </td></tr>
    <tr><td>male  </td><td>66    </td><td>BE    </td></tr>
    <tr><td>female</td><td>61    </td><td>BSCS  </td></tr>
    <tr><td>female</td><td>48    </td><td>MA    </td></tr>
    <tr><td>male  </td><td>25    </td><td>MA    </td></tr>
    <tr><td>female</td><td>49    </td><td>BE    </td></tr>
    <tr><td>male  </td><td>52    </td><td>ME    </td></tr>
    <tr><td>female</td><td>57    </td><td>MA    </td></tr>
    <tr><td>female</td><td>35    </td><td>MA    </td></tr>
</tbody>
</table></div>



<p>The above code gives the following output, which shows a newly created dataframe.<br></p>
<pre><code>  gender       age        degree
   male           40         MA
   female       57         BSCS
   male           66         BE
   female       61         BSCS
   female       48        MA
   male           25         MA
   female       49         BE
   male        52        ME
   female       57         MA
   female       35         MA
</code></pre><p>Let&#39;s sort the dataframe in the ascending order by using <code>order()</code> based on the variable age.</p>
<pre><code class="lang-python">order(final.data$age)
</code></pre>
<p>The above code gives the following output:<br></p>
<pre><code>6 10 3 9 5 8 4 2 7 1
</code></pre><p>Since age 25 is at index 6 followed by age 35 at index 10 and similarly, all the age-related values are arranged in ascending order.</p>
<p>The code below contains the <code>[]</code> order with variable age, is used to arrange in ascending order where the gender, along with degree information is also printed.</p>
<pre><code class="lang-python">final.data[order(final.data$age),]
</code></pre>
<div class="output_wrapper"><table>
<thead><tr><th></th><th scope=col>gender</th><th scope=col>age</th><th scope=col>degree</th></tr></thead>
<tbody>
    <tr><th scope=row>6</th><td>male  </td><td>25    </td><td>MA    </td></tr>
    <tr><th scope=row>10</th><td>female</td><td>35    </td><td>MA    </td></tr>
    <tr><th scope=row>1</th><td>male  </td><td>40    </td><td>MA    </td></tr>
    <tr><th scope=row>5</th><td>female</td><td>48    </td><td>MA    </td></tr>
    <tr><th scope=row>7</th><td>female</td><td>49    </td><td>BE    </td></tr>
    <tr><th scope=row>8</th><td>male  </td><td>52    </td><td>ME    </td></tr>
    <tr><th scope=row>2</th><td>female</td><td>57    </td><td>BSCS  </td></tr>
    <tr><th scope=row>9</th><td>female</td><td>57    </td><td>MA    </td></tr>
    <tr><th scope=row>4</th><td>female</td><td>61    </td><td>BSCS  </td></tr>
    <tr><th scope=row>3</th><td>male  </td><td>66    </td><td>BE    </td></tr>
</tbody>
</table></div>



<p>The above code gives the following output:<br></p>
<pre><code>gender    age    degree
6    male    25    MA
10    female    35    MA
1    male    40    MA
5    female    48    MA
7    female    49    BE
8    male    52    ME
2    female    57    BSCS
9    female    57    MA
4    female    61    BSCS
3    male    66    BE
</code></pre><p>The output above shows that age is arranged in ascending order along with its corresponding gender and degree information is obtained.</p>
<h2 id="congratulations">Congratulations</h2>
<p>Congratulations, you have made it to the end of this tutorial!<br></p>
<p>You&#39;ve learned about sorting using <code>order()</code> with its argument with the examples and sorting vector using different parameters and final example, which contains sorting of the dataframe.<br></p>
<p>To know more detail about this topic, you can visit <a href="https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/order">Order function in R</a></p>
<p>If you would like to learn more about R, take DataCamp&#39;s <a href="https://www.datacamp.com/courses/free-introduction-to-r">Introduction to R course</a>.</p>


https://www.datacamp.com/community/tutorials/sorting-in-r

In this tutorial, you'll learn about sorting using order(). We will cover how to sort vectors using different parameters, sorting dataframes, and more.

Sorting in R using order()

<p><b>Pyspark = Python + Apache Spark</b></p>
<p>Apache Spark is a new and open-source framework used in the big data industry for real-time processing and batch processing. It supports different languages, like Python, Scala, Java, and R.</p>
<p>Apache Spark is initially written in a Java Virtual Machine(JVM) language called Scala, whereas Pyspark is like a Python API which contains a library called Py4J. This allows dynamic interaction with JVM objects.</p>
<center>
<a target=“_blank” href="https://www.datacamp.com/learn/python/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1583330891/python_3_qgciu1.png" /></a>
</center>   

<h2 id="windows-installation">Windows Installation</h2>
<p>The installation which is going to be shown is for the Windows Operating System. It consists of the installation of Java with the environment variable and Apache Spark with the environment variable.</p>
<p>The recommended pre-requisite installation is Python, which is done from <a href="https://www.python.org/downloads/">here</a>.</p>
<h2 id="java-installation">Java installation</h2>
<ol>
<li>Go to <a href="https://www.oracle.com/java/technologies/javase-jdk13-downloads.html">Download Java JDK</a>.<br>
Visit Oracle&#39;s website for the download of the Java Development Kit(JDK).<br><br></li>
<li>Move to download section consisting of operating system Windows, and in my case, it&#39;s Windows Offline(64-bit). The installer file will be downloaded.
<img src="https://i.imgur.com/9SpClJs.png" width="500" height="500"><br><br></li>
<li>Open the installer file, and the download begins.
<img src="https://i.imgur.com/thvLwKx.png" width="500" height="500"><br><br></li>
<li><p>Go to &quot;Command Prompt&quot; and type &quot;java -version&quot; to know the version and know  whether it is installed or not.
<img src="https://i.imgur.com/UBKQIES.png" width="500" height="500"></p>
</li>
<li><p>Add the Java path
<img src="https://i.imgur.com/lU2y7xE.png" width="500" height="500"></p>
</li>
<li>Go to the search bar and &quot;EDIT THE ENVIRONMENT VARIABLES.
<img SRC="https://i.imgur.com/NU0LqoR.png" width="500" height="500"></li>
<li>Click into the &quot;Environment Variables&#39;
<img src="https://i.imgur.com/SNRZyyW.png" width="500" height="500"></li>
<li>Click into &quot;New&quot; to create your new Environment variable.
<img src="https://i.imgur.com/HAs9InC.png" width="500" height="500"></li>
<li>Use Variable Name as &quot;JAVA_HOME&#39; and your Variable Value as &#39;C:\Program Files (x86)\Java\jdk1.8.0_251&#39;. This is your location of the Java file. Click &#39;OK&#39; after you&#39;ve finished the process.
<img src="https://i.imgur.com/tkUM5Pu.png" width="500" height="500"></li>
<li>Let&#39;s add the User variable and select &#39;Path&#39; and click &#39;New&#39; to create it.
<img src="https://i.imgur.com/sJZlilZ.png" width="500" height="500"></li>
<li>Add the Variable name as &#39;PATH&#39; and path value as &#39;C:\Program Files (x86)\Java\jdk1.8.0_251\bin&#39;, which is your location of Java bin file. Click &#39;OK&#39; after you&#39;ve finished the process.
<img src="https://i.imgur.com/wopPx9j.png" width="500" height="500"></li>
</ol>
<p><b>Note:</b> You can locate your Java file by going to C drive,  which is C:\Program Files (x86)\Java\jdk1.8.0_251&#39; if you&#39;ve not changed location during the download.
<img src="https://i.imgur.com/SOWtRR6.png" width="500" height="500"></p>
<h3 id="installing-pyspark">Installing Pyspark</h3>
<ol>
<li><p>Head over to the <a href="https://spark.apache.org/downloads.html">Spark homepage</a>.</p>
</li>
<li><p>Select the Spark release and package type as following and download the .tgz file.</p>
</li>
</ol>
<p><img src="https://i.imgur.com/KlKb21g.png">
<img src="https://i.imgur.com/As7McNw.png"></p>
<p>You can make a new folder called &#39;spark&#39; in the C directory and extract the given file by using &#39;Winrar&#39;, which will be helpful afterward.</p>
<h3 id="download-and-setup-winutils-exe">Download and setup winutils.exe</h3>
<p>Go to <a href="https://github.com/steveloughran/winutils">Winutils</a> choose your previously downloaded Hadoop version, then download the winutils.exe file by going inside &#39;bin&#39;. The link to my Hadoop version is: <a href="https://github.com/steveloughran/winutils/blob/master/hadoop-2.7.1/bin/winutils.exe">https://github.com/steveloughran/winutils/blob/master/hadoop-2.7.1/bin/winutils.exe</a></p>
<p>Make a new folder called &#39;winutils&#39; and inside of it create again a new folder called &#39;bin&#39;.Then put the file recently download &#39;winutils&#39; inside it.</p>
<h3 id="environment-variables">Environment variables</h3>
<ol>
<li>Let&#39;s create a new environment where variable name as &quot;hadoop_home&quot; and variable value to be the location of winutils, which is &quot;C:\winutils&quot; and click &quot;OK&quot;.<br>
<img src="https://i.imgur.com/mwUJqro.png"><br></li>
<li>For spark, also let&#39;s create a new environment where the variable name is &quot;Spark_home&quot; and the variable value to be the location of spark, which is &quot;C:\spark&quot; and click &quot;OK&quot;.<br>
<img src="https://i.imgur.com/aE1lQqq.png"><br></li>
<li>Finally, double click the &#39;path&#39; and change the following as done below where a new path is created &quot;%Spark_Home%\bin&#39; is added and click &quot;OK&quot;.<br>
<img src="https://i.imgur.com/eKDy5iB.png"></li>
</ol>
<h3 id="finalizing-pyspark-installation">Finalizing Pyspark Installation</h3>
<ol>
<li>Open Command Prompt and type the following command.<br>
<img src="https://i.imgur.com/KAVautu.png"><br></li>
<li>Once everything is successfully done, the following message is obtained.<br>
<img src="https://i.imgur.com/1O9Hj9d.png" width="700" height="700"></li>
</ol>
<h2 id="linux-installation">Linux Installation</h2>
<p>The installation which is going to be shown is for the Linux Operating System. It consists of the installation of Java with the environment variable along with Apache Spark and the environment variable.</p>
<p>The recommended pre-requisite installation is Python, which is done from <a href="https://www.python.org/downloads/source/">here</a>.</p>
<h3 id="java-installation">Java Installation</h3>
<ol>
<li>Go to <a href="https://www.oracle.com/java/technologies/javase-jdk13-downloads.html">Download Java JDK</a>.<br>
Visit Oracle&#39;s website for the download of the Java Development Kit (JDK).<br><br></li>
<li>Move to the download section consisting of the operating system Linux and download it according to your system requirement.<br>
<img src="https://i.imgur.com/8W8zNLv.png" width = "700" height = "700"><br></li>
<li>Save the file and click &quot;Ok&quot; to save in your local machine.<br>
<img src="https://i.imgur.com/gQF2eXb.png" width = "700" height = "700"></li>
<li>Go to your terminal and check the recently downloaded file using &#39;ls&#39; command.<br>
<img src="https://i.imgur.com/OqaIa1m.png"><br></li>
<li>Install the package using the following command, which will install the debian package of java, which is recently downloaded.
<img src="https://i.imgur.com/AwZ3hk4.png"></li>
<li>Finally, you can check your java version using &#39;java --version&#39; command.<br>
<img src="https://i.imgur.com/m3sCj1C.png"><br></li>
<li>For configuring environment variables, let&#39;s open the &#39;gedit&#39; text editor using the following command.<br>
<img src="https://i.imgur.com/nU9yraN.png"><br></li>
<li>Let&#39;s make the change by providing the following information where the &#39;Java&#39; path is specified.<br>
<img src="https://i.imgur.com/N5RtFrI.png"><br></li>
<li>To make a final change, let&#39;s type the following command.
<img src="https://i.imgur.com/07T8aJd.png"></li>
</ol>
<h2 id="installing-spark">Installing Spark</h2>
<ol>
<li>Head over to the <a href="https://spark.apache.org/downloads.html">Spark homepage</a>.</li>
<li>Select the Spark release and package type as following and download the .tgz file.
<img src="https://i.imgur.com/KlKb21g.png"><br>
<img src="https://i.imgur.com/As7McNw.png"><br></li>
<li>Save the file to your local machine and click &#39;Ok&#39;.<br>
<img src="https://i.imgur.com/HeDHHGA.png"><br></li>
<li>Open your terminal and go to the recently downloaded file.<br>
<img src="https://i.imgur.com/RW4svhQ.png"></li>
<li>Let&#39;s extract the file using the following command.<br>
<img src="https://i.imgur.com/9GYmv5C.png"></li>
<li>After extracting the file, the new file is created and shown using the list(&#39;ls&#39;) command.<br>
<img src="https://i.imgur.com/XQPkqhp.png"><br></li>
</ol>
<h3 id="configuring-environment-variable-in-linux">Configuring Environment Variable in Linux</h3>
<ol>
<li>Let&#39;s open the &#39;bashrc&#39; file using &#39;vim editor&#39; by the command &#39;vim ~/.bashrc&#39;.<br>
<img src="https://i.imgur.com/yToxRFy.png"><br></li>
<li>Provide the following information according to your suitable path on your computer. In my case, the following were the required path to my Spark location, Python path, and Java path. Also, first press &#39;Esc&#39; and then type &quot;:wq&quot; to save and exit from vim.<br>
<img src="https://i.imgur.com/AuGyKm5.png"><br></li>
<li>To make a final change, save, and exit. This results in accessing the pyspark command everywhere in the directory.
<img src="https://i.imgur.com/xlEUe7d.png"><br></li>
<li>Open pyspark using &#39;pyspark&#39; command, and the final message will be shown as below.
<img src="https://i.imgur.com/NVzj0eL.png">
<img src="https://i.imgur.com/Zeq2AYQ.png"></li>
</ol>
<h2 id="mac-installation">Mac Installation</h2>
<p>The installation which is going to be shown is for the Mac Operating System. It consists of the installation of Java with the environment variable along with Apache Spark and the environment variable.</p>
<p>The recommended pre-requisite installation is Python, which is done from <a href="https://www.python.org/downloads/mac-osx/">here</a>.</p>
<h3 id="java-installation">Java Installation</h3>
<ol>
<li>Go to <a href="https://www.oracle.com/java/technologies/javase-jdk13-downloads.html">Download Java JDK</a>.<br>
Visit Oracle&#39;s website for the download of the Java Development Kit (JDK).<br><br></li>
<li>Move to download section consisting of the operating system Linux and download according to your system requirement.<br>
<img src="https://i.imgur.com/yZS58VJ.png" width = "700" height = "700"><br></li>
<li>The installation of Java can be confirmed by using <code>$java --showversion</code> in the Terminal.</li>
</ol>
<h3 id="installing-apache-spark">Installing Apache Spark</h3>
<ol>
<li>Head over to the <a href="https://spark.apache.org/downloads.html">Spark homepage</a>.</li>
<li>Select the Spark release and package type as following and download the .tgz file.
<img src="https://i.imgur.com/KlKb21g.png"><br>
<img src="https://i.imgur.com/As7McNw.png"><br></li>
<li>Save the file to your local machine and click &#39;Ok&#39;.<br></li>
<li>Let&#39;s extract the file using the following command.<br>
<code>$ tar -xzf spark-2.4.6-bin-hadoop2.7.tgz</code></li>
</ol>
<h3 id="configuring-environment-variable-for-apache-spark-and-python">Configuring Environment Variable for Apache Spark and Python</h3>
<p>You need to open the ~/.bashrc or  ~/.zshrc file depending upon your current Mac version.</p>
<pre><code>export SPARK_HOME=&quot;/Downloads/spark&quot;
export PATH=$SPARK_HOME/bin:$PATH
export PYSPARK_PYTHON=python3
</code></pre><p>Open pyspark using &#39;pyspark&#39; command, and the final message will be shown as below.
<img src="https://i.imgur.com/9Mw3906.png"></p>
<h2 id="congratulations">Congratulations</h2>
<p>Congratulations, you have made it to the end of this tutorial!</p>
<p>In this tutorial, you&#39;ve learned about the installation of Pyspark, starting the installation of Java along with Apache Spark and managing the environment variables in Windows, Linux, and Mac Operating System.</p>
<p>If you would like to learn more about Pyspark, take DataCamp&#39;s <a href="https://www.datacamp.com/courses/introduction-to-pyspark">Introduction to Pyspark</a>.</p>


https://www.datacamp.com/community/tutorials/installation-of-pyspark

This tutorial will demonstrate the installation of Pyspark and hot to manage the environment variables in Windows, Linux, and Mac Operating System.

(Tutorial) Installation of Pyspark (All operating systems)

Installation of Pyspark (All operating systems)

<p>You can write your very own Python functions using the def keyword, function headers, docstrings, and function bodies. However, there&#39;s a quicker way to write functions on the fly, and these are called <code>lambda</code> functions because you use the keyword <code>lambda</code>.</p>
<p>Some function definitions are simple enough that they can be converted to a lambda function. By doing this, you write fewer lines of code, which is pretty awesome and will come in handy, especially when you&#39;re writing and maintaining big programs.</p>
<h2 id="-lambda-function"><code>lambda</code> Function</h2>
<p>Here we rewrite our function <code>raise_to_power</code> as a lambda function. After the keyword <code>lambda</code>, we specify the names of the arguments; then, we use a colon followed by the expression that specifies what we wish the function to return.</p>
<pre><code class="lang-python">raise_to_power = lambda x, y: x ** y

raise_to_power(2, 3)
</code></pre>
<pre><code>8
</code></pre><p>As mentioned, the <code>lambda</code> functions allow you to write functions in a quick and dirty way, so we wouldn&#39;t advise you to use them all the time, but there are situations when they come in handy, like the example below.</p>
<h2 id="-map-and-lambda-function"><code>map()</code> and <code>lambda</code> Function</h2>
<p>The <code>map</code> function takes two arguments, a function and a sequence such as a list and applies the function over all the elements of the sequence. We can pass <code>lambda</code> function to the <code>map</code> without even naming them, and in this case, we refer to them as anonymous functions.</p>
<p>In this example, we use <code>map()</code> on the <code>lambda</code> function, which squares all elements of the list, and we store the result in <code>square_all</code>.</p>
<pre><code class="lang-python">nums = [48, 6, 9, 21, 1]

square_all = map(lambda num: num ** 2, nums)

print(square_all)
</code></pre>
<pre><code>&lt;map object at 0x103e065c0&gt;
</code></pre><p>Printing <code>square_all</code> reveals that its a <code>map</code> object, so to see what it returns, we use list to turn it into a list and print the result.</p>
<pre><code class="lang-Python">print(list(square_all))
</code></pre>
<pre><code>[2304, 36, 81, 441, 1]
</code></pre><h2 id="interactive-example-of-writing-a-lambda-function">Interactive Example of Writing a <code>lambda</code> Function</h2>
<p>The below function <code>echo_word</code> takes 2 parameters: a string value, <code>word1</code>, and an integer value <code>echo</code>. It returns a string that is a concatenation of <code>echo</code> copies of <code>word1</code>.</p>
<p>You will convert the above simple function into a lambda function.</p>
<pre><code class="lang-Python">def echo_word(word1, echo):
    &quot;&quot;&quot;Concatenate echo copies of word1.&quot;&quot;&quot;
    words = word1 * echo
    return words
</code></pre>
<p>In the following example, you will:</p>
<ul>
<li>Define a lambda function <code>echo_word</code> using the variables <code>word1</code> and <code>echo</code>. Replicate what the original function definition for <code>echo_word()</code> does above.</li>
<li>Call <code>echo_word()</code> with the string argument <code>&#39;hey&#39;</code> and the value <code>5</code>, in that order. Assign the call to <code>result</code>. Finally, print the <code>result</code> variable.</li>
</ul>
<pre><code class="lang-Python"># Define echo_word as a lambda function: echo_word
echo_word = (lambda word1, echo: word1 * echo)

# Call echo_word: result
result = echo_word(&#39;hey&#39;, 5)

# Print result
print(result)
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<pre><code>heyheyheyheyhey
</code></pre><p><a href="https://campus.datacamp.com/courses/python-data-science-toolbox-part-1/lambda-functions-and-error-handling?ex=3">Try it for yourself</a>.</p>
<p>To learn more about Lambda functions, please see this video from our course <a href="https://campus.datacamp.com/courses/python-data-science-toolbox-part-1/lambda-functions-and-error-handling?ex=1">Python Data Science Toolbox (Part 1)</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/xTtwkfGLGus" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://learn.datacamp.com/courses/python-data-science-toolbox-part-1">Python Data Science Toolbox (Part 1)</a> course by Hugo Bowne-Anderson.</p>


https://www.datacamp.com/community/tutorials/python-lambda

(Tutorial) Python lambda

Python lambda

<p><code>pip</code> is a standard package manager used to install and maintain packages for Python. The Python standard library comes with a collection of built-in functions and built-in packages.</p>
<p>Data science packages like <strong>scikit-learn</strong> and <strong>statsmodel</strong> are <strong>NOT</strong> part of the Python standard library. They can be installed through <code>pip</code>, the standard package manager for Python, via the <strong>command line</strong>.</p>
<h2 id="pip-documentation">Pip Documentation</h2>
<p><code>Pip</code> has a variety of commands and option flags designed to manage Python packages.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1596502711/pip3_qsxf4u.png" /></center>

<p>You can print the <code>pip</code> version the same way you print the Python version. It is important that the <code>pip</code> version is compatible with the Python version. Here we see that <code>pip</code> 19.1.1 is compatible with Python 3.5.2.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1596502711/pip4_op1jgm.png" /></center>


<h2 id="upgrading-pip">Upgrading Pip</h2>
<p>If <code>pip</code> is giving you an upgrade warning, you can upgrade using pip itself:</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1596502711/pip5_hapljf.png" /></center>


<h2 id="viewing-a-pip-list">Viewing a Pip List</h2>
<p>Before you make any installs, it is a good idea to see what is already installed. You can use <code>pip list</code> in the command line, and it will display the Python packages in your current working environment in alphabetical order.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1596502711/pip6_e4dqrf.png" /></center>


<h2 id="installing-the-scikit-learn-package">Installing the <code>scikit-learn</code> Package</h2>
<p>In the following example, you will learn how you can install the <code>scikit-learn</code> package, which will install the other necessary dependencies.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1596502711/pip7_yesydf.png" /></center>

<p>You may notice from the logs that more then the <code>scikit-learn</code> package is being installed. This is because <code>pip</code> will install any other packages that <code>scikit-learn</code> depends on. These other packages are called dependencies.</p>
<h2 id="installing-a-specific-package-version">Installing a Specific Package Version</h2>
<p><code>pip</code> will always install the latest version, so if you wish to install an older version of <code>scikit-learn</code>, all you need to do is specify it in the installation statement use a double equal sign:</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1596502711/pip8_u1r1wi.png" /></center>

<h2 id="upgrading-packages">Upgrading Packages</h2>
<p>If the package you are looking to use is already installed but simply out of date. You can update the package in a similar way we upgraded <code>pip</code> above.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1597687262/Screen_Shot_2020-08-17_at_1.57.22_PM_hwre4z.png" /></center>

<p>This upgrade will also upgrade any necessary dependency packages as well, automatically.</p>
<h2 id="installing-and-upgrading-the-scikit-learn-and-statsmodel-package">Installing and Upgrading the <code>scikit-learn</code> and <code>statsmodel</code> Package</h2>
<p>To <code>pip install</code> more than one Python package, the packages can be listed in line with the same <code>pip install</code> command as long as they are separated with spaces. Here we are installing both <code>scikit-learn</code> and the <code>statsmodel</code> package in one line of code.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1597687262/Screen_Shot_2020-08-17_at_1.58.12_PM_bnw7wy.png" /></center>

<p>You can also upgrade multiple packages in one line of code.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1597687262/Screen_Shot_2020-08-17_at_1.58.19_PM_zpapbk.png" /></center>

<h2 id="installing-packages-with-requirements-txt-">Installing Packages With <code>requirements.txt</code></h2>
<p>If you want to install many packages at once, you can save them one package per line in a text file called <code>requirements.txt</code>. If we preview the file, it looks like this:</p>
<p><cneter><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1597687262/Screen_Shot_2020-08-17_at_1.58.56_PM_clqcim.png" /></center></p>
<p>It is conventional for Python package developers to create a <code>requirements.txt</code> file in their Github repositories listing all dependencies for <code>pip</code> to find and install.</p>
<p>The <code>-r</code> option flag in <code>pip</code> allows <code>pip install</code> to install packages from the file specified after the option flag. Keep in mind that naming this file <code>requirements.txt</code> is conventional but not required.</p>
<p>Using our examples, <code>pip install -r requirements.txt</code> will have the same effect as <code>pip install scikit-learn statsmodel</code>. Typing out each package could get messy if you needed to install ten packages. Using the <code>requirements.txt</code> file is much cleaner.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1597687262/Screen_Shot_2020-08-17_at_2.00.05_PM_cbbiqu.png" /></center>

<h2 id="interactive-example-of-installing-python-dependencies">Interactive Example of Installing Python Dependencies</h2>
<p>In the following example, you will work through the setup process for making sure your Python environment has the proper library dependencies installed prior to executing a Python model script.</p>
<p>You will instantiate the <code>requirements.txt</code> document and add the <code>scikit-learn</code> library to the <code>requirements.txt</code> file.</p>
<pre><code class="lang-Pip"># Add scikit-learn to the requirements.txt file
echo &quot;scikit-learn&quot; &gt; requirements.txt

# Preview file content
cat requirements.txt
</code></pre>
<p>When we run the code above, it produces the following result:</p>
<pre><code>$ # Add scikit-learn to the requirements.txt file
$ echo &quot;scikit-learn&quot; &gt; requirements.txt
$
$ # Preview file content
$ cat requirements.txt
scikit-learn
</code></pre><p><a href="https://campus.datacamp.com/courses/data-processing-in-shell/data-pipeline-on-the-command-line?ex=6">Try it for yourself</a>.</p>
<p>To learn more about using pip in the command line, please see this video from our course <a href="https://campus.datacamp.com/courses/data-processing-in-shell/data-pipeline-on-the-command-line?ex=4">Data Processing in Shell</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/h3D-wC7XfI0" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://learn.datacamp.com/courses/data-processing-in-shell">Data Processing in Shell</a> course by Susan Sun.</p>


https://www.datacamp.com/community/tutorials/python-install-pip

(Tutorial) Python Install Pip

Python Install Pip

<p>Often, a DataFrame will contain columns that are not useful to your analysis. Such columns should be dropped from the DataFrame to make it easier for you to focus on the remaining columns.</p>
<p>The columns can be removed by specifying label names and corresponding axis, or by specifying index or column names directly. When using a multi-index, labels on different levels can be removed by specifying the level.</p>
<h2 id="-drop-method"><code>.drop()</code> Method</h2>
<p>Let&#39;s compare missing value counts with the shape of the dataframe. You will notice that the <code>county_name</code> column contains as many missing values as rows, meaning that it only contains missing values.</p>
<pre><code class="lang-python">ri.isnull().sum()
</code></pre>
<pre><code>state                            0
stop_date                        0
stop_time                        0
county_name                  91741
driver_gender                 5205
driver_race                   5202
...
</code></pre><pre><code class="lang-python">ri.shape
</code></pre>
<pre><code>91741, 15
</code></pre><p>Since it contains no useful information, this column can be dropped using the <code>.drop()</code> method.</p>
<p>Besides specifying the column name, you need to specify that you are dropping from the columns axis and that you want the operation to occur in place, which avoids an assignment statement as shown below:</p>
<pre><code class="lang-python">ri.drop(&#39;county_name&#39;,
  axis=&#39;columns&#39;, inplace=True)
</code></pre>
<h2 id="-dropna-method"><code>.dropna()</code> Method</h2>
<p>The <code>.dropna()</code> method is a great way to drop rows based on the presence of missing values in that row.</p>
<p>For example, using the dataset above, let&#39;s assume the stop_date and stop_time columns are critical to our analysis, and thus a row is useless to us without that data.</p>
<pre><code class="lang-python">ri.head()
</code></pre>
<pre><code>    state   stop_date    stop_time    driver_gender   driver_race
0      RI  2005-01-04        12:55                M         White
1      RI  2005-01-23        23:15                M         White
2      RI  2005-02-17        04:15                M         White
3      RI  2005-02-20        17:15                M         White
4      RI  2005-02-24        01:20                F         White
</code></pre><p>We can tell pandas to drop all rows that have a missing value in either the <code>stop_date</code> or <code>stop_time</code> column. Because we specify a subset, the <code>.dropna()</code> method only takes these two columns into account when deciding which rows to drop.</p>
<pre><code class="lang-python">ri.dropna(subset=[&#39;stop_date&#39;, &#39;stop_time&#39;], inplace=True)
</code></pre>
<h2 id="interactive-example-of-dropping-columns">Interactive Example of Dropping Columns</h2>
<p>In this example, you will drop the <code>county_name</code> column because it only contains missing values, and you&#39;ll drop the <code>state</code> column because all of the traffic stops took place in one state (Rhode Island). Thus, these columns can be dropped because they contain no useful information. The number of missing values in each column has been printed to the console for you.</p>
<ul>
<li>Examine the DataFrame&#39;s <code>.shape</code> to find out the number of rows and columns.</li>
<li>Drop both the <code>county_name</code> and <code>state</code> columns by passing the column names to the <code>.drop()</code> method as a list of strings.</li>
<li>Examine the <code>.shape</code> again to verify that there are now two fewer columns.</li>
</ul>
<pre><code class="lang-python"># Examine the shape of the DataFrame
print(ri.shape)

# Drop the &#39;county_name&#39; and &#39;state&#39; columns
ri.drop([&#39;county_name&#39;, &#39;state&#39;], axis=&#39;columns&#39;, inplace=True)

# Examine the shape of the DataFrame (again)
print(ri.shape)
</code></pre>
<p>When you run the above code, it produces the following result:</p>
<pre><code>(91741, 15)
(91741, 13)
</code></pre><p><a href="https://campus.datacamp.com/courses/analyzing-police-activity-with-pandas/preparing-the-data-for-analysis?ex=3">Try it for yourself</a>.</p>
<p>To learn more about dropping columns in pandas, please see this video from our course, <a href="https://campus.datacamp.com/courses/analyzing-police-activity-with-pandas/preparing-the-data-for-analysis?ex=1">Analyzing Police Activity with pandas</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/NEsgHxl-NX0?start=155" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://learn.datacamp.com/courses/introduction-to-data-visualization-with-ggplot2">Analyzing Police Activity with pandas</a> course by Kevin Markham.</p>


https://www.datacamp.com/community/tutorials/pandas-drop-column

(Tutorial) How To Drop  Columns in Pandas

How To Drop  Columns in Pandas

<p>Numpy&#39;s random module, a suite of functions based on pseudorandom number generation. Random means something that can not be predicted logically.</p>
<h2 id="-np-random-seed-function"><code>np.random.seed()</code> Function</h2>
<p>In this example, you will simulate a coin flip. You will use the function <code>np.random()</code>, which draws a number between 0 and 1 such that all numbers in this interval are equally likely to occur.</p>
<p>If the number you draw is less than 0.5, which has a 50% chance of happening, you say heads and tails otherwise. This type of result where results are either True (Heads) or False (Tails) is referred to as Bernoulli trial.</p>
<p>The pseudorandom number works by starting with an integer called a seed and then generates numbers in succession. The same seed gives the same sequence of random numbers, hence the name &quot;pseudo&quot; random number generation. If you want to have reproducible code, it is good to seed the random number generator using the <code>np.random.seed()</code> function.</p>
<p>To do the coin flips, you import <code>NumPy</code>, seed the random number generator, and then draw four random numbers. You can specify how many random numbers you want with the <code>size</code> keyword.</p>
<pre><code class="lang-Python">import numpy as np
np.random.seed(42)
random_numbers = np.random.random(size=4)
random_numbers
</code></pre>
<pre><code>array([0.3745012, 0.95071431, 0.73199394, 0.59865848])
</code></pre><p>The first number you get is less than 0.5, so it is heads while the remaining three are tails. You can show this explicitly using the less than operation, which gives you an array with boolean values, True for heads while False for tails.</p>
<pre><code class="lang-Python">heads = random_numbers &lt; 0.5
heads
</code></pre>
<pre><code>array([True, False, False, False], dtype=bool)
</code></pre><p>Finally, you can compute the number of heads by summing the array of booleans <code>heads</code>, because in numerical context, Python treats True as one and False as zero.</p>
<pre><code class="lang-Python">np.sum(heads)
</code></pre>
<pre><code>1
</code></pre><h2 id="simulating-four-random-coin-flips">Simulating Four Random Coin Flips</h2>
<p>In the following example, we want to know the probability of getting four heads if we were to repeat the four flips of the coin over and over again. We can do this with a for loop.</p>
<p>We first initialize the count to zero. We then do repeat 10,000 repeats of the four-flip trials. If a given trial had four heads, we would increase the count.</p>
<p>So what is the probability of getting all four heads? It&#39;s the number of times you got all heads, divided by the total number of trials. The result is about 0.06.</p>
<pre><code class="lang-Python">n_all_heads = 0

# Initialize number of 4-heads trials
for _ in range(10000):
         heads = np.random.random(size=4) &lt; 0.5
         n_heads = np.sum(heads)
         if n_heads == 4:
             n_all_heads += 1

n_all_heads / 10000
</code></pre>
<pre><code>0.0621
</code></pre><h2 id="interactive-example-of-generating-random-numbers">Interactive Example of Generating Random Numbers</h2>
<p>In this example, we&#39;ll generate lots of random numbers between zero and one, and then plot a histogram of the results. If the numbers are truly random, all bars in the histogram should be of (close to) equal height.</p>
<p>For this example you will need to:</p>
<ul>
<li>Seed the random number generator with <code>np.random.seed</code> using the seed <code>42</code>.</li>
<li>Initialize an empty array, random_numbers, of 100,000 entries to store the <code>random numbers</code>. Make sure you use <code>np.empty(100000)</code> to do this.</li>
<li>Write a <code>for</code> loop to draw 100,000 random numbers using <code>np.random.random()</code>, storing them in the <code>random_numbers</code> array. To do so, loop over <code>range(100000)</code>.</li>
<li>Plot a histogram of <code>random_numbers</code>. It is not necessary to label the axes in this case because we are just checking the random number generator.</li>
</ul>
<pre><code class="lang-Python"># Seed the random number generator
np.random.seed(42)

# Initialize random numbers: random_numbers
random_numbers = np.empty(100000)

# Generate random numbers by looping over range(100000)
for i in range(100000):
    random_numbers[i] = np.random.random()

# Plot a histogram
_ = plt.hist(random_numbers)

# Show the plot
plt.show()
</code></pre>
<p>When we run the above code, it produces the following result:</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1597694766/nprand2_wwnymj.png" /></center>

<p><a href="https://campus.datacamp.com/courses/statistical-thinking-in-python-part-1/thinking-probabilistically-discrete-variables?ex=5">Try it for yourself</a>.</p>
<p>To learn more about random number generators and hacker statistics, please see this video from our course <a href="https://campus.datacamp.com/courses/statistical-thinking-in-python-part-1/thinking-probabilistically-discrete-variables?ex=4">Statistical Thinking in Python (Part 1)</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/NX4eCKJHb5M?start=50" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

<p>This content is taken from DataCamp’s <a href="https://www.datacamp.com/courses/statistical-thinking-in-python-part-1">Statistical Thinking in Python (Part 1)</a> course by Justin Bois.</p>


https://www.datacamp.com/community/tutorials/numpy-random

(Tutorial) Random Number Generator Using Numpy

Random Number Generator Using Numpy

<p>Python&#39;s rename column is a method used to change the column names with pandas&#39; rename function. It&#39;s useful when you load a tabular dataset that has no column names or if you want to assign different names to specific columns.</p>
<h2 id="syntax">Syntax</h2>
<p>Its syntax is given as:</p>
<pre><code class="lang-python">df.rename(columns = d, inplace = False)
</code></pre>
<p>Where <code>d</code> is a dictionary, and the keys are the columns you want to change. The values are the new names for these columns. The code <code>inplace = False</code> means the result would be stored in a new DataFrame instead of the original one.</p>
<h2 id="renaming-columns">Renaming Columns</h2>
<p>You have the following dataset called <code>df2_melted</code>.</p>
<pre><code class="lang-python">print(df2_melted)
</code></pre>
<pre><code>        Country variable     value
0   Afghanistan    Y1980  21.48678
1       Albania    Y1980  25.22533
2       Algeria    Y1980  22.25703
3   Afghanistan    Y1981  21.46552
4       Albania    Y1981  25.23981
5       Algeria    Y1981  22.34745
6   Afghanistan    Y1982  21.45145
7       Albania    Y1982  25.25636
8       Algeria    Y1982  22.43647
9   Afghanistan    Y1983  21.43822
10      Albania    Y1983  25.27176
11      Algeria    Y1983  22.52105
</code></pre><p>With this dataset, we want to do the following:</p>
<ul>
<li>Import <code>pandas</code> as pd.</li>
<li>Rename the <code>variable</code> column of <code>df2_melted</code> to <code>Year</code> and the <code>value</code> column to <code>Income</code> and assign it to <code>df2_tidy</code>.</li>
<li>Finally, print the updated dataframe.</li>
</ul>
<pre><code class="lang-python"># Import pandas
import pandas as pd

# Rename the columns of df2_melted: df2_tidy
df2_tidy = df2_melted.rename(columns = {&#39;variable&#39;: &#39;Year&#39;, &#39;value&#39;: &#39;Income&#39;}, inplace = False)

# Print out df2_tidy
print(df2_tidy)
</code></pre>
<p>When you run the above code, it produces the following result:</p>
<pre><code>        Country   Year    Income
0   Afghanistan  Y1980  21.48678
1       Albania  Y1980  25.22533
2       Algeria  Y1980  22.25703
3   Afghanistan  Y1981  21.46552
4       Albania  Y1981  25.23981
5       Algeria  Y1981  22.34745
6   Afghanistan  Y1982  21.45145
7       Albania  Y1982  25.25636
8       Algeria  Y1982  22.43647
9   Afghanistan  Y1983  21.43822
10      Albania  Y1983  25.27176
11      Algeria  Y1983  22.52105
</code></pre><p><a href="https://campus.datacamp.com/courses/tidy-data-in-python-mini-course/tidy-data-in-python?ex=3">Try it for yourself</a>.</p>
<p>To learn more about Tidy Data and Messy Data, please check out the interactive exercise from our course <a href="https://campus.datacamp.com/courses/tidy-data-in-python-mini-course/tidy-data-in-python?ex=1">Tidy Data in Python</a>.</p>
<p>This content is taken from DataCamp’s <a href="https://learn.datacamp.com/courses/regular-expressions-in-python">Tidy Data in Python</a> course by Vincent Lan.</p>


https://www.datacamp.com/community/tutorials/python-rename-column

community

datacamp

Tutorials