http://datacamp-community-prod.s3.amazonaws.com/acfa4325-1d43-4542-8ce4-bea2d287db10

This PySpark cheat sheet with code samples covers the basics like initializing Spark in Python, loading data, sorting, and repartitioning.

PySpark Cheat Sheet: Spark in Python

http://datacamp-community-prod.s3.amazonaws.com/ba1fe95a-8b70-4d2f-95b0-bc954e9071b0

This NumPy cheat sheet helps you to tackle the basics of this Python scientific computing library!

NumPy Cheat Sheet: Data Analysis in Python

https://www.datacamp.com/community/podcast/successful-digital-transformation-puts-people-first

<p><em>Bhavin Patel, Director of Analytics and Innovation at Johnson &amp; Johnson, discusses how to drive success with digital and data transformation, why investing in people is the most important aspect of becoming a digital-first organization, how data science accelerates outcomes in supply chain, and more.</em></p>

Bhavin Patel, Director of Analytics and Innovation at Johnson & Johnson, discusses how to drive success with digital and data transformation and more.

Successful Digital Transformation Puts People First

https://www.datacamp.com/blog/successful-digital-transformation-puts-people-first

https://www.datacamp.com/community/podcast/the-gradual-process-of-building-a-data-strategy

<p>Vijay Yadav, Director of Quantitative Sciences &amp; Head of Data Science at Center for Mathematical Sciences at <a target="_blank" rel="noopener noreferrer" href="https://www.linkedin.com/company/merck/">Merck</a> breaks down the different pillars of a scalable data strategy.</p>

Vijay Yadav, Director of Quantitative Sciences & Head of Data Science at Center for Mathematical Sciences at Merck breaks down the different pillars of a scalable data strategy.

The Gradual Process of Building a Data Strategy

https://www.datacamp.com/blog/the-gradual-process-of-building-a-data-strategy

https://www.datacamp.com/community/news/creating-smart-cities-with-data-science-ofs0psrh048

Amen Ra Mashariki deep dives into the state of data literacy in government agencies, the use cases he worked on to make the City of New York smarter, and more!

Creating Smart Cities With Data Science

https://www.datacamp.com/community/news/spacy-cheat-sheet-advanced-nlp-in-python-nryt9kwegmb

Check out the first official spaCy cheat sheet! A handy two-page reference to the most important concepts and features.

spaCy Cheat Sheet: Advanced NLP in Python

https://www.datacamp.com/community/news/python-for-finance-algorithmic-trading-vrfjxb6qk2

This Python for Finance tutorial introduces you to the basics of algorithmic trading in Python

Python For Finance: Algorithmic Trading

https://www.datacamp.com/community/news/object-oriented-programming-in-python-tutorial-tt4da4d8rac

Object-Oriented programming is a widely used concept to write powerful applications. As a data scientist, you will be required to write applications to process your data, among a range of other things.  In this tutorial, you will discover the basics of object-oriented programming in Python.

Object-Oriented Programming in Python: Tutorial

https://www.datacamp.com/community/blog/how-data-culture-unlock-digital-transformation

Chief Data & Innovation Officer at Gulf Bank, Mai AlOwaish, outlines how a people and data-first approach can unlock the value of digital transformation for organizations across industries.

How Data & Culture Unlock Digital Transformation

<center><a target="_blank" href="https://www.datacamp.com/groups/business"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1624651228/schedule_a_demo_bfhlx1.png" /></a></center><br>
<p>In a recent episode of <a target="_blank" href="https://www.datacamp.com/community/blog/the-data-storytelling-skills-data-teams-need">DataFramed</a>, Andy Cotgrave, technical evangelist at Tableau, shared the importance of data storytelling in driving change with analytics. In this two-part blog post, we deep dive into seven concrete tips that Andy provided on telling better data stories.</p>
<p>Read <a target="_blank" href="https://www.datacamp.com/community/blog/seven-tricks-for-better-data-storytelling-1">Part I</a> if you haven't already!</p>
<h2 id="4-use-a-narrative-structure-in-data-stories-">4. Use a narrative structure in data stories.</h2>
<p>Stories generally follow a common arc – the protagonist faces a challenge, goes on to resolve it, then returns to normalcy. An example of this arc is the Hero’s Journey.</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1643320567/image8_grp6ly.png" />Figure 1: The hero’s journey</center><br>

<p>The audience is <a target="_blank" href="https://www.forbes.com/sites/kateharrison/2015/01/20/a-good-presentation-is-about-data-and-story/?sh=7599eca6450f">22 times more likely</a> to remember a fact when told a story. Clearly, data visualizations stand to benefit from adopting the age-old narrative structure.</p>
<p>Adopting the narrative structure, a data story starts with a background on the current situation, continues with evidence that builds up to the central insight, and ends with a call to action. (Figure 2)</p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1643320742/image4_nnm9k9.png" />Figure 2. The narrative structure of a data story</center><br>


<p>An excellent example of a data story, the <a target="_blank" href="https://www.gatesfoundation.org/goalkeepers/report/2021-report/#LongTermInvestmentsPayOff">Goalkeeper’s Report by the Gates Foundation</a> demonstrated how the pandemic exacerbated inequality. (Figures 3A, 3B, and 3C)</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1643320251/poverty_s2wd18.gif" />Figure 3A: To set the scene, the data story first established the impact of COVID-19 on poverty-stricken countries. [<a target="_blank" href="https://www.gatesfoundation.org/goalkeepers/report/2021-report/#LongTermInvestmentsPayOff">Source</a>]</center><br>

<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1643320508/image7_bmdtdx.png" />Figure 3B: The data story provided rising insights by highlighting the widening chasm between the haves and the have-nots pre-pandemic and post-pandemic.</center><br>

<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1643319219/image2_lyw5ro.png" />Figure 3C: The data story closes with a call to action–for greater investments in women, communities, and innovations.</center>

<h2 id="5-adapt-data-stories-to-the-medium">5. Adapt data stories to the medium</h2>
<p>It is tempting to reuse existing charts across multiple mediums. Data storytellers who do so are doing themselves a disservice.</p>
<p>Charts that are not optimized for their medium might fail to deliver their message effectively. A chart originally designed for email might contain minuscule texts that cannot be read on a screen.</p>
<p>The Economist portrayed how data stories can be adapted from print media to social media. The printed chart (designed for serious readers) was dense in information. It was streamlined for Instagram (designed for casual scrollers).</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1643320591/image5_qjtkiu.png" />Figure 4: The Economist’s printed chart (left) is adapted for social media (right) [<a target="_blank" href="https://medium.economist.com/charting-new-territory-7f5afb293270">Source</a>]</center>

<h2 id="6-declutter-your-data-stories">6. Declutter your data stories</h2>
<p>Clutter makes data stories harder to read. The additional cognitive load distracts the audience from the main message.</p>
<p>The onus is on the data storyteller to identify and remove superfluous elements that do not contribute to the intended message. The result of decluttering is striking.</p>
<p>In particular, our whitepaper “<a target="_blank" href="https://www.datacamp.com/resources/whitepapers/8-rules-for-better-data-storytelling">8 Rules for Better Data Storytelling</a>” suggests data storytellers to:</p>
<ul>
<li>Remove chart borders</li>
<li>Remove gridlines or axes</li>
<li>Clean up axis labels</li>
<li>Label data directly (as opposed to using a legend)</li>
<li>Remove data markers</li>
<li>Use special effects (bold, underline, italics, shadows) sparingly</li>
</ul>
<h2 id="7-explain-data-stories-in-stages">7. Explain data stories in stages</h2>
<p>When presenting complex data stories, one practical tip is to reveal the elements of the chart gradually. When the information is presented in digestible pieces, the audience is more likely to engage with the presentation.</p>
<p>A prime example is the talk “<a target="_blank" href="https://www.youtube.com/watch?v=jbkSRLYSojo">The Joy of Stats</a>” by Hans Rosling. Watch how Rosling masterfully told his data story.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/jbkSRLYSojo" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<h2 id="data-storytelling-is-here-to-stay">Data storytelling is here to stay</h2>
<p>Data storytelling will soon <a target="_blank" href="https://www.forbes.com/sites/brentdykes/2020/01/22/why-data-storytellers-will-define-the-next-decade-of-data/?sh=56a32d24fb9c">become a necessary skill set</a> for all in the coming decade as data insights become a cornerstone of organizations. To learn more tips and tricks on data storytelling, listen to Andy’s podcast, or learn from his most recent webinar on DataCamp.</p>
<br><center><a target="_blank" href="https://www.datacamp.com/groups/business"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1624651228/schedule_a_demo_bfhlx1.png" /></a></center>

https://www.datacamp.com/community/blog/seven-tricks-for-better-data-storytelling-2

Seven Tricks for Better Data Storytelling: Part II

<h2>Conceptos básicos de Python</h2><p>Una introducción a los conceptos básicos de Python. Aprende como utilizar Python de forma interactiva y a tráves de una consola. Crea tus primeras variables y familiarícese con los tipos de datos básicos en Python.</p><h2>Listas de Python</h2><p>Aprende a almacenar, acceder y manipular datos en una lista: el primer paso hacia un trabajo eficiente con gran cantidad de datos.</p><h2>Funciones y paquetes</h2><p>Para aprovechar el código que los desarrolladores de Python han escrito, aprenderás sobre funciones, métodos y paquetes. ¡Esto te ayudará a reducir la cantidad de código que necesitas para resolver problemas!</p><h2>NumPy</h2><p>NumPy es un poderoso paquete de Python para hacer ciencia de datos eficientemente. Aprende como trabajar con arrays de NumPy, una alternativa más rápida y poderosa a las listas, y toma tus primeros pasos en la exploración de datos.</p>

https://www.datacamp.com/community/open-courses/introduccion-a-python

Aprenderás acerca de poderosas maneras de almacenar y manipular datos, así como también geniales herramientas de ciencia de datos para comenzar tus propios análisis. 

Introducción a Python

<h2>Introduction</h2><p>Introduction to data.world</p><h2>Working with datasets</h2><p>Working with data.world datasets</p><h2>Working with multiple datasets</h2><p>Working with multiple datasets</p><h2>Querying with data.world</h2><p>Querying with data.world</p><h2>Wrap up & next steps</h2><p>Setup your own data.world account and apply what you've learned and more!</p>

https://www.datacamp.com/community/open-courses/intro-to-dataworld-in-python

data.world allows data people to discover and share cool data. In this course, you'll learn to use the data.world Python SDK to pull data and metadata into your environment.

Intro to data.world in Python

<h2 id="table-of-contents-">Table of Contents:</h2>
<ul>
<li><a href="#what-is-the-normal-equation-">What is the Normal Equation?</li>
<li><a href="#the-normal-equation-vs-gradient-descent">The Normal Equation vs Gradient Descent<ul>
<li><a href="#gradient-descent">Gradient Descent</li>
<li><a href="#how-do-they-differ-">How do they differ?</li>
</ul>
</li>
<li><a href="#deciphering-when-to-use-the-normal-equation">Deciphering When to Use the Normal Equation</li>
<li><a href="#the-normal-equation-from-scratch-in-python">The Normal Equation from Scratch in Python</li>
<li><a href="#frequently-asked-questions-faqs-">FAQs</li>
</ul>
<p>Most problems we encounter have several ways they can be solved. For example, if we wanted to get from one side of a room to another, we may decide to walk around the room until we arrive at the opposing side, or we could just cut across.</p>
<p>The normal equation is just an emphasis of this concept. It’s just another way to solve a problem. What problem did you ask? We’ll cover that in the remainder of this article. For now, all you need to know is that it&#39;s an effective approach that can help you save lots of time when implementing linear regression under certain conditions. </p>
<p>Let’s dive deeper…</p>
<h2 id="what-is-the-normal-equation-">What is the Normal Equation?</h2>
<p>The normal equation is a closed-form solution used to find the value of θ that minimizes the cost function. Another way to describe the normal equation is as a one-step algorithm used to analytically find the coefficients that minimize the loss function. Both descriptions work, but what exactly do they mean? We will start with linear regression.</p>
<p>Linear regression makes a prediction, y_hat, by computing the weighted sum of input features plus a bias term. Mathematically it can be represented as follows:</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1649238228/image5_iengil.png"></center>
<br>
<p>Where θ represents the parameters and n is the number of features.</p>
<p>Essentially, all that occurs in the above equation is the dot product of θ, and x is being summed. Thus, a more concise way to represent this is to use its vectorized form:</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1649238228/image6_t3omdh.png"></center>
<br>
<p>h(θ) is the hypothesis function.</p>
<p>Given this approximate target function, we can use our model to make predictions. To determine if our model has learned well, it’s important we measure the performance of our model on the training data. For this purpose, we compute a loss function. The goal of the training process is to find the values of theta (θ) that minimize the loss function.  </p>
<p>Here’s how we can represent our loss function mathematically:</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1649238227/image3_p2c9gn.png"></center>
<br>
<p>In the above equation, theta (θ) is a n + 1 dimensional vector, and our loss function is a function of the vector value. Consequently, the partial derivative of the loss function, J, has to be taken with respect to every parameter of θ_j in turn. All of them must equal zero. Following this process and solving for all of the values of θ from θ_0 to θ_n will result in the values of θ that minimize the loss function.</p>
<p>Working through the solution to the parameters θ_0 to θ_n using the process described above results in an extremely involved derivation procedure. There is indeed a faster solution.</p>
<p>Take a look at the formula for the normal equation:</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1649238228/image4_sckrl4.png"></center>

<p>Where:</p>
<p>θ → The parameters that minimize the loss function
X → The input feature values for each instance
y → The vector of output values for each instance</p>
<h2 id="the-normal-equation-vs-gradient-descent">The Normal Equation vs Gradient Descent</h2>
<p>While both methods seek to find the parameters theta (θ) that minimize the loss function, the method of approach differs greatly between the two solutions.</p>
<p>Since we’ve already covered how the normal equation works in ‘<a href="#what-is-the-normal-equation-">What is the Normal Equation?</a>’, in this section, we will briefly touch on gradient descent and then provide ways in which the two techniques differ.</p>
<h3 id="gradient-descent">Gradient Descent</h3>
<p>Gradient descent is one of the most used machine learning algorithms in machine learning. It’s deployed to iteratively find the parameters theta (θ) that minimize the loss function.</p>
<p>The process starts by first evaluating the model&#39;s performance. Next, the partial derivative is calculated from the loss function which is used to reference the slope at its current point. Lastly, we take steps proportional to the negative gradient to make a descent to the minimum of the loss function by updating the current set of parameters - see formula below.</p>
<br><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1649238227/image1_b93ju9.png"></center>
<br>
<p>This process is repeated until convergence at the minimum of the loss function.</p>
<h3 id="how-do-they-differ-">How do they Differ?</h3>
<p>The most obvious way in which the normal equation differs from gradient descent is that it’s analytical. Gradient descent takes an iterative approach which means our parameters are updated gradually until convergence. Another subtle difference baked into this is that gradient descent requires us to define a learning rate that controls the size of the steps taken towards the minimum of the loss function. The normal equation doesn’t require us to define a learning rate because we are not taking iterative steps - we get the results directly.</p>
<p>Also, feature scaling is not required when we use the normal equation approach; we typically perform feature scaling to ensure our features have a similar range of values. This is because gradient descent is sensitive to the ranges of our data points. Failing to normalize our features when we use gradient descent may introduce skewness into the contour plot of the loss function, but the normal equation does not suffer from this problem.</p>
<h2 id="deciphering-when-to-use-the-normal-equation">Deciphering When to Use the Normal Equation</h2>
<p>The best way to know if you should use the normal equation over gradient descent is to understand its disadvantages.</p>
<p>Computing the normal equation becomes computationally challenging when the number of features in our dataset is large. The reason for this is that in order to solve for the parameters θ, the term (X’ X)^-1 must be computed. Computing X’ X produces an n x n matrix and for most computer implementations, converting a matrix grows approximately as the cube of the dimensions of the matrix. This means the inverse operation runs in O(n^3) runtime complexity which makes the normal equation run extremely slow when n is very large - learn more about <a target="_blank" href="https://en.wikipedia.org/wiki/Time_complexity">time complexity</a>.</p>
<p>Thus, it’s best to use gradient descent when the number of features in the dataset is large. Andrew Ng, a prominent machine learning and AI expert, recommends you should consider using gradient descent when the number of features, n, is greater than 10,000. For 10,000 features or less, you may be better off using the normal equation since you’re not required to select a value for the learning rate which means you have one less hyperparameter to tune.</p>
<h2 id="the-normal-equation-from-scratch-in-python">The Normal Equation from Scratch in Python</h2>
<p>Let’s generate a regression problem to test this equation:</p>
<pre><code class="lang-python">import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_regression

# Generate a regression problem
X, y = make_regression(
    n_samples=100,
    n_features=2,
    n_informative=2,
    noise = 10,
    random_state=25
    )

# Visualize feature at index 1 vs target
plt.subplots(figsize=(8, 5))
plt.scatter(X[:, 1], y, marker=&#39;o&#39;)
plt.xlabel(&quot;Feature at Index 1&quot;)
plt.ylabel(&quot;Target&quot;)
plt.show()
</code></pre><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1649238227/image2_c75gme.png"></center>
<br>
<p>Here’s where we will implement the normal equation:</p>
<pre><code class="lang-python"># adds x0 = 1 to each instance
X_b = np.concatenate([np.ones((len(X), 1)), X], axis=1)
# calculate normal equation
theta_best = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)
# best values for theta
intercept, *coef = theta_best
print(f&quot;Intercept: {intercept}\n\
Coefficients: {coef}&quot;)

Intercept: 0.35921242677977794
Coefficients: [6.129199175400593, 96.44309685893134]
</code></pre><p>Let’s put our model to the test by making a prediction:</p>
<pre><code class="lang-python"># making a new sample
new_sample = np.array([[-2, 0.25]])
# adding a bias term to the instance
new_sample_b = np.concatenate([np.ones((len(new_sample), 1)), new_sample], axis=1)
# predicting the value of our new sample
new_sample_pred = new_sample_b.dot(theta_best)
print(f&quot;Prediction: {new_sample_pred}&quot;)
Prediction: [12.21158829]
</code></pre><p>Whenever you implement a machine learning algorithm from scratch, it’s always helpful to have a method of validating your solution; <a target="_blank" href="https://scikit-learn.org/">Scikit-learn</a> is one of the most popular machine learning libraries in Python. It features several implementations of different algorithms, including linear regression, which we will be using to validate our normal equation.</p>
<pre><code class="lang-python">from sklearn.linear_model import LinearRegression

lr = LinearRegression()
lr.fit(X, y)

print(f&quot;Intercept: {lr.intercept_}\n\
Coefficients: {lr.coef_}&quot;)
print(f&quot;Prediction: {lr.predict(new_sample)}&quot;)

Intercept: 0.3592124267797807
Coefficients: [ 6.12919918 96.44309686]
Prediction: [12.21158829]
</code></pre><p>The solutions are approximately equal so we can confirm our solution is correct.</p>
<h2 id="frequently-asked-questions-faqs-">Frequently Asked Questions (FAQs)</h2>
<h4 id="what-is-the-normal-equation-in-machine-learning-">What is the normal equation in machine learning?</h4>
<p>The normal equation is an analytical approach to finding the value of θ that minimizes the loss function without having to iterate.</p>
<h4 id="when-should-i-use-the-normal-equation-instead-of-gradient-descent-">When should I use the normal equation instead of gradient descent?</h4>
<p>It’s better to use the normal equation when we have fewer features. Computing the normal equation becomes computationally challenging when the number of features in our dataset is large.</p>
<h4 id="can-the-normal-equation-be-used-for-logistic-regression-">Can the normal equation be used for logistic regression?</h4>
<p>Unfortunately no. There’s only one conditional model in classification theory that has a closed-form solution - linear regression.</p>
<h4 id="what-s-the-difference-between-the-normal-equation-and-gradient-descent-">What’s the difference between the normal equation and gradient descent?</h4>
<p>The obvious difference between the normal equation and gradient descent is that the normal equation uses an analytical approach to find the minimum of the loss function, whereas gradient descent uses an iterative approach. Another difference is that you don’t have to tune a learning rate to compute the normal equation since only one step is taken to find the parameters θ.</p>


https://www.datacamp.com/community/tutorials/tutorial-normal-equation-for-linear-regression

Learn what the normal equation is and how can you use it to build machine learning models. 

Normal Equation for Linear Regression Tutorial

https://www.datacamp.com/community/tutorials/tutorial-how-to-execute-sql-queries-in-r-and-python

Learn easy and effective ways to run SQL queries in Python and R for data analysis and database management. 

community

datacamp

python

News (103)

Creating Smart Cities With Data Science

Python For Finance: Algorithmic Trading

Creating Smart Cities With Data Science

Tutorials (242)

Normal Equation for Linear Regression Tutorial

How to Execute SQL Queries in Python and R Tutorial

Cheat Sheets (15)

PySpark Cheat Sheet: Spark in Python

Open Courses (5)

Introducción a Python

Intro to data.world in Python

Podcast (39)

Successful Digital Transformation Puts People First (with Bhavin Patel)

Blog Posts (200)

How Data & Culture Unlock Digital Transformation

Tech Thoughts (0)