https://www.datacamp.com/community/podcast/critical-thinking-in-data-science

<p>Debbie Berebichez, Chief Data Scientist at Metis, speaks with Hugo about what type of critical thinking skills are necessary in data science.</p>

Debbie Berebichez speaks with Hugo about the importance of critical thinking in data science.

Critical Thinking in Data Science

https://www.datacamp.com/community/blog/critical-thinking-in-data-science

https://www.datacamp.com/community/podcast/data-science-gambling-bookmaking

<p>Marco Blume, Trading Director at Pinnacle Sports talks about the role of data science in large-scale bets and bookmaking, how Marco is training an army of data scientists and much more.</p>

Marco Blume, Trading Director at Pinnacle Sports talks about the role of data science in large-scale bets and bookmaking.

Data Science, Gambling and Bookmaking

https://www.datacamp.com/community/blog/data-science-gambling-bookmaking

https://www.datacamp.com/community/news/weapons-of-math-destruction-with-cathy-oneil-du7aqb297we

Cathy and Hugo discuss the current lack of fairness in artificial intelligence, how societal biases are perpetuated by algorithms and how both transparency and auditability of algorithms will be necessary for a fairer future.

Weapons of Math Destruction (with Cathy O'Neil)

https://www.datacamp.com/community/news/testing-multiple-interventions-in-a-single-experiment-2jedgabir95

A reader recently inquired about functions in simstudy that could generate data for a balanced multi-factorial design. I had to report that nothing really exists. A few weeks later, a colleague of mine asked if I could help estimate the appropriate sample size for a study that plans to use a...

Testing multiple interventions in a single experiment

https://www.datacamp.com/community/news/16-useful-advices-for-aspiring-data-scientists-5ggav9rqq3u

Data is being generated exponentially and those who can understand and extract value from it are needed more than ever. The hard-earned lessons and joy about data and models from these thoughtful practitioners would be tremendously useful if you aspire to join the next generation of data scientists.

16 Useful Advices for Aspiring Data Scientists

https://www.datacamp.com/community/news/logistic-regression-in-r-tutorial-pfa5sd77cy

Discover all about logistic regression: how it differs from linear regression, how to fit and evaluate these models it in R with the glm() function and more!

Logistic Regression in R Tutorial

https://www.datacamp.com/community/news/network-analysis-in-r-centrality-measures-gqj0ns7618r

Explore the definition of centrality, learn what different types of centrality measures exist in network analysis and pick the best one for a given network!

Network Analysis in R: Centrality Measures

<p><center><a href="https://www.datacamp.com/courses/"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1603467080/Python_vs._R_for_Data_Science__What_s_the_Difference__klxnwh.png" /></a></center><br></p>
<p>If you’re new to data science, or your organization is, you’ll need to pick a language to analyze your data and a thoughtful way to make that decision. Full disclosure: While I can write Python, my background is mostly in the R community—but I&#39;ll try my best to be non-partisan.</p>
<p>The good news is that you don&#39;t need to sweat the decision too hard: both Python and R have vast software ecosystems and communities, so either language is suitable for almost any data science task.</p>
<p>The two most commonly used programming language indexes, <a href="https://www.tiobe.com/tiobe-index">TIOBE</a> and <a href="https://spectrum.ieee.org/computing/software/the-top-programming-languages-2019">IEEE Spectrum</a>, rank the most popular programming languages. They use different criteria for popularity, which explains the differences in the results (TIOBE is entirely based on search engine results; IEEE Spectrum also includes community and social media data sources like Stack Overflow, Reddit, and Twitter). Of the languages on each list that are commonly used for data science, both indexes list Python as the most popular language for data science, followed by R. MATLAB and SAS come in third and fourth place, respectively.</p>
<p>Now that we&#39;ve established that Python and R are both good, popular choices, there are a few factors that may sway your decision one way or the other.</p>
<h2 id="what-language-do-your-colleagues-use-">What language do your colleagues use?</h2>
<p>The most important factor in deciding which programming language to use is knowing which language your colleagues use, since the benefits of being able to share code with your colleagues and maintaining a simpler software stack outweigh any benefits of one language over another.</p>
<h2 id="who-is-working-with-data-">Who is working with data?</h2>
<p>Python was originally developed as a programming language for software development (the data science tools were added later), so people with a computer science or software development background often find Python comes more naturally to them. That is, the transition from other popular programming languages like Java or C++ to Python is easier than the transition from those languages to R.</p>
<p>R has a set of packages known as the Tidyverse, which provide powerful yet easy-to-learn tools for importing, manipulating, visualizing, and reporting on data. Using these tools, people without any programming or data science experience (at least anecdotally) can become productive more quickly than in Python. If you want to test this for yourself, try taking <a href="https://www.datacamp.com/courses/introduction-to-the-tidyverse">Introduction to the Tidyverse</a>, which introduces R&#39;s <a href="https://dplyr.tidyverse.org/">dplyr</a> and <a href="https://ggplot2.tidyverse.org/">ggplot2</a> packages, and <a href="https://www.datacamp.com/courses/introduction-to-data-science-in-python">Introduction to Data Science in Python</a>, which introduces Python&#39;s <a href="https://pandas.pydata.org/">pandas</a> and <a href="https://matplotlib.org/">Matplotlib</a> packages, and see which you prefer.</p>
<p>Verdict: If data science in your organization will primarily be conducted by a dedicated team with programming experience, Python has a slight advantage. If you have many employees who don&#39;t have a data science or programming background, but who still need to work with data, R has a slight advantage.</p>
<h2 id="what-tasks-are-you-performing-">What tasks are you performing?</h2>
<p>While Python and R can basically both do any data science task you can think of, there are some areas where one language is stronger than the other.</p>
<div class="output_wrapper"><table>
<thead>
<tr>
<th>Where Python Excels</th>
<th>Where R Excels</th>
</tr>
</thead>
<tbody>
<tr>
<td>The majority of <strong>deep learning</strong> research is done in Python, so tools such as <a href="https://keras.io/">Keras</a> and <a href="https://pytorch.org/">PyTorch</a> have &quot;Python-first&quot; development. You can learn about these topics in <a href="https://www.datacamp.com/courses/deep-learning-with-keras-in-python">Introduction to Deep Learning in Keras</a> and <a href="https://www.datacamp.com/courses/deep-learning-with-pytorch">Introduction to Deep Learning in PyTorch</a>.</td>
<td>A lot of <strong>statistical modeling</strong> research is conducted in R, so there&#39;s a wider variety of model types to choose from. If you regularly have questions about the best way to model data, R is the better option. DataCamp has a large selection of courses on <a href="https://www.datacamp.com/search?facets%5Btechnology%5D%5B%5D=r&amp;facets%5Btopic%5D%5B%5D=Probability+%26+Statistics">statistics with R</a>.</td>
</tr>
<tr>
<td>Another area where Python has an edge over R is with <strong>deploying models into other pieces of software</strong>. Since Python is a general purpose programming language, you can write the whole application in Python and then including your Python-based model is seamless. We cover deploying models in <a href="https://www.datacamp.com/courses/designing-machine-learning-workflows-in-python">Designing Machine Learning Workflows in Python</a> and <a href="https://www.datacamp.com/courses/building-data-engineering-pipelines-in-python">Building Data Engineering Pipelines in Python</a>.</td>
<td>The other big trick up R&#39;s sleeve is easy <strong>dashboard creation</strong> using <a href="https://shiny.rstudio.com/">Shiny</a>. This enables people without much technical experience to create and publish dashboards to share with their colleagues. Python&#39;s <a href="https://plot.ly/dash/">Dash</a> is an alternative, but not as mature. You can learn about Shiny in <a href="https://www.datacamp.com/courses/building-web-applications-with-shiny-in-r">Building Web Applications with Shiny in R</a> and <a href="https://www.datacamp.com/courses/building-web-applications-in-r-with-shiny-case-studies">Building Web Applications with Shiny in R: Case Studies</a>.</td>
</tr>
</tbody>
</table></div>
<p>This list is far from exhaustive and experts endlessly debate which tasks can be done better in one language or another. Again, there is more good news: Python programmers and R programmers borrow good ideas from each other a lot. For example, Python&#39;s <a href="https://plotnine.readthedocs.io/en/stable/">plotnine</a> data visualization package was inspired by R&#39;s <a href="https://ggplot2.tidyverse.org/">ggplot2</a> package, and R&#39;s <a href="https://rvest.tidyverse.org/">rvest</a> web scraping package was inspired by Python&#39;s <a href="https://www.crummy.com/software/BeautifulSoup/">BeautifulSoup</a> package. So eventually the best ideas from either language make their way into the other.</p>
<p>If you’re too impatient to wait for a particular feature in your language of choice, it&#39;s also worth noting that there is excellent language interoperability between Python and R. That is, you can run R code from Python using the <a href="https://rpy2.bitbucket.io/">rpy2</a> package, and you can run Python code from R using <a href="https://rstudio.github.io/reticulate/">reticulate</a>. That means that all the features present in one language can be accessed from the other language. For example, the <a href="https://keras.rstudio.com/">R version</a> of deep learning package <a href="https://keras.io/">Keras</a> actually calls Python. Likewise, <a href="https://f0nzie.github.io/rTorch/">rTorch</a> calls <a href="https://pytorch.org/">PyTorch</a>.</p>
<h2 id="what-do-your-competitors-use-">What do your competitors use?</h2>
<p>If you work at a business that is growing fast and want to recruit top employees, it&#39;s worth doing some opposition research to see what technologies your competitors are using. After all, your new hires will be productive more quickly if they don&#39;t have to learn a new language.</p>
<h2 id="summary">Summary</h2>
<p>Programming language wars are mostly excuses for people to promote their favorite language and have fun trolling people who use something else. So I want to be clear that I&#39;m not interested in starting another argument on the internet about Python versus R for data science.</p>
<p>I hope I&#39;ve convinced you that, while both Python and R are good choices for data science, factors like employee background, the problems you work on, and the culture of your industry can guide your decision.</p>


https://www.datacamp.com/community/blog/when-to-use-python-or-r

Here’s how to determine when to use the two most popular languages for data science: Python and R. 

Python vs. R for Data Science: What’s the Difference?

Debbie Berebichez speaks with Hugo about what type of critical thinking skills are necessary in data science.

<h2>Introduction to Bayesian thinking</h2><p>This chapter introduces the idea of discrete probability models and Bayesian learning. You'll express your opinion about plausible models by defining a prior probability distribution, you'll observe new information, and then, you'll update your opinion about the models by applying Bayes' theorem.</p><h2>Learning about a binomial probability</h2><p>This chapter describes learning about a population proportion using discrete and continuous models. You'll use a beta curve to represent prior opinion about the proportion, take a sample and observe the number of successes and failures, and construct a beta posterior curve that combines both the information in the prior and in the sample. You'll then use the beta posterior curve to draw inferences about the population proportion.</p><h2>Learning about a normal mean</h2><p>This chapter introduces Bayesian learning about a population mean. You'll sample from a normal population with an unknown mean and a known standard deviation,  construct a normal prior to reflect your opinion about the location of the mean before sampling, and see that the posterior distribution also has a normal form with updated values of the mean and standard deviation. You'll also get more practice  drawing inferences from the posterior distribution, only this time, about a population mean.</p><h2>Bayesian comparisons</h2><p>Suppose you're interested in comparing proportions from two populations. You take a random sample from each population and you want to learn about the difference in proportions. This chapter will illustrate the use of discrete and continuous priors to do this kind of inference. You'll use a Bayesian regression approach to learn about a mean or the difference in means when the sampling standard deviation is unknown.</p>

https://www.datacamp.com/community/open-courses/beginning-bayes-in-r

Discover Bayesian statistics with R: represent your beliefs with the language of probability, collect some data and update your beliefs based on the evidence contained in the data.

Beginning Bayes in R

<h2>Lab 1 - Intro to probability and data</h2><p>Questions for Lab 1</p><h2>Lab 2 - Introduction to data</h2><p>Questions for Lab 2</p><h2>Lab 3 - Probability</h2><p>Questions for Lab 3</p>

https://www.datacamp.com/community/open-courses/introduction-to-probability-and-data-labs

This interactive DataCamp course complements the Coursera course Introduction to Probability and Data by Mine Çetinkaya-Rundel. 

Introduction to Probability and Data - Labs

<p>Nearly every data science project involves studying correlations among variables. A correlation is the statistical summary of the relationship between two sets of variables. It is a core part of <a target="_blank" href="https://www.datacamp.com/courses/exploratory-data-analysis-in-python">data exploratory analysis</a>, and is a critical aspect of numerous advanced machine learning techniques. If you are considering breaking into data science, sooner or later in your data science journey you will need to learn correlation.</p>
<p>In this tutorial, we will explain what correlation is and its relevance when conducting data science projects. We will also have a look at the different correlation coefficients we can use to measure the strength and direction of the relationship between variables. We will use Python code to provide practical examples of how to measure and visualize correlations in a simple manner.</p>
<ul>
<li><a href="#pearson-s-correlation-coefficient">What is Correlation?</a></li>
<li><a href="#correlation-coefficients">Correlation coefficients</a>
  <ul>
<li><a href="#pearson-s-correlation-coefficient">Pearson's correlation coefficient</a></li>
<li><a href="#other-correlation-coefficients">Other correlation coefficients</a></li>
</ul>
</li>
<li><a href="#calculating-correlation-in-python">Calculating correlation in Python</a></li>
<li><a href="#correlation-does-not-imply-causation">Correlation does not imply causation</a></li>
<li><a href="#conclusion">Conclusion</a></li>
</ul>
<h2 id="what-is-correlation-">What is Correlation?</h2>
<p>Correlation is the statistical analysis of the relationship or dependency between two variables. Correlation allows us to study both the strength and direction of the relationship between two sets of variables.</p>
<p>Studying correlation can be very useful in many data science tasks. First, it is a key component in data exploratory analysis; the initial study we conduct in our data to see how it looks, to summarize its main characteristics, and to discover partners and anomalies.</p>
<p>Second, correlations have many real-world applications. They can help us answer questions, such as whether there is a link between democracy and economic growth, or whether the use of cars correlates to the level of air pollution.</p>
<p>Finally, the study of correlation is critical in the field of machine learning. For example, some algorithms will not work properly if two or more variables are closely related, usually known as <a target="_blank" href="https://campus.datacamp.com/courses/inference-for-linear-regression-in-r/building-on-inference-in-simple-linear-regression?ex=4">multicollinearity</a>. Correlation is also the basis of <a target="_blank" href="https://www.datacamp.com/community/tutorials/principal-component-analysis-in-python">Principal Component Analysis</a>, a linear dimensionality reduction technique which is very useful in machine learning projects.</p>
<p>There are different types of correlation:</p>
<ul>
<li>Positive Correlation: Two variables are said to be positively correlated when their values move in the same direction. For example, in the image below, as the value for X increases, so does the value for Y at a constant rate:</li>
</ul>
<centre><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1645797213/image2_xkne9e.png" alt="Positive Correlation"></centre>

<ul>
<li>Neutral Correlation: No relationship in the change of variables X and Y. In this case, the values are completely random and do not show any sign of correlation, as shown in the following image:</li>
</ul>
<centre><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1645797214/image1_aee1ws.png" alt="Neutral Correlation"></centre>

<ul>
<li>Negative Correlation: Finally, variables X and Y will be negatively correlated when their values change in opposite directions, so here as the value for X increases, the value for Y decreases at a constant rate:</li>
</ul>
<centre><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1645797214/image8_mw98sy.png" alt="Negative Correlation"></centre>

<p>Plotting our data is the fastest and most effective way to discover the type of correlation between variables. The typical way to visualize the dependencies between two variables is with a scatterplot. A scatterplot is a graph in which the x-axis is the value of the first variable, and the y-axis is the value of the second variable. The last three images were types of scatterplots. Check out this <a target="_blank" href="https://campus.datacamp.com/courses/introduction-to-data-science-in-python/different-types-of-plots?ex=1">video tutorial</a> to see how to create a scatterplot in Python.</p>
<p>However, sometimes we will need not only to visualize correlation, but also to measure how strong the correlation is. This is where correlation coefficients come into play.</p>
<h2 id="correlation-coefficients">Correlation coefficients</h2>
<p>A correlation coefficient is a statistical summary that measures the strength and direction to which two variables are associated with one another.</p>
<p>One of the advantages of correlation coefficients is that they estimate the correlation between two variables in a standardized way, meaning that the value of the coefficient will always be on the same scale, ranging from -1.0 to 1.0.</p>
<p>There are several correlation coefficients. Choosing one or another depends on what we know about the structure of the relationship between the variables. However, data scientists normally stick to the most famous one: Pearson's correlation coefficient.</p>
<h3 id="pearson-s-correlation-coefficient">Pearson's correlation coefficient</h3>
<p>The Pearson's correlation coefficient (r) is a score that measures the strength of a linear relationship between two variables. It is calculated by dividing the covariance of variables X and Y by the product of the standard deviation of each variable, as shown in the following formula:</p>
<centre><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1645797214/image4_xdyzit.png" alt="Pearson's correlation coefficient"></centre>

<p>The coefficient is based on two assumptions. First, it assumes that the variables follow a normal or gaussian distribution. If the data is not normally distributed, then other coefficients may be more reliable.</p>
<p>Second, it assumes that there is a linear relationship between the two variables, meaning that the changes in the data can be modelled using a linear function (i.e. their values increase or decrease simultaneously at a constant rate). If the relationship between the two variables is closer to some straight line, then their (linear) correlation is stronger and the absolute value of Pearson's correlation coefficient is higher. For example, in the next image, all the data points can be perfectly modeled using a straight line, resulting in a correlation coefficient equal to 1.0.  </p>
<centre><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1645797215/image7_ijaikw.png" alt="Pearson's correlation coefficient 2"></centre>

<p>A coefficient of -1.0 indicates a perfect negative correlation, whereas a coefficient of 1.0 shows a perfect positive correlation. On the contrary, a coefficient of 0.0 indicates that there is no linear correlation between the variables.</p>
<p>Interpreting the coefficient becomes more difficult when its value is approaching zero. Several rules of thumb have been proposed to interpret the values of the coefficient. One of the most popular was formulated by Dennis E. Hinkle and his co-authors in their book <a target="_blank" href="https://www.amazon.es/Applied-Statistics-Behavioural-Sciences-Dennis/dp/0618124055">Applied  Statistics for the  Behavioral  Sciences</a>:</p>
<centre><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1645797214/image3_yzl0dq.png" alt="Rule of Thumb for Interpreting Correlation Coefficient. Hinkle, Wiersma, &amp; Jurs."></centre>

<centre>Rule of  Thumb for Interpreting Correlation Coefficient. Hinkle, Wiersma, &amp; Jurs.</centre>

<p>It is important to highlight that a Pearson's correlation coefficient close to 0.0 only indicates that there is no linear relationship between the variables. But it could happen that there is a strong, non-linear relationship instead. If a relationship between two variables is nonlinear, the rate of increase or decrease may change as one variable changes, drawing a curve-like pattern in the data. Such a trend could be better modeled by a nonlinear function, such as a quadratic or cubic function.</p>
<p>For example, the following image shows how the happiness level of workers varies depending on the hours worked per week. We can see that as the number of hours increases, happiness also goes up, but it starts declining beyond 35 hours approximately. The linear model (red line) is not  well-suited to fit the data. Instead, a  quadratic model (blue line), also known as a polynomial model with a degree of 2, seems a much better candidate to fit the curvy shape of the data. When the relationship is nonlinear,  correlation generally underestimates how strong it is. This is the case in our example, where the coefficient is equal to 0.127, telling little about the nonlinear relationship between happiness and hours worked.</p>
<centre><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1645797214/image10_os0yru.png" alt="Rule of Thumb for Interpreting Correlation Coefficient. Hinkle, Wiersma, &amp; Jurs. 2"></centre>

<h3 id="other-correlation-coefficients">Other correlation coefficients</h3>
<p>The biggest limitation of Pearson's correlation coefficient is that it assumes a linear relationship between the two variables. However, more often than not, variables are associated with a nonlinear relationship. Also, the data at hand may have a non-Gaussian distribution. To address this shortcoming, statisticians have proposed other types of estimates, namely the Spearman's rho and Kendall's tau correlation coefficients.</p>
<p>While Pearson's coefficient assumes linear relationships, Spearman's and Kendall's coefficients assume that the relationship between the variables is just monotonic. In a monotonic relationship, the variables tend to move in the same relative direction, but not necessarily at a constant rate. In other words, every linear relationship is monotonic, but not every monotonic relationship is linear.</p>
<centre><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1645797214/image6_hiawoh.png" alt="Pearson's coefficient"></centre>

<p>Instead of calculating the coefficients using the covariance and standard deviations of the variables, they are based on the relative rank of the values of the variables. This particularity makes them robust estimates against extreme values, and suitable to handle certain types of nonlinearities.</p>
<p>As with the Pearson correlation coefficient, the scores of Spearman's and Kendall's coefficients are between -1 and 1 for perfectly negatively correlated variables and perfectly positively correlated, respectively.</p>
<p>However, data scientists can generally stick to Pearson's correlation coefficient. The appeal of rank-based estimates is mostly for smaller data sets and specific hypothesis tests.</p>
<h2 id="calculating-correlation-in-python">Calculating correlation in Python</h2>
<p>There are various Python packages that can help us measure correlation. In this section, we will focus on the correlation functions available in three well-known packages: SciPy, NumPy,  and pandas.</p>
<p>To try the functions, imagine we want to study the relationship between work experience (measured in years) and salary (measured in dollars) in a small yet successful startup comprising 16 workers.</p>
<pre><code class="lang-python">experience = [1, 3, 4, 5, 5, 6, 7, 10, 11, 12, 15, 20, 25, 28, 30,35]

salary = [20000, 30000, 40000, 45000, 55000, 60000, 80000, 100000, 130000, 150000, 200000, 230000, 250000, 300000, 350000, 400000]
</code></pre><p>As we can see in the next graph, there is a strong positive correlation between the two variables, meaning that employees with more experience tend to have higher salaries.</p>
<centre><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1645797214/image11_ywwjae.png" alt="there is a strong positive correlation between the two variable"></centre>

<p>SciPy is a great library to perform statistical operations. The scipy.stats module includes the pearsonr(x, y) function to calculate Pearson's correlation coefficient between two data samples.</p>
<pre><code class="lang-python">import scipy.stats as stats

corr, _ = stats.pearsonr (experience, salary)
corr
0.9929845761480398
</code></pre><p>We could calculate Spearman's and Kendall's coefficient in the same fashion:</p>
<pre><code class="lang-python">spearman_corr, _ = stats.spearmanr(experience, salary)
spearman_corr
0.9992644353546791

kendall_corr, _ = stats.kendalltau(experience, salary)
kendall_corr
0.9958246164193105
</code></pre><p>NumPy is a popular package that offers an extensive collection of advanced mathematical functions, including np.corrcoef() that returns a matrix of Pearson's correlation coefficients:</p>
<pre><code class="lang-python">import numpy as np

np.corrcoef(experience, salary)
array([[1.        , 0.99298458],
       [0.99298458, 1.        ]])
</code></pre><p>A correlation matrix is a table showing correlation coefficients between variables. Each cell in the table shows the correlation between two variables. The diagonal of the matrix includes the coefficients between each variable and itself, which is always equal to 1.0. The other values in the matrix represent the correlation between experience and salary. In this case, as we are only calculating correlation for two variables, the values are the same.</p>
<p>Finally, we will usually need to calculate correlation for our variables stored in pandas DataFrames. Imagine we have our DataFrame with information about the workers of the startup:</p>
<centre><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1645797214/image5_ylmaib.png" alt="will usually need to calculate correlation for our variables stored in pandas DataFrames."></centre>

<p>If we wanted to calculate the correlation between two columns, we could use the pandas method .corr(), as follows:</p>
<pre><code class="lang-python">import pandas as pd

df[&#39;experience&#39;].corr(df[&#39;salary&#39;])
0.9929845761480398
</code></pre><p>The .corr() includes the parameter "method", which can be used to calculate the three correlation coefficients. By default, it calculated Pearson's.</p>
<pre><code class="lang-python">print(df[&#39;experience&#39;].corr(df[&#39;salary&#39;], method=&#39;spearman&#39;))
print(df[&#39;experience&#39;].corr(df[&#39;salary&#39;], method=&#39;kendall&#39;))
0.9992644353546791
0.9958246164193105
</code></pre><p>In case we wanted to explore the correlation between all the pairs of variables, we could simply use the .corr() method directly to our DataFrame, which results again in a correlation matrix with the coefficient of all the pairs of variables:</p>
<pre><code class="lang-python">df.corr()
</code></pre><centre><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1645797214/image12_idwtae.png" alt="coefficient of all the pairs of variables:"></centre>

<p>The correlation matrix can be very big and difficult to interpret if our DataFrame has many columns. To extract the insights of our matrix in a more effective way, we could use a heatmap; a data visualization technique where each value is represented by a color, according to its intensity in a given scale. The fastest way to create a heatmap is  by using the function heatmap(), available in the seaborn library:</p>
<pre><code class="lang-python">import seaborn as sns

sns.heatmap(df.corr(), vmin=-1, vmax=1,
annot=True,cmap=&quot;rocket_r&quot;)
plt.show()
</code></pre><centre><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1645797214/image9_dmi5yg.png" alt="seaborn library:"></centre>

<h2 id="correlation-does-not-imply-causation">Correlation does not imply causation</h2>
<p>We could not finish this tutorial without mentioning an important remark: correlation does not imply causation.</p>
<p>The correlation only quantifies the strength and the direction of the relationship between two variables. There might be a strong correlation between two variables, but it does not allow us to conclude that one causes the other. When strong correlations are not causal, we call them spurious correlations.</p>
<p>Two scenarios could lead to spurious correlations:</p>
<ul>
<li>First, it could happen that there might be a third, "confounding" variable that is causing the two variables. Imagine there is a strong positive correlation between ice cream consumption and the number of heatstrokes in a given population. One may come to the absurd conclusion that eating ice cream is causing heatstrokes. However, what happens is both variables depend on a third one, most likely, the temperature. The higher the temperature, the more people will love to have ice cream, and also the more people will be likely to have a heatstroke.</li>
<li>A second reason to be cautious with correlation is that dependencies between variables <a href="https://www.tylervigen.com/spurious-correlations">could be simply due to chance</a>. This happens all the time, but more likely when we deal with small samples.</li>
</ul>
<p>Finally, another reason to take correlation with caution is the world we live in is complex. A stronger correlation between two variables can easily lead us to conclude that one caused the other. The belief that a given effect has only one cause is called monocausality, but this is rarely the case. Several variables can be behind a certain effect. Thus, in order to consider causality, we first need to assume all the elements of a system and evaluate them not only quantitatively (i.e. through statistical methods) but also qualitatively.</p>
<p>For these reasons, before making any conclusion about relationships in our data, it is important to conduct a more thorough and detailed analysis to identify possible spurious correlations.</p>
<h2 id="conclusion">Conclusion</h2>
<p>We hope you enjoyed this tutorial. Correlation is an important topic that every aspiring data scientist should learn. At DataCamp, we have a wide number of courses of statistics where we deal with correlation, causation, and other key statistical concepts, both with <a target="_blank" href="https://www.datacamp.com/search?q=&amp;tab=courses&amp;facets%5Btechnology%5D%5B%5D=Python&amp;facets%5Btopic%5D%5B%5D=Probability+%26+Statistics">Python</a> and <a target="_blank" href="https://www.datacamp.com/search?q=&amp;tab=courses&amp;facets%5Btechnology%5D%5B%5D=R&amp;facets%5Btopic%5D%5B%5D=Probability+%26+Statistics">R</a>.</p>
<p>Once you have a solid understanding, we recommend you try <a target="_blank" href="https://www.datacamp.com/workspace">DataCamp Workspace</a>, an online environment to write code and apply what you have learned.</p>


https://www.datacamp.com/community/tutorials/tutorial-datails-on-correlation

A tutorial to understand what correlation is and why it is important for every aspiring data scientist to know it.


Python Details on Correlation Tutorial

https://www.datacamp.com/community/tutorials/python-statistics-data-science

A list of Python resources for the eight statistics topics that you need to know to excel in data science

40+ resources for Python statistical analysis. Understand statistics with Python and topics that you need to know to excel in data science today!

community

datacamp

statistical modeling

News (38)

Weapons of Math Destruction (with Cathy O'Neil)

Logistic Regression in R Tutorial

Weapons of Math Destruction (with Cathy O'Neil)

Tutorials (26)

Python Details on Correlation Tutorial

40+ Python Statistics For Data Science Resources

Cheat Sheets (0)

Open Courses (5)

Beginning Bayes in R

Introduction to Probability and Data - Labs

Podcast (18)

Critical Thinking in Data Science (with Debbie Berebichez)

Blog Posts (33)

Python vs. R for Data Science: What’s the Difference?

Tech Thoughts (0)