Data Engineering Episodes

Data Engineering Articles

Data Engineering Tutorials | Read, Learn, & Grow Your Skills

Data Engineering Cheat Sheets

<h2 id="about-spacy">About spaCy</h2>
<p><a href="https://spacy.io">spaCy</a> is a free, open-source library for advanced Natural Language Processing (NLP) in Python. It's designed specifically for production use and helps you build applications that process and "understand" large volumes of text.</p>
<p>To learn more about spaCy, take my DataCamp course <a href="https://www.datacamp.com/courses/advanced-nlp-with-spacy">"Advanced NLP with spaCy"</a>. You can download the Cheat Sheet <a href="http://datacamp-community-prod.s3.amazonaws.com/b5685b85-c4de-4987-926d-b999d0f5a8b6">here</a>!</p>
<center><a href="http://datacamp-community-prod.s3.amazonaws.com/b5685b85-c4de-4987-926d-b999d0f5a8b6"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1625838105/spaCy_Cheat_Sheet-_Advanced_NLP_in_Python_tdcgca.png" /></a></center>
<h2 id="getting-started">Getting started</h2>
<pre><code class="lang-python">$ pip install spacy
</code></pre>
<pre><code class="lang-python">import spacy
</code></pre>
<h2 id="statistical-models">Statistical models</h2>
<h3 id="download-statistical-models">Download statistical models</h3>
<p>Predict part-of-speech tags, dependency labels, named entities and more. <a href="https://spacy.io/models">See here</a> for available models.</p>
<pre><code class="lang-bash">$ python -m spacy download en_core_web_sm
</code></pre>
<h3 id="check-that-your-installed-models-are-up-to-date">Check that your installed models are up to date</h3>
<pre><code class="lang-bash">$ python -m spacy validate
</code></pre>
<h3 id="loading-statistical-models">Loading statistical models</h3>
<pre><code class="lang-python">import spacy
# Load the installed model "en_core_web_sm"
nlp = spacy.load("en_core_web_sm")
</code></pre>
<h2 id="documents-tokens-and-spans">Documents, tokens and spans</h2>
<h3 id="processing-text">Processing text</h3>
<p>Processing text with the <code>nlp</code> object returns a <code>Doc</code> object that holds all information about the tokens, their linguistic features and their relationships.</p>
<pre><code class="lang-python">doc = nlp("This is a text")
</code></pre>
<h3 id="accessing-token-attributes">Accessing token attributes</h3>
<pre><code class="lang-python">doc = nlp("This is a text")
# Token texts
[token.text for token in doc]
# ['This', 'is', 'a', 'text']
</code></pre>
<h2 id="spans">Spans</h2>
<h3 id="accessing-spans">Accessing spans</h3>
<p>Span indices are <strong>exclusive</strong>. So <code>doc[2:4]</code> is a span starting at token 2, up to &ndash; but not including! &ndash; token 4.</p>
<pre><code class="lang-python">doc = nlp("This is a text")
span = doc[2:4]
span.text
# 'a text'
</code></pre>
<h3 id="creating-a-span-manually">Creating a span manually</h3>
<pre><code class="lang-python"># Import the Span object
from spacy.tokens import Span
# Create a Doc object
doc = nlp("I live in New York")
# Span for "New York" with label GPE (geopolitical)
span = Span(doc, 3, 5, label="GPE")
span.text
# 'New York'
</code></pre>
<h2 id="linguistic-features">Linguistic features</h2>
<p>Attributes return label IDs. For string labels, use the attributes with an underscore. For example, <code>token.pos_</code>.</p>
<h2 id="part-of-speech-tags-predicted-by-statistical-model-">Part-of-speech tags (predicted by statistical model)</h2>
<pre><code class="lang-python">doc = nlp("This is a text.")
# Coarse-grained part-of-speech tags
[token.pos_ for token in doc]
# ['DET', 'VERB', 'DET', 'NOUN', 'PUNCT']
# Fine-grained part-of-speech tags
[token.tag_ for token in doc]
# ['DT', 'VBZ', 'DT', 'NN', '.']
</code></pre>
<h2 id="syntactic-dependencies-predicted-by-statistical-model-">Syntactic dependencies (predicted by statistical model)</h2>
<pre><code class="lang-python">doc = nlp("This is a text.")
# Dependency labels
[token.dep_ for token in doc]
# ['nsubj', 'ROOT', 'det', 'attr', 'punct']
# Syntactic head token (governor)
[token.head.text for token in doc]
# ['is', 'is', 'text', 'is', 'is']
</code></pre>
<h2 id="named-entities-predicted-by-statistical-model-">Named Entities (predicted by statistical model)</h2>
<pre><code class="lang-python">doc = nlp("Larry Page founded Google")
# Text and label of named entity span
[(ent.text, ent.label_) for ent in doc.ents]
# [('Larry Page', 'PERSON'), ('Google', 'ORG')]
</code></pre>
<h2 id="sentences-usually-needs-the-dependency-parser-">Sentences (usually needs the dependency parser)</h2>
<pre><code class="lang-python">doc = nlp("This a sentence. This is another one.")
# doc.sents is a generator that yields sentence spans
[sent.text for sent in doc.sents]
# ['This is a sentence.', 'This is another one.']
</code></pre>
<h2 id="base-noun-phrases-needs-the-tagger-and-parser-">Base noun phrases (needs the tagger and parser)</h2>
<pre><code class="lang-python">doc = nlp("I have a red car")
# doc.noun_chunks is a generator that yields spans
[chunk.text for chunk in doc.noun_chunks]
# ['I', 'a red car']
</code></pre>
<h2 id="label-explanations">Label explanations</h2>
<pre><code class="lang-python">spacy.explain("RB")
# 'adverb'
spacy.explain("GPE")
# 'Countries, cities, states'
</code></pre>
<h2 id="visualizing">Visualizing</h2>
<p>⚠️ If you're in a Jupyter notebook, use <code>displacy.render</code>. Otherwise, use <code>displacy.serve</code> to start a web server and show the visualization in your browser.</p>
<pre><code class="lang-python">from spacy import displacy
</code></pre>
<h3 id="visualize-dependencies">Visualize dependencies</h3>
<pre><code class="lang-python">doc = nlp("This is a sentence")
displacy.render(doc, style="dep")
</code></pre>
<p><img src="https://user-images.githubusercontent.com/13643239/52234132-38e76a00-28c1-11e9-82e6-e5d6b4a7f339.png" alt="displacy_dep" width="640" /></p>
<h3 id="visualize-named-entities">Visualize named entities</h3>
<pre><code class="lang-python">doc = nlp("Larry Page founded Google")
displacy.render(doc, style="ent")
</code></pre>
<p><img src="https://user-images.githubusercontent.com/13643239/52234151-443a9580-28c1-11e9-831a-9adee3ec87b1.png" alt="displacy_ent" width="422" /></p>
<h2 id="word-vectors-and-similarity">Word vectors and similarity</h2>
<p>⚠️ To use word vectors, you need to install the larger models ending in <code>md</code> or <code>lg</code> , for <code>example en_core_web_lg</code>.</p>
<h3 id="comparing-similarity">Comparing similarity</h3>
<pre><code class="lang-python">doc1 = nlp("I like cats")
doc2 = nlp("I like dogs")
# Compare 2 documents
doc1.similarity(doc2)
# Compare 2 tokens
doc1[2].similarity(doc2[2])
# Compare tokens and spans
doc1[0].similarity(doc2[1:3])
</code></pre>
<h3 id="accessing-word-vectors">Accessing word vectors</h3>
<pre><code class="lang-python"># Vector as a numpy array
doc = nlp("I like cats")
# The L2 norm of the token's vector
doc[2].vector
doc[2].vector_norm
</code></pre>
<h2 id="pipeline-components">Pipeline components</h2>
<p>Functions that take a <code>Doc</code> object, modify it and return it.</p>
<p><img src="https://user-images.githubusercontent.com/13643239/55229632-dbff9480-521d-11e9-8499-efb2a9c948db.png" alt="pipeline" width="682" /></p>
<h3 id="pipeline-information">Pipeline information</h3>
<pre><code class="lang-python">nlp = spacy.load("en_core_web_sm")
nlp.pipe_names
# ['tagger', 'parser', 'ner']
nlp.pipeline
# [('tagger', &lt;spacy.pipeline.Tagger&gt;),
# ('parser', &lt;spacy.pipeline.DependencyParser&gt;),
# ('ner', &lt;spacy.pipeline.EntityRecognizer&gt;)]
</code></pre>
<h3 id="custom-components">Custom components</h3>
<pre><code class="lang-python"># Function that modifies the doc and returns it
def custom_component(doc):
 print("Do something to the doc here!")
 return doc

# Add the component first in the pipeline
nlp.add_pipe(custom_component, first=True)
</code></pre>
<p>Components can be added <code>first</code>, <code>last</code> (default), or <code>before</code> or <code>after</code> an existing component.</p>
<h2 id="extension-attributes">Extension attributes</h2>
<p>Custom attributes that are registered on the global <code>Doc</code>, <code>Token</code> and <code>Span</code> classes and become available as <code>._</code>.</p>
<pre><code class="lang-python">from spacy.tokens import Doc, Token, Span
doc = nlp("The sky over New York is blue")
</code></pre>
<h3 id="attribute-extensions-with-default-value-">Attribute extensions (with default value)</h3>
<pre><code class="lang-python"># Register custom attribute on Token class
Token.set_extension("is_color", default=False)
# Overwrite extension attribute with default value
doc[6]._.is_color = True
</code></pre>
<h3 id="property-extensions-with-getter-setter-">Property extensions (with getter &amp; setter)</h3>
<pre><code class="lang-python"># Register custom attribute on Doc class
get_reversed = lambda doc: doc.text[::-1]
Doc.set_extension("reversed", getter=get_reversed)
# Compute value of extension attribute with getter
doc._.reversed
# 'eulb si kroY weN revo yks ehT'
</code></pre>
<h3 id="method-extensions-callable-method-">Method extensions (callable method)</h3>
<pre><code class="lang-python"># Register custom attribute on Span class
has_label = lambda span, label: span.label_ == label
Span.set_extension("has_label", method=has_label)
# Compute value of extension attribute with method
doc[3:5].has_label("GPE")
# True
</code></pre>
<h2 id="rule-based-matching">Rule-based matching</h2>
<h3 id="using-the-matcher">Using the Matcher</h3>
<pre><code class="lang-python"># Matcher is initialized with the shared vocab
from spacy.matcher import Matcher
# Each dict represents one token and its attributes
matcher = Matcher(nlp.vocab)
# Add with ID, optional callback and pattern(s)
pattern = [{"LOWER": "new"}, {"LOWER": "york"}]
matcher.add('CITIES', None, pattern)
# Match by calling the matcher on a Doc object
doc = nlp("I live in New York")
matches = matcher(doc)
# Matches are (match_id, start, end) tuples
for match_id, start, end in matches:
     # Get the matched span by slicing the Doc
     span = doc[start:end]
     print(span.text)
# 'New York'
</code></pre>
<h3 id="token-patterns">Token patterns</h3>
<pre><code class="lang-python"># "love cats", "loving cats", "loved cats"
pattern1 = [{"LEMMA": "love"}, {"LOWER": "cats"}]
# "10 people", "twenty people"
pattern2 = [{"LIKE_NUM": True}, {"TEXT": "people"}]
# "book", "a cat", "the sea" (noun + optional article)
pattern3 = [{"POS": "DET", "OP": "?"}, {"POS": "NOUN"}]
</code></pre>
<h3 id="operators-and-quantifiers">Operators and quantifiers</h3>
<p>Can be added to a token dict as the <code>"OP"</code> key.</p>
<div class="output_wrapper">
<table>
<thead>
<tr>
<th>OP</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>!</code></td>
<td>Negate pattern and match exactly 0 times.</td>
</tr>
<tr>
<td><code>?</code></td>
<td>Make pattern optional and match 0 or 1 times.</td>
</tr>
<tr>
<td><code>+</code></td>
<td>Require pattern to match 1 or more times.</td>
</tr>
<tr>
<td><code>*</code></td>
<td>Allow pattern to match 0 or more times.</td>
</tr>
</tbody>
</table>
</div>
<h2 id="glossary">spaCy Glossary</h2>
<div class="output_wrapper">
<table>
<thead>
<tr>
<th>Name</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Tokenization</strong></td>
<td>Segmenting text into words, punctuation etc.</td>
</tr>
<tr>
<td><strong>Lemmatization</strong></td>
<td>Assigning the base forms of words, for example: "was" &rarr; "be" or "rats" &rarr; "rat".</td>
</tr>
<tr>
<td><strong>Sentence Boundary Detection</strong></td>
<td>Finding and segmenting individual sentences.</td>
</tr>
<tr>
<td><strong>Part-of-speech (POS) Tagging</strong></td>
<td>Assigning word types to tokens like verb or noun.</td>
</tr>
<tr>
<td><strong>Dependency Parsing</strong></td>
<td>Assigning syntactic dependency labels, describing the relations between individual tokens, like subject or object.</td>
</tr>
<tr>
<td><strong>Named Entity Recognition (NER)</strong></td>
<td>Labeling named "real-world" objects, like persons, companies or locations.</td>
</tr>
<tr>
<td><strong>Text Classification</strong></td>
<td>Assigning categories or labels to a whole document, or parts of a document.</td>
</tr>
<tr>
<td><strong>Statistical model</strong></td>
<td>Process for making predictions based on examples.</td>
</tr>
<tr>
<td><strong>Training</strong></td>
<td>Updating a statistical model with new examples.</td>
</tr>
</tbody>
</table>
</div>

Check out the first official spaCy cheat sheet! A handy two-page reference to the most important concepts and features.

spaCy Cheat Sheet: Advanced NLP in Python

<p>You most probably will know by now that data storytelling, accomplished by data visualization, amongst other things, is an essential skill for every data scientist: after you have turned the raw data into understanding, insights and knowledge, you also need to communicate these findings effectively to your audience.</p>
<p>For most beginners, the first Python data visualization library that they use is, naturally, Matplotlib. It is a Python 2D plotting library that enables users to make publication-quality figures. It is quite an extensive library where a <a href="https://www.datacamp.com/community/blog/python-matplotlib-cheat-sheet">cheat sheet</a> will definitely come in handy when you're learning, but when you manage to use this library effectively, you'll also be able to get insights and work better with other packages, such as Pandas, that intend to build more plotting integration with Matplotlib as time goes on.</p>
<p>Another package that you'll be able to tackle easily is Seaborn, the statistical data visualization library of Python.</p>
<p>DataCamp has created a Seaborn cheat sheet for those who are ready to get started with this data visualization library with the help of a handy one-page reference.</p>
<p>You'll see that this cheat sheet presents you with the five basic steps that you can go through to make beautiful statistical graphs in Python.</p>
<p>Check out the infographic by clicking on the button below:</p>
<p><a href="http://datacamp-community-prod.s3.amazonaws.com/263130e2-2c92-4348-a356-9ed9b5034247"><img style="width: 800px; height: 419px;" src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1625839419/Python_Seaborn_Cheat_Sheet_q074wv.png" alt="Seaborn Python cheat sheet" /></a></p>
<p>This cheat sheet will walk you through the five steps that you need to go through to make these plots: you'll see how you can load in data, set the figure aesthetics, plot, customize and eventually, show or save your plot with Seaborn.</p>
<p>What might have looked difficult before will definitely be more clear once you start using this cheat sheet! Use it in combination with the <a href="http://seaborn.pydata.org/examples/index.html">Seaborn Gallery</a>, the <a href="http://seaborn.pydata.org/index.html">documentation</a>, and our <a href="https://www.datacamp.com/community/tutorials/seaborn-python-tutorial">Seaborn tutorial</a>.</p>
<p>Also, don't miss out on our other cheat sheets for data science that cover <a href="https://www.datacamp.com/community/blog/python-scipy-cheat-sheet/">SciPy</a>, <a href="https://www.datacamp.com/community/blog/python-numpy-cheat-sheet/">NumPy</a>, <a href="https://www.datacamp.com/community/blog/scikit-learn-cheat-sheet/">Scikit-Learn</a>, <a href="https://www.datacamp.com/community/blog/bokeh-cheat-sheet-python/">Bokeh</a>, <a href="https://www.datacamp.com/community/blog/python-pandas-cheat-sheet/">Pandas</a> and the <a href="https://www.datacamp.com/community/tutorials/python-data-science-cheat-sheet-basics/">Python basics</a>.</p>
<h2>Learn more about Seaborn</h2>
<p><a href="datacamp.com/courses/introduction-to-data-visualization-with-seaborn">Introduction to Data Visualization with Seaborn</a></p>
<p><a href="https://datacamp.com/courses/intermediate-data-visualization-with-seaborn">Intermediate Data Visualization with Seaborn</a><br /><br /></p>

This Python Seaborn cheat sheet with code samples guides you through the data visualization library that is based on Matplotlib.

Data Engineering Cheat Sheets

spaCy Cheat Sheet: Advanced NLP in Python

Python Seaborn Cheat Sheet