1. Local and global thought patterns
While we might not be Twitter fans, we have to admit that it has a huge influence on the world (who doesn't know about Trump's tweets). Twitter data is not only gold in terms of insights, but Twitter-storms are available for analysis in near real-time. This means we can learn about the big waves of thoughts and moods around the world as they arise.
As any place filled with riches, Twitter has security guards blocking us from laying our hands on the data right away ⛔️ Some authentication steps (really straightforward) are needed to call their APIs for data collection. Since our goal today is learning to extract insights from data, we have already gotten a green-pass from security ✅ Our data is ready for usage in the datasets folder — we can concentrate on the fun part! 🕵️♀️🌎
Twitter provides both global and local trends. Let's load and inspect data for topics that were hot worldwide (WW) and in the United States (US) at the moment of query — snapshot of JSON response from the call to Twitter's GET trends/place API.
Note: Here is the documentation for this call, and here a full overview on Twitter's APIs.
# Loading json module
import json
# Load WW_trends and US_trends data into the the given variables respectively
WW_trends = json.loads(open('datasets/WWTrends.json').read())
US_trends = json.loads(open('datasets/USTrends.json').read())
# Inspecting data by printing out WW_trends and US_trends variables
WW_trendsUS_trends2. Prettifying the output
Our data was hard to read! Luckily, we can resort to the json.dumps() method to have it formatted as a pretty JSON string.
# Pretty-printing the results. First WW and then US trends.
print("WW trends:")
print(json.dumps(WW_trends,indent = 1))
print("\n", "US trends:")
print(json.dumps(US_trends,indent = 1))3. Finding common trends
🕵️♀️ From the pretty-printed results (output of the previous task), we can observe that:
We have an array of trend objects having: the name of the trending topic, the query parameter that can be used to search for the topic on Twitter-Search, the search URL and the volume of tweets for the last 24 hours, if available. (The trends get updated every 5 mins.)
At query time #BeratKandili, #GoodFriday and #WeLoveTheEarth were trending WW.
"tweet_volume" tell us that #WeLoveTheEarth was the most popular among the three.
Results are not sorted by "tweet_volume".
There are some trends which are unique to the US.
It’s easy to skim through the two sets of trends and spot common trends, but let's not do "manual" work. We can use Python’s set data structure to find common trends — we can iterate through the two trends objects, cast the lists of names to sets, and call the intersection method to get the common names between the two sets.
# Extracting all the WW trend names from WW_trends
world_trends = set([WW_trends[0]['trends'][i]['name'] for i in range(len(WW_trends[0]['trends']))])
# Extracting all the US trend names from US_trends
us_trends = set([US_trends[0]['trends'][i]['name'] for i in range(len(US_trends[0]['trends']))])
# Getting the intersection of the two sets of trends
common_trends = world_trends.intersection(us_trends)
# Inspecting the data
world_trendsus_trendsprint (len(common_trends), "common trends:", common_trends)4. Exploring the hot trend
🕵️♀️ From the intersection (last output) we can see that, out of the two sets of trends (each of size 50), we have 11 overlapping topics. In particular, there is one common trend that sounds very interesting: #WeLoveTheEarth — so good to see that Twitteratis are unanimously talking about loving Mother Earth! 💚
Note: We could have had no overlap or a much higher overlap; when we did the query for getting the trends, people in the US could have been on fire obout topics only relevant to them.
We have found a hot-trend, #WeLoveTheEarth. Now let's see what story it is screaming to tell us!
If we query Twitter's search API with this hashtag as query parameter, we get back actual tweets related to it. We have the response from the search API stored in the datasets folder as 'WeLoveTheEarth.json'. So let's load this dataset and do a deep dive in this trend.
# Loading the data
tweets = json.loads(open('datasets/WeLoveTheEarth.json').read())
# Inspecting some tweets
tweets[0:2]5. Digging deeper
🕵️♀️ Printing the first two tweet items makes us realize that there’s a lot more to a tweet than what we normally think of as a tweet — there is a lot more than just a short text!
But hey, let's not get overwhemled by all the information in a tweet object! Let's focus on a few interesting fields and see if we can find any hidden insights there.
# Extracting the text of all the tweets from the tweet object
texts = [tweet['text'] for tweet in tweets]
# Extracting screen names of users tweeting about #WeLoveTheEarth
names = [user_mentions['screen_name'] for tweet in tweets for user_mentions in tweet['entities']['user_mentions']]
# Extracting all the hashtags being used when talking about this topic
hashtags = [hashtag['text'] for tweet in tweets for hashtag in tweet['entities']['hashtags']]
# Inspecting the first 10 results
print(json.dumps(texts[0:10], indent=1),"\n")json.dumps(names[0:10], indent=1)json.dumps(hashtags[0:10], indent=1)