http://datacamp-community-prod.s3.amazonaws.com/02213cb4-b391-4516-adcd-57243ced8eed

This PySpark SQL cheat sheet is your handy companion to Apache Spark DataFrames in Python and includes code samples. 

PySpark Cheat Sheet: Spark DataFrames in Python

http://datacamp-community-prod.s3.amazonaws.com/acfa4325-1d43-4542-8ce4-bea2d287db10

This PySpark cheat sheet with code samples covers the basics like initializing Spark in Python, loading data, sorting, and repartitioning.

PySpark Cheat Sheet: Spark in Python

https://www.datacamp.com/community/podcast/beyond-the-language-wars-r-python-for-the-modern-data-scientist

<p>In this episode of DataFramed, we speak with Rick Scavetta and Boyan Angelov about their new book, Python and R for the Modern Data Scientist: The Best of Both Worlds, and how it dawns the start of a new bilingual data science community. &nbsp;</p>

In this episode of DataFramed, we speak with Rick Scavetta and Boyan Angelov about their new book, Python and R for the Modern Data Scientist: The Best of Both Worlds, and how it dawns the start of a new bilingual data science community.  

Beyond the Language Wars: R & Python for the Modern Data Scientist

https://www.datacamp.com/community/blog/beyond-the-language-wars-r-python-for-the-modern-data-scientist

https://www.datacamp.com/community/podcast/the-path-to-building-data-cultures

<p>In this episode of DataFramed, Adel speaks with Sudaman Thoppan Mohanchandralal, Regional Chief Data, and Analytics Officer at Allianz Benelux on the importance of building data cultures, and his experiences operationalizing data culture transformation programs.</p>

Adel speaks with Sudaman Thoppan Mohanchandralal, Regional Chief Data, and Analytics Officer at Allianz Benelux on the importance of building data cultures, and his experiences operationalizing data culture transformation programs.

The Path to Building Data Cultures

https://www.datacamp.com/community/blog/the-path-to-building-data-cultures

https://www.datacamp.com/community/news/creating-smart-cities-with-data-science-ofs0psrh048

Amen Ra Mashariki deep dives into the state of data literacy in government agencies, the use cases he worked on to make the City of New York smarter, and more!

Creating Smart Cities With Data Science

https://www.datacamp.com/community/news/cricpy-takes-a-swing-at-the-odis-nohaj97lw3

No computer has ever been designed that is ever aware of what it’s doing; but most of the time, we aren’t either.” Marvin Minksy “The competent programmer is fully aware of the limited size of his own skull. He therefore approaches his task with full humility, and avoids clever tricks like the...

Cricpy takes a swing at the ODIs

https://www.datacamp.com/community/news/data-science-nuclear-engineering-and-the-open-source-with-katy-huff-dye1i4ngg8e

Data science, nuclear engineering, the importance of interdisciplinary data science and the open source.

Data Science, Nuclear Engineering and the Open Source (with Katy Huff)

https://www.datacamp.com/community/news/data-science-at-buzzfeed-and-the-digital-media-landscape-50nqxmvpwed

How does data science help Buzzfeed achieve online virality? What type of mass online experiments do data scientists at BuzzFeed run for this purpose? What products do they develop to make all of this easy and intuitive for content producers? 

Data Science at BuzzFeed and the Digital Media Landscape

<br><center><a target="_blank" href="https://www.datacamp.com/groups/business"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1624651228/schedule_a_demo_bfhlx1.png" /></a></center><br>
<p>In 2021, we witnessed the impact of COVID-19 on accelerating digital transformation. This rapid digitization has led to an explosion in the amount of data organizations are generating. As organizations look to reap the value of this data in 2022, they will adopt new technologies, upskill their workforce, and embrace new ways of work. Here are nine data science trends and predictions to expect in 2022.</p>
<h2 id="-1-organizations-accelerate-culture-transformation-programs">#1 Organizations accelerate culture transformation programs</h2>
<p>Data culture is the collective behaviors and beliefs of people who value, practice and encourage the use of data to improve decision-making. It sets the foundation and mindset for companies to effectively capture value from their ever-growing datasets.</p>
<p>Unfortunately, the <a target="_blank" href="https://c6abb8db-514c-4f5b-b5a1-fc710f1e464e.filesusr.com/ugd/e5361a_d59b4629443945a0b0661d494abb5233.pdf">lack of data culture</a> is a major roadblock in an organization’s roadmap to becoming data-driven.   </p>
<center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1642644861/image2_t1r0vh.png" /></center>

<p>In 2022, we anticipate chief data officers (CDO) to focus on the key building blocks of data culture, which include proper data governance, data literacy programs, and nurturing a data-driven mindset. These are catalysts for organization-wide data-driven decision-making processes.</p>
<h2 id="-2-organizations-will-scale-data-governance">#2 Organizations will scale data governance</h2>
<p>The growing demand for self-serve analytics fueled the need for compliant, actionable, and high-quality data. Yet, the challenge of measuring and maintaining data quality scales in tandem with the size and complexity of the datasets. As such, companies are adapting their data governance strategies.</p>
<p>One such strategy is to adopt <a target="_blank" href="https://www.montecarlodata.com/what-is-data-observability/">data observability</a> in data pipelines. Simply put, data observability aims to identify, troubleshoot, and resolve data issues in near real-time.</p>
<p>In 2022 and beyond, more companies will scale their data governance programs and adopt new and modern tools to monitor and detect data quality issues.</p>
<h2 id="-3-nlp-ushers-in-a-new-generation-of-low-code-data-tools">#3: NLP ushers in a new generation of low-code data tools</h2>
<p>NLP has grown tremendously in the past few years, thanks to the arms race for ever-larger large language models (LLM) like T5, GPT-3, and Megatron-Turing NLG.</p>
<p><center><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1642644862/image3_mt06mh.png" />Large language models get larger over the years (<a href="https://huggingface.co/blog/large-language-models">Source</a>)</center></p>
<p>LLMs are pushing the limits of what NLP can do. The most recent models surprised the community with their ability to generate various types of texts (like computer code and guitar tabs) without any explicit training.</p>
<p>Such NLP models have the potential to usher in the age of low-code and no-code tools. Today, <a target="_blank" href="https://blogs.microsoft.com/ai/from-conversation-to-code-microsoft-introduces-its-first-product-features-powered-by-gpt-3/">Microsoft’s Power App</a> allows non-technical users to build apps with natural speech. Such tools will continue to lower the barriers to coding and facilitate the rise of citizen developers and citizen data scientists within organizations.</p>
<h2 id="-4-l-d-becomes-a-fabric-of-company-culture">#4 L&amp;D becomes a fabric of company culture</h2>
<p>Driven by the C-suite’s concern over their <a target="_blank" href="https://www.pwc.com/gx/en/issues/reinventing-the-future/take-on-tomorrow/covid-disparity-evaluation.html">organization’s lack of key skills</a>, companies will continue to invest heavily in learning and development (L&amp;D) programs in 2022. The benefits of such L&amp;D programs are obvious–the World Economic Forum predicted that <a target="_blank" href="https://www.pwc.com/gx/en/issues/upskilling/upskilling-for-shared-prosperity.html">38% of GDP can be gained from upskilling</a> by 2030.</p>
<p>As the workforce continues to cope with the pandemic, we expect companies to allocate their L&amp;D budgets on virtual learning ecosystems that facilitate effective learning and provide communities of practice. Companies looking to improve their employees’ data literacy at scale can leverage existing L&amp;D programs as internal data science skill academies.</p>
<h2 id="data-trend-5-mlops-will-continue-to-mature-within-organizations">Data Trend #5:  MLOps will continue to mature within organizations</h2>
<p>MLOps is a set of practices that combine machine learning, data engineering, and DevOps. It includes standardized processes that automate the machine learning workflow.</p>
<p>Companies can only extract value from machine learning at scale with production-level AI systems. This explains why the demand for MLOps is expected to grow immensely. In fact, the industry is estimated to be valued at <a target="_blank" href="https://www.cognilytica.com/2020/04/02/infographic-the-rapid-growth-of-mlops/">$126.1 billion</a> by 2025.</p>
<p>In the upcoming year, MLOps tools like <a target="_blank" href="https://www.kubeflow.org/">KubeFlow</a> and <a target="_blank" href="https://mlflow.org/">MLFlow</a> will continue to mature. It is only a matter of time before they become a staple for all data science teams.  </p>
<h2 id="data-trend-6-responsible-ai-becomes-more-operationalized">Data Trend #6: Responsible AI becomes more operationalized</h2>
<p>Unfortunately, many existing AI systems today are rife with hidden biases. As such, regulators <a target="_blank" href="https://www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/what-the-draft-european-union-ai-regulations-mean-for-business">in the EU</a> are planning to hold such AI accountable, and many are expected to follow suit. Companies must ensure that their AI systems remain fair and responsible. Those that fail to do so risk damaging their reputations and are culpable for exacerbating inequality.</p>
<p>That is why companies are increasingly operationalizing principles of Responsible AI to ensure that AI remains fair, interpretable, privacy-preserving, and secure. An example of a framework is PwC’s <a target="_blank" href="https://www.pwc.com/gx/en/issues/data-and-analytics/artificial-intelligence/what-is-responsible-ai.html">Responsible AI Toolkit</a>, which addresses the various dimensions of Responsible AI.</p>
<h2 id="data-trend-7-rise-of-the-data-mesh">Data Trend #7: Rise of the data mesh</h2>
<p>Most data architectures today are in the form of data lakes. That might change soon as a new form of data architecture addresses the weakness of data lakes.</p>
<p>A new alternative, coined by Zhamak Dehghani is called the <a target="_blank" href="https://martinfowler.com/articles/data-monolith-to-mesh.html">data mesh</a>. A data mesh has distributed “data products”–each handled by a cross-functional team of data engineers and product owners. Adopting a data mesh architecture allows companies to <a target="_blank" href="https://www2.deloitte.com/nl/nl/pages/strategy-analytics-and-ma/articles/from-data-mess-to-a-data-mesh.html">deliver data faster and achieve greater business domain agility</a>.</p>
<p>Soon, as the pains of using data lakes become more acute, companies will begin experimenting with data mesh, as did <a target="_blank" href="https://databricks.com/session_eu20/data-mesh-in-practice-how-europes-leading-online-platform-for-fashion-goes-beyond-the-data-lake">Zalando</a> and <a target="_blank" href="https://medium.com/intuit-engineering/intuits-data-mesh-strategy-778e3edaa017">Intuit</a>.</p>
<h2 id="data-trend-8-new-generation-of-tooling-will-improve-the-data-team-s-productivity">Data Trend #8: New generation of tooling will improve the data team’s productivity</h2>
<p>Various data science productivity tools emerged in 2021 and will continue to gain popularity in the years to come. These tools reduce the need for manual work and allow data scientists to perform higher-value tasks.</p>
<p>Such productivity tools include AutoML tools (like <a target="_blank" href="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html">H2O AutoML</a> and <a target="_blank" href="https://www.automl.org/automl/autopytorch/">Auto PyTorch</a>), which automates the process of machine learning model selection and even hyperparameter tuning.</p>
<p>A multitude of <a target="_blank" href="https://blogs.nvidia.com/blog/2021/06/08/what-is-synthetic-data/">synthetic data generation tools</a> have also been on the rise. Their ability in creating balanced and labeled datasets at scale is particularly attractive for data-hungry companies.</p>
<p>Large data science teams might also find collaboration tools like <a href="https://databricks.com/solutions/data-science">Databricks</a> and <a target="_blank" href="https://www.datacamp.com/workspace">DataCamp Workspace</a> useful. These tools allow data scientists to collaborate asynchronously in data exploration and ML modeling.</p>
<br><center><a target="_blank" href="https://www.datacamp.com/workspace"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/v1642779150/chrome-capture_krnwqo.gif" /></a></center>
<h2 id="data-trend-9-the-talent-crunch-and-flexible-work-will-broaden-and-improve-the-search-for-data-talent">Data Trend #9: The talent crunch and flexible work will broaden and improve the search for data talent</h2>
<p>As the Great Resignation rages on, the talent crunch becomes ever more severe. This phenomenon prompts organizations to rethink how they recruit and retain data talent.</p>
<p>In particular, we expect companies to prioritize skills over zip codes in their hiring policies, as evident from the <a target="_blank" href="https://www.linkedin.com/business/talent/blog/talent-strategy/global-data-shows-surge-in-remote-work">280% increase in remote job postings</a> on LinkedIn since March 2020. Moreover, as FAANG companies embrace the work-from-home arrangement, we expect other tech companies to follow suit and provide their employees with flexible working options too.</p>
<p>For more on our data trends and predictions for 2022, download the free white paper, or register for our upcoming webinar where we’ll deep-dive into what’s coming in data science in the new year!</p>
<br><center><a target="_blank" href="https://www.datacamp.com/groups/business"><img src="https://res.cloudinary.com/dyd911kmh/image/upload/f_auto,q_auto:best/v1624651228/schedule_a_demo_bfhlx1.png" /></a></center>


https://www.datacamp.com/community/blog/2022-data-trends-and-predictions

2022 data trends and predictions

https://www.datacamp.com/community/tutorials/tutorial-cloudera-hadoop-tutorial

Learn about Hadoop ecosystem, the architectures and how to start with Cloudera. 

Cloudera Hadoop Tutorial

https://www.datacamp.com/community/tutorials/generate-data-trumania

Discover Python Trumania, a scenario-based random dataset generator library. Learn how to generate a synthetic and random dataset in this step-by-step tutorial. 

Step-by-step tutorial to use Python Trumania a random dataset generator. Learn how to generate synthetic datasets today!

community

datacamp

big data

News (37)

Creating Smart Cities With Data Science

Creating Smart Cities With Data Science

Creating Smart Cities With Data Science

Tutorials (4)

Cloudera Hadoop Tutorial

Generating Realistic Random Datasets with Trumania

Cheat Sheets (2)

PySpark Cheat Sheet: Spark DataFrames in Python

Open Courses (0)

Podcast (9)

Beyond the Language Wars: R & Python for the Modern Data Scientist (with Rick Scavetta and Boyan Angelov)

Blog Posts (34)

2022 data trends and predictions

Tech Thoughts (0)