Skip to content
0

Unveiling Trends in Renewable Energy 🌍🔋

The Data Scientist Master

📖 Background

The race to net-zero emissions is heating up. As nations work to combat climate change and meet rising energy demands, renewable energy has emerged as a cornerstone of the clean transition. Solar, wind, and hydro are revolutionizing how we power our lives. Some countries are leading the charge, while others are falling behind. But which nations are making the biggest impact? What’s driving their success? And what lessons can we learn to accelerate green energy transition?

As a data scientist at NextEra Energy, one of the world’s leading renewable energy providers, your role is to move beyond exploration, into prediction. Using a rich, real-world dataset, you’ll build models to forecast renewable energy production, drawing on indicators like GDP, population, carbon emissions, and policy metrics.

With the world watching, your model could help shape smarter investments, forward-thinking policies, and a faster transition to clean energy. 🔮⚡🌱

💾 The data

Your team has gathered a global renewable energy dataset ("Training_set_augumented.csv") covering energy production, investments, policies, and economic factors shaping renewable adoption worldwide:

🌍 Basic Identifiers

  • Country – Country name
  • Year – Calendar year (YYYY)
  • Energy Type – Type of renewable energy (e.g., Solar, Wind)
⚡ Energy Metrics
  • Production (GWh) – Renewable energy produced (Gigawatt-hours)
  • Installed Capacity (MW) – Installed renewable capacity (Megawatts)
  • Investments (USD) – Total investment in renewables (US Dollars)
  • Energy Consumption (GWh) – Total national energy use
  • Energy Storage Capacity (MWh) – Capacity of energy storage systems
  • Grid Integration Capability (Index) – Scale of 0–1; ability to handle renewables in grid
  • Electricity Prices (USD/kWh) – Average cost of electricity
  • Energy Subsidies (USD) – Government subsidies for energy sector
  • Proportion of Energy from Renewables (%) – Share of renewables in total energy mix
🧠 Innovation & Tech
  • R&D Expenditure (USD) – R&D spending on renewables
  • Renewable Energy Patents – Number of patents filed
  • Innovation Index (Index) – Global innovation score (0–100)
💰 Economy & Policy
  • GDP (USD) – Gross domestic product
  • Population – Total population
  • Government Policies – Number of policies supporting renewables
  • Renewable Energy Targets – Whether national targets are in place (1 = Yes, 0 = No)
  • Public-Private Partnerships in Energy – Number of active collaborations
  • Energy Market Liberalization (Index) – Scale of 0–1
🧑‍🤝‍🧑 Social & Governance
  • Ease of Doing Business (Score) – World Bank index (0–100)
  • Regulatory Quality – Governance score (-2.5 to 2.5)
  • Political Stability – Governance score (-2.5 to 2.5)
  • Control of Corruption – Governance score (-2.5 to 2.5)
🌿 Environment & Resources
  • CO2 Emissions (MtCO2) – Emissions in million metric tons
  • Average Annual Temperature (°C) – Country’s avg. temp
  • Solar Irradiance (kWh/m²/day) – Solar energy availability
  • Wind Speed (m/s) – Average wind speed
  • Hydro Potential (Index) – Relative hydropower capability (0–1)
  • Biomass Availability (Tons/year) – Total available biomass

💪 Challenge

As a data scientist at NextEra Energy, your task is to use the Training Set (80% of the data) to train a powerful machine learning model that can predict renewable energy production (GWh). Once your model is trained, you will use it to generate predictions for the Test Set, which does not include the target (Production (GWh)) but has an additional ID column.

🚀 Your Task:

  1. Train Your Model:

    • Use the Training Set, which contains all features and the target (Production (GWh)), to build and fine-tune your model.
    • Explore, clean, and transform the data as needed.

  2. Generate Predictions:

    • Use your trained model to make predictions for the Test Set (20%), which has all the features except Production (GWh).
    • The Test Set also has an ID column, which uniquely identifies each row.

  3. Submit Your Results:

    • Save your predictions as a CSV file with exactly two columns:

      • ID: Directly from the Test Set (must match exactly).
      • Predicted Production (GWh): Your model’s predictions for each row.

🌐 Ready to Start?

  • Download the Training Set and Test Set.
  • Build, train, and test your model.
  • Submit your predictions. 🚀

🔎 Your model won’t just generate predictions — it will uncover underlying drivers of renewable energy production and reveal where the biggest gains can be made!

🧑‍⚖️ Judging Criteria

Your submission will be evaluated using a hybrid system, combining Model Accuracy (80%) and Community Votes (20%).

📊 1. Model Accuracy (80%)

  • Your submission will be scored using Root Mean Squared Error (RMSE), which measures how close your predictions are to the actual values in our hidden test set.
  • The lower your RMSE, the better your model’s performance.
✅ Submission Instructions:

  • First, submit your Datalab workbook.

  • Then, submit your predictions as a .csv file via this Google Form.

  • Your file must contain exactly two columns:

    • ID: Directly from the Test Set (must match exactly).
    • Predicted Production (GWh): Your model’s predictions for each row.
✅ Submission Example:
IDPredicted Production (GWh)
150200.34
267820.78
345210.55
......
✅ Important:
  • Use the same email address for the Google Form as the one associated with your DataCamp account. This is how we will link your submission to your Datalab workbook.
  • Only submissions in the correct format will be accepted and scored.
  • We will automatically check for formatting errors (missing IDs, extra IDs, or invalid columns).

✍️ 2. Community Votes (20%)

  • Once the competition ends, you will be able to view the top submissions from other participants.
  • Vote for the most insightful, creative, or well-explained solutions.

✅ Checklist before publishing

  • Rename your workspace to make it descriptive of your work. N.B. you should leave the notebook name as notebook.ipynb.
  • Remove redundant cells like the introduction to data science notebooks, so the workbook is focused on your story.
  • Check that all the cells run without error.

⏳ Data is the new fuel - let’s generate insights and electrify the future!

🌍 Renewable Energy Production Prediction

DataCamp Competition Portfolio Project

📋 Executive Summary

This project analyzes global renewable energy production patterns and builds predictive models to forecast energy generation. Using advanced machine learning techniques and comprehensive feature engineering, we aim to identify key drivers of renewable energy success worldwide.

📚 Table of Contents

  1. Data Loading & Initial Exploration (Invalid URL)
  2. Exploratory Data Analysis (Invalid URL)
  3. Feature Engineering (Invalid URL)
  4. Model Development (Invalid URL)
  5. Model Evaluation & Selection (Invalid URL)
  6. Predictions & Submission (Invalid URL)
  7. Business Insights (Invalid URL)
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.linear_model import Ridge, ElasticNet
from sklearn.preprocessing import StandardScaler, RobustScaler, LabelEncoder
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
from sklearn.feature_selection import SelectKBest, f_regression
import xgboost as xgb
import warnings
warnings.filterwarnings('ignore')

# Set style for beautiful plots
plt.style.use('seaborn-v0_8')
sns.set_palette("husl")

print("🚀 Environment setup complete! Ready to predict renewable energy production.")

## 1. Data Loading & Initial Exploration {#data-loading}

# Load the datasets
print("📊 Loading datasets...")
train_df = pd.read_csv('Training_set_augmented.csv')
test_df = pd.read_csv('Public_Test_Set.csv')  # Assuming test set filename

print(f"✅ Training set shape: {train_df.shape}")
print(f"✅ Test set shape: {test_df.shape}")

# Display basic information
print("\n📋 Dataset Overview:")
train_df.info()

# First look at the data
print("\n👀 First 5 rows of training data:")
train_df.head()

# Check for missing values
print("\n🔍 Missing Values Analysis:")
missing_train = train_df.isnull().sum()
missing_test = test_df.isnull().sum()

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))

# Training set missing values
missing_train_pct = (missing_train / len(train_df)) * 100
missing_train_pct = missing_train_pct[missing_train_pct > 0].sort_values(ascending=False)

if not missing_train_pct.empty:
    ax1.barh(range(len(missing_train_pct)), missing_train_pct.values, color='coral')
    ax1.set_yticks(range(len(missing_train_pct)))
    ax1.set_yticklabels(missing_train_pct.index)
    ax1.set_xlabel('Missing Percentage (%)')
    ax1.set_title('Missing Values - Training Set')
else:
    ax1.text(0.5, 0.5, 'No Missing Values!', ha='center', va='center', fontsize=16, color='green')
    ax1.set_title('Missing Values - Training Set')

# Test set missing values
missing_test_pct = (missing_test / len(test_df)) * 100
missing_test_pct = missing_test_pct[missing_test_pct > 0].sort_values(ascending=False)

if not missing_test_pct.empty:
    ax2.barh(range(len(missing_test_pct)), missing_test_pct.values, color='lightblue')
    ax2.set_yticks(range(len(missing_test_pct)))
    ax2.set_yticklabels(missing_test_pct.index)
    ax2.set_xlabel('Missing Percentage (%)')
    ax2.set_title('Missing Values - Test Set')
else:
    ax2.text(0.5, 0.5, 'No Missing Values!', ha='center', va='center', fontsize=16, color='green')
    ax2.set_title('Missing Values - Test Set')

plt.tight_layout()
plt.show()

# Summary statistics
print("\n📊 Summary Statistics:")
train_df.describe()

2. Exploratory Data Analysis {#eda}

2.1 Target Variable Analysis

Analyze the target variable

fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(15, 12))

# Distribution of Production
ax1.hist(train_df['Production (GWh)'], bins=50, alpha=0.7, color='skyblue', edgecolor='black')
ax1.set_xlabel('Production (GWh)')
ax1.set_ylabel('Frequency')
ax1.set_title('📊 Distribution of Renewable Energy Production')
ax1.grid(True, alpha=0.3)

# Log-transformed distribution
ax2.hist(np.log1p(train_df['Production (GWh)']), bins=50, alpha=0.7, color='lightgreen', edgecolor='black')
ax2.set_xlabel('Log(Production + 1)')
ax2.set_ylabel('Frequency')
ax2.set_title('📊 Log-Transformed Production Distribution')
ax2.grid(True, alpha=0.3)

# Box plot by Energy Type
energy_types = train_df['Energy Type'].value_counts().head(8).index
filtered_df = train_df[train_df['Energy Type'].isin(energy_types)]
sns.boxplot(data=filtered_df, x='Energy Type', y='Production (GWh)', ax=ax3)
ax3.set_xticklabels(ax3.get_xticklabels(), rotation=45)
ax3.set_title('📊 Production by Energy Type')

# Production over time
yearly_production = train_df.groupby('Year')['Production (GWh)'].sum()
ax4.plot(yearly_production.index, yearly_production.values, marker='o', linewidth=2, markersize=6)
ax4.set_xlabel('Year')
ax4.set_ylabel('Total Production (GWh)')
ax4.set_title('📈 Total Renewable Energy Production Over Time')
ax4.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

print(f"🎯 Target Variable Statistics:")
print(f"Mean Production: {train_df['Production (GWh)'].mean():,.2f} GWh")
print(f"Median Production: {train_df['Production (GWh)'].median():,.2f} GWh")
print(f"Standard Deviation: {train_df['Production (GWh)'].std():,.2f} GWh")
print(f"Skewness: {train_df['Production (GWh)'].skew():.2f}")

2.2 Geographic and Energy Type Analysis

Top producing countries

top_countries = train_df.groupby('Country')['Production (GWh)'].sum().sort_values(ascending=False).head(15)

fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(15, 12))

# Top producing countries
colors = plt.cm.viridis(np.linspace(0, 1, len(top_countries)))
bars = ax1.barh(range(len(top_countries)), top_countries.values, color=colors)
ax1.set_yticks(range(len(top_countries)))
ax1.set_yticklabels(top_countries.index)
ax1.set_xlabel('Total Production (GWh)')
ax1.set_title('🌍 Top 15 Renewable Energy Producing Countries')
ax1.grid(True, alpha=0.3)

# Add value labels on bars
for i, (bar, value) in enumerate(zip(bars, top_countries.values)):
    ax1.text(value + max(top_countries.values) * 0.01, i, f'{value:,.0f}', 
             va='center', ha='left', fontweight='bold')

# Energy type breakdown
energy_production = train_df.groupby('Energy Type')['Production (GWh)'].sum().sort_values(ascending=False)
colors_energy = plt.cm.Set3(np.linspace(0, 1, len(energy_production)))

wedges, texts, autotexts = ax2.pie(energy_production.values, labels=energy_production.index, 
                                   autopct='%1.1f%%', colors=colors_energy, startangle=90)
ax2.set_title('⚡ Renewable Energy Production by Type')

# Enhance pie chart text
for autotext in autotexts:
    autotext.set_color('black')
    autotext.set_fontweight('bold')

plt.tight_layout()
plt.show()

2.3 Correlation Analysis

Select numeric columns for correlation analysis

numeric_cols = train_df.select_dtypes(include=[np.number]).columns
correlation_matrix = train_df[numeric_cols].corr()

# Create a more focused correlation heatmap
fig, ax = plt.subplots(figsize=(16, 14))

# Create mask for upper triangle
mask = np.triu(np.ones_like(correlation_matrix, dtype=bool))

# Generate heatmap
sns.heatmap(correlation_matrix, mask=mask, annot=True, cmap='RdYlBu_r', center=0,
            square=True, fmt='.2f', cbar_kws={'shrink': 0.8}, ax=ax)
ax.set_title('🔗 Feature Correlation Matrix', fontsize=16, pad=20)

plt.tight_layout()
plt.show()

# Show strongest correlations with target
target_corr = correlation_matrix['Production (GWh)'].abs().sort_values(ascending=False)
print("🎯 Strongest correlations with Production:")
print(target_corr.head(10))
‌
‌
‌