Skip to content

Homework 1: Introduction for MLOps Zoomcamp 2024

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

Q1. Downloading the data

We'll use the same NYC taxi dataset, but instead of "Green Taxi Trip Records", we'll use "Yellow Taxi Trip Records".

Download the data for January and February 2023.

Read the data for January. How many columns are there?

urls = ["https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2023-01.parquet",
       "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2023-02.parquet"]
raw_dfs = [pd.read_parquet(url) for url in urls]

print(f"Number of columns in January data: {raw_dfs[0].shape[1]}")
raw_dfs = [df[['PULocationID', 'DOLocationID', 'tpep_dropoff_datetime', 'tpep_pickup_datetime']] for df in raw_dfs]

Q2. Computing duration

Now let's compute the duration variable. It should contain the duration of a ride in minutes.

What's the standard deviation of the trips duration in January?

def calc_duration(df):
    return (df['tpep_dropoff_datetime'] - df['tpep_pickup_datetime']).dt.total_seconds() / 60

raw_dfs[0] = (raw_dfs[0]
 .assign(duration=calc_duration)
)

duration_std = (raw_dfs[0]
 .duration
 .std()
)
print(f"Duration std: {duration_std:.2f}")

Q3. Dropping outliers

Next, we need to check the distribution of the duration variable. There are some outliers. Let's remove them and keep only the records where the duration was between 1 and 60 minutes (inclusive).

What fraction of the records left after you dropped the outliers?

jan_df = (raw_dfs[0]
.query("duration >= 1 and duration < 60")
)

outlier_frac = jan_df.shape[0] / raw_dfs[0].shape[0]
print(f"Fraction after removing outliers: {outlier_frac:.2f}")

Q4. One-hot encoding

Let's apply one-hot encoding to the pickup and dropoff location IDs. We'll use only these two features for our model.

  • Turn the dataframe into a list of dictionaries (remember to re-cast the ids to strings - otherwise it will label encode them)
  • Fit a dictionary vectorizer
  • Get a feature matrix from it

What's the dimensionality of this matrix (number of columns)?

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction import DictVectorizer
from sklearn.preprocessing import FunctionTransformer

# Define a transformer to convert DataFrame to a list of dictionaries
def df_to_dicts(df):
    return df[['PULocationID', 'DOLocationID']].astype(str).to_dict(orient='records')

# Initialize the DictVectorizer
vectorizer = DictVectorizer(sparse=True)

# Create a pipeline with a single step that converts the DataFrame to a list of dictionaries and then applies DictVectorizer
pipeline = Pipeline([
    ('transformer', FunctionTransformer(df_to_dicts, validate=False)),
    ('vectorizer', vectorizer)
])

# Selecting the features and target variable
X = jan_df[['PULocationID', 'DOLocationID']]
y = jan_df['duration']

# Fitting the pipeline to the data
X_transformed = pipeline.fit_transform(X)

X_transformed.shape

Q5. Training a model

Now let's use the feature matrix from the previous step to train a model.

  • Train a plain linear regression model with default parameters
  • Calculate the RMSE of the model on the training data

What's the RMSE on train?

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
import numpy as np

# Train the model
model = LinearRegression()
model.fit(X_transformed, y)

# Predict on the training data
y_pred = model.predict(X_transformed)

# Calculate RMSE
rmse = np.sqrt(mean_squared_error(y, y_pred))
print(f"RMSE: {rmse:.2f}")

Q6. Evaluating the model

Now let's apply this model to the validation dataset (February 2023).

What's the RMSE on validation?

feb_df = (raw_dfs[1]
          .assign(duration=calc_duration)
          #.query("duration >= 1 and duration < 60")
          .loc[:,['PULocationID', 'DOLocationID', 'duration']]
)

val_X = feb_df[['PULocationID', 'DOLocationID']]
val_y = feb_df['duration']

# Fitting the pipeline to the data
val_X_transformed = pipeline.fit_transform(val_X)
val_X_transformed.shape
‌
‌
‌