Skip to content

🏦 Banks are battling frauds with machine learning models, but changing data patterns can weaken these defenses. London's Poundbank needs your help to figure out why their fraud detection models aren't as accurate anymore.

Poundbank recommends the nannyml library for monitoring machine learning models, which is also their tool of choice.

The data

They have provided you with a reference(test data) and analysis set(production data). A summary and preview are provided below.

reference.csv and analysis.csv

ColumnDescription
'timestamp'Date of the transaction.
'time_since_login_min'Time since the user logged in to the app.
'transaction_amount'The amount of Pounds(£) that users sent to another account.
'transaction_type'Transaction type:
  • CASH-OUT - Withdrawing money from an account.
  • PAYMENT - Transaction where a payment is made to a third party.
  • CASH-IN - This is the opposite of a cash-out. It involves depositing money into an account.
  • TRANSFER - Transaction which involves moving funds from one account to another.
'is_first_transaction'A binary indicator denoting if the transaction is the user's first (1 for the first transaction, 0 otherwise).
'user_tenure_months'The duration in months since the user's account was created or since they became a member.
'is_fraud'A binary label indicating whether the transaction is fraudulent (1 for fraud, 0 otherwise).
'predicted_fraud_proba'The probability assigned by a detection model indicates the likelihood of a fraudulent transaction.
'predicted_fraud'The predicted classification label is calculated based on predicted fraud probability by the detection model (1 for predicted fraud, 0 otherwise).
# Re-run this cell to install nannyml
!pip install nannyml
# Re-run this cell
# Import required libraries
import pandas as pd
import nannyml as nml
nml.disable_usage_logging()

reference = pd.read_csv("reference.csv")
analysis = pd.read_csv("analysis.csv")
reference.head()
# Start coding here
# Use as many cells as you need
analysis.head()
cont_columns = ['is_fraud', 'predicted_fraud_proba', 'time_since_login_min', 'transaction_amount', 'user_tenure_months']

columns = [x for x in reference.columns if x not in cont_columns]
unseen_calc = nml.UnseenValuesCalculator(column_names=columns,chunk_period='m',timestamp_column_name='timestamp')
unseen_calc.fit(reference)
unseen_calc.calculate(analysis).plot()
analysis['is_first_transaction'] = analysis['is_first_transaction'].astype(int)
fraud_estimator =(
    nml.CBPE(
        y_pred='predicted_fraud',
        y_pred_proba='predicted_fraud_proba',
        y_true='is_fraud',
        timestamp_column_name='timestamp',
        problem_type='classification_binary',
        chunk_period='m',
        metrics=['accuracy']
    )
    )
fraud_estimator.fit(reference)
est_results = fraud_estimator.estimate(analysis)
check_performance = (nml.PerformanceCalculator(
    y_pred='predicted_fraud',
        y_pred_proba='predicted_fraud_proba',
        y_true='is_fraud',
        timestamp_column_name='timestamp',
        problem_type='classification_binary',
        chunk_period='m',
        metrics=['accuracy']
)
                    )
check_performance.fit(reference_data=reference)
calc_results = check_performance.calculate(analysis)
est_results.compare(other=calc_results).plot()
months_with_performance_alerts = ['april_2019','may_2019','june_2019']
# reference['is_first_transaction'] =reference["is_first_transaction"].astype(int)
‌
‌
‌