{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Gabriele Iocco

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Previsione di opportunità di Cross Sell di assicurazioni

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Costruisco un modello predittivo in grado di prevedere se gli assicurati dell'anno passato potrebbero essere interessati ad acquistare anche un'assicurazione per il proprio veicolo.

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Importo le librerie necessarie

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import seaborn as sns\n", "import warnings\n", "\n", "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n", "from sklearn.compose import ColumnTransformer\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.impute import SimpleImputer\n", "from sklearn.ensemble import RandomForestClassifier\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score, classification_report, confusion_matrix\n", "from sklearn.model_selection import GridSearchCV\n", "from imblearn.pipeline import Pipeline as ImbPipeline\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Carico il dataset e analizzo i dati

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "BASE_URL = \"D:/Anaconda/Cross_Sell/\"\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_cross_sell = pd.read_csv(BASE_URL+\"insurance_cross_sell.csv\")\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
Out[5]:
\n
\n
(381109, 12)
\n
\n
\n
\n
" ] } } ], "source": [ "df_cross_sell.shape\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
Out[6]:
\n
\n
id                      381109\nGender                  381109\nAge                     381109\nDriving_License         381109\nRegion_Code             381109\nPreviously_Insured      381109\nVehicle_Age             381109\nVehicle_Damage          381109\nAnnual_Premium          381109\nPolicy_Sales_Channel    381109\nVintage                 381109\nResponse                381109\ndtype: int64
\n
\n
\n
\n
" ] } } ], "source": [ "df_cross_sell.count()\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
Out[7]:
\n
\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
idGenderAgeDriving_LicenseRegion_CodePreviously_InsuredVehicle_AgeVehicle_DamageAnnual_PremiumPolicy_Sales_ChannelVintageResponse
01Male44128.00> 2 YearsYes40454.026.02171
12Male7613.001-2 YearNo33536.026.01830
23Male47128.00> 2 YearsYes38294.026.0271
34Male21111.01< 1 YearNo28619.0152.02030
45Female29141.01< 1 YearNo27496.0152.0390
\n
\n
\n
\n
\n
" ] } } ], "source": [ "df_cross_sell.head()\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
\nInformazioni sul dataset:\n<class 'pandas.core.frame.DataFrame'>\nRangeIndex: 381109 entries, 0 to 381108\nData columns (total 12 columns):\n #   Column                Non-Null Count   Dtype  \n---  ------                --------------   -----  \n 0   id                    381109 non-null  int64  \n 1   Gender                381109 non-null  object \n 2   Age                   381109 non-null  int64  \n 3   Driving_License       381109 non-null  int64  \n 4   Region_Code           381109 non-null  float64\n 5   Previously_Insured    381109 non-null  int64  \n 6   Vehicle_Age           381109 non-null  object \n 7   Vehicle_Damage        381109 non-null  object \n 8   Annual_Premium        381109 non-null  float64\n 9   Policy_Sales_Channel  381109 non-null  float64\n 10  Vintage               381109 non-null  int64  \n 11  Response              381109 non-null  int64  \ndtypes: float64(3), int64(6), object(3)\nmemory usage: 34.9+ MB\nNone\n
\n
\n
\n
\n
" ] } } ], "source": [ "print(\"\\nInformazioni sul dataset:\")\n", "print(df_cross_sell.info())\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
Out[9]:
\n
\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
idAgeDriving_LicenseRegion_CodePreviously_InsuredAnnual_PremiumPolicy_Sales_ChannelVintageResponse
count381109.000000381109.000000381109.000000381109.000000381109.000000381109.000000381109.000000381109.000000381109.000000
mean190555.00000038.8225840.99786926.3888070.45821030564.389581112.034295154.3473970.122563
std110016.83620815.5116110.04611013.2298880.49825117213.15505754.20399583.6713040.327936
min1.00000020.0000000.0000000.0000000.0000002630.0000001.00000010.0000000.000000
25%95278.00000025.0000001.00000015.0000000.00000024405.00000029.00000082.0000000.000000
50%190555.00000036.0000001.00000028.0000000.00000031669.000000133.000000154.0000000.000000
75%285832.00000049.0000001.00000035.0000001.00000039400.000000152.000000227.0000000.000000
max381109.00000085.0000001.00000052.0000001.000000540165.000000163.000000299.0000001.000000
\n
\n
\n
\n
\n
" ] } } ], "source": [ "df_cross_sell.describe()\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
Out[10]:
\n
\n
Gender\nMale      206089\nFemale    175020\nName: count, dtype: int64
\n
\n
\n
\n
" ] } } ], "source": [ "df_cross_sell[\"Gender\"].value_counts()\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
Out[11]:
\n
\n
Vehicle_Damage\nYes    192413\nNo     188696\nName: count, dtype: int64
\n
\n
\n
\n
" ] } } ], "source": [ "df_cross_sell[\"Vehicle_Damage\"].value_counts()\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
Out[12]:
\n
\n
Vehicle_Age\n1-2 Year     200316\n< 1 Year     164786\n> 2 Years     16007\nName: count, dtype: int64
\n
\n
\n
\n
" ] } } ], "source": [ "df_cross_sell[\"Vehicle_Age\"].value_counts()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Controllo se ci sono valori mancanti o nulli

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Valori mancanti per colonna:\nid                      0\nGender                  0\nAge                     0\nDriving_License         0\nRegion_Code             0\nPreviously_Insured      0\nVehicle_Age             0\nVehicle_Damage          0\nAnnual_Premium          0\nPolicy_Sales_Channel    0\nVintage                 0\nResponse                0\ndtype: int64\n
\n
\n
\n
\n
" ] } } ], "source": [ "print(\"Valori mancanti per colonna:\")\n", "print(df_cross_sell.isna().sum())\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Valori nulli per colonna:\nid                      0\nGender                  0\nAge                     0\nDriving_License         0\nRegion_Code             0\nPreviously_Insured      0\nVehicle_Age             0\nVehicle_Damage          0\nAnnual_Premium          0\nPolicy_Sales_Channel    0\nVintage                 0\nResponse                0\ndtype: int64\n
\n
\n
\n
\n
" ] } } ], "source": [ "print(\"Valori nulli per colonna:\")\n", "print(df_cross_sell.isnull().sum())\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
Out[15]:
\n
\n
Response\n0    334399\n1     46710\nName: count, dtype: int64
\n
\n
\n
\n
" ] } } ], "source": [ "counts_response = df_cross_sell[\"Response\"].value_counts()\n", "counts_response\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Acquirenti che hanno risposto positivamente alla proposta = 46710 (12.26%)\nAcquirenti che non hanno risposto positivamente alla proposta = 334399 (87.74%)\n
\n
\n
\n
\n
" ] } } ], "source": [ "print(f\"Acquirenti che hanno risposto positivamente alla proposta = {counts_response[1]} ({counts_response[1]/counts_response.sum()*100:.2f}%)\")\n", "print(f\"Acquirenti che non hanno risposto positivamente alla proposta = {counts_response[0]} ({counts_response[0]/counts_response.sum()*100:.2f}%)\")\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "class_distribution=(df_cross_sell['Response'].value_counts(normalize=True))\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Response\n0    0.877437\n1    0.122563\nName: proportion, dtype: float64\n
\n
\n
\n
\n
" ] } } ], "source": [ "print(class_distribution)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "plt.figure(figsize=(10, 6))\n", "class_distribution.plot(kind='bar')\n", "plt.title('Distribuzione delle classi (Response)')\n", "plt.xlabel('Classi')\n", "plt.ylabel('Percentuale')\n", "plt.xticks(rotation=0)\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Modello di regressione logistica

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Faccio il drop della colonna id e separo la colonna 'Response' dal resto del dataset

X conterrà tutte le feature (variabili indipendenti) che useremo per prevedere 'Response'

y è la variabile target (dipendente) che stiamo cercando di prevedere

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_cross_sell.drop('id', axis=1, inplace=True)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "X = df_cross_sell.drop('Response', axis=1) # dati in formato pandas\n", "y = df_cross_sell['Response']\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Divisione in train, val e test set, definizione delle colonne e creazione del preprocessor

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "# Funzione di valutazione\n", "def evaluate_model(y_true, y_pred, y_pred_proba, model_name):\n", " results = {\n", " \"Accuracy\": accuracy_score(y_true, y_pred),\n", " \"Precision\": precision_score(y_true, y_pred),\n", " \"Recall\": recall_score(y_true, y_pred),\n", " \"F1-score\": f1_score(y_true, y_pred),\n", " \"AUC-ROC\": roc_auc_score(y_true, y_pred_proba)\n", " }\n", " \n", " print(f\"\\nEvaluation for {model_name}:\")\n", " for metric, value in results.items():\n", " print(f\"{metric}: {value:.4f}\")\n", " \n", " print(\"\\nConfusion Matrix:\")\n", " print(confusion_matrix(y_true, y_pred))\n", " print(\"\\nClassification Report:\")\n", " print(classification_report(y_true, y_pred))\n", " \n", " return results\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "RANDOM_SEED = 42\n", "X_train_lr, X_test_lr, y_train_lr, y_test_lr = train_test_split(X, y, test_size=0.15, random_state=RANDOM_SEED)\n", "X_train_lr, X_val_lr, y_train_lr, y_val_lr = train_test_split(X_train_lr, y_train_lr, test_size=0.17, random_state=RANDOM_SEED)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
Out[29]:
\n
\n
(268871, 10)
\n
\n
\n
\n
" ] } } ], "source": [ "X_train_lr.shape # set di addestramento\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
Out[30]:
\n
\n
(55071, 10)
\n
\n
\n
\n
" ] } } ], "source": [ "X_val_lr.shape # set di validazione \n", " # serve a valutare le performance del modello durante il processo di addestramento\n", " # aiuta a prevenire l'overfitting, aiuta a regolare gli iperparametri\n", " # fornisce una stima delle performance del modello su dati non visti\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
Out[31]:
\n
\n
(57167, 10)
\n
\n
\n
\n
" ] } } ], "source": [ "X_test_lr.shape # set di test. Usato per la valutazione finale del modello\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "numeric_features = ['Age', 'Annual_Premium', 'Vintage', 'Driving_License', 'Region_Code', 'Previously_Insured', 'Policy_Sales_Channel']\n", "categorical_features = ['Gender', 'Vehicle_Age', 'Vehicle_Damage']\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Preprocessor e one-hot encoding

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# il preprocessor serve per standardizzare il processo di preparazione, pulizia e trasformazione dei dati\n", "# assicura che i dati siano trattati allo stesso modo in fase di training, validazione e testing\n", "# il one-hot encoding è una tecnica di codifica dei dati categorici\n", "preprocessor = ColumnTransformer(\n", " transformers=[\n", " ('num', StandardScaler(), numeric_features),\n", " ('cat', OneHotEncoder(drop='first', sparse_output=False, handle_unknown='ignore'), categorical_features)\n", " ])\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# la pipeline automatizza il flusso di lavoro\n", "# combina più step di preprocessing e il modello in un unico oggetto\n", "# prevenzione del data leakage, codice più pulito\n", "pipeline_lr = Pipeline([\n", " ('preprocessor', preprocessor),\n", " ('classifier', LogisticRegression(class_weight='balanced', random_state=RANDOM_SEED, max_iter=1000))\n", "])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Addestro il modello

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
Out[36]:
\n
\n
Pipeline(steps=[('preprocessor',\n                 ColumnTransformer(transformers=[('num', StandardScaler(),\n                                                  ['Age', 'Annual_Premium',\n                                                   'Vintage', 'Driving_License',\n                                                   'Region_Code',\n                                                   'Previously_Insured',\n                                                   'Policy_Sales_Channel']),\n                                                 ('cat',\n                                                  OneHotEncoder(drop='first',\n                                                                handle_unknown='ignore',\n                                                                sparse_output=False),\n                                                  ['Gender', 'Vehicle_Age',\n                                                   'Vehicle_Damage'])])),\n                ('classifier',\n                 LogisticRegression(class_weight='balanced', max_iter=1000,\n                                    random_state=42))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
\n
\n
\n
\n
" ] } } ], "source": [ "pipeline_lr.fit(X_train_lr, y_train_lr)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from termcolor import colored\n", "\n", "def print_colored_title(text):\n", " light_green = '\\033[38;2;000;204;000m' \n", " bold = '\\033[1m'\n", " reset = '\\033[0m'\n", " print(f\"{light_green}{bold}{text}{reset}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Predizioni sul set di training

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Performance sul set di training:\n\nEvaluation for Logistic Regression (Training):\nAccuracy: 0.6386\nPrecision: 0.2494\nRecall: 0.9730\nF1-score: 0.3971\nAUC-ROC: 0.8354\n\nConfusion Matrix:\n[[139714  96277]\n [   887  31993]]\n\nClassification Report:\n              precision    recall  f1-score   support\n\n           0       0.99      0.59      0.74    235991\n           1       0.25      0.97      0.40     32880\n\n    accuracy                           0.64    268871\n   macro avg       0.62      0.78      0.57    268871\nweighted avg       0.90      0.64      0.70    268871\n\n
\n
\n
\n
\n
" ] } } ], "source": [ "y_pred_train_lr = pipeline_lr.predict(X_train_lr)\n", "y_pred_proba_train_lr = pipeline_lr.predict_proba(X_train_lr)[:, 1]\n", "\n", "print(\"Performance sul set di training:\")\n", "train_results_lr = evaluate_model(y_train_lr, y_pred_train_lr, y_pred_proba_train_lr, \"Logistic Regression (Training)\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Con una Precision del 24,94% il modello porta a molti falsi positivi

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Predizioni sul set di validazione

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
\nPerformance sul set di validazione:\n\nEvaluation for Logistic Regression (Validation):\nAccuracy: 0.6429\nPrecision: 0.2498\nRecall: 0.9752\nF1-score: 0.3977\nAUC-ROC: 0.8387\n\nConfusion Matrix:\n[[28910 19503]\n [  165  6493]]\n\nClassification Report:\n              precision    recall  f1-score   support\n\n           0       0.99      0.60      0.75     48413\n           1       0.25      0.98      0.40      6658\n\n    accuracy                           0.64     55071\n   macro avg       0.62      0.79      0.57     55071\nweighted avg       0.90      0.64      0.70     55071\n\n
\n
\n
\n
\n
" ] } } ], "source": [ "y_pred_val_lr = pipeline_lr.predict(X_val_lr)\n", "y_pred_proba_val_lr = pipeline_lr.predict_proba(X_val_lr)[:, 1]\n", "\n", "print(\"\\nPerformance sul set di validazione:\")\n", "val_results_lr = evaluate_model(y_val_lr, y_pred_val_lr, y_pred_proba_val_lr, \"Logistic Regression (Validation)\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Il modello mostra coerenza tra training e validazione, suggerendo una buona generalizzazione

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Faccio previsioni sul set di test per valutare le performance finali del modello dopo che è stato addestrato

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Logistic Regression:\n\nEvaluation for Logistic Regression:\nAccuracy: 0.6413\nPrecision: 0.2557\nRecall: 0.9727\nF1-score: 0.4049\nAUC-ROC: 0.8403\n\nConfusion Matrix:\n[[29686 20309]\n [  196  6976]]\n\nClassification Report:\n              precision    recall  f1-score   support\n\n           0       0.99      0.59      0.74     49995\n           1       0.26      0.97      0.40      7172\n\n    accuracy                           0.64     57167\n   macro avg       0.62      0.78      0.57     57167\nweighted avg       0.90      0.64      0.70     57167\n\n
\n
\n
\n
\n
" ] } } ], "source": [ "y_pred_test_lr = pipeline_lr.predict(X_test_lr)\n", "y_pred_proba_test_lr = pipeline_lr.predict_proba(X_test_lr)[:, 1]\n", "\n", "print(\"Logistic Regression:\")\n", "test_results_lr = evaluate_model(y_test_lr, y_pred_test_lr, y_pred_proba_test_lr, \"Logistic Regression\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Il modello è eccellente nell'identificare i falsi positivi dati dall'alto Recall ma avendo una bassa Precision include molti falsi positivi

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Visualizzazione delle matrici di confusione

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "import matplotlib.pyplot as plt\n", "from sklearn.metrics import ConfusionMatrixDisplay\n", "\n", "def plot_confusion_matrix(y_true, y_pred, title):\n", " fig, ax = plt.subplots(figsize=(10, 6))\n", " ConfusionMatrixDisplay.from_predictions(y_true, y_pred, ax=ax, cmap='Blues', values_format='d')\n", " \n", " precision = precision_score(y_true, y_pred)\n", " recall = recall_score(y_true, y_pred)\n", " \n", " plt.text(0, -0.15, f\"Precision: {precision:.4f}\", transform=ax.transAxes)\n", " plt.text(1, -0.15, f\"Recall: {recall:.4f}\", transform=ax.transAxes, ha='right')\n", " \n", " plt.title(title)\n", " plt.tight_layout()\n", " plt.show()\n", "\n", "# Matrice di confusione per il set di training\n", "plot_confusion_matrix(y_train_lr, y_pred_train_lr, 'Matrice di Confusione (Training)')\n", "\n", "# Matrice di confusione per il set di validazione\n", "plot_confusion_matrix(y_val_lr, y_pred_val_lr, 'Matrice di Confusione (Validazione)')\n", "\n", "# Matrice di confusione per il set di test\n", "plot_confusion_matrix(y_test_lr, y_pred_test_lr, 'Matrice di Confusione (Test)')\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Cambio la soglia di decisione

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Set di training

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Performance sul set di training:\n\nEvaluation for Logistic Regression (Training):\nAccuracy: 0.7477\nPrecision: 0.2967\nRecall: 0.7764\nF1-score: 0.4294\nAUC-ROC: 0.8354\n\nConfusion Matrix:\n[[175495  60496]\n [  7353  25527]]\n\nClassification Report:\n              precision    recall  f1-score   support\n\n           0       0.96      0.74      0.84    235991\n           1       0.30      0.78      0.43     32880\n\n    accuracy                           0.75    268871\n   macro avg       0.63      0.76      0.63    268871\nweighted avg       0.88      0.75      0.79    268871\n\n
\n
\n
\n
\n
" ] } } ], "source": [ "y_pred_train_lr_mod = np.where(y_pred_proba_train_lr>0.65,1,0)\n", "y_pred_proba_train_lr_mod = pipeline_lr.predict_proba(X_train_lr)[:, 1]\n", "\n", "print(\"Performance sul set di training:\")\n", "train_results_lr_mod = evaluate_model(y_train_lr, y_pred_train_lr_mod, y_pred_proba_train_lr_mod, \"Logistic Regression (Training)\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Set di validazione

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
\nPerformance sul set di validazione:\n\nEvaluation for Logistic Regression (Validation):\nAccuracy: 0.7500\nPrecision: 0.2971\nRecall: 0.7816\nF1-score: 0.4305\nAUC-ROC: 0.8387\n\nConfusion Matrix:\n[[36101 12312]\n [ 1454  5204]]\n\nClassification Report:\n              precision    recall  f1-score   support\n\n           0       0.96      0.75      0.84     48413\n           1       0.30      0.78      0.43      6658\n\n    accuracy                           0.75     55071\n   macro avg       0.63      0.76      0.64     55071\nweighted avg       0.88      0.75      0.79     55071\n\n
\n
\n
\n
\n
" ] } } ], "source": [ "y_pred_val_lr_mod = np.where(y_pred_proba_val_lr>0.65,1,0)\n", "y_pred_proba_val_lr_mod = pipeline_lr.predict_proba(X_val_lr)[:, 1]\n", "\n", "print(\"\\nPerformance sul set di validazione:\")\n", "val_results_lr_mod = evaluate_model(y_val_lr, y_pred_val_lr_mod, y_pred_proba_val_lr_mod, \"Logistic Regression (Validation)\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Set di test

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Logistic Regression:\n\nEvaluation for Logistic Regression:\nAccuracy: 0.7526\nPrecision: 0.3083\nRecall: 0.7817\nF1-score: 0.4422\nAUC-ROC: 0.8403\n\nConfusion Matrix:\n[[37418 12577]\n [ 1566  5606]]\n\nClassification Report:\n              precision    recall  f1-score   support\n\n           0       0.96      0.75      0.84     49995\n           1       0.31      0.78      0.44      7172\n\n    accuracy                           0.75     57167\n   macro avg       0.63      0.77      0.64     57167\nweighted avg       0.88      0.75      0.79     57167\n\n
\n
\n
\n
\n
" ] } } ], "source": [ "y_pred_test_lr_mod = np.where(y_pred_proba_test_lr>0.65,1,0)\n", "y_pred_proba_test_lr_mod = pipeline_lr.predict_proba(X_test_lr)[:, 1]\n", "\n", "print(\"Logistic Regression:\")\n", "test_results_lr_mod = evaluate_model(y_test_lr, y_pred_test_lr_mod, y_pred_proba_test_lr_mod, \"Logistic Regression\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Visualizzazione delle matrici di confusione

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "def plot_confusion_matrix(y_true, y_pred, title):\n", " fig, ax = plt.subplots(figsize=(10, 6))\n", " ConfusionMatrixDisplay.from_predictions(y_true, y_pred, ax=ax, cmap='Blues', values_format='d')\n", " \n", " precision = precision_score(y_true, y_pred)\n", " recall = recall_score(y_true, y_pred)\n", " \n", " plt.text(0, -0.15, f\"Precision: {precision:.4f}\", transform=ax.transAxes)\n", " plt.text(1, -0.15, f\"Recall: {recall:.4f}\", transform=ax.transAxes, ha='right')\n", " \n", " plt.title(title)\n", " plt.tight_layout()\n", " plt.show()\n", "\n", "plot_confusion_matrix(y_train_lr, y_pred_train_lr_mod, 'Matrice di Confusione (Training)')\n", "\n", "plot_confusion_matrix(y_val_lr, y_pred_val_lr_mod, 'Matrice di Confusione (Validazione)')\n", "\n", "plot_confusion_matrix(y_test_lr, y_pred_test_lr_mod, 'Matrice di Confusione (Test)')\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Confronto diretto delle metriche

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
\nSet         Accuracy    F1-score    AUC-ROC     Precision   Recall      \n----------------------------------------------------------------------\nTraining    0.6386      0.3971      0.8354      0.2494      0.9730      \nValidation  0.6429      0.3977      0.8387      0.2498      0.9752      \nTest        0.6413      0.4049      0.8403      0.2557      0.9727      \nTrain 0.65  0.7477      0.4294      0.8354      0.2967      0.7764      \nVal 0.65    0.7500      0.4305      0.8387      0.2971      0.7816      \nTest 0.65   0.7526      0.4422      0.8403      0.3083      0.7817      \n\n
\n
\n
\n
\n
" ] } } ], "source": [ "results_lr = f\"\"\"\n", "{'Set':<12}{'Accuracy':<12}{'F1-score':<12}{'AUC-ROC':<12}{'Precision':<12}{'Recall':<12}\n", "{'-'*70}\n", "{'Training ':<12}{train_results_lr['Accuracy']:<12.4f}{train_results_lr['F1-score']:<12.4f}{train_results_lr['AUC-ROC']:<12.4f}{train_results_lr['Precision']:<12.4f}{train_results_lr['Recall']:<12.4f}\n", "{'Validation ':<12}{val_results_lr['Accuracy']:<12.4f}{val_results_lr['F1-score']:<12.4f}{val_results_lr['AUC-ROC']:<12.4f}{val_results_lr['Precision']:<12.4f}{val_results_lr['Recall']:<12.4f}\n", "{'Test ':<12}{test_results_lr['Accuracy']:<12.4f}{test_results_lr['F1-score']:<12.4f}{test_results_lr['AUC-ROC']:<12.4f}{test_results_lr['Precision']:<12.4f}{test_results_lr['Recall']:<12.4f}\n", "{'Train 0.65 ':<12}{train_results_lr_mod['Accuracy']:<12.4f}{train_results_lr_mod['F1-score']:<12.4f}{train_results_lr_mod['AUC-ROC']:<12.4f}{train_results_lr_mod['Precision']:<12.4f}{train_results_lr_mod['Recall']:<12.4f}\n", "{'Val 0.65 ':<12}{val_results_lr_mod['Accuracy']:<12.4f}{val_results_lr_mod['F1-score']:<12.4f}{val_results_lr_mod['AUC-ROC']:<12.4f}{val_results_lr_mod['Precision']:<12.4f}{val_results_lr_mod['Recall']:<12.4f}\n", "{'Test 0.65 ':<12}{test_results_lr_mod['Accuracy']:<12.4f}{test_results_lr_mod['F1-score']:<12.4f}{test_results_lr_mod['AUC-ROC']:<12.4f}{test_results_lr_mod['Precision']:<12.4f}{test_results_lr_mod['Recall']:<12.4f}\n", "\"\"\"\n", "print(results_lr)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Dopo varie prove ho impostato la soglia di decisione a 0.65, facendo così aumentano l'accuracy, F1-score e Precision. La Recall scende ma è ancora buona

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Confronto delle curve ROC per mostrare il trade-off tra il tasso di veri positivi e il tasso di falsi positivi

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "from sklearn.metrics import roc_curve, auc\n", "\n", "\n", " # Curva ROC soglia 0.5\n", "\n", "def plot_roc_curve(y_true, y_pred_proba, label, color):\n", " fpr, tpr, thresholds = roc_curve(y_true, y_pred_proba)\n", " roc_auc = auc(fpr, tpr)\n", " plt.plot(fpr, tpr, label=f'{label} (AUC = {roc_auc:.2f})', color=color)\n", " return fpr, tpr, thresholds\n", "\n", "plt.figure(figsize=(10, 6))\n", "\n", "# Curve ROC per i diversi set con soglia a 0.5\n", "fpr_train, tpr_train, thresholds_train = plot_roc_curve(y_train_lr, y_pred_proba_train_lr, 'Training', 'darkblue')\n", "fpr_val, tpr_val, thresholds_val = plot_roc_curve(y_val_lr, y_pred_proba_val_lr, 'Validation', 'darkorange')\n", "fpr_test, tpr_test, thresholds_test = plot_roc_curve(y_test_lr, y_pred_proba_test_lr, 'Test', 'purple')\n", "\n", "# Linea di riferimento (random classifier)\n", "plt.plot([0, 1], [0, 1], color='#C0C0C0', linestyle='--')\n", "\n", "# Funzione per trovare il punto più vicino a una data soglia\n", "def find_nearest(array, value):\n", " return (np.abs(array - value)).argmin()\n", "\n", "# Punti per le soglie specifiche\n", "soglie = [0.5, 0.65] # Aggiungo le soglie che ho usato\n", "colori = ['#A7C7E7', '#FFCC99'] \n", "\n", "for soglia, colore in zip(soglie, colori):\n", " # Trovo il punto più vicino alla soglia per ogni set\n", " idx_train = find_nearest(thresholds_train, soglia)\n", " idx_val = find_nearest(thresholds_val, soglia)\n", " idx_test = find_nearest(thresholds_test, soglia)\n", " \n", " # Plotto i punti\n", " plt.plot(fpr_test[idx_test], tpr_test[idx_test], 'o', color=colore, markersize=10, label=f'Soglia {soglia}')\n", "\n", "plt.xlabel('Falsi Positivi')\n", "plt.ylabel('Veri Positivi')\n", "plt.title('ROC Curve con Soglie')\n", "plt.legend(loc=\"lower right\")\n", "plt.show()\n", "\n", "\n", "\n", "\n", " # Curva ROC soglia 0.65\n", "\n", "def plot_roc_curve(y_true, y_pred_proba, label, color):\n", " fpr, tpr, thresholds = roc_curve(y_true, y_pred_proba)\n", " roc_auc = auc(fpr, tpr)\n", " plt.plot(fpr, tpr, label=f'{label} (AUC = {roc_auc:.2f})', color=color)\n", " return fpr, tpr, thresholds\n", "\n", "plt.figure(figsize=(10, 6))\n", "\n", "# Curve ROC per i diversi set con soglia a 0.65\n", "fpr_train, tpr_train, thresholds_train = plot_roc_curve(y_train_lr, y_pred_proba_train_lr_mod, 'Training', 'blue')\n", "fpr_val, tpr_val, thresholds_val = plot_roc_curve(y_val_lr, y_pred_proba_val_lr_mod, 'Validation', 'orange')\n", "fpr_test, tpr_test, thresholds_test = plot_roc_curve(y_test_lr, y_pred_proba_test_lr_mod, 'Test', 'green')\n", "\n", "# Linea di riferimento (random classifier)\n", "plt.plot([0, 1], [0, 1], color='#C0C0C0', linestyle='--')\n", "\n", "# Funzione per trovare il punto più vicino a una data soglia\n", "def find_nearest(array, value):\n", " return (np.abs(array - value)).argmin()\n", "\n", "# Punti per le soglie specifiche\n", "soglie = [0.5, 0.65] # Aggiungo le soglie che ho usato\n", "colori = ['#A7C7E7', '#FFCC99'] \n", "\n", "for soglia, colore in zip(soglie, colori):\n", " # Trovo il punto più vicino alla soglia per ogni set\n", " idx_train = find_nearest(thresholds_train, soglia)\n", " idx_val = find_nearest(thresholds_val, soglia)\n", " idx_test = find_nearest(thresholds_test, soglia)\n", " \n", " # Plotto i punti\n", " plt.plot(fpr_test[idx_test], tpr_test[idx_test], 'o', color=colore, markersize=10, label=f'Soglia {soglia}')\n", "\n", "plt.xlabel('Falsi Positivi')\n", "plt.ylabel('Veri Positivi')\n", "plt.title('ROC Curve con Soglie')\n", "plt.legend(loc=\"lower right\")\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Ho inglobato il codice delle due curve in un'unica cella in modo da poter confrontare meglio i plot

Essendo le curve molto simili, in un unico grafico si andrebbero a sovrapporre

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Analisi delle differenze nelle distribuzioni di probabilità

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "warnings.filterwarnings('ignore')\n", "\n", "plt.figure(figsize=(10, 6))\n", "sns.kdeplot(y_pred_proba_train_lr_mod, fill=True, color=\"blue\", label=\"Train\")\n", "sns.kdeplot(y_pred_proba_val_lr_mod, fill=True, color=\"orange\", label=\"Validation\")\n", "sns.kdeplot(y_pred_proba_test_lr_mod, fill=True, color=\"black\", label=\"Test\")\n", "\n", "plt.axvline(x=0.65, color='red', linestyle='--', label='Soglia 0.65')\n", "\n", "plt.xlabel('Probabilità predette')\n", "plt.ylabel('Density')\n", "plt.title('Distribuzione delle probabilità predette')\n", "plt.legend()\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

La distribuzione a tre picchi suggerisce che il modello sta discriminando tra diverse categorie di clienti, i poco propensi, i moderatamente propensi e altamente propensi.

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Scatter plot

Serve per visualizzare la distribuzione di probabilità, identificare i falsi positivi e negativi, valutare l'impatto delle soglie di decisione

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "def plot_probability_scatter(y_true, y_pred_proba):\n", " plt.figure(figsize=(10, 6))\n", " \n", " # Creo un array di indici\n", " indices = np.arange(len(y_true))\n", " \n", " # Scatter plot per la classe negativa (0)\n", " plt.scatter(indices[y_true == 0], y_pred_proba[y_true == 0], \n", " color='blue', alpha=0.5, label='Classe Negativa')\n", " \n", " # Scatter plot per la classe positiva (1)\n", " plt.scatter(indices[y_true == 1], y_pred_proba[y_true == 1], \n", " color='red', alpha=0.5, label='Classe Positiva')\n", " \n", " # Linea per la soglia\n", " plt.axhline(y=0.65, color='black', linestyle='--', label='Soglia 0.65')\n", " \n", " plt.title('Scatter Plot delle Probabilità Predette')\n", " plt.xlabel('Indice dell\\'istanza')\n", " plt.ylabel('Probabilità Predetta')\n", " plt.legend()\n", "\n", " plt.show()\n", "\n", "plot_probability_scatter(y_test_lr, y_pred_proba_test_lr_mod)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

TP = punti rossi sopra la soglia, TN = punti blu sotto la soglia, FP = punti blu sopra la soglia, FN = punti rossi sotto la soglia

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Confronto delle metriche per diverse soglie di decisione

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "from sklearn.metrics import precision_recall_curve\n", "\n", "def plot_precision_recall_vs_threshold(y_true, y_pred, test_name):\n", " precisions, recalls, thresholds = precision_recall_curve(y_true, y_pred)\n", " \n", " plt.figure(figsize=(10, 6))\n", " plt.plot(thresholds, precisions[:-1], \"blue\", label=\"Precision\")\n", " plt.plot(thresholds, recalls[:-1], \"orange\", label=\"Recall\")\n", " plt.xlabel(\"Threshold\")\n", " plt.legend(loc=\"upper left\")\n", " plt.ylim([0, 1])\n", " plt.title(f\"Precision and Recall vs. Threshold ({test_name})\")\n", "\n", "plot_precision_recall_vs_threshold(y_train_lr, y_pred_proba_train_lr_mod, \"Train\")\n", "plot_precision_recall_vs_threshold(y_val_lr, y_pred_proba_val_lr_mod, \"Validation\")\n", "plot_precision_recall_vs_threshold(y_test_lr, y_pred_proba_test_lr_mod, \"Test\")\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Analizzo gli errori

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Training - Numero di errori: 67849 su 268871 esempi (25.23%)\nValidation - Numero di errori: 13766 su 55071 esempi (25.00%)\nTest - Numero di errori: 14143 su 57167 esempi (24.74%)\n
\n
\n
\n
\n
" ] } } ], "source": [ "def analyze_errors(y_true, y_pred, test_name):\n", " errors = y_true != y_pred\n", " error_rate = errors.sum() / len(y_true)\n", " print(f\"{test_name} - Numero di errori: {errors.sum()} su {len(y_true)} esempi ({error_rate:.2%})\")\n", " return errors\n", "\n", "train_errors_lr = analyze_errors(y_train_lr, y_pred_train_lr_mod, \"Training\")\n", "val_errors_lr = analyze_errors(y_val_lr, y_pred_val_lr_mod, \"Validation\")\n", "test_errors_lr = analyze_errors(y_test_lr, y_pred_test_lr_mod, \"Test\")\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
\nTraining Error Analysis:\nFalse Positives: 60496 (25.63% of actual negatives)\nFalse Negatives: 7353 (22.36% of actual positives)\n\nValidation Error Analysis:\nFalse Positives: 12312 (25.43% of actual negatives)\nFalse Negatives: 1454 (21.84% of actual positives)\n\nTest Error Analysis:\nFalse Positives: 12577 (25.16% of actual negatives)\nFalse Negatives: 1566 (21.83% of actual positives)\n
\n
\n
\n
\n
" ] } } ], "source": [ "def analyze_error_types(y_true, y_pred, test_name):\n", " cm = confusion_matrix(y_true, y_pred)\n", " tn, fp, fn, tp = cm.ravel()\n", " print(f\"\\n{test_name} Error Analysis:\")\n", " print(f\"False Positives: {fp} ({fp/(fp+tn):.2%} of actual negatives)\")\n", " print(f\"False Negatives: {fn} ({fn/(fn+tp):.2%} of actual positives)\")\n", "\n", "analyze_error_types(y_train_lr, y_pred_train_lr_mod, \"Training\")\n", "analyze_error_types(y_val_lr, y_pred_val_lr_mod, \"Validation\")\n", "analyze_error_types(y_test_lr, y_pred_test_lr_mod, \"Test\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

I tassi di errore sono simili in tutti e tre i set, quindi il modello generalizza bene, però il numero di falsi positivi è troppo grande

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Bar plot

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "plt.figure(figsize=(10, 6))\n", "bars = plt.bar(['Training', 'Validation', 'Test'], \n", " [train_errors_lr.mean(), val_errors_lr.mean(), test_errors_lr.mean()],\n", " yerr=[train_errors_lr.std(), val_errors_lr.std(), test_errors_lr.std()],\n", " capsize=5, color=['#A7C7E7', '#FFCC99', '#C0C0C0'])\n", "\n", "plt.title('Confronto del Tasso di Errore tra i test')\n", "plt.ylabel('Tasso di Errore')\n", "\n", "# Aggiungo etichette con i valori sopra ciascuna barra\n", "for i, bar in enumerate(bars):\n", " height = bar.get_height()\n", " plt.text(bar.get_x() + bar.get_width()/2., height,\n", " f'{height:.4f}',\n", " ha='center', va='bottom')\n", "\n", "\n", "\n", "plt.tight_layout() # Assicura che tutto si adatti bene nella figura\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Intervallo di confidenza

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Accuratezza: 0.7526\nIntervallo di confidenza al 95%: (0.7490648538668198, 0.7561391974215109)\n
\n
\n
\n
\n
" ] } } ], "source": [ "from scipy.stats import norm\n", "\n", "accuracy_lr = accuracy_score(y_test_lr, y_pred_test_lr_mod)\n", "n_lr = len(y_test_lr)\n", "std_error_lr = np.sqrt(accuracy_lr * (1 - accuracy_lr) / n_lr)\n", "confidence_interval_lr = norm.interval(0.95, loc=accuracy_lr, scale=std_error_lr)\n", "\n", "print(f\"Accuratezza: {accuracy_lr:.4f}\")\n", "print(f\"Intervallo di confidenza al 95%: {confidence_interval_lr}\")\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "accuracy_lr = 0.7526\n", "confidence_interval_lr = (0.7490648538668198, 0.7561391974215109) \n", "\n", "plt.figure(figsize=(10, 6))\n", "\n", "# Calcolo l'errore come la differenza tra il valore di accuratezza e i limiti dell'intervallo di confidenza\n", "yerr = [[accuracy_lr - confidence_interval_lr[0]], [confidence_interval_lr[1] - accuracy_lr]]\n", "\n", "plt.bar(['Accuratezza'], [accuracy_lr], yerr=yerr, capsize=5)\n", "\n", "plt.title('Accuratezza con Intervallo di Confidenza al 95%')\n", "plt.ylabel('Accuratezza')\n", "plt.ylim(0, 1) # Assumendo che l'accuratezza sia tra 0 e 1\n", "\n", "# Aggiungo l'etichetta con il valore esatto sopra la barra\n", "plt.text(0, accuracy_lr, f'{accuracy_lr:.4f}', ha='center', va='bottom')\n", "\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

L'accuratezza del modello si trova tra il 74.90% e il 75.61%. Essendo l'intervallo stretto la stima è abbastanza precisa

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Importanza delle feature per determinare il contributo di ogni feature alla preformance del modello

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
\nFeature Importance:\n                  feature  importance\n10     Vehicle_Damage_Yes    1.996013\n5      Previously_Insured    1.985682\n8    Vehicle_Age_< 1 Year    1.136031\n0                     Age    0.390421\n9   Vehicle_Age_> 2 Years    0.195323\n6    Policy_Sales_Channel    0.123041\n7             Gender_Male    0.108362\n3         Driving_License    0.056416\n1          Annual_Premium    0.034451\n2                 Vintage    0.004401\n4             Region_Code    0.001927\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "feature_names_lr = (numeric_features + \n", " pipeline_lr.named_steps['preprocessor']\n", " .named_transformers_['cat']\n", " .get_feature_names_out(categorical_features).tolist())\n", "\n", "feature_importance_lr = pd.DataFrame({\n", " 'feature': feature_names_lr,\n", " 'importance': abs(pipeline_lr.named_steps['classifier'].coef_[0])\n", "}).sort_values('importance', ascending=False)\n", "\n", "print(\"\\nFeature Importance:\")\n", "print(feature_importance_lr)\n", "\n", "N = 10 # Numero di top feature da visualizzare\n", "plt.figure(figsize=(10, 6))\n", "sns.barplot(x='importance', y='feature', data=feature_importance_lr.head(N))\n", "plt.title(f'Top {N} Feature Importances (Logistic Regression 0.65)')\n", "plt.tight_layout()\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Decision boundary

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "# Le feature che voglio visualizzare\n", "feature1 = 'Annual_Premium'\n", "feature2 = 'Vintage'\n", "\n", "# Griglia di punti\n", "x0, x1 = X[feature1], X[feature2]\n", "xx, yy = np.meshgrid(np.linspace(x0.min()-1, x0.max()+1, 100),\n", " np.linspace(x1.min()-1, x1.max()+1, 100))\n", "\n", "# Preparo i dati per la predizione\n", "X_plot = np.c_[xx.ravel(), yy.ravel()]\n", "\n", "# Creo un DataFrame con le stesse colonne di X originale\n", "X_plot_df_lr = pd.DataFrame(X_plot, columns=[feature1, feature2])\n", "for col in X.columns:\n", " if col not in X_plot_df_lr.columns:\n", " X_plot_df_lr[col] = X[col].mode()[0] # Uso il valore più frequente per le altre feature\n", "\n", "# Predizioni\n", "Z = pipeline_lr.predict_proba(X_plot_df_lr)[:, 1]\n", "Z = Z.reshape(xx.shape)\n", "\n", "plt.figure(figsize=(10, 6))\n", "plt.contourf(xx, yy, Z, alpha=0.4, cmap='RdYlBu')\n", "scatter = plt.scatter(X[feature1], X[feature2], c=y, alpha=0.8, cmap='RdYlBu')\n", "plt.xlabel(feature1)\n", "plt.ylabel(feature2)\n", "plt.title(f'Decision Boundary - {feature1} vs {feature2}')\n", "plt.colorbar(scatter)\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

La maggior parte dei clienti ha un premio annuale inferiore a 100.000

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Secondo modello - Random Forest

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "X_train_rf, X_test_rf, y_train_rf, y_test_rf = train_test_split(X, y, test_size=0.15, random_state=RANDOM_SEED)\n", "X_train_rf, X_val_rf, y_train_rf, y_val_rf = train_test_split(X_train_rf, y_train_rf, test_size=0.17, random_state=RANDOM_SEED)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pipeline_rf = Pipeline([\n", " ('preprocessor', preprocessor), # Usa il preprocessor esistente\n", " ('classifier', RandomForestClassifier(class_weight='balanced', random_state=RANDOM_SEED, n_jobs=-1))\n", "])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Addestro il modello

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
Out[85]:
\n
\n
Pipeline(steps=[('preprocessor',\n                 ColumnTransformer(transformers=[('num', StandardScaler(),\n                                                  ['Age', 'Annual_Premium',\n                                                   'Vintage', 'Driving_License',\n                                                   'Region_Code',\n                                                   'Previously_Insured',\n                                                   'Policy_Sales_Channel']),\n                                                 ('cat',\n                                                  OneHotEncoder(drop='first',\n                                                                handle_unknown='ignore',\n                                                                sparse_output=False),\n                                                  ['Gender', 'Vehicle_Age',\n                                                   'Vehicle_Damage'])])),\n                ('classifier',\n                 RandomForestClassifier(class_weight='balanced', n_jobs=-1,\n                                        random_state=42))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
\n
\n
\n
\n
" ] } } ], "source": [ "pipeline_rf.fit(X_train_rf, y_train_rf)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Predizioni sul set di training

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Performance sul set di training:\n\nEvaluation for Random Forest (Training):\nAccuracy: 0.9998\nPrecision: 0.9990\nRecall: 0.9997\nF1-score: 0.9993\nAUC-ROC: 1.0000\n\nConfusion Matrix:\n[[235959     32]\n [    11  32869]]\n\nClassification Report:\n              precision    recall  f1-score   support\n\n           0       1.00      1.00      1.00    235991\n           1       1.00      1.00      1.00     32880\n\n    accuracy                           1.00    268871\n   macro avg       1.00      1.00      1.00    268871\nweighted avg       1.00      1.00      1.00    268871\n\n
\n
\n
\n
\n
" ] } } ], "source": [ "y_pred_train_rf = pipeline_rf.predict(X_train_rf)\n", "y_pred_proba_train_rf = pipeline_rf.predict_proba(X_train_rf)[:, 1]\n", "\n", "print(\"Performance sul set di training:\")\n", "train_results_rf = evaluate_model(y_train_rf, y_pred_train_rf, y_pred_proba_train_rf, \"Random Forest (Training)\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Predizioni sul set di validazione

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
\nPerformance sul set di validazione:\n\nEvaluation for Random Forest (Validation):\nAccuracy: 0.8704\nPrecision: 0.3713\nRecall: 0.1042\nF1-score: 0.1628\nAUC-ROC: 0.8337\n\nConfusion Matrix:\n[[47238  1175]\n [ 5964   694]]\n\nClassification Report:\n              precision    recall  f1-score   support\n\n           0       0.89      0.98      0.93     48413\n           1       0.37      0.10      0.16      6658\n\n    accuracy                           0.87     55071\n   macro avg       0.63      0.54      0.55     55071\nweighted avg       0.83      0.87      0.84     55071\n\n
\n
\n
\n
\n
" ] } } ], "source": [ "y_pred_val_rf = pipeline_rf.predict(X_val_rf)\n", "y_pred_proba_val_rf = pipeline_rf.predict_proba(X_val_rf)[:, 1]\n", "\n", "print(\"\\nPerformance sul set di validazione:\")\n", "val_results_rf = evaluate_model(y_val_rf, y_pred_val_rf, y_pred_proba_val_rf, \"Random Forest (Validation)\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Faccio previsioni sul set di test per valutare le performance finali del modello dopo che è stato addestrato

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Random Forest Regression:\n\nEvaluation for Random Forest Regression:\nAccuracy: 0.8672\nPrecision: 0.3903\nRecall: 0.1039\nF1-score: 0.1641\nAUC-ROC: 0.8377\n\nConfusion Matrix:\n[[48831  1164]\n [ 6427   745]]\n\nClassification Report:\n              precision    recall  f1-score   support\n\n           0       0.88      0.98      0.93     49995\n           1       0.39      0.10      0.16      7172\n\n    accuracy                           0.87     57167\n   macro avg       0.64      0.54      0.55     57167\nweighted avg       0.82      0.87      0.83     57167\n\n
\n
\n
\n
\n
" ] } } ], "source": [ "y_pred_test_rf = pipeline_rf.predict(X_test_rf)\n", "y_pred_proba_test_rf = pipeline_rf.predict_proba(X_test_rf)[:, 1]\n", "\n", "print(\"Random Forest Regression:\")\n", "test_results_rf = evaluate_model(y_test_rf, y_pred_test_rf, y_pred_proba_test_rf, \"Random Forest Regression\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Confronto diretto delle metriche

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
\nRiepilogo delle metriche principali:\nSet       Accuracy    F1-score    AUC-ROC     Precision   Recall      \n----------------------------------------------------------------------\nTraining  0.9998      0.9993      1.0000      0.9990      0.9997      \nValidation0.8704      0.1628      0.8337      0.3713      0.1042      \nTest      0.8672      0.1641      0.8377      0.3903      0.1039      \n\n
\n
\n
\n
\n
" ] } } ], "source": [ "results_rf = f\"\"\"\n", "Riepilogo delle metriche principali:\n", "{'Set':<10}{'Accuracy':<12}{'F1-score':<12}{'AUC-ROC':<12}{'Precision':<12}{'Recall':<12}\n", "{'-'*70}\n", "{'Training':<10}{train_results_rf['Accuracy']:<12.4f}{train_results_rf['F1-score']:<12.4f}{train_results_rf['AUC-ROC']:<12.4f}{train_results_rf['Precision']:<12.4f}{train_results_rf['Recall']:<12.4f}\n", "{'Validation':<10}{val_results_rf['Accuracy']:<12.4f}{val_results_rf['F1-score']:<12.4f}{val_results_rf['AUC-ROC']:<12.4f}{val_results_rf['Precision']:<12.4f}{val_results_rf['Recall']:<12.4f}\n", "{'Test':<10}{test_results_rf['Accuracy']:<12.4f}{test_results_rf['F1-score']:<12.4f}{test_results_rf['AUC-ROC']:<12.4f}{test_results_rf['Precision']:<12.4f}{test_results_rf['Recall']:<12.4f}\n", "\"\"\"\n", "print(results_rf)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Nel set di training abbiamo delle prestazioni quasi perfette, nel test di validazione e test invece abbiamo prestazioni peggiori, indice di overfitting

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Visualizzazione delle matrici di confusione

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "plot_confusion_matrix(y_train_rf, y_pred_train_rf, 'Matrice di Confusione (Training)')\n", "\n", "plot_confusion_matrix(y_val_rf, y_pred_val_rf, 'Matrice di Confusione (Validazione)')\n", "\n", "plot_confusion_matrix(y_test_rf, y_pred_test_rf, 'Matrice di Confusione (Test)')\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Confronto delle curve ROC

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "plt.figure(figsize=(10, 6))\n", "\n", "# Curve ROC per i diversi set\n", "fpr_train, tpr_train, thresholds_train = plot_roc_curve(y_train_rf, y_pred_proba_train_rf, 'Training', 'darkblue')\n", "fpr_val, tpr_val, thresholds_val = plot_roc_curve(y_val_rf, y_pred_proba_val_rf, 'Validation', 'darkorange')\n", "fpr_test, tpr_test, thresholds_test = plot_roc_curve(y_test_rf, y_pred_proba_test_rf, 'Test', 'purple')\n", "\n", "# Linea di riferimento (random classifier)\n", "plt.plot([0, 1], [0, 1], color='#C0C0C0', linestyle='--')\n", "\n", "# Funzione per trovare il punto più vicino a una data soglia\n", "def find_nearest(array, value):\n", " return (np.abs(array - value)).argmin()\n", "\n", " \n", " # Plotto i punti\n", " plt.plot(fpr_test[idx_test], tpr_test[idx_test], 'o', color=colore, markersize=10, label=f'Soglia {soglia}')\n", "\n", "plt.xlabel('Falsi Positivi')\n", "plt.ylabel('Veri Positivi')\n", "plt.title('ROC Curve con Soglie')\n", "plt.legend(loc=\"lower right\")\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Analisi delle differenze nelle distribuzioni di probabilità

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "warnings.filterwarnings('ignore')\n", "\n", "plt.figure(figsize=(10, 6))\n", "sns.kdeplot(y_pred_proba_train_rf, fill=True, color=\"blue\", label=\"Train\")\n", "sns.kdeplot(y_pred_proba_val_rf, fill=True, color=\"orange\", label=\"Validation\")\n", "sns.kdeplot(y_pred_proba_test_rf, fill=True, color=\"black\", label=\"Test\")\n", "\n", "plt.xlabel('Predicted Probability')\n", "plt.ylabel('Density')\n", "plt.title('Distribution of Predicted Probabilities')\n", "plt.legend()\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Confronto delle metriche per diverse soglie di decisione

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "from sklearn.metrics import precision_recall_curve\n", "\n", "def plot_precision_recall_vs_threshold(y_true, y_scores, test_name):\n", " precisions, recalls, thresholds = precision_recall_curve(y_true, y_scores)\n", " \n", " plt.figure(figsize=(10, 6))\n", " plt.plot(thresholds, precisions[:-1], \"b--\", label=\"Precision\")\n", " plt.plot(thresholds, recalls[:-1], \"g-\", label=\"Recall\")\n", " plt.xlabel(\"Threshold\")\n", " plt.legend(loc=\"upper left\")\n", " plt.ylim([0, 1])\n", " plt.title(f\"Precision and Recall vs. Threshold ({test_name})\")\n", "\n", "plot_precision_recall_vs_threshold(y_train_rf, y_pred_proba_train_rf, \"Train\")\n", "plot_precision_recall_vs_threshold(y_val_rf, y_pred_proba_val_rf, \"Validation\")\n", "plot_precision_recall_vs_threshold(y_test_rf, y_pred_proba_test_rf, \"Test\")\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Analizzo gli errori

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Training - Numero di errori: 43 su 268871 esempi (0.02%)\nValidation - Numero di errori: 7139 su 55071 esempi (12.96%)\nTest - Numero di errori: 7591 su 57167 esempi (13.28%)\n
\n
\n
\n
\n
" ] } } ], "source": [ "def analyze_errors(y_true, y_pred, test_name):\n", " errors = y_true != y_pred\n", " error_rate = errors.sum() / len(y_true)\n", " print(f\"{test_name} - Numero di errori: {errors.sum()} su {len(y_true)} esempi ({error_rate:.2%})\")\n", " return errors\n", "\n", "train_errors_rf = analyze_errors(y_train_rf, y_pred_train_rf, \"Training\")\n", "val_errors_rf = analyze_errors(y_val_rf, y_pred_val_rf, \"Validation\")\n", "test_errors_rf = analyze_errors(y_test_rf, y_pred_test_rf, \"Test\")\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
\nTraining Error Analysis:\nFalse Positives: 32 (0.01% of actual negatives)\nFalse Negatives: 11 (0.03% of actual positives)\n\nValidation Error Analysis:\nFalse Positives: 1175 (2.43% of actual negatives)\nFalse Negatives: 5964 (89.58% of actual positives)\n\nTest Error Analysis:\nFalse Positives: 1164 (2.33% of actual negatives)\nFalse Negatives: 6427 (89.61% of actual positives)\n
\n
\n
\n
\n
" ] } } ], "source": [ "def analyze_error_types(y_true, y_pred, test_name):\n", " cm = confusion_matrix(y_true, y_pred)\n", " tn, fp, fn, tp = cm.ravel()\n", " print(f\"\\n{test_name} Error Analysis:\")\n", " print(f\"False Positives: {fp} ({fp/(fp+tn):.2%} of actual negatives)\")\n", " print(f\"False Negatives: {fn} ({fn/(fn+tp):.2%} of actual positives)\")\n", "\n", "analyze_error_types(y_train_rf, y_pred_train_rf, \"Training\")\n", "analyze_error_types(y_val_rf, y_pred_val_rf, \"Validation\")\n", "analyze_error_types(y_test_rf, y_pred_test_rf, \"Test\")\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "plt.figure(figsize=(10, 6))\n", "bars = plt.bar(['Training', 'Validation', 'Test'], \n", " [train_errors_rf.mean(), val_errors_rf.mean(), test_errors_rf.mean()],\n", " yerr=[train_errors_rf.std(), val_errors_rf.std(), test_errors_rf.std()],\n", " capsize=5, color=['#A7C7E7', '#FFCC99', '#C0C0C0'])\n", "\n", "plt.title('Confronto del Tasso di Errore tra i test')\n", "plt.ylabel('Tasso di Errore')\n", "\n", "# Aggiungoe etichette con i valori sopra ciascuna barra\n", "for i, bar in enumerate(bars):\n", " height = bar.get_height()\n", " plt.text(bar.get_x() + bar.get_width()/2., height,\n", " f'{height:.4f}',\n", " ha='center', va='bottom')\n", "\n", "plt.tight_layout()\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Intervallo di confidenza

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Accuratezza: 0.8672\nIntervallo di confidenza al 95%: (0.8644318706448486, 0.8699953338612476)\n
\n
\n
\n
\n
" ] } } ], "source": [ "accuracy_rf = accuracy_score(y_test_rf, y_pred_test_rf)\n", "n_rf = len(y_test_rf)\n", "std_error_rf = np.sqrt(accuracy_rf * (1 - accuracy_rf) / n_rf)\n", "confidence_interval_rf = norm.interval(0.95, loc=accuracy_rf, scale=std_error_rf)\n", "\n", "print(f\"Accuratezza: {accuracy_rf:.4f}\")\n", "print(f\"Intervallo di confidenza al 95%: {confidence_interval_rf}\")\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "plt.figure(figsize=(10, 6))\n", "\n", "# Calcolo l'errore come la differenza tra il valore di accuratezza e i limiti dell'intervallo di confidenza\n", "yerr_rf = [[accuracy_rf - confidence_interval_rf[0]], [confidence_interval_rf[1] - accuracy_rf]]\n", "\n", "plt.bar(['Accuratezza'], [accuracy_rf], yerr=yerr, capsize=5)\n", "\n", "plt.title('Accuratezza con Intervallo di Confidenza al 95%')\n", "plt.ylabel('Accuratezza')\n", "plt.ylim(0, 1) # Assumendo che l'accuratezza sia tra 0 e 1\n", "\n", "# Aggiungo l'etichetta con il valore esatto sopra la barra\n", "plt.text(0, accuracy_rf, f'{accuracy_rf:.4f}', ha='center', va='bottom')\n", "\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Importanza delle feature

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
\nFeature Importance:\n                  feature  importance\n10     Vehicle_Damage_Yes    0.199428\n2                 Vintage    0.190459\n1          Annual_Premium    0.166639\n5      Previously_Insured    0.136071\n0                     Age    0.116371\n4             Region_Code    0.085140\n6    Policy_Sales_Channel    0.066108\n8    Vehicle_Age_< 1 Year    0.026329\n7             Gender_Male    0.007918\n9   Vehicle_Age_> 2 Years    0.004882\n3         Driving_License    0.000655\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
" ] } } ], "source": [ "feature_names_rf = (numeric_features + \n", " pipeline_rf.named_steps['preprocessor']\n", " .named_transformers_['cat']\n", " .get_feature_names_out(categorical_features).tolist())\n", "\n", "feature_importance_rf = pd.DataFrame({\n", " 'feature': feature_names_rf,\n", " 'importance': pipeline_rf.named_steps['classifier'].feature_importances_\n", "}).sort_values('importance', ascending=False)\n", "\n", "print(\"\\nFeature Importance:\")\n", "print(feature_importance_rf)\n", "\n", "\n", "N = 10 # Numero di top feature da visualizzare\n", "plt.figure(figsize=(10, 6))\n", "sns.barplot(x='importance', y='feature', data=feature_importance_rf.head(N))\n", "plt.title(f'Top {N} Feature Importances (Random Forest)')\n", "plt.tight_layout()\n", "plt.show()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Modelli SMOTE e Undersampling

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "from sklearn.model_selection import train_test_split\n", "from imblearn.over_sampling import SMOTE\n", "from imblearn.under_sampling import RandomUnderSampler\n", "\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15, random_state=RANDOM_SEED)\n", "X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.17, random_state=RANDOM_SEED)\n", "\n", "def evaluate_model_due(y_true, y_pred, y_pred_proba):\n", " return {\n", " 'Accuracy': accuracy_score(y_true, y_pred),\n", " 'Precision': precision_score(y_true, y_pred),\n", " 'Recall': recall_score(y_true, y_pred),\n", " 'F1-score': f1_score(y_true, y_pred),\n", " 'AUC-ROC': roc_auc_score(y_true, y_pred_proba)\n", " }\n", "\n", "def create_and_evaluate_models(X_train, y_train, X_val, y_val, X_test, y_test, model_class, model_name):\n", " base_model = model_class(random_state=RANDOM_SEED) # Istanza del modello base\n", " # base_model è il parametro che verrà passato\n", " # alle funzioni LogisticRegression e RandomForestClassifier\n", " # durante la creazione e valutazione dei modelli successivamente\n", " \n", " # Creo i modelli\n", " smote = ImbPipeline([\n", " ('preprocessor', preprocessor),\n", " ('smote', SMOTE(random_state=RANDOM_SEED)), # Bilanciamento\n", " ('classifier', base_model) # Classificatore\n", " ])\n", " \n", " under = ImbPipeline([\n", " ('preprocessor', preprocessor),\n", " ('undersampler', RandomUnderSampler(random_state=RANDOM_SEED)),\n", " ('classifier', base_model)\n", " ])\n", "\n", " # Dizionario \n", " models = {\n", " f'{model_name} SMOTE': smote,\n", " f'{model_name} Undersampling': under\n", " }\n", "\n", " # Inizializzazione del dizionario vuoto\n", " results = {}\n", " \n", " # Iterazione su ciascun modello del dizionario\n", " for name, model in models.items():\n", " model.fit(X_train, y_train) # Addestramento\n", " \n", " y_train_pred = model.predict(X_train) # Previsioni e calcolo delle probabilità\n", " y_train_pred_proba = model.predict_proba(X_train)[:, 1]\n", " train_results = evaluate_model_due(y_train, y_train_pred, y_train_pred_proba) # Valutazione delle prestazioni\n", " \n", " y_val_pred = model.predict(X_val)\n", " y_val_pred_proba = model.predict_proba(X_val)[:, 1]\n", " val_results = evaluate_model_due(y_val, y_val_pred, y_val_pred_proba)\n", " \n", " y_test_pred = model.predict(X_test)\n", " y_test_pred_proba = model.predict_proba(X_test)[:, 1]\n", " test_results = evaluate_model_due(y_test, y_test_pred, y_test_pred_proba)\n", "\n", " # Memorizzazione dei risultati nel dizionario\n", " results[name] = {\n", " 'model': model,\n", " 'training': train_results,\n", " 'validation': val_results,\n", " 'test': test_results\n", " }\n", " \n", "return results\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Creo un dizionario con i risultati per Logistic Regression 0.65 e Random Forest Standard che poi verranno\n", "# usati per fare le comparazioni\n", "results_lr_065 = {\n", " 'Accuracy': test_results_lr_mod['Accuracy'],\n", " 'F1-score': test_results_lr_mod['F1-score'],\n", " 'AUC-ROC': test_results_lr_mod['AUC-ROC'],\n", " 'Precision': test_results_lr_mod['Precision'],\n", " 'Recall': test_results_lr_mod['Recall']\n", "}\n", "\n", "\n", "results_rf = {\n", " 'Accuracy': test_results_rf['Accuracy'],\n", " 'F1-score': test_results_rf['F1-score'],\n", " 'AUC-ROC': test_results_rf['AUC-ROC'],\n", " 'Precision': test_results_rf['Precision'],\n", " 'Recall': test_results_rf['Recall']\n", "}\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Creazione e valutazione dei modelli di Regressione Logistica e Random Forest\n", "lr_results = create_and_evaluate_models(X_train, y_train, X_val, y_val, X_test, y_test, \n", " LogisticRegression, \"Logistic Regression\")\n", "\n", "rf_results = create_and_evaluate_models(X_train, y_train, X_val, y_val, X_test, y_test, \n", " RandomForestClassifier, \"Random Forest\")\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Combino i risultati\n", "all_results = {**lr_results, **rf_results}\n", "\n", "comparison_train = pd.DataFrame({name: results['training'] for name, results in all_results.items()}).transpose()\n", "\n", "# Creo il confronto per il set di validazione\n", "comparison_val = pd.DataFrame({name: results['validation'] for name, results in all_results.items()}).transpose()\n", "\n", "# Creo il DataFrame per i risultati del test\n", "comparison_test = pd.DataFrame({name: results['test'] for name, results in all_results.items()}).transpose()\n", "\n", "# Aggiungo le righe di summary per Random Forest e Logistic Regression 0.65\n", "comparison_test.loc['Logistic Regression 0.65'] = pd.Series(results_lr_065)\n", "comparison_test.loc['Random Forest Standard'] = pd.Series(results_rf)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n
Risultati sul set di training:\n                                   Accuracy  Precision    Recall  F1-score  \\\nLogistic Regression SMOTE          0.638968   0.249483  0.972111  0.397063   \nLogistic Regression Undersampling  0.638503   0.249404  0.973388  0.397070   \nRandom Forest SMOTE                0.999874   0.999483  0.999483  0.999483   \nRandom Forest Undersampling        0.763303   0.340641  0.999909  0.508165   \n\n                                    AUC-ROC  \nLogistic Regression SMOTE          0.835329  \nLogistic Regression Undersampling  0.835420  \nRandom Forest SMOTE                1.000000  \nRandom Forest Undersampling        0.967654  \n\n\nRisultati sul set di validazione:\n                                   Accuracy  Precision    Recall  F1-score  \\\nLogistic Regression SMOTE          0.643188   0.249807  0.974166  0.397646   \nLogistic Regression Undersampling  0.642643   0.249712  0.975668  0.397649   \nRandom Forest SMOTE                0.836847   0.327040  0.330430  0.328726   \nRandom Forest Undersampling        0.712789   0.280117  0.876239  0.424522   \n\n                                    AUC-ROC  \nLogistic Regression SMOTE          0.838584  \nLogistic Regression Undersampling  0.838833  \nRandom Forest SMOTE                0.833170  \nRandom Forest Undersampling        0.838147  \n\n\nRisultati sul set di test di tutti i modelli esaminati:\n                                   Accuracy  Precision    Recall  F1-score  \\\nLogistic Regression SMOTE          0.641646   0.255724  0.971695  0.404892   \nLogistic Regression Undersampling  0.641227   0.255661  0.972950  0.404921   \nRandom Forest SMOTE                0.835902   0.343843  0.339096  0.341453   \nRandom Forest Undersampling        0.714573   0.290127  0.881344  0.436548   \nLogistic Regression 0.65           0.752602   0.308310  0.781651  0.442201   \nRandom Forest Standard             0.867214   0.390257  0.103876  0.164079   \n\n                                    AUC-ROC  \nLogistic Regression SMOTE          0.840149  \nLogistic Regression Undersampling  0.840037  \nRandom Forest SMOTE                0.835023  \nRandom Forest Undersampling        0.841565  \nLogistic Regression 0.65           0.840257  \nRandom Forest Standard             0.837689  \n
\n
\n
\n
\n
" ] } } ], "source": [ "from termcolor import colored\n", "\n", "train_text = \"Risultati sul set di training:\"\n", "val_text = \"Risultati sul set di validazione:\"\n", "test_text = \"Risultati sul set di test di tutti i modelli esaminati:\"\n", "\n", "def print_colored_title(text):\n", " light_green = '\\033[38;2;000;204;000m' \n", " bold = '\\033[1m'\n", " reset = '\\033[0m'\n", " print(f\"{light_green}{bold}{text}{reset}\")\n", "\n", "\n", "print_colored_title(train_text)\n", "print(comparison_train)\n", "print(\"\\n\") \n", "\n", "print_colored_title(val_text)\n", "print(comparison_val)\n", "print(\"\\n\") \n", "\n", "print_colored_title(test_text)\n", "print(comparison_test)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Logistic Regression con SMOTE e Undersampling hanno la Recall alta, identificano un numero elevato di casi positivi ma inglobano anche i falsi positivi.

Random Forest Undersampling ha una buona Recall ma una bassa Precision inglobando falsi positivi.

Random Forest SMOTE ha una bassa Recall, perde molti veri positivi.

Random Forest Standard ha una Recall bassissima, perde molti veri positivi.

Logistic Regression 0.65 ha una buona Recall, una buona Accuracy e F1-score tra le più alte

\n
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

ROC Curve

\n
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "output_type": "display_data", "metadata": {}, "data": { "text/html": [ "
\n
\n
\n
\n
\n
\n
\n\"No\n
\n
\n
\n
\n
\n
\nValori AUC-ROC:\nLogistic Regression SMOTE: 0.8400\nLogistic Regression Undersampling: 0.8400\nRandom Forest SMOTE: 0.8416\nRandom Forest Undersampling: 0.8416\nRandom Forest Standard: 0.8377\nLogistic Regression 0.65: 0.8403\n
\n
\n
\n
\n
" ] } } ], "source": [ "def plot_roc_curve(y_true, y_pred_proba, label, color):\n", " fpr, tpr, thresholds = roc_curve(y_true, y_pred_proba)\n", " roc_auc = auc(fpr, tpr)\n", " plt.plot(fpr, tpr, label=f'{label} (AUC = {roc_auc:.2f})', color=color)\n", " return fpr, tpr, thresholds\n", "\n", "plt.figure(figsize=(10, 6))\n", "\n", "# Curve ROC per i modelli SMOTE e Undersampling\n", "for name, results in all_results.items():\n", " try:\n", " model = results['model']\n", " y_pred_proba = model.predict_proba(X_test)[:, 1]\n", " plot_roc_curve(y_test, y_pred_proba, name, None) # None per colore casuale\n", " except Exception as e:\n", " pass # Silenziosamente ignora eventuali errori\n", "\n", "# Curve ROC per RF standard e LR standard (solo test)\n", "fpr_rf, tpr_rf, _ = plot_roc_curve(y_test_rf, y_pred_proba_test_rf, 'Random Forest Standard (Test)', 'purple')\n", "fpr_lr, tpr_lr, _ = plot_roc_curve(y_test_lr, y_pred_proba_test_lr, 'Logistic Regression 0.65 (Test)', 'darkgreen')\n", "\n", "# Linea di riferimento (random classifier)\n", "plt.plot([0, 1], [0, 1], color='#C0C0C0', linestyle='--', label='Random Chance')\n", "\n", "# Funzione per trovare il punto più vicino a una data soglia\n", "def find_nearest(array, value):\n", " return (np.abs(array - value)).argmin()\n", "\n", "# Aggiungo il punto per la soglia specifica (solo per LR standard)\n", "soglie = [0.65]\n", "colori = ['#A7C7E7', '#FFCC99']\n", "for soglia, colore in zip(soglie, colori):\n", " idx_lr = find_nearest(_, soglia) # _ contiene i thresholds dell'ultima chiamata a plot_roc_curve (LR)\n", " plt.plot(fpr_lr[idx_lr], tpr_lr[idx_lr], 'o', color=colore, markersize=10, label=f'LR Soglia {soglia}')\n", "\n", "plt.xlabel('False Positive Rate')\n", "plt.ylabel('True Positive Rate')\n", "plt.title('ROC Curves - All Models')\n", "plt.legend(loc=\"lower right\")\n", "plt.grid(True)\n", "plt.show()\n", "\n", "print(\"\\nValori AUC-ROC:\")\n", "for name, results in all_results.items():\n", " model = results['model']\n", " y_pred_proba = model.predict_proba(X_test)[:, 1]\n", " roc_auc = auc(roc_curve(y_test, y_pred_proba)[0], roc_curve(y_test, y_pred_proba)[1])\n", " print(f\"{name}: {roc_auc:.4f}\")\n", "\n", "print(f\"Random Forest Standard: {auc(fpr_rf, tpr_rf):.4f}\")\n", "print(f\"Logistic Regression 0.65: {auc(fpr_lr, tpr_lr):.4f}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n

Per il nostro caso studio credo che il modello Logistic Regression con soglia di decisione a 0.65 sia un buon compromesso

\n
" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }