Dictionnary Encoding tutorial

This tutorial shows how to use simple python dictionnaries to reverse data preprocessing and display explicit labels

Data from Kaggle Titanic

Content : - Encode data with dictionary - Build a Binary Classifier (Random Forest) - Using Shapash - Show inversed data

import numpy as np
import pandas as pd
from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split

Load titanic Data

from shapash.data.data_loader import data_loading

titan_df, titan_dict = data_loading('titanic')
del titan_df['Name']

titan_df.head()

	Survived	Pclass	Sex	Age	SibSp	Parch	Fare	Embarked	Title
PassengerId
1	0	Third class	male	22.0	1	0	7.25	Southampton	Mr
2	1	First class	female	38.0	1	0	71.28	Cherbourg	Mrs
3	1	Third class	female	26.0	0	0	7.92	Southampton	Miss
4	1	First class	female	35.0	1	0	53.10	Southampton	Mrs
5	0	Third class	male	35.0	0	0	8.05	Southampton	Mr

Prepare data for the model

Create Target

y = titan_df['Survived']
X = titan_df.drop('Survived', axis=1)

Build dict tranformation and reversed dict

#construct new variable
X['new_embarked'] = X.apply(lambda x : 1 if x.Embarked in ['Southampton','Cherbourg'] else 2 if x.Embarked in 'Queenstown' else 3, axis = 1)
#Construct the reversed dict
transfo_embarked = {'col': 'new_embarked',
                    'mapping': pd.Series(data=[1, 2, np.nan], index=['Southampton-Cherbourg', 'Queenstown','missing']),
                    'data_type': 'object'}

#construct new variable
X['new_ages'] = X.apply(lambda x : 1 if x.Age <= 25 else 2 if x.Age <= 40 else 3, axis = 1)
#Construct the reversed dict
transfo_age = dict()
transfo_age = {'col': 'new_ages',
                'mapping': pd.Series(data=[1, 2, 3, np.nan], index=['-25 years', '26-40 years', '+40 years','missing']),
                'data_type': 'object'}

#put transformation into list
encoder = [transfo_age,transfo_embarked]

X.head(4)

	Pclass	Sex	Age	SibSp	Parch	Fare	Embarked	Title	new_embarked	new_ages
PassengerId
1	Third class	male	22.0	1	0	7.25	Southampton	Mr	1	1
2	First class	female	38.0	1	0	71.28	Cherbourg	Mrs	1	2
3	Third class	female	26.0	0	0	7.92	Southampton	Miss	1	2
4	First class	female	35.0	1	0	53.10	Southampton	Mrs	1	2

Fit a model

X = X[['new_embarked','new_ages','Fare','Parch','Age']]

Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, train_size=0.75, random_state=1)

clf = XGBClassifier(n_estimators=200,min_child_weight=2).fit(Xtrain,ytrain)
clf.fit(Xtrain, ytrain)

XGBClassifier(base_score=0.5, booster=None, colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,
              importance_type='gain', interaction_constraints=None,
              learning_rate=0.300000012, max_delta_step=0, max_depth=6,
              min_child_weight=2, missing=nan, monotone_constraints=None,
              n_estimators=200, n_jobs=0, num_parallel_tree=1,
              objective='binary:logistic', random_state=0, reg_alpha=0,
              reg_lambda=1, scale_pos_weight=1, subsample=1, tree_method=None,
              validate_parameters=False, verbosity=None)

Using Shapash

from shapash import SmartExplainer

xpl = SmartExplainer(model=clf, preprocessing=encoder)

xpl.compile(x=Xtest,
y_target=ytest, # Optional: allows to display True Values vs Predicted Values
)

Backend: Shap TreeExplainer

Visualize data in pandas

xpl.x_init.head(4)

	new_embarked	new_ages	Fare	Parch	Age
PassengerId
863	Southampton-Cherbourg	+40 years	25.93	0	48.0
224	Southampton-Cherbourg	26-40 years	7.90	0	29.5
85	Southampton-Cherbourg	-25 years	10.50	0	17.0
681	Queenstown	26-40 years	8.14	0	29.5

xpl.x_encoded.head(4)

	new_embarked	new_ages	Fare	Parch	Age
PassengerId
863	1	3	25.93	0	48.0
224	1	2	7.90	0	29.5
85	1	1	10.50	0	17.0
681	2	2	8.14	0	29.5