Decision Trees (DTs) are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. We’re going to try to predict the species of penguins based on their body measurements.
Firstly we’ll import the .csv file and take a look at the data.
import pandas as pd df = pd.read_csv(r'C:\Users\hanna\Downloads\penguins.csv') df.head(5)
df.info() and df.decribe() are simple and awesome ways of checking the data available.
<class 'pandas.core.frame.DataFrame'> RangeIndex: 344 entries, 0 to 343 Data columns (total 7 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 species 344 non-null object 1 island 344 non-null object 2 culmen_length_mm 342 non-null float64 3 culmen_depth_mm 342 non-null float64 4 flipper_length_mm 342 non-null float64 5 body_mass_g 342 non-null float64 6 sex 334 non-null object dtypes: float64(4), object(3) memory usage: 18.9+ KB None culmen_length_mm culmen_depth_mm flipper_length_mm body_mass_g count 342.000000 342.000000 342.000000 342.000000 mean 43.921930 17.151170 200.915205 4201.754386 std 5.459584 1.974793 14.061714 801.954536 min 32.100000 13.100000 172.000000 2700.000000 25% 39.225000 15.600000 190.000000 3550.000000 50% 44.450000 17.300000 197.000000 4050.000000 75% 48.500000 18.700000 213.000000 4750.000000 max 59.600000 21.500000 231.000000 6300.000000
Thankfully I already know this dataset so I’m going to move straight to dropping null values as we won’t need those.
species 0 island 0 culmen_length_mm 0 culmen_depth_mm 0 flipper_length_mm 0 body_mass_g 0 sex 0 dtype: int64
Seaborn is a fantastic library for visualising datasets.
import seaborn as sns sns.boxplot(x="culmen_length_mm",data=df,color ="orange");
As we know we want to look at species I want to know how many of each species we have in the dataset.
Adelie 146 Gentoo 120 Chinstrap 68 Name: species, dtype: int64
.corr is a great way to instantly see the correlation between numerical columns. 1 is completely correlated. Anything negative is negatively correlated.
corr = df.corr() corr
If you only have a few columns to view then the seaborn heatmap is a great visual tool for correlation:
sns.heatmap(corr,vmin=-1, vmax=1, center=0, cmap=sns.diverging_palette(220, 20, as_cmap=True), square=True, annot = True );
Splitting out the features so we know what we want to use to predict. We want the feature names to predict the species and the class names in the tree to be the species names. In this case we’re going to use all of the numerical columns.
feature_names = ['culmen_length_mm', 'culmen_depth_mm', 'flipper_length_mm', 'body_mass_g'] X = df[feature_names] y = df['species'] class_names = df['species'].unique()
Training the Model
Now it’s time to train the model: We import the DecisionTree Classifier. We have chosen 10 leaves, and a max depth of 2. (Later on we can tweak the min_samples_leaf and the max_depth if we want to change the tree size to better suit the dataset). We are going to fit the X and y as defined above.
from sklearn.tree import DecisionTreeClassifier clf = DecisionTreeClassifier( min_samples_leaf=10, max_depth=2 ) clf.fit(X, y)
Now we can import Tree and we can see a little tree below!
from sklearn import tree tree.plot_tree(clf);
The gini index (or gini impurity) calculates the amount of probability of a specific feature that is classified incorrectly when selected randomly.
The sample size is the sample size of penguins that make up each node.
The value is how many samples of that node fall into each category (Adelie, Chinstrap, Gentoo).
The true values go to the left (as you look at the tree) and the false values go to the right. You can see this easier below where the Adelie node that begins the tree grows down to the left. The false values form the first node go to the right and that’s where we can see the Gentoo node. We don’t see Chinstrap until the next layer down.
The tree is pretty good but this is where matplotlib comes in handy for visualising things… The deeper the colour, the better the match. These colours look pretty good to me.
import matplotlib.pyplot as plt fig, axes = plt.subplots( nrows=1, ncols=1, figsize=(3, 4), dpi=300 ) tree.plot_tree( clf, feature_names=feature_names, class_names=class_names, filled=True, fontsize=4 );
Now we can make our own predictions if we wish. We can input our own measurements into the model and see what it comes out with. The prediction is in the order of: ‘culmen_length_mm’, ‘culmen_depth_mm’, ‘flipper_length_mm’, ‘body_mass_g’
The model prediction is that these measurements are for a ‘Gentoo’ penguin:
clf.predict([ [40, 16, 220, 3500] ])
Now it’s time to evaluate the model:
We’ll take a sample of 10 rows. Sample_X is the feature names as we defined earlier. Sample_y is the species name. Let’s have a look at them…
sample = df.sample(10) sample_X = sample[feature_names] sample_y = sample['species'] sample_X
257 Gentoo 260 Gentoo 34 Adelie 102 Adelie 157 Chinstrap 221 Gentoo 150 Adelie 172 Chinstrap 105 Adelie 95 Adelie Name: species, dtype: object
Now we’re going to predict the penguin species from the measurements we’ve added as variables (sample_X) and then compare them against our known species names for those values (sample_y) and see if it gets them right.
predictions = clf.predict(sample_X) predictions
array(['Chinstrap', 'Gentoo', 'Adelie', 'Adelie', 'Chinstrap', 'Gentoo', 'Adelie', 'Adelie', 'Adelie', 'Chinstrap'], dtype=object)
It’s only a small sample but this shows that the model predictions using the feature_names matched the known species names of the penguins. It’s a good start.
predictions == sample_y
257 False 260 True 34 True 102 True 157 True 221 True 150 True 172 False 105 True 95 False Name: species, dtype: bool
We can see above that there were incorrect predictions.
Now it’s time to split the data to check that the decision tree works.
We’ll train the model on 80% and test it on 20% of the data.
from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split( df[feature_names], df['species'], test_size=0.2 )
model = DecisionTreeClassifier( max_depth=2, min_samples_leaf=12, random_state=1 ) model.fit(X_train, y_train)
DecisionTreeClassifier(max_depth=2, min_samples_leaf=12, random_state=1)
ground_truth = y_test predictions = model.predict(X_test) predictions == ground_truth
310 True 336 True 248 True 57 True 305 False ... 174 True 1 True 110 True 79 True 90 True Name: species, Length: 67, dtype: bool
Time to score the model out of 1.
Now we can look at the classification report for each of the penguin species (again with a maximum value of 1). We can see from below that the model works best with Adelie penguins.
from sklearn.metrics import classification_report print(classification_report( y_test, predictions, labels=class_names ))
precision recall f1-score support Adelie 0.85 0.92 0.88 24 Chinstrap 0.65 0.93 0.76 14 Gentoo 1.00 0.72 0.84 29 accuracy 0.84 67 macro avg 0.83 0.86 0.83 67 weighted avg 0.87 0.84 0.84 67
Finally, we’ll take a look at the confusion matrix…Thankfully, it’s not as confusing as it may seem.
from sklearn.metrics import confusion_matrix confusion_matrix( y_test, predictions, labels=class_names )
array([[22, 2, 0], [ 1, 13, 0], [ 3, 5, 21]], dtype=int64)
The above array is represented here in this diagram of a confusion matrix. The order of the array L-R and Top to bottom is Adelie, Chinstrap and Gentoo. The true positives we can see from the visual representation are 22 Adelie, 13 Chinstrap and 21 Gentoo. The model managed to predict all of the Gentoo penguins correctly!
If you want to know more about SKlearn Decision Trees and what else you can do with them then visit: https://scikit-learn.org/stable/modules/tree.html
To find this dataset visit: https://www.kaggle.com/parulpandey/penguin-dataset-the-new-iris