Library Tutorial

The Journey to Building the Best Classifier: A Regular Story

It was an overcast Wednesday morning when Alex decided to tackle a classification challenge. Armed with the sklearn datasets and a toolbox of custom Python functions from the classifierpromax package, Alex set out to build the best possible model to classify data efficiently and effectively.

import pandas as pd
import numpy as np
from classifierpromax.ClassifierTrainer import ClassifierTrainer
from classifierpromax.ClassifierOptimizer import ClassifierOptimizer
from classifierpromax.FeatureSelector import FeatureSelector
from classifierpromax.ResultHandler import ResultHandler
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn import datasets

Loading the Dataset

Alex started by loading a familiar dataset from sklearn. For this project, the Iris dataset served as the training grounds—a well-known dataset for classifying different types of iris flowers.

# Load the dataset
iris = datasets.load_iris()
X, y = pd.DataFrame(iris.data, columns=iris.feature_names), pd.Series(iris.target)

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=421)
pd.concat([X_train, y_train], axis=1)

	sepal length (cm)	sepal width (cm)	petal length (cm)	petal width (cm)	0
90	5.5	2.6	4.4	1.2	1
29	4.7	3.2	1.6	0.2	0
65	6.7	3.1	4.4	1.4	1
82	5.8	2.7	3.9	1.2	1
135	7.7	3.0	6.1	2.3	2
...	...	...	...	...	...
30	4.8	3.1	1.6	0.2	0
79	5.7	2.6	3.5	1.0	1
19	5.1	3.8	1.5	0.3	0
53	5.5	2.3	4.0	1.3	1
87	6.3	2.3	4.4	1.3	1

75 rows × 5 columns

With the data ready, Alex built a preprocessing pipeline to ensure the input features were standardized.

# Preprocessing pipeline
preprocessor = make_pipeline(StandardScaler())

Training the Initial Models

The first step in Alex’s workflow was training baseline models using the ClassifierTrainer function. This function automatically trained several models, including Logistic Regression, SVC, and Random Forest, and evaluated them using cross-validation metrics.

trained_models, initial_scores = ClassifierTrainer(preprocessor, X_train, y_train, seed=421)
ResultHandler(initial_scores)

	dummy		logreg		svc		random_forest
	mean	std	mean	std	mean	std	mean	std
fit_time	0.002787	2.413336e-04	0.006328	0.000476	0.004342	0.000713	0.096318	0.000433
score_time	0.007225	3.381562e-04	0.010718	0.000100	0.009658	0.002247	0.012941	0.000052
test_accuracy	0.333333	0.000000e+00	0.933333	0.081650	0.946667	0.055777	0.933333	0.047140
train_accuracy	0.346667	7.453560e-03	0.953333	0.013944	0.966667	0.020412	1.000000	0.000000
test_precision	0.111111	1.551584e-17	0.939333	0.079588	0.951111	0.054772	0.948063	0.033914
train_precision	0.120222	5.093266e-03	0.953569	0.013877	0.967508	0.019637	1.000000	0.000000
test_recall	0.333333	0.000000e+00	0.933333	0.081650	0.946667	0.055777	0.933333	0.047140
train_recall	0.346667	7.453560e-03	0.953333	0.013944	0.966667	0.020412	1.000000	0.000000
test_f1	0.166667	0.000000e+00	0.932929	0.082477	0.946397	0.055859	0.932088	0.049128
train_f1	0.178519	6.625387e-03	0.953296	0.013956	0.966621	0.020468	1.000000	0.000000

As Alex examined the results, they noticed that Random Forest showed promise, but its performance could likely improve with feature selection and hyperparameter tuning.

Selecting Features for Simplicity

With a sense that the dataset contained redundant features, Alex decided to use the FeatureSelector function to refine the models further. They opted for Recursive Feature Elimination (RFE) to identify and select the most informative features.

feature_selected_models = FeatureSelector(
    preprocessor, trained_models, X_train, y_train, method='RFE', n_features_to_select=2
)

Now equipped with models that only used the top two features, Alex could already see improvements in simplicity and interpretability. But there was one final step to maximize the models’ potential.

Optimizing Model Hyperparameters

The models were performing well, but Alex wanted them to be great. Enter the ClassifierOptimizer function. This function searched for the best hyperparameters using RandomizedSearchCV, tuning each model to its optimal configuration.

optimized_models, optimized_scores = ClassifierOptimizer(
    feature_selected_models, X_train, y_train, scoring='f1', n_iter=10, random_state=421
)
ResultHandler(optimized_scores)

Training logreg...

Best parameters for logreg: {'logisticregression__C': np.float64(1.1904538755038707), 'logisticregression__class_weight': 'balanced'}

Training svc...

Best parameters for svc: {'svc__C': np.float64(0.4621351227119489), 'svc__class_weight': None}

Training random_forest...

Best parameters for random_forest: {'randomforestclassifier__max_depth': 6, 'randomforestclassifier__n_estimators': 25}

	logreg		svc		random_forest
	mean	std	mean	std	mean	std
fit_time	0.016752	0.000316	0.007162	0.000186	0.343617	0.012400
score_time	0.011257	0.000176	0.007564	0.000063	0.009763	0.000181
test_accuracy	0.946667	0.055777	0.946667	0.055777	0.933333	0.047140
train_accuracy	0.950000	0.011785	0.953333	0.007454	1.000000	0.000000
test_precision	0.959175	0.040828	0.951556	0.054714	0.948063	0.033914
train_precision	0.950475	0.011793	0.953927	0.007123	1.000000	0.000000
test_recall	0.946667	0.055777	0.946667	0.055777	0.933333	0.047140
train_recall	0.950000	0.011785	0.953333	0.007454	1.000000	0.000000
test_f1	0.945556	0.057790	0.946667	0.055779	0.932088	0.049128
train_f1	0.950006	0.011785	0.953321	0.007461	1.000000	0.000000

After a few minutes of computation, the function returned the optimized models and their performance metrics. Alex was thrilled to see that the Random Forest classifier now had the best F1 score among all models.

Summarizing the Results

To present the results clearly, Alex used the ResultHandler function to compile the performance metrics and hyperparameters into a tidy DataFrame.

results_df = ResultHandler(initial_scores, optimized_scores)
results_df

	dummy_baseline	logreg_baseline	logreg_optimized	random_forest_baseline	random_forest_optimized	svc_baseline	svc_optimized
fit_time	0.002787	0.006328	0.016752	0.096318	0.343617	0.004342	0.007162
score_time	0.007225	0.010718	0.011257	0.012941	0.009763	0.009658	0.007564
test_accuracy	0.333333	0.933333	0.946667	0.933333	0.933333	0.946667	0.946667
train_accuracy	0.346667	0.953333	0.950000	1.000000	1.000000	0.966667	0.953333
test_precision	0.111111	0.939333	0.959175	0.948063	0.948063	0.951111	0.951556
train_precision	0.120222	0.953569	0.950475	1.000000	1.000000	0.967508	0.953927
test_recall	0.333333	0.933333	0.946667	0.933333	0.933333	0.946667	0.946667
train_recall	0.346667	0.953333	0.950000	1.000000	1.000000	0.966667	0.953333
test_f1	0.166667	0.932929	0.945556	0.932088	0.932088	0.946397	0.946667
train_f1	0.178519	0.953296	0.950006	1.000000	1.000000	0.966621	0.953321

Testing the Final Model

The optimized Random Forest model was deployed for testing on the test dataset. Alex measured its performance and confirmed that it generalized well beyond the training data.

# Testing the final Random Forest model
final_model = optimized_models['random_forest']
y_pred = final_model.predict(X_test)

print(classification_report(y_test, y_pred))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        25
           1       0.96      0.96      0.96        26
           2       0.96      0.96      0.96        24

    accuracy                           0.97        75
   macro avg       0.97      0.97      0.97        75
weighted avg       0.97      0.97      0.97        75

Reflection

By combining tools like ClassifierTrainer, FeatureSelector, ClassifierOptimizer, and ResultHandler, Alex had created an efficient, repeatable, and transparent machine learning pipeline. It was a productive day of problem-solving, leaving Alex eager to tackle their next project.