1. Introduction
In this article we use the Bayesian Optimization (BO) package to determine hyperparameters for a 2D convolutional neural network classifier with Keras.
2. Using Bayesian Optimization
CORRECTION:
In the code below dict_params should be:
dict_params = {'num_cnn_blocks':int(num_cnn_blocks),
'num_filters':16*int(num_filters),
'kernel_size':int(kernel_size),
'num_dense_nodes':64*int(num_dense_nodes),
'dense_nodes_divisor':2*int(dense_nodes_divisor),
'batch_size':32*int(batch_size),
'drop_out':drop_out}
The integer multipliers should have been outside of the cast to integer. BO only supports real valued intervals, so to obtain discrete integers, we must cast the real values to integers then multiply by an integer. for num_filters, we have (1, 4.001) (see pbounds below) as the range of real values used by BO. To convert to discrete values of 16, 32, 48, 64, we use the revised coding above. This will not cause errors in the code, as Keras only requires the use of integers for variables such as filters. Using 17 as the possible value for filters is not erroneous, but it is not the intended possible value of 16, so we did not rerun the code. We apologize for this error.
model_name = ml_algo_name + '_' + str(np.random.uniform(0,1,))[2:9]
# variable parameters
dict_params = {'num_cnn_blocks':int(num_cnn_blocks),
'num_filters':int(16*num_filters),
'kernel_size':int(kernel_size),
'num_dense_nodes':int(64*num_dense_nodes),
'dense_nodes_divisor':int(2*dense_nodes_divisor),
'batch_size':int(32*batch_size),
'drop_out':drop_out}
# start of cnn coding
input_tensor = Input(shape=input_shape)
# 1st cnn block
x = BatchNormalization()(input_tensor)
x = Activation('relu')(x)
x = Conv2D(filters=dict_params['num_filters'],
kernel_size=dict_params['kernel_size'],
strides=1, padding='same')(x)
#x = MaxPooling2D()(x)
x = Dropout(dict_params['drop_out'])(x)
# additional cnn blocks
for iblock in range(dict_params['num_cnn_blocks'] - 1):
x = BatchNormalization()(x)
x = Activation('relu')(x)
x = Conv2D(filters=dict_params['num_filters'],
kernel_size=dict_params['kernel_size'],
strides=1, padding='same')(x)
x = MaxPooling2D()(x)
x = Dropout(dict_params['drop_out'])(x)
# mlp
x = Flatten()(x)
x = Dense(dict_params['num_dense_nodes'], activation='relu')(x)
x = Dropout(dict_params['drop_out'])(x)
x = Dense(dict_params['num_dense_nodes']//dict_params['dense_nodes_divisor'],
activation='relu')(x)
output_tensor = Dense(number_of_classes, activation='softmax')(x)
# instantiate and compile model
cnn_model = Model(inputs=input_tensor, outputs=output_tensor)
opt = Adam(lr=0.00025) # default = 0.001
cnn_model.compile(loss='categorical_crossentropy',
optimizer=opt, metrics=['accuracy'])
# callbacks for early stopping and for learning rate reducer
callbacks_list = [EarlyStopping(monitor='val_loss', patience=early_stop_epochs),
ReduceLROnPlateau(monitor='val_loss', factor=0.1,
patience=learning_rate_epochs,
verbose=0, mode='auto', min_lr=1.0e-6),
ModelCheckpoint(filepath=results_dir + model_name + '.h5',
monitor='val_loss', save_best_only=True)]
# fit the model
h = cnn_model.fit(x=xcalib, y=ycalib,
batch_size=dict_params['batch_size'],
epochs=maximum_epochs,
validation_split=0.25,
shuffle=True, verbose=0,
callbacks=callbacks_list)
# record actual best epochs and valid loss here, added to bayes opt parameter df below
list_early_stop_epochs.append(len(h.history['val_loss']) - early_stop_epochs)
validation_loss = np.min(h.history['val_loss'])
list_validation_loss.append(validation_loss)
list_saved_model_name.append(model_name)
# bayes opt is a maximization algorithm, to minimize validation_loss, return 1-this
bayes_opt_score = 1.0 - validation_loss
return bayes_opt_score
# end of cnn_function()
optimizer = BayesianOptimization(f=cnn_function,
pbounds={'num_cnn_blocks':(2, 4.001),
'num_filters':(1, 4.001), # *16
'kernel_size':(2, 4.001),
'num_dense_nodes':(1, 16.001), # *64
'dense_nodes_divisor':(1, 4.001), # *2
'batch_size':(1, 4.001), # *32
'drop_out': (0.05, 0.5)},
verbose=2)
optimizer.maximize(init_points=num_random_points, n_iter=num_iterations)
print('nbest result:', optimizer.max)
We construct a simple CNN as we are using the MNIST handwritten digits data set, so we do not require sophisticated architectures such as DenseNet, ResNet, etc. dict_params contains all of the variables that will be exposed to BO. Note that since BO only accepts continuous numbers, if we want integers, we must use casting to obtain them. However, when we want integers that are evenly spaced with any value other than 1, we set the multiple (16 for number of filters) and the continuous interval (1, 4.001) (pbounds), so that we get [16, 32, 48, 64]. We put this in dict_params so that it is easier to use in the Keras CNN coding.
This code fragment is a function, cnn_function(num_cnn_blocks, num_filters, kernel_size, num_dense_nodes, dense_nodes_divisor, batch_size, drop_out), nested inside of another function (see the Code section at the end of this article). Within this function is an implicit loop enacted by BO n_iter times. Any information from the CNN result that we wish to save can be done so by appending it to a list and synchronizing with the BO diagnostic output.
Note the use of model_name = ml_algo_name + ‘_’ + str(np.random.uniform(0,1,))[2:9] within the implicit loop along with ModelCheckpoint from Keras. ModelCheckpoint will save the best model encountered so far during a loop over epochs. By using the same name for filepath that is unique for each BO loop and also saving the name in a list, we ensure that we can match up the validation loss with the model name. This is later used in ensembling by setting a threshold for validation loss.
A reminder: Bayesian Optimization is a maximization algorithm. Thus we record 1.0 – validation_loss.
See Hyperparameter Search With Bayesian Optimization for Scikit-learn Classification and Ensembling for an explanation of the other BO parameters.
To collect all results from BO, we have
list_dfs = []
counter = 0
for result in optimizer.res:
df_temp = pd.DataFrame.from_dict(data=result['params'], orient='index',
columns=['trial' + str(counter)]).T
df_temp['bayes opt error'] = result['target']
df_temp['epochs'] = list_early_stop_epochs[counter]
df_temp['validation_loss'] = list_validation_loss[counter]
df_temp['model_name'] = list_saved_model_name[counter]
list_dfs.append(df_temp)
counter = counter + 1
df_results = pd.concat(list_dfs, axis=0)
df_results.to_pickle(results_dir + 'df_bayes_opt_results_parameters.pkl')
df_results.to_csv(results_dir + 'df_bayes_opt_results_parameters.csv')
optimizer.res is a list of dictionaries. The code above puts each dictionary into a DataFrame then stacks them vertically into a single DataFrame. optimizer.max yields the best result (maximum target), it can also be obtained from the single DataFrame.
3. Ensembling and Results
To obtain an ensemble, we apply a threshold on the validation loss, load the corresponding saved models, then process production data (unseen data) to obtain class probabilities, NOT the classes themselves. These probabilities are then averaged and the final predicted class is determined via plurality voting.
accepted_models_num = 0
list_predicted_prob = []
num_models = df_params.shape[0]
for i in range(num_models):
if df_params.loc[df_params.index[i],'validation_loss'] < threshold:
model_name = df_params.loc[df_params.index[i],'model_name']
model_final = load_model(models_directory + model_name + '.h5')
# these are class probabilities
list_predicted_prob.append(model_final.predict(xprod))
accepted_models_num = accepted_models_num + 1
# compute mean probabilities
mean_probabilities = np.mean(list_predicted_prob, axis=0)
# compute predicted class
# argmax uses 1st ocurrance in case of a tie
y_predicted_class = np.argmax(mean_probabilities, axis=1)
CNN Hyperparameters
| batch_size | dense_nodes_divisor | drop_out | kernel_size | num_cnn_blocks | num_dense_nodes | num_filters | bayes opt error | epochs | validation_loss | model_name | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| trial24 | 3.3389 | 4.0010 | 0.0500 | 4.0010 | 4.0010 | 4.8344 | 4.0010 | 0.9700 | 24 | 0.0300 | cnn_7466261 |
| trial21 | 2.5018 | 1.0000 | 0.0500 | 4.0010 | 4.0010 | 11.0982 | 4.0010 | 0.9661 | 14 | 0.0339 | cnn_0522347 |
| trial37 | 4.0010 | 1.0000 | 0.0500 | 4.0010 | 4.0010 | 4.4970 | 4.0010 | 0.9643 | 17 | 0.0357 | cnn_5051812 |
| trial47 | 1.2830 | 2.0793 | 0.0500 | 2.1005 | 4.0010 | 7.0846 | 2.6164 | 0.9624 | 16 | 0.0376 | cnn_1950449 |
| trial42 | 4.0010 | 1.9473 | 0.0500 | 4.0010 | 4.0010 | 5.0035 | 1.7908 | 0.9621 | 17 | 0.0379 | cnn_0833026 |
| trial35 | 3.7624 | 3.9981 | 0.0500 | 3.1735 | 4.0010 | 1.8905 | 1.0849 | 0.9609 | 43 | 0.0391 | cnn_8772981 |
| trial18 | 3.3421 | 4.0010 | 0.5000 | 4.0010 | 4.0010 | 15.3680 | 4.0010 | 0.9606 | 24 | 0.0394 | cnn_2551120 |
| trial9 | 3.6915 | 2.3017 | 0.2560 | 3.1245 | 3.4552 | 2.4900 | 3.7501 | 0.9604 | 25 | 0.0396 | cnn_4124185 |
| trial14 | 4.0010 | 4.0010 | 0.0500 | 2.0000 | 4.0010 | 16.0010 | 1.0000 | 0.9601 | 38 | 0.0399 | cnn_0094828 |
| trial34 | 4.0010 | 3.6555 | 0.0500 | 2.0000 | 4.0010 | 14.8151 | 3.5020 | 0.9594 | 34 | 0.0406 | cnn_2174602 |
| trial16 | 4.0010 | 1.0000 | 0.0500 | 2.0000 | 4.0010 | 16.0010 | 4.0010 | 0.9594 | 16 | 0.0406 | cnn_8600071 |
| trial15 | 4.0010 | 1.0000 | 0.0500 | 4.0010 | 4.0010 | 1.0000 | 1.0000 | 0.9594 | 32 | 0.0406 | cnn_9779994 |
| trial23 | 1.0000 | 1.0000 | 0.0500 | 4.0010 | 4.0010 | 5.1663 | 1.0000 | 0.9590 | 21 | 0.0410 | cnn_5389027 |
| trial33 | 1.0000 | 4.0010 | 0.0500 | 2.4149 | 4.0010 | 8.8738 | 1.0000 | 0.9589 | 34 | 0.0411 | cnn_7401566 |
| trial29 | 4.0010 | 3.2454 | 0.0500 | 3.9663 | 4.0010 | 13.4776 | 1.3420 | 0.9584 | 29 | 0.0416 | cnn_7782207 |
| trial25 | 1.0000 | 4.0010 | 0.0500 | 2.0000 | 4.0010 | 16.0010 | 4.0010 | 0.9584 | 16 | 0.0416 | cnn_1088887 |
| trial49 | 4.0010 | 1.3321 | 0.0500 | 3.7722 | 4.0010 | 13.0380 | 4.0010 | 0.9582 | 17 | 0.0418 | cnn_1277738 |
| trial11 | 3.2915 | 2.2746 | 0.3246 | 3.1829 | 3.8379 | 8.6781 | 1.4269 | 0.9577 | 35 | 0.0423 | cnn_8345552 |
| trial3 | 3.9304 | 2.9875 | 0.2520 | 2.5634 | 3.2125 | 2.5510 | 1.5212 | 0.9571 | 32 | 0.0429 | cnn_1375661 |
| trial8 | 3.2834 | 3.8950 | 0.1270 | 2.9211 | 3.8746 | 8.5245 | 3.7471 | 0.9556 | 10 | 0.0444 | cnn_2874261 |
| trial30 | 4.0010 | 1.0000 | 0.0500 | 2.0000 | 4.0010 | 5.4644 | 1.0000 | 0.9552 | 45 | 0.0448 | cnn_7897795 |
| trial6 | 3.3851 | 1.5713 | 0.4678 | 3.0353 | 3.4220 | 13.1056 | 1.3557 | 0.9541 | 41 | 0.0459 | cnn_1827607 |
| trial12 | 1.0000 | 1.0000 | 0.0500 | 2.0000 | 4.0010 | 1.0000 | 1.0654 | 0.9537 | 38 | 0.0463 | cnn_2804524 |
| trial17 | 4.0010 | 4.0010 | 0.0534 | 2.0000 | 4.0010 | 1.0000 | 4.0010 | 0.9527 | 32 | 0.0473 | cnn_7216191 |
| trial41 | 2.2563 | 3.4701 | 0.2850 | 2.0160 | 3.8976 | 4.3858 | 1.0014 | 0.9525 | 41 | 0.0475 | cnn_5766945 |
| trial4 | 3.4131 | 3.8578 | 0.0629 | 2.8113 | 3.7743 | 7.3985 | 2.2179 | 0.9511 | 8 | 0.0489 | cnn_6586571 |
| trial38 | 1.0324 | 3.7249 | 0.0763 | 3.8798 | 3.8725 | 10.5557 | 3.9936 | 0.9493 | 6 | 0.0507 | cnn_2038463 |
| trial43 | 3.9163 | 3.9900 | 0.1196 | 2.0881 | 3.9042 | 3.3694 | 3.9648 | 0.9489 | 15 | 0.0511 | cnn_2407983 |
| trial40 | 3.2168 | 1.4162 | 0.0536 | 2.2749 | 3.9170 | 12.1045 | 1.0488 | 0.9471 | 17 | 0.0529 | cnn_3200254 |
| trial19 | 1.0000 | 2.0569 | 0.5000 | 2.0000 | 4.0010 | 5.4079 | 4.0010 | 0.9463 | 31 | 0.0537 | cnn_2272907 |
| trial32 | 1.0338 | 3.2372 | 0.0557 | 3.1871 | 3.8629 | 2.6796 | 3.9573 | 0.9461 | 4 | 0.0539 | cnn_8375640 |
| trial36 | 1.6279 | 1.2807 | 0.0650 | 4.0002 | 2.2150 | 3.1657 | 1.2277 | 0.9390 | 4 | 0.0610 | cnn_2177580 |
| trial0 | 1.3710 | 3.5067 | 0.2024 | 3.3116 | 2.4735 | 7.7148 | 3.1912 | 0.9381 | 16 | 0.0619 | cnn_4534673 |
| trial45 | 4.0010 | 4.0010 | 0.0500 | 4.0010 | 2.0000 | 16.0010 | 1.0000 | 0.9356 | 16 | 0.0644 | cnn_6611043 |
| trial26 | 4.0010 | 1.0000 | 0.5000 | 4.0010 | 4.0010 | 15.9693 | 1.0000 | 0.9303 | 38 | 0.0697 | cnn_7027454 |
| trial20 | 1.0000 | 1.9328 | 0.0500 | 4.0010 | 2.0000 | 1.4661 | 4.0010 | 0.9301 | 2 | 0.0699 | cnn_4494844 |
| trial1 | 2.5818 | 1.9285 | 0.3844 | 3.0367 | 2.5969 | 8.3888 | 2.8445 | 0.9300 | 16 | 0.0700 | cnn_4664828 |
| trial5 | 2.1604 | 1.6733 | 0.0673 | 3.9997 | 2.8536 | 14.7074 | 3.2052 | 0.9299 | 5 | 0.0701 | cnn_4679929 |
| trial48 | 2.9484 | 1.0000 | 0.0500 | 4.0010 | 2.0000 | 10.9270 | 1.0000 | 0.9296 | 5 | 0.0704 | cnn_4994236 |
| trial2 | 1.5486 | 1.5576 | 0.4545 | 2.0458 | 2.8034 | 13.4613 | 3.8463 | 0.9229 | 24 | 0.0771 | cnn_4649820 |
| trial46 | 4.0010 | 1.0000 | 0.0500 | 2.0000 | 2.0000 | 16.0010 | 1.0000 | 0.9224 | 14 | 0.0776 | cnn_0834311 |
| trial7 | 1.1423 | 1.3225 | 0.3457 | 2.4675 | 2.9744 | 12.7185 | 2.8241 | 0.9211 | 25 | 0.0789 | cnn_3976373 |
| trial27 | 4.0010 | 4.0010 | 0.0500 | 2.0000 | 2.0000 | 16.0010 | 4.0010 | 0.9208 | 7 | 0.0792 | cnn_5418930 |
| trial31 | 4.0010 | 4.0010 | 0.0500 | 4.0010 | 2.0000 | 1.0000 | 4.0010 | 0.9193 | 9 | 0.0807 | cnn_2041164 |
| trial13 | 1.0000 | 4.0010 | 0.5000 | 4.0010 | 4.0010 | 1.0000 | 1.0305 | 0.9129 | 39 | 0.0871 | cnn_2932650 |
| trial28 | 1.5008 | 4.0010 | 0.0500 | 2.0000 | 2.0000 | 1.0000 | 1.0000 | 0.9104 | 17 | 0.0896 | cnn_3575582 |
| trial22 | 3.9836 | 1.2313 | 0.4435 | 2.1642 | 3.9619 | 1.1961 | 2.5067 | 0.9061 | 22 | 0.0939 | cnn_6597575 |
| trial10 | 3.7392 | 2.7750 | 0.4556 | 2.4944 | 2.0160 | 6.6550 | 3.1538 | 0.8937 | 8 | 0.1063 | cnn_0709821 |
| trial39 | 1.0000 | 4.0010 | 0.5000 | 2.0000 | 4.0010 | 16.0010 | 1.0000 | 0.8935 | 57 | 0.1065 | cnn_1597172 |
| trial44 | 4.0010 | 4.0010 | 0.5000 | 2.6860 | 4.0010 | 11.1510 | 1.0000 | 0.8296 | 56 | 0.1704 | cnn_4295834 |
balanced accuracy score = 0.9939
accuracy score = 0.994
number of accepted models = 26 for threshold = 0.05
4. Code
import os
import numpy as np
import pandas as pd
from bayes_opt import BayesianOptimization
from keras.layers import Dense, Conv2D, BatchNormalization
from keras.layers import MaxPooling2D
from keras.layers import Input, Flatten, Dropout
from keras.layers import Activation
from keras.optimizers import Adam
from keras.callbacks import ReduceLROnPlateau, EarlyStopping, ModelCheckpoint
from keras.models import Model, load_model
from keras.utils import to_categorical
from sklearn.model_selection import train_test_split
from sklearn.metrics import balanced_accuracy_score, accuracy_score
from sklearn.metrics import confusion_matrix, classification_report
import time
from pathlib import Path
import sys
sys.path.append(YOUR DIRECTORY)
import confusion_matrix_plot
# *******************************************************************
# raw shapes: x:(num pts, 784), y:(num pts,)
def get_data_mnist(data_type, data_dir):
data_dictionary = {}
if data_type == 'calibration':
# x data - reshape = (60000, 28, 28, 1) - channels last
x_calib_raw = np.load(data_dir + 'x_mnist_calibration.npy', allow_pickle=True)
x_calib_raw = x_calib_raw.astype('float32')/255.0 # normalize
x_calib = np.reshape(x_calib_raw, (x_calib_raw.shape[0], 28, 28, 1))
# y data - shape = (50000, ) -> (50000, 10)
y_calib = to_categorical(np.load(data_dir + 'y_mnist_calibration.npy',
allow_pickle=True))
# split
x_calib1, x_calib2, y_calib1, y_calib2 = train_test_split(x_calib, y_calib,
train_size=0.5, shuffle=True, stratify=y_calib)
data_dictionary['x_calib1'] = x_calib1
data_dictionary['y_calib1'] = y_calib1
data_dictionary['x_calib2'] = x_calib2
data_dictionary['y_calib2'] = y_calib2
elif data_type == 'production':
# x data - reshape = (10000, 28, 28, 1) - channels last
x_prod_raw = np.load(data_dir + 'x_mnist_production.npy', allow_pickle=True)
x_prod_raw = x_prod_raw.astype('float32')/255.0 # normalize
x_prod = np.reshape(x_prod_raw, (x_prod_raw.shape[0], 28, 28, 1))
# y data - shape = (10000, )
y_prod = np.load(data_dir + 'y_mnist_production.npy', allow_pickle=True)
data_dictionary['x_prod'] = x_prod
data_dictionary['y_prod'] = y_prod
else:
raise NameError
return data_dictionary
# end of get_data_mnist()
# *******************************************************************
def create_save_models_bayes_opt(xcalib, ycalib,
num_random_points, num_iterations,
results_dir, ml_algo_name):
input_shape = xcalib.shape[1:] # (28, 28, 1) - channels last
number_of_classes = ycalib.shape[1]
# parameters for cnn that do NOT have to be saved
maximum_epochs = 1000
early_stop_epochs = 10
learning_rate_epochs = 5
# parameters that change for each iteration that must be saved
list_early_stop_epochs = []
list_validation_loss = []
list_saved_model_name = []
start_time_total = time.time()
def cnn_function(num_cnn_blocks, num_filters, kernel_size,
num_dense_nodes, dense_nodes_divisor, batch_size, drop_out):
model_name = ml_algo_name + '_' + str(np.random.uniform(0,1,))[2:9]
# variable parameters
dict_params = {'num_cnn_blocks':int(num_cnn_blocks),
'num_filters':int(16*num_filters),
'kernel_size':int(kernel_size),
'num_dense_nodes':int(64*num_dense_nodes),
'dense_nodes_divisor':int(2*dense_nodes_divisor),
'batch_size':int(32*batch_size),
'drop_out':drop_out}
# start of cnn coding
input_tensor = Input(shape=input_shape)
# 1st cnn block
x = BatchNormalization()(input_tensor)
x = Activation('relu')(x)
x = Conv2D(filters=dict_params['num_filters'],
kernel_size=dict_params['kernel_size'],
strides=1, padding='same')(x)
#x = MaxPooling2D()(x)
x = Dropout(dict_params['drop_out'])(x)
# additional cnn blocks
for iblock in range(dict_params['num_cnn_blocks'] - 1):
x = BatchNormalization()(x)
x = Activation('relu')(x)
x = Conv2D(filters=dict_params['num_filters'],
kernel_size=dict_params['kernel_size'],
strides=1, padding='same')(x)
x = MaxPooling2D()(x)
x = Dropout(dict_params['drop_out'])(x)
# mlp
x = Flatten()(x)
x = Dense(dict_params['num_dense_nodes'], activation='relu')(x)
x = Dropout(dict_params['drop_out'])(x)
x = Dense(dict_params['num_dense_nodes']//dict_params['dense_nodes_divisor'],
activation='relu')(x)
output_tensor = Dense(number_of_classes, activation='softmax')(x)
# instantiate and compile model
cnn_model = Model(inputs=input_tensor, outputs=output_tensor)
opt = Adam(lr=0.00025) # default = 0.001
cnn_model.compile(loss='categorical_crossentropy',
optimizer=opt, metrics=['accuracy'])
# callbacks for early stopping and for learning rate reducer
callbacks_list = [EarlyStopping(monitor='val_loss', patience=early_stop_epochs),
ReduceLROnPlateau(monitor='val_loss', factor=0.1,
patience=learning_rate_epochs,
verbose=0, mode='auto', min_lr=1.0e-6),
ModelCheckpoint(filepath=results_dir + model_name + '.h5',
monitor='val_loss', save_best_only=True)]
# fit the model
h = cnn_model.fit(x=xcalib, y=ycalib,
batch_size=dict_params['batch_size'],
epochs=maximum_epochs,
validation_split=0.25,
shuffle=True, verbose=0,
callbacks=callbacks_list)
# record actual best epochs and valid loss here, added to bayes opt parameter df below
list_early_stop_epochs.append(len(h.history['val_loss']) - early_stop_epochs)
validation_loss = np.min(h.history['val_loss']) # h.history['val_loss']
list_validation_loss.append(validation_loss)
list_saved_model_name.append(model_name)
# bayes opt is a maximization algorithm, to minimize validation_loss, return 1-this
bayes_opt_score = 1.0 - validation_loss
return bayes_opt_score
# end of cnn_function()
optimizer = BayesianOptimization(f=cnn_function,
pbounds={'num_cnn_blocks':(2, 4.001),
'num_filters':(1, 4.001), # *16
'kernel_size':(2, 4.001),
'num_dense_nodes':(1, 16.001), # *64
'dense_nodes_divisor':(1, 4.001), # *2
'batch_size':(1, 4.001), # *32
'drop_out': (0.05, 0.5)},
verbose=2)
optimizer.maximize(init_points=num_random_points, n_iter=num_iterations)
print('nbest result:', optimizer.max)
elapsed_time_total = (time.time()-start_time_total)/60
print('\n\ntotal elapsed time =',elapsed_time_total,' minutes')
# optimizer.res is a list of dicts
list_dfs = []
counter = 0
for result in optimizer.res:
df_temp = pd.DataFrame.from_dict(data=result['params'], orient='index',
columns=['trial' + str(counter)]).T
df_temp['bayes opt error'] = result['target']
df_temp['epochs'] = list_early_stop_epochs[counter]
df_temp['validation_loss'] = list_validation_loss[counter]
df_temp['model_name'] = list_saved_model_name[counter]
list_dfs.append(df_temp)
counter = counter + 1
df_results = pd.concat(list_dfs, axis=0)
df_results.to_pickle(results_dir + 'df_bayes_opt_results_parameters.pkl')
df_results.to_csv(results_dir + 'df_bayes_opt_results_parameters.csv')
# end of create_save_models_bayes_opt()
# *******************************************************************
def make_final_predictions(xprod, yprod,
list_class_names, models_directory,
save_directory, df_params,
threshold, ml_name):
# apply threshold
accepted_models_num = 0
list_predicted_prob = []
num_models = df_params.shape[0]
for i in range(num_models):
if df_params.loc[df_params.index[i],'validation_loss'] < threshold:
model_name = df_params.loc[df_params.index[i],'model_name']
model_final = load_model(models_directory + model_name + '.h5')
# these are class probabilities
list_predicted_prob.append(model_final.predict(xprod))
accepted_models_num = accepted_models_num + 1
# compute mean probabilities
mean_probabilities = np.mean(list_predicted_prob, axis=0)
# compute predicted class
# argmax uses 1st ocurrance in case of a tie
y_predicted_class = np.argmax(mean_probabilities, axis=1)
# compute and save error measures
# print info to file
stdout_default = sys.stdout
sys.stdout = open(save_directory + ml_name + '_prediction_results.txt','w')
print('balanced accuracy score =',balanced_accuracy_score(yprod, y_predicted_class))
print('accuracy score =',accuracy_score(yprod, y_predicted_class))
print('number of accepted models =',accepted_models_num,' for threshold =',threshold)
print('\nclassification report:')
print(classification_report(yprod, y_predicted_class, digits=3, output_dict=False))
print('\nraw confusion matrix:')
cm_raw = confusion_matrix(yprod, y_predicted_class)
print(cm_raw)
print('\nconfusion matrix normalized by prediction:')
cm_pred = confusion_matrix(yprod, y_predicted_class, normalize='pred')
print(cm_pred)
print('\nconfusion matrix normalized by true:')
cm_true = confusion_matrix(yprod, y_predicted_class, normalize='true')
print(cm_true)
sys.stdout = stdout_default
# plot and save confustion matrices
figure_size = (12, 8)
number_of_decimals = 4
confusion_matrix_plot.confusion_matrix_save_and_plot(cm_raw,
list_class_names, save_directory, 'Confusion Matrix',
ml_name + '_confusion_matrix', False, None, 30, figure_size,
number_of_decimals)
confusion_matrix_plot.confusion_matrix_save_and_plot(cm_pred,
list_class_names, save_directory, 'Confusion Matrix Normalized by Prediction',
ml_name + '_confusion_matrix_norm_by_prediction', False, 'pred',
30, figure_size, number_of_decimals)
confusion_matrix_plot.confusion_matrix_save_and_plot(cm_true,
list_class_names, save_directory, 'Confusion Matrix Normalized by Actual',
ml_name + '_confusion_matrix_norm_by_true', False, 'true',
30, figure_size, number_of_decimals)
# end of make_final_predictions()
# *******************************************************************
if __name__ == '__main__':
ml_algorithm_name = 'cnn'
dataset_name = 'mnist'
file_name_stub = ml_algorithm_name + '_' + dataset_name + '_bayes_opt'
calculation_type = 'production' #'calibration' 'production'
base_directory = YOUR DIRECTORY
data_directory = base_directory + 'data/'
results_directory_stub = base_directory + file_name_stub + '/'
if not Path(results_directory_stub).is_dir():
os.mkdir(results_directory_stub)
# fixed parameters
threshold_validation_loss = 0.05
total_number_of_iterations = 50
number_of_random_points = 12 # random searches to start opt process
# this is # of bayes iters, thus total=this + # of random pts
number_of_iterations = total_number_of_iterations - number_of_random_points
# read data
dict_data = get_data_mnist(calculation_type, data_directory)
print('\n*** starting at',pd.Timestamp.now())
# calibration
if calculation_type == 'calibration':
# using 1/2 of calib data
x_calib = dict_data['x_calib1']
y_calib = dict_data['y_calib1']
results_directory = results_directory_stub + calculation_type + '/'
if not Path(results_directory).is_dir():
os.mkdir(results_directory)
create_save_models_bayes_opt(x_calib, y_calib,
number_of_random_points, number_of_iterations,
results_directory, ml_algorithm_name)
# production
elif calculation_type == 'production':
# get saved parameters
models_dir = results_directory_stub + 'calibration/'
df_parameters = pd.read_pickle(models_dir + 'df_bayes_opt_results_parameters.pkl')
results_directory = results_directory_stub + calculation_type + '/'
if not Path(results_directory).is_dir():
os.mkdir(results_directory)
x_prod = dict_data['x_prod']
y_prod = dict_data['y_prod']
num_classes = np.unique(y_prod).shape[0]
class_names_list = []
for i in range(num_classes):
class_names_list.append('class ' + str(i))
make_final_predictions(x_prod, y_prod, class_names_list,
models_dir, results_directory,
df_parameters,
threshold_validation_loss, ml_algorithm_name)
else:
print('\ninvalid calculation type:',calculation_type)
raise NameError
