1. Introduction
In this article we use the Bayesian Optimization (BO) package to determine hyperparameters for a 2D convolutional neural network classifier with Keras.
2. Using Bayesian Optimization
CORRECTION:
In the code below dict_params should be:
dict_params = {'num_cnn_blocks':int(num_cnn_blocks), 'num_filters':16*int(num_filters), 'kernel_size':int(kernel_size), 'num_dense_nodes':64*int(num_dense_nodes), 'dense_nodes_divisor':2*int(dense_nodes_divisor), 'batch_size':32*int(batch_size), 'drop_out':drop_out}
The integer multipliers should have been outside of the cast to integer. BO only supports real valued intervals, so to obtain discrete integers, we must cast the real values to integers then multiply by an integer. for num_filters, we have (1, 4.001) (see pbounds below) as the range of real values used by BO. To convert to discrete values of 16, 32, 48, 64, we use the revised coding above. This will not cause errors in the code, as Keras only requires the use of integers for variables such as filters. Using 17 as the possible value for filters is not erroneous, but it is not the intended possible value of 16, so we did not rerun the code. We apologize for this error.
model_name = ml_algo_name + '_' + str(np.random.uniform(0,1,))[2:9] # variable parameters dict_params = {'num_cnn_blocks':int(num_cnn_blocks), 'num_filters':int(16*num_filters), 'kernel_size':int(kernel_size), 'num_dense_nodes':int(64*num_dense_nodes), 'dense_nodes_divisor':int(2*dense_nodes_divisor), 'batch_size':int(32*batch_size), 'drop_out':drop_out} # start of cnn coding input_tensor = Input(shape=input_shape) # 1st cnn block x = BatchNormalization()(input_tensor) x = Activation('relu')(x) x = Conv2D(filters=dict_params['num_filters'], kernel_size=dict_params['kernel_size'], strides=1, padding='same')(x) #x = MaxPooling2D()(x) x = Dropout(dict_params['drop_out'])(x) # additional cnn blocks for iblock in range(dict_params['num_cnn_blocks'] - 1): x = BatchNormalization()(x) x = Activation('relu')(x) x = Conv2D(filters=dict_params['num_filters'], kernel_size=dict_params['kernel_size'], strides=1, padding='same')(x) x = MaxPooling2D()(x) x = Dropout(dict_params['drop_out'])(x) # mlp x = Flatten()(x) x = Dense(dict_params['num_dense_nodes'], activation='relu')(x) x = Dropout(dict_params['drop_out'])(x) x = Dense(dict_params['num_dense_nodes']//dict_params['dense_nodes_divisor'], activation='relu')(x) output_tensor = Dense(number_of_classes, activation='softmax')(x) # instantiate and compile model cnn_model = Model(inputs=input_tensor, outputs=output_tensor) opt = Adam(lr=0.00025) # default = 0.001 cnn_model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy']) # callbacks for early stopping and for learning rate reducer callbacks_list = [EarlyStopping(monitor='val_loss', patience=early_stop_epochs), ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=learning_rate_epochs, verbose=0, mode='auto', min_lr=1.0e-6), ModelCheckpoint(filepath=results_dir + model_name + '.h5', monitor='val_loss', save_best_only=True)] # fit the model h = cnn_model.fit(x=xcalib, y=ycalib, batch_size=dict_params['batch_size'], epochs=maximum_epochs, validation_split=0.25, shuffle=True, verbose=0, callbacks=callbacks_list) # record actual best epochs and valid loss here, added to bayes opt parameter df below list_early_stop_epochs.append(len(h.history['val_loss']) - early_stop_epochs) validation_loss = np.min(h.history['val_loss']) list_validation_loss.append(validation_loss) list_saved_model_name.append(model_name) # bayes opt is a maximization algorithm, to minimize validation_loss, return 1-this bayes_opt_score = 1.0 - validation_loss return bayes_opt_score # end of cnn_function() optimizer = BayesianOptimization(f=cnn_function, pbounds={'num_cnn_blocks':(2, 4.001), 'num_filters':(1, 4.001), # *16 'kernel_size':(2, 4.001), 'num_dense_nodes':(1, 16.001), # *64 'dense_nodes_divisor':(1, 4.001), # *2 'batch_size':(1, 4.001), # *32 'drop_out': (0.05, 0.5)}, verbose=2) optimizer.maximize(init_points=num_random_points, n_iter=num_iterations) print('nbest result:', optimizer.max)
We construct a simple CNN as we are using the MNIST handwritten digits data set, so we do not require sophisticated architectures such as DenseNet, ResNet, etc. dict_params contains all of the variables that will be exposed to BO. Note that since BO only accepts continuous numbers, if we want integers, we must use casting to obtain them. However, when we want integers that are evenly spaced with any value other than 1, we set the multiple (16 for number of filters) and the continuous interval (1, 4.001) (pbounds), so that we get [16, 32, 48, 64]. We put this in dict_params so that it is easier to use in the Keras CNN coding.
This code fragment is a function, cnn_function(num_cnn_blocks, num_filters, kernel_size, num_dense_nodes, dense_nodes_divisor, batch_size, drop_out), nested inside of another function (see the Code section at the end of this article). Within this function is an implicit loop enacted by BO n_iter times. Any information from the CNN result that we wish to save can be done so by appending it to a list and synchronizing with the BO diagnostic output.
Note the use of model_name = ml_algo_name + ‘_’ + str(np.random.uniform(0,1,))[2:9] within the implicit loop along with ModelCheckpoint from Keras. ModelCheckpoint will save the best model encountered so far during a loop over epochs. By using the same name for filepath that is unique for each BO loop and also saving the name in a list, we ensure that we can match up the validation loss with the model name. This is later used in ensembling by setting a threshold for validation loss.
A reminder: Bayesian Optimization is a maximization algorithm. Thus we record 1.0 – validation_loss.
See Hyperparameter Search With Bayesian Optimization for Scikit-learn Classification and Ensembling for an explanation of the other BO parameters.
To collect all results from BO, we have
list_dfs = [] counter = 0 for result in optimizer.res: df_temp = pd.DataFrame.from_dict(data=result['params'], orient='index', columns=['trial' + str(counter)]).T df_temp['bayes opt error'] = result['target'] df_temp['epochs'] = list_early_stop_epochs[counter] df_temp['validation_loss'] = list_validation_loss[counter] df_temp['model_name'] = list_saved_model_name[counter] list_dfs.append(df_temp) counter = counter + 1 df_results = pd.concat(list_dfs, axis=0) df_results.to_pickle(results_dir + 'df_bayes_opt_results_parameters.pkl') df_results.to_csv(results_dir + 'df_bayes_opt_results_parameters.csv')
optimizer.res is a list of dictionaries. The code above puts each dictionary into a DataFrame then stacks them vertically into a single DataFrame. optimizer.max yields the best result (maximum target), it can also be obtained from the single DataFrame.
3. Ensembling and Results
To obtain an ensemble, we apply a threshold on the validation loss, load the corresponding saved models, then process production data (unseen data) to obtain class probabilities, NOT the classes themselves. These probabilities are then averaged and the final predicted class is determined via plurality voting.
accepted_models_num = 0 list_predicted_prob = [] num_models = df_params.shape[0] for i in range(num_models): if df_params.loc[df_params.index[i],'validation_loss'] < threshold: model_name = df_params.loc[df_params.index[i],'model_name'] model_final = load_model(models_directory + model_name + '.h5') # these are class probabilities list_predicted_prob.append(model_final.predict(xprod)) accepted_models_num = accepted_models_num + 1 # compute mean probabilities mean_probabilities = np.mean(list_predicted_prob, axis=0) # compute predicted class # argmax uses 1st ocurrance in case of a tie y_predicted_class = np.argmax(mean_probabilities, axis=1)
CNN Hyperparameters
batch_size | dense_nodes_divisor | drop_out | kernel_size | num_cnn_blocks | num_dense_nodes | num_filters | bayes opt error | epochs | validation_loss | model_name | |
---|---|---|---|---|---|---|---|---|---|---|---|
trial24 | 3.3389 | 4.0010 | 0.0500 | 4.0010 | 4.0010 | 4.8344 | 4.0010 | 0.9700 | 24 | 0.0300 | cnn_7466261 |
trial21 | 2.5018 | 1.0000 | 0.0500 | 4.0010 | 4.0010 | 11.0982 | 4.0010 | 0.9661 | 14 | 0.0339 | cnn_0522347 |
trial37 | 4.0010 | 1.0000 | 0.0500 | 4.0010 | 4.0010 | 4.4970 | 4.0010 | 0.9643 | 17 | 0.0357 | cnn_5051812 |
trial47 | 1.2830 | 2.0793 | 0.0500 | 2.1005 | 4.0010 | 7.0846 | 2.6164 | 0.9624 | 16 | 0.0376 | cnn_1950449 |
trial42 | 4.0010 | 1.9473 | 0.0500 | 4.0010 | 4.0010 | 5.0035 | 1.7908 | 0.9621 | 17 | 0.0379 | cnn_0833026 |
trial35 | 3.7624 | 3.9981 | 0.0500 | 3.1735 | 4.0010 | 1.8905 | 1.0849 | 0.9609 | 43 | 0.0391 | cnn_8772981 |
trial18 | 3.3421 | 4.0010 | 0.5000 | 4.0010 | 4.0010 | 15.3680 | 4.0010 | 0.9606 | 24 | 0.0394 | cnn_2551120 |
trial9 | 3.6915 | 2.3017 | 0.2560 | 3.1245 | 3.4552 | 2.4900 | 3.7501 | 0.9604 | 25 | 0.0396 | cnn_4124185 |
trial14 | 4.0010 | 4.0010 | 0.0500 | 2.0000 | 4.0010 | 16.0010 | 1.0000 | 0.9601 | 38 | 0.0399 | cnn_0094828 |
trial34 | 4.0010 | 3.6555 | 0.0500 | 2.0000 | 4.0010 | 14.8151 | 3.5020 | 0.9594 | 34 | 0.0406 | cnn_2174602 |
trial16 | 4.0010 | 1.0000 | 0.0500 | 2.0000 | 4.0010 | 16.0010 | 4.0010 | 0.9594 | 16 | 0.0406 | cnn_8600071 |
trial15 | 4.0010 | 1.0000 | 0.0500 | 4.0010 | 4.0010 | 1.0000 | 1.0000 | 0.9594 | 32 | 0.0406 | cnn_9779994 |
trial23 | 1.0000 | 1.0000 | 0.0500 | 4.0010 | 4.0010 | 5.1663 | 1.0000 | 0.9590 | 21 | 0.0410 | cnn_5389027 |
trial33 | 1.0000 | 4.0010 | 0.0500 | 2.4149 | 4.0010 | 8.8738 | 1.0000 | 0.9589 | 34 | 0.0411 | cnn_7401566 |
trial29 | 4.0010 | 3.2454 | 0.0500 | 3.9663 | 4.0010 | 13.4776 | 1.3420 | 0.9584 | 29 | 0.0416 | cnn_7782207 |
trial25 | 1.0000 | 4.0010 | 0.0500 | 2.0000 | 4.0010 | 16.0010 | 4.0010 | 0.9584 | 16 | 0.0416 | cnn_1088887 |
trial49 | 4.0010 | 1.3321 | 0.0500 | 3.7722 | 4.0010 | 13.0380 | 4.0010 | 0.9582 | 17 | 0.0418 | cnn_1277738 |
trial11 | 3.2915 | 2.2746 | 0.3246 | 3.1829 | 3.8379 | 8.6781 | 1.4269 | 0.9577 | 35 | 0.0423 | cnn_8345552 |
trial3 | 3.9304 | 2.9875 | 0.2520 | 2.5634 | 3.2125 | 2.5510 | 1.5212 | 0.9571 | 32 | 0.0429 | cnn_1375661 |
trial8 | 3.2834 | 3.8950 | 0.1270 | 2.9211 | 3.8746 | 8.5245 | 3.7471 | 0.9556 | 10 | 0.0444 | cnn_2874261 |
trial30 | 4.0010 | 1.0000 | 0.0500 | 2.0000 | 4.0010 | 5.4644 | 1.0000 | 0.9552 | 45 | 0.0448 | cnn_7897795 |
trial6 | 3.3851 | 1.5713 | 0.4678 | 3.0353 | 3.4220 | 13.1056 | 1.3557 | 0.9541 | 41 | 0.0459 | cnn_1827607 |
trial12 | 1.0000 | 1.0000 | 0.0500 | 2.0000 | 4.0010 | 1.0000 | 1.0654 | 0.9537 | 38 | 0.0463 | cnn_2804524 |
trial17 | 4.0010 | 4.0010 | 0.0534 | 2.0000 | 4.0010 | 1.0000 | 4.0010 | 0.9527 | 32 | 0.0473 | cnn_7216191 |
trial41 | 2.2563 | 3.4701 | 0.2850 | 2.0160 | 3.8976 | 4.3858 | 1.0014 | 0.9525 | 41 | 0.0475 | cnn_5766945 |
trial4 | 3.4131 | 3.8578 | 0.0629 | 2.8113 | 3.7743 | 7.3985 | 2.2179 | 0.9511 | 8 | 0.0489 | cnn_6586571 |
trial38 | 1.0324 | 3.7249 | 0.0763 | 3.8798 | 3.8725 | 10.5557 | 3.9936 | 0.9493 | 6 | 0.0507 | cnn_2038463 |
trial43 | 3.9163 | 3.9900 | 0.1196 | 2.0881 | 3.9042 | 3.3694 | 3.9648 | 0.9489 | 15 | 0.0511 | cnn_2407983 |
trial40 | 3.2168 | 1.4162 | 0.0536 | 2.2749 | 3.9170 | 12.1045 | 1.0488 | 0.9471 | 17 | 0.0529 | cnn_3200254 |
trial19 | 1.0000 | 2.0569 | 0.5000 | 2.0000 | 4.0010 | 5.4079 | 4.0010 | 0.9463 | 31 | 0.0537 | cnn_2272907 |
trial32 | 1.0338 | 3.2372 | 0.0557 | 3.1871 | 3.8629 | 2.6796 | 3.9573 | 0.9461 | 4 | 0.0539 | cnn_8375640 |
trial36 | 1.6279 | 1.2807 | 0.0650 | 4.0002 | 2.2150 | 3.1657 | 1.2277 | 0.9390 | 4 | 0.0610 | cnn_2177580 |
trial0 | 1.3710 | 3.5067 | 0.2024 | 3.3116 | 2.4735 | 7.7148 | 3.1912 | 0.9381 | 16 | 0.0619 | cnn_4534673 |
trial45 | 4.0010 | 4.0010 | 0.0500 | 4.0010 | 2.0000 | 16.0010 | 1.0000 | 0.9356 | 16 | 0.0644 | cnn_6611043 |
trial26 | 4.0010 | 1.0000 | 0.5000 | 4.0010 | 4.0010 | 15.9693 | 1.0000 | 0.9303 | 38 | 0.0697 | cnn_7027454 |
trial20 | 1.0000 | 1.9328 | 0.0500 | 4.0010 | 2.0000 | 1.4661 | 4.0010 | 0.9301 | 2 | 0.0699 | cnn_4494844 |
trial1 | 2.5818 | 1.9285 | 0.3844 | 3.0367 | 2.5969 | 8.3888 | 2.8445 | 0.9300 | 16 | 0.0700 | cnn_4664828 |
trial5 | 2.1604 | 1.6733 | 0.0673 | 3.9997 | 2.8536 | 14.7074 | 3.2052 | 0.9299 | 5 | 0.0701 | cnn_4679929 |
trial48 | 2.9484 | 1.0000 | 0.0500 | 4.0010 | 2.0000 | 10.9270 | 1.0000 | 0.9296 | 5 | 0.0704 | cnn_4994236 |
trial2 | 1.5486 | 1.5576 | 0.4545 | 2.0458 | 2.8034 | 13.4613 | 3.8463 | 0.9229 | 24 | 0.0771 | cnn_4649820 |
trial46 | 4.0010 | 1.0000 | 0.0500 | 2.0000 | 2.0000 | 16.0010 | 1.0000 | 0.9224 | 14 | 0.0776 | cnn_0834311 |
trial7 | 1.1423 | 1.3225 | 0.3457 | 2.4675 | 2.9744 | 12.7185 | 2.8241 | 0.9211 | 25 | 0.0789 | cnn_3976373 |
trial27 | 4.0010 | 4.0010 | 0.0500 | 2.0000 | 2.0000 | 16.0010 | 4.0010 | 0.9208 | 7 | 0.0792 | cnn_5418930 |
trial31 | 4.0010 | 4.0010 | 0.0500 | 4.0010 | 2.0000 | 1.0000 | 4.0010 | 0.9193 | 9 | 0.0807 | cnn_2041164 |
trial13 | 1.0000 | 4.0010 | 0.5000 | 4.0010 | 4.0010 | 1.0000 | 1.0305 | 0.9129 | 39 | 0.0871 | cnn_2932650 |
trial28 | 1.5008 | 4.0010 | 0.0500 | 2.0000 | 2.0000 | 1.0000 | 1.0000 | 0.9104 | 17 | 0.0896 | cnn_3575582 |
trial22 | 3.9836 | 1.2313 | 0.4435 | 2.1642 | 3.9619 | 1.1961 | 2.5067 | 0.9061 | 22 | 0.0939 | cnn_6597575 |
trial10 | 3.7392 | 2.7750 | 0.4556 | 2.4944 | 2.0160 | 6.6550 | 3.1538 | 0.8937 | 8 | 0.1063 | cnn_0709821 |
trial39 | 1.0000 | 4.0010 | 0.5000 | 2.0000 | 4.0010 | 16.0010 | 1.0000 | 0.8935 | 57 | 0.1065 | cnn_1597172 |
trial44 | 4.0010 | 4.0010 | 0.5000 | 2.6860 | 4.0010 | 11.1510 | 1.0000 | 0.8296 | 56 | 0.1704 | cnn_4295834 |
balanced accuracy score = 0.9939
accuracy score = 0.994
number of accepted models = 26 for threshold = 0.05



4. Code
import os import numpy as np import pandas as pd from bayes_opt import BayesianOptimization from keras.layers import Dense, Conv2D, BatchNormalization from keras.layers import MaxPooling2D from keras.layers import Input, Flatten, Dropout from keras.layers import Activation from keras.optimizers import Adam from keras.callbacks import ReduceLROnPlateau, EarlyStopping, ModelCheckpoint from keras.models import Model, load_model from keras.utils import to_categorical from sklearn.model_selection import train_test_split from sklearn.metrics import balanced_accuracy_score, accuracy_score from sklearn.metrics import confusion_matrix, classification_report import time from pathlib import Path import sys sys.path.append(YOUR DIRECTORY) import confusion_matrix_plot # ******************************************************************* # raw shapes: x:(num pts, 784), y:(num pts,) def get_data_mnist(data_type, data_dir): data_dictionary = {} if data_type == 'calibration': # x data - reshape = (60000, 28, 28, 1) - channels last x_calib_raw = np.load(data_dir + 'x_mnist_calibration.npy', allow_pickle=True) x_calib_raw = x_calib_raw.astype('float32')/255.0 # normalize x_calib = np.reshape(x_calib_raw, (x_calib_raw.shape[0], 28, 28, 1)) # y data - shape = (50000, ) -> (50000, 10) y_calib = to_categorical(np.load(data_dir + 'y_mnist_calibration.npy', allow_pickle=True)) # split x_calib1, x_calib2, y_calib1, y_calib2 = train_test_split(x_calib, y_calib, train_size=0.5, shuffle=True, stratify=y_calib) data_dictionary['x_calib1'] = x_calib1 data_dictionary['y_calib1'] = y_calib1 data_dictionary['x_calib2'] = x_calib2 data_dictionary['y_calib2'] = y_calib2 elif data_type == 'production': # x data - reshape = (10000, 28, 28, 1) - channels last x_prod_raw = np.load(data_dir + 'x_mnist_production.npy', allow_pickle=True) x_prod_raw = x_prod_raw.astype('float32')/255.0 # normalize x_prod = np.reshape(x_prod_raw, (x_prod_raw.shape[0], 28, 28, 1)) # y data - shape = (10000, ) y_prod = np.load(data_dir + 'y_mnist_production.npy', allow_pickle=True) data_dictionary['x_prod'] = x_prod data_dictionary['y_prod'] = y_prod else: raise NameError return data_dictionary # end of get_data_mnist() # ******************************************************************* def create_save_models_bayes_opt(xcalib, ycalib, num_random_points, num_iterations, results_dir, ml_algo_name): input_shape = xcalib.shape[1:] # (28, 28, 1) - channels last number_of_classes = ycalib.shape[1] # parameters for cnn that do NOT have to be saved maximum_epochs = 1000 early_stop_epochs = 10 learning_rate_epochs = 5 # parameters that change for each iteration that must be saved list_early_stop_epochs = [] list_validation_loss = [] list_saved_model_name = [] start_time_total = time.time() def cnn_function(num_cnn_blocks, num_filters, kernel_size, num_dense_nodes, dense_nodes_divisor, batch_size, drop_out): model_name = ml_algo_name + '_' + str(np.random.uniform(0,1,))[2:9] # variable parameters dict_params = {'num_cnn_blocks':int(num_cnn_blocks), 'num_filters':int(16*num_filters), 'kernel_size':int(kernel_size), 'num_dense_nodes':int(64*num_dense_nodes), 'dense_nodes_divisor':int(2*dense_nodes_divisor), 'batch_size':int(32*batch_size), 'drop_out':drop_out} # start of cnn coding input_tensor = Input(shape=input_shape) # 1st cnn block x = BatchNormalization()(input_tensor) x = Activation('relu')(x) x = Conv2D(filters=dict_params['num_filters'], kernel_size=dict_params['kernel_size'], strides=1, padding='same')(x) #x = MaxPooling2D()(x) x = Dropout(dict_params['drop_out'])(x) # additional cnn blocks for iblock in range(dict_params['num_cnn_blocks'] - 1): x = BatchNormalization()(x) x = Activation('relu')(x) x = Conv2D(filters=dict_params['num_filters'], kernel_size=dict_params['kernel_size'], strides=1, padding='same')(x) x = MaxPooling2D()(x) x = Dropout(dict_params['drop_out'])(x) # mlp x = Flatten()(x) x = Dense(dict_params['num_dense_nodes'], activation='relu')(x) x = Dropout(dict_params['drop_out'])(x) x = Dense(dict_params['num_dense_nodes']//dict_params['dense_nodes_divisor'], activation='relu')(x) output_tensor = Dense(number_of_classes, activation='softmax')(x) # instantiate and compile model cnn_model = Model(inputs=input_tensor, outputs=output_tensor) opt = Adam(lr=0.00025) # default = 0.001 cnn_model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy']) # callbacks for early stopping and for learning rate reducer callbacks_list = [EarlyStopping(monitor='val_loss', patience=early_stop_epochs), ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=learning_rate_epochs, verbose=0, mode='auto', min_lr=1.0e-6), ModelCheckpoint(filepath=results_dir + model_name + '.h5', monitor='val_loss', save_best_only=True)] # fit the model h = cnn_model.fit(x=xcalib, y=ycalib, batch_size=dict_params['batch_size'], epochs=maximum_epochs, validation_split=0.25, shuffle=True, verbose=0, callbacks=callbacks_list) # record actual best epochs and valid loss here, added to bayes opt parameter df below list_early_stop_epochs.append(len(h.history['val_loss']) - early_stop_epochs) validation_loss = np.min(h.history['val_loss']) # h.history['val_loss'] list_validation_loss.append(validation_loss) list_saved_model_name.append(model_name) # bayes opt is a maximization algorithm, to minimize validation_loss, return 1-this bayes_opt_score = 1.0 - validation_loss return bayes_opt_score # end of cnn_function() optimizer = BayesianOptimization(f=cnn_function, pbounds={'num_cnn_blocks':(2, 4.001), 'num_filters':(1, 4.001), # *16 'kernel_size':(2, 4.001), 'num_dense_nodes':(1, 16.001), # *64 'dense_nodes_divisor':(1, 4.001), # *2 'batch_size':(1, 4.001), # *32 'drop_out': (0.05, 0.5)}, verbose=2) optimizer.maximize(init_points=num_random_points, n_iter=num_iterations) print('nbest result:', optimizer.max) elapsed_time_total = (time.time()-start_time_total)/60 print('\n\ntotal elapsed time =',elapsed_time_total,' minutes') # optimizer.res is a list of dicts list_dfs = [] counter = 0 for result in optimizer.res: df_temp = pd.DataFrame.from_dict(data=result['params'], orient='index', columns=['trial' + str(counter)]).T df_temp['bayes opt error'] = result['target'] df_temp['epochs'] = list_early_stop_epochs[counter] df_temp['validation_loss'] = list_validation_loss[counter] df_temp['model_name'] = list_saved_model_name[counter] list_dfs.append(df_temp) counter = counter + 1 df_results = pd.concat(list_dfs, axis=0) df_results.to_pickle(results_dir + 'df_bayes_opt_results_parameters.pkl') df_results.to_csv(results_dir + 'df_bayes_opt_results_parameters.csv') # end of create_save_models_bayes_opt() # ******************************************************************* def make_final_predictions(xprod, yprod, list_class_names, models_directory, save_directory, df_params, threshold, ml_name): # apply threshold accepted_models_num = 0 list_predicted_prob = [] num_models = df_params.shape[0] for i in range(num_models): if df_params.loc[df_params.index[i],'validation_loss'] < threshold: model_name = df_params.loc[df_params.index[i],'model_name'] model_final = load_model(models_directory + model_name + '.h5') # these are class probabilities list_predicted_prob.append(model_final.predict(xprod)) accepted_models_num = accepted_models_num + 1 # compute mean probabilities mean_probabilities = np.mean(list_predicted_prob, axis=0) # compute predicted class # argmax uses 1st ocurrance in case of a tie y_predicted_class = np.argmax(mean_probabilities, axis=1) # compute and save error measures # print info to file stdout_default = sys.stdout sys.stdout = open(save_directory + ml_name + '_prediction_results.txt','w') print('balanced accuracy score =',balanced_accuracy_score(yprod, y_predicted_class)) print('accuracy score =',accuracy_score(yprod, y_predicted_class)) print('number of accepted models =',accepted_models_num,' for threshold =',threshold) print('\nclassification report:') print(classification_report(yprod, y_predicted_class, digits=3, output_dict=False)) print('\nraw confusion matrix:') cm_raw = confusion_matrix(yprod, y_predicted_class) print(cm_raw) print('\nconfusion matrix normalized by prediction:') cm_pred = confusion_matrix(yprod, y_predicted_class, normalize='pred') print(cm_pred) print('\nconfusion matrix normalized by true:') cm_true = confusion_matrix(yprod, y_predicted_class, normalize='true') print(cm_true) sys.stdout = stdout_default # plot and save confustion matrices figure_size = (12, 8) number_of_decimals = 4 confusion_matrix_plot.confusion_matrix_save_and_plot(cm_raw, list_class_names, save_directory, 'Confusion Matrix', ml_name + '_confusion_matrix', False, None, 30, figure_size, number_of_decimals) confusion_matrix_plot.confusion_matrix_save_and_plot(cm_pred, list_class_names, save_directory, 'Confusion Matrix Normalized by Prediction', ml_name + '_confusion_matrix_norm_by_prediction', False, 'pred', 30, figure_size, number_of_decimals) confusion_matrix_plot.confusion_matrix_save_and_plot(cm_true, list_class_names, save_directory, 'Confusion Matrix Normalized by Actual', ml_name + '_confusion_matrix_norm_by_true', False, 'true', 30, figure_size, number_of_decimals) # end of make_final_predictions() # ******************************************************************* if __name__ == '__main__': ml_algorithm_name = 'cnn' dataset_name = 'mnist' file_name_stub = ml_algorithm_name + '_' + dataset_name + '_bayes_opt' calculation_type = 'production' #'calibration' 'production' base_directory = YOUR DIRECTORY data_directory = base_directory + 'data/' results_directory_stub = base_directory + file_name_stub + '/' if not Path(results_directory_stub).is_dir(): os.mkdir(results_directory_stub) # fixed parameters threshold_validation_loss = 0.05 total_number_of_iterations = 50 number_of_random_points = 12 # random searches to start opt process # this is # of bayes iters, thus total=this + # of random pts number_of_iterations = total_number_of_iterations - number_of_random_points # read data dict_data = get_data_mnist(calculation_type, data_directory) print('\n*** starting at',pd.Timestamp.now()) # calibration if calculation_type == 'calibration': # using 1/2 of calib data x_calib = dict_data['x_calib1'] y_calib = dict_data['y_calib1'] results_directory = results_directory_stub + calculation_type + '/' if not Path(results_directory).is_dir(): os.mkdir(results_directory) create_save_models_bayes_opt(x_calib, y_calib, number_of_random_points, number_of_iterations, results_directory, ml_algorithm_name) # production elif calculation_type == 'production': # get saved parameters models_dir = results_directory_stub + 'calibration/' df_parameters = pd.read_pickle(models_dir + 'df_bayes_opt_results_parameters.pkl') results_directory = results_directory_stub + calculation_type + '/' if not Path(results_directory).is_dir(): os.mkdir(results_directory) x_prod = dict_data['x_prod'] y_prod = dict_data['y_prod'] num_classes = np.unique(y_prod).shape[0] class_names_list = [] for i in range(num_classes): class_names_list.append('class ' + str(i)) make_final_predictions(x_prod, y_prod, class_names_list, models_dir, results_directory, df_parameters, threshold_validation_loss, ml_algorithm_name) else: print('\ninvalid calculation type:',calculation_type) raise NameError