Previous articles in this series:

  1. Multilayer Perceptrons
  2. MLP Models
  3. Results
  4. Code

1. Multilayer Perceptrons

We use a random hyperparameter search along with cross validation to create multiple 2 hidden layer neural networks. The final ensemble is created by choosing the 10 best models and taking the median predicted output. As our target is win fraction, we convert to wins and losses to compare to those of the 2018-2019 NBA season. Our goal is to use underlying statistics, in this case we use per game data, to determine if a team has over or under performed.

For a detailed description of the data, see the two articles referenced above. Briefly, we used per game offensive and defensive data from the 2008-2009 season to the 2018-2019 season from Basketball Reference. Both a standard scaler and minmax scaler are used to create two separate datasets. Note that scaling is performed for each individual season.

2. MLP Models

Below are the hyperparameters of the 10 best models.
 

Standard Scaler MLP Hyperparameters

optimizerlayer_2_nodes_numlayer_1_nodes_numl2_regularizerl1_regularizerdropoutbatch_sizeactivationModel NameMetricMean CV ErrorMax CV Epochs
Adagrad17340.000100.3532elumlp_standard_scaler_27mean_squared_error0.001153334662
Adagrad17170.000100.2532elumlp_standard_scaler_0mean_squared_error0.0011711058926
Adagrad34340.000100.332elumlp_standard_scaler_162mean_squared_error0.0011747929827
Adagrad17340.000100.4532elumlp_standard_scaler_10mean_squared_error0.0013440113788
Adagrad17680.000100.2564elumlp_standard_scaler_114mean_squared_error0.0013493246570
Adagrad17170.00010064linearmlp_standard_scaler_154mean_squared_error0.00135397131149
Adagrad343400.00010.432linearmlp_standard_scaler_144mean_squared_error0.0013580421641
Adagrad17340.00010.0001032linearmlp_standard_scaler_135mean_squared_error0.00137742361388
Adagrad17170.00100.564linearmlp_standard_scaler_14mean_squared_error0.00138055991625
Adam34340.00100.364linearmlp_standard_scaler_13mean_squared_error0.0013824059757

 

MinMax Scaler MLP Hyperparameters

optimizerlayer_2_nodes_numlayer_1_nodes_numl2_regularizerl1_regularizerdropoutbatch_sizeactivationModel NameMetricMean CV ErrorMax CV Epochs
Adagrad176800.00010.532relumlp_minmax_scaler_119mean_squared_error0.00130499111990
Adam17170.000100.232relumlp_minmax_scaler_114mean_squared_error0.0013332827320
Adam17170.00100.4532relumlp_minmax_scaler_60mean_squared_error0.0013726814462
Adam17680.00100.232relumlp_minmax_scaler_115mean_squared_error0.0014109208422
Adagrad17170.00100.132relumlp_minmax_scaler_207mean_squared_error0.00151300021794
Adam17170.000100.3532relumlp_minmax_scaler_243mean_squared_error0.0015283889324
Adam68680.0010.00010.264relumlp_minmax_scaler_96mean_squared_error0.0015425326825
Adam34340.000100.3532elumlp_minmax_scaler_175mean_squared_error0.0015479742532
Adam34680.0010.00010.164relumlp_minmax_scaler_178mean_squared_error0.00155862628
Nadam34680.00010.00010.164relumlp_minmax_scaler_233mean_squared_error0.0015693607628

 

3. Results

Standard Scaler MLP Errors

Model NameMean Squared ErrorMedian Absolute Error
mlp_standard_scaler_270.0021510.037584
mlp_standard_scaler_00.0020590.032815
mlp_standard_scaler_1620.0019820.032537
mlp_standard_scaler_100.0016870.033868
mlp_standard_scaler_1140.0018320.033277
mlp_standard_scaler_1540.0017940.024584
mlp_standard_scaler_1440.0018570.031286
mlp_standard_scaler_1350.0020990.029827
mlp_standard_scaler_140.0016280.028434
mlp_standard_scaler_130.0016860.027493
Ensemble Median0.0018250.028304

 

MinMax Scaler MLP Errors

Model NameMean Squared ErrorMedian Absolute Error
mlp_minmax_scaler_1190.0024100.027469
mlp_minmax_scaler_1140.0041900.037902
mlp_minmax_scaler_600.0035750.036797
mlp_minmax_scaler_1150.0023350.034938
mlp_minmax_scaler_2070.0023050.030048
mlp_minmax_scaler_2430.0033670.041560
mlp_minmax_scaler_960.0030470.029131
mlp_minmax_scaler_1750.0014170.030938
mlp_minmax_scaler_1780.0024360.037883
mlp_minmax_scaler_2330.0027330.033085
Ensemble Median0.0022450.034716

 

Standard Scaler Ensemble Results: 2018-2019 Season

TeamWinsLossesWin FractionPredicted Win FractionPredicted WinsPredicted LossesActual - Predicted Wins
Atlanta Hawks29530.3540.28924585
Boston Celtics49330.5980.6625428-5
Brooklyn Nets42400.5120.47539433
Charlotte Hornets39430.4760.47339430
Chicago Bulls22600.2680.24420622
Cleveland Cavaliers19630.2320.22018641
Dallas Mavericks33490.4020.4693844-5
Denver Nuggets54280.6590.64052302
Detroit Pistons41410.5000.44837454
Golden State Warriors57250.6950.7316022-3
Houston Rockets53290.6460.64553290
Indiana Pacers48340.5850.6405230-4
Los Angeles Clippers48340.5850.48740428
Los Angeles Lakers37450.4510.43135472
Memphis Grizzlies33490.4020.4283547-2
Miami Heat39430.4760.5284339-4
Milwaukee Bucks60220.7320.7996616-6
Minnesota Timberwolves36460.4390.4944141-5
New Orleans Pelicans33490.4020.4553745-4
New York Knicks17650.2070.20417650
Oklahoma City Thunder49330.5980.57247352
Orlando Magic42400.5120.50041411
Philadelphia 76ers51310.6220.56446365
Phoenix Suns19630.2320.20016663
Portland Trail Blazers53290.6460.62851312
Sacramento Kings39430.4760.45737452
San Antonio Spurs48340.5850.56546362
Toronto Raptors58240.7070.68756262
Utah Jazz50320.6100.6545428-4
Washington Wizards32500.3900.3973349-1


 

MinMax Scaler Ensemble Results: 2018-2019 Season

TeamWinsLossesWin FractionPredicted Win FractionPredicted WinsPredicted LossesActual - Predicted Wins
Atlanta Hawks29530.3540.24420629
Boston Celtics49330.5980.6375230-3
Brooklyn Nets42400.5120.45337455
Charlotte Hornets39430.4760.38331518
Chicago Bulls22600.2680.22518644
Cleveland Cavaliers19630.2320.22518641
Dallas Mavericks33490.4020.4233547-2
Denver Nuggets54280.6590.64553291
Detroit Pistons41410.5000.44737454
Golden State Warriors57250.6950.7486121-4
Houston Rockets53290.6460.6795626-3
Indiana Pacers48340.5850.6345230-4
Los Angeles Clippers48340.5850.50742406
Los Angeles Lakers37450.4510.37030527
Memphis Grizzlies33490.4020.37931512
Miami Heat39430.4760.47839430
Milwaukee Bucks60220.7320.71759231
Minnesota Timberwolves36460.4390.4493745-1
New Orleans Pelicans33490.4020.39432501
New York Knicks17650.2070.2231864-1
Oklahoma City Thunder49330.5980.57947352
Orlando Magic42400.5120.47639433
Philadelphia 76ers51310.6220.61951310
Phoenix Suns19630.2320.22218641
Portland Trail Blazers53290.6460.61450323
Sacramento Kings39430.4760.40333496
San Antonio Spurs48340.5850.6225131-3
Toronto Raptors58240.7070.70157251
Utah Jazz50320.6100.6795626-6
Washington Wizards32500.3900.34829533


 

4. Code

Import modules.

import os
import pandas as pd
import numpy as np
import time
from pathlib import Path
from keras.layers import Input, Dense
from keras.layers import Dropout
from keras.models import Model
from keras.models import load_model
from keras.callbacks import EarlyStopping
from keras import regularizers
from sklearn.model_selection import ParameterGrid
from sklearn.model_selection import ParameterSampler
from sklearn.metrics import mean_squared_error
from sklearn.metrics import median_absolute_error

Generate random hyperparameters.

def parameters_mlp(grid_type, num_samples, num_inputs):    
    batch_size = [32, 64]
    dropout = np.linspace(0.0, 0.5, num=11, endpoint=True)
    activation = ['relu','elu','linear']
    optimizer = ['Adam','Nadam','Adadelta','Adagrad']
    l1_regularizer = [0.0, 0.0001, 0.001, 0.01]
    l2_regularizer = [0.0, 0.0001, 0.001, 0.01]    
    layer_1_nodes_num = [int(num_inputs/2), num_inputs, 2*num_inputs]
    layer_2_nodes_num = layer_1_nodes_num
        
    grid = {'batch_size':batch_size, 'dropout':dropout, 'activation':activation, 
            'optimizer':optimizer, 
            #'learning_rate':learning_rate,
            'l1_regularizer':l1_regularizer, 'l2_regularizer':l2_regularizer,
            'layer_1_nodes_num':layer_1_nodes_num, 'layer_2_nodes_num':layer_2_nodes_num}
   
    if grid_type == 'deterministic':
        parameter_grid = list(ParameterGrid(grid))
    elif grid_type == 'stochastic':
        parameter_grid = list(ParameterSampler(grid, n_iter=num_samples))
    else:
        print('\ninvalid grid_type in parameters():',grid_type)
        raise NameError
        
    return parameter_grid

Construct the MLP.

def mlp_model_construction(dict_params_mlp, num_inputs, num_outputs, loss_type):
    metric = loss_type
                 
    l1_reg = dict_params_mlp['l1_regularizer']
    l2_reg = dict_params_mlp['l2_regularizer']
             
    input_tensor = Input(shape=(num_inputs,))
    
    layer_1_output = Dense(dict_params_mlp['layer_1_nodes_num'], 
                           activation=dict_params_mlp['activation'],
                           kernel_regularizer=regularizers.l1_l2(l1=l1_reg, l2=l2_reg),
                     activity_regularizer=regularizers.l1_l2(l1=l1_reg, l2=l2_reg))(input_tensor)
    num_nodes_layer_2=min(dict_params_mlp['layer_2_nodes_num'],dict_params_mlp['layer_1_nodes_num'])
    dict_params_mlp['layer_2_nodes_num'] = num_nodes_layer_2
    layer_2_output = Dense(num_nodes_layer_2, 
                           activation=dict_params_mlp['activation'],
                           kernel_regularizer=regularizers.l1_l2(l1=l1_reg, l2=l2_reg),
                     activity_regularizer=regularizers.l1_l2(l1=l1_reg, l2=l2_reg))(layer_1_output)
        
    output_tensor = Dense(num_outputs)(layer_2_output)
       
    model_mlp = Model(input_tensor, output_tensor)
     
    model_mlp.compile(optimizer=dict_params_mlp['optimizer'], loss=loss_type, metrics=[metric])

    return model_mlp

Apply cross validation to a given MLP, only save if the mean cross validation mean squared error is less than the error threshold and refit using all calibration data. Return cv error and the maximum epochs (we use early stopping).

def nn_cv(x_calib, y_calib, cv_folds, metric,
          nn_model, nn_model_copy, batchsize, num_epochs, stop_epochs, verbosity,
          models_dir, model_name, threshold):
      
    callbacks_list = [EarlyStopping(monitor='val_loss', patience=stop_epochs)]
    
    cv_num = int(x_calib.shape[0]/cv_folds)
    list_cv_metrics = []
    list_cv_epochs = []
    for fold in range(cv_folds):
        # get train/valid
        x_train = np.vstack((x_calib[:cv_num*fold], x_calib[cv_num*(fold+1):]))        
        y_train = np.hstack((y_calib[:cv_num*fold], y_calib[cv_num*(fold+1):]))
         
        x_valid = x_calib[cv_num*fold:cv_num*(fold+1)]
        y_valid = y_calib[cv_num*fold:cv_num*(fold+1)]
        
        # fit the model
        h = nn_model.fit(x_train, y_train, epochs=num_epochs, 
                         batch_size=batchsize, validation_data=(x_valid, y_valid), 
                         callbacks=callbacks_list, verbose=verbosity)
     
        # collect cv stats
        list_cv_metrics.append(np.min(h.history['val_' + metric]))
        epoch_best_early_stop = len(h.history['val_loss']) - stop_epochs
        list_cv_epochs.append(epoch_best_early_stop)
                 
    dict_return = {}
    mean_cv_error = np.mean(list_cv_metrics)
    max_cv_epochs = np.max(list_cv_epochs)
    if mean_cv_error < threshold:
        # fit the model (use copy) with all calib data
        h_refit = nn_model_copy.fit(x_calib, y_calib, epochs=max_cv_epochs, 
                                    batch_size=batchsize, verbose=0)
        nn_model_copy.save(models_dir + model_name + '.h5')
        dict_return['Model Name'] = model_name
        dict_return['Metric'] = metric
        dict_return['Mean CV Error'] = mean_cv_error
        dict_return['Max CV Epochs'] = max_cv_epochs
     
    return dict_return

Read in the best num_models, apply production data (2018-2019 season), compute median output.

def production_results(models_dir, results_dir, num_models,
                       df_prod, x_prod, y_prod):
    
    dfparamresults = pd.read_pickle(models_dir + 'df_parameter_results.pkl')
    if dfparamresults.shape[0] > 0:
        dfparamresults.sort_values(by='Mean CV Error', ascending=True, inplace=True)
        num_models = min(num_models, dfparamresults.shape[0])
        
        if num_models > 0:
            dfparamresults = dfparamresults[:num_models]
                        
            df_production_errors = pd.DataFrame(index=['Mean Squared Error', 
                                                       'Median Absolute Error'])
            # get results
            list_predictions = []
            for i in range(num_models):
                model_name = dfparamresults.loc[dfparamresults.index[i],'Model Name']
                
                model_final = load_model(models_dir + model_name + '.h5')
                   
                y_prod_predict = model_final.predict(x_prod)
                list_predictions.append(y_prod_predict)
                
                mse = mean_squared_error(y_prod, y_prod_predict)
                mae = median_absolute_error(y_prod, y_prod_predict)
                df_production_errors[model_name] = [mse, mae]
                                
            # compute ensemble info 
            print('\nlist_predictions shape =',len(list_predictions),len(list_predictions[0]))
            array_predictions = np.hstack((list_predictions))
            print('\narray_predictions shape =',array_predictions.shape)
            median_prediction = np.median(array_predictions, axis=1)
            print('\ny_prod shape =',y_prod.shape,'\tmedian_prediction shape =',median_prediction.shape)
            mse = mean_squared_error(y_prod, median_prediction)
            mae = median_absolute_error(y_prod, median_prediction)
            df_production_errors['Ensemble Median'] = [mse, mae]
            df_production_errors.to_pickle(results_dir + 'df_production_errors_ensemble.pkl')
            df_production_errors.to_csv(results_dir + 'df_production_errors_ensemble.csv')
                    
            # all predicted results are ensemble median
            df_production_results = pd.DataFrame(data=df_prod.values, index=df_prod.index, 
                                                 columns=df_prod.columns)
            games = df_production_results['Wins'].values + df_production_results['Losses'].values
            df_production_results['Predicted Win Fraction'] = median_prediction
            predicted_wins = np.around(median_prediction*games, decimals=0)
            df_production_results['Predicted Wins'] = predicted_wins
            df_production_results['Predicted Losses'] = games - predicted_wins
            actual_wins = df_production_results['Wins'].values
            df_production_results['Actual - Predicted Wins'] = actual_wins - predicted_wins
            df_production_results.to_pickle(results_dir + 'df_production_results_ensemble.pkl')
            df_production_results.to_csv(results_dir + 'df_production_results_ensemble.csv')

Code for getting data. Main code for calibration and production. Note that the maximum possible number of models created is set by size_of_parameter_list but max_time_minutes limits the total run time.

def get_data(data_dir, type_of_data):
    
    dictionary_data = {}
    
    dictionary_data['x_calibration'] = np.load(data_dir + 'x_calibration_' + type_of_data + '.npy')
    dictionary_data['y_calibration'] = np.load(data_dir + 'y_calibration_' + type_of_data + '.npy')
    
    dictionary_data['x_production'] = np.load(data_dir + 'x_production_' + type_of_data + '.npy')
    dictionary_data['y_production'] = np.load(data_dir + 'y_production.npy')
    
    return dictionary_data

# end of get_data()
    
# *******************************************************************
    
if __name__ == '__main__':
        
    production_season = '2018_2019'
    
    calculation_type = 'production'  # production cv
        
    list_data_type = ['standard_scaler', 'minmax_scaler']
    
    base_directory = YOUR DIRECTORY
    data_directory = base_directory + 'team_per_game_data/'    
      
    # limits and params
    metric_type = 'mean_squared_error'
    verbose = 0
    max_time_minutes = 200 
    error_threshold = 0.002  # 0.0015
    epochs = 2000
    cross_valid_folds = 3
    checkpoint_epochs_stop = 10
    size_of_parameter_list = 1000
    number_of_models = 10

    # loops over types
    nn_type = 'mlp'
    for data_type in list_data_type:
        
        dict_data = get_data(data_directory, data_type)
        xcalib = dict_data['x_calibration']
        ycalib = dict_data['y_calibration']
        
        try:
            number_of_outputs = ycalib.shape[1]
        except:
            number_of_outputs = 1  # ycalib.shape[1]
            
        number_of_inputs = xcalib.shape[1]
        
        if calculation_type == 'cv':
            
            models_directory = base_directory + 'models_' + nn_type + '_' + data_type + '/'
            if not Path(models_directory).is_dir():
                os.mkdir(models_directory)
                
            # set up parameters
            if nn_type == 'mlp':
                list_param_dict = parameters_mlp('stochastic', 
                                  size_of_parameter_list, number_of_inputs)
            
            print('\n\n*** starting', nn_type,' ',data_type)
            print('at',pd.Timestamp.now())
            start_time = time.time()
            # loop over parameters
            list_dict_results = []
            counter = 0
            for dictionary_parameter in list_param_dict:   
                
                modelname = nn_type + '_' + data_type + '_' + str(counter)
                counter = counter + 1
                
                model_nn = mlp_model_construction(dictionary_parameter, number_of_inputs, 
                                                  number_of_outputs, metric_type)
                                    
                dict_nn_results = nn_cv(xcalib, ycalib, cross_valid_folds, metric_type,
                                        model_nn, model_nn, 
                                        dictionary_parameter['batch_size'], epochs, 
                                        checkpoint_epochs_stop, verbose,
                                        models_directory, modelname, error_threshold)
                                
                if len(dict_nn_results) > 0:  # add epochs and modelname to param dict and save it
                    dictionary_parameter.update(dict_nn_results)
                    list_dict_results.append(dictionary_parameter)
                    
                # check elapsed time for early stopping
                elapsed_time_minutes = (time.time() - start_time)/60
                if elapsed_time_minutes > max_time_minutes:
                    break
            
            print('\n\n*** ending', nn_type,' ',data_type)
            print('at',pd.Timestamp.now())
            print('elapsed time =',(time.time()-start_time)/60,' min')
            print('counter =',counter)
            
            # collect results in a df, then save
            df_param_results = pd.DataFrame(data=list_dict_results)
            df_param_results.to_pickle(models_directory + 'df_parameter_results.pkl')
            df_param_results.to_csv(models_directory + 'df_parameter_results.csv')
            list_dict_results.clear()

            
        elif calculation_type == 'production':
            
            df_prod = pd.read_pickle(data_directory + 'dfwinloss_' + production_season + '.pkl')
            
            xprod = dict_data['x_production']
            yprod = dict_data['y_production']
                
            models_directory = base_directory + 'models_' + nn_type + '_' + data_type + '/'
            
            results_directory = models_directory + 'results/'
            if not Path(results_directory).is_dir():
                os.mkdir(results_directory)
                
            production_results(models_directory, results_directory, number_of_models,
                               df_prod, xprod, yprod)
            
        else:
            raise NameError

Pin It on Pinterest