'val_acc' not available while running 'train_single_model.ipynb'

[abhiray92]

2020-11-11 18:45:17,086 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 18:45:17,296 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 18:45:17,297 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
H1 {'loss': [1.3797682523727417, 0.6599361300468445, 0.6564924120903015, 0.5561164021492004, 0.4747893810272217, 0.3267454504966736, 0.3100120425224304, 0.1791907399892807, 0.29043635725975037, 0.1876538246870041, 0.1521572321653366, 0.22007215023040771, 0.06935647875070572, 0.09439448267221451, 0.20947016775608063, 0.27807852625846863, 0.15932726860046387, 0.11780016869306564, 0.09338890016078949, 0.13629283010959625], 'accuracy': [0.5182186365127563, 0.6396760940551758, 0.6518218517303467, 0.71659916639328, 0.7773279547691345, 0.8663967847824097, 0.8704453706741333, 0.931174099445343, 0.8866396546363831, 0.9271255135536194, 0.9392712712287903, 0.898785412311554, 0.9797570705413818, 0.9554656147956848, 0.9271255135536194, 0.8866396546363831, 0.9433198571205139, 0.9433198571205139, 0.9554656147956848, 0.9473684430122375], 'val_loss': [0.9056219458580017, 0.8241466879844666, 0.7785531282424927, 0.5593583583831787, 0.7367925047874451, 0.8970717191696167, 0.5637212991714478, 0.6786322593688965, 0.9303268790245056, 0.7068821787834167, 1.7561753988265991, 1.0093439817428589, 0.7377994060516357, 0.8516809344291687, 0.9103375673294067, 0.982994794845581, 1.294120192527771, 1.17690110206604, 0.7544564008712769, 0.7987895607948303], 'val_accuracy': [0.5032258033752441, 0.5032258033752441, 0.5032258033752441, 0.7096773982048035, 0.6516128778457642, 0.6903225779533386, 0.7225806713104248, 0.7096773982048035, 0.6838709712028503, 0.7612903118133545, 0.6709677577018738, 0.7225806713104248, 0.7870967984199524, 0.7806451320648193, 0.6193548440933228, 0.7225806713104248, 0.7032257914543152, 0.7483870983123779, 0.7870967984199524, 0.7354838848114014]}
stopped_epoch1 19
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-10-eae6ea118bfc> in <module>
     15                             verbose=True)
     16 
---> 17 architecture.train_model()
     18 
     19 gc.collect()

D:\***\Deep-Neural-Networks-for-Video-Classification\deepvideoclassification\architectures.py in train_model(self, epochs, patience)
    928         print('H1', history1.history)
    929         print('stopped_epoch1',stopped_epoch1)
--> 930         print(len(history1.history['val_acc']))
    931         print(history1.history['val_acc'][stopped_epoch1])
    932 

KeyError: 'val_acc'

[alxcnwy]

weird. can you please print history1 and post the output here?

[abhiray92]

I tried printing 'history1', however, I believe the problem is with the 'fit' method. 'history1' cannot be printed, because it throws the exception in the 'fit' method itself.

[alxcnwy]

the error you posted above is a missing key in the history dictionary.

Can you post the fit exception?

[abhiray92]

This is the complete error -

2020-11-11 19:12:20,777 [MainThread  ] [INFO ]  Loading data
2020-11-11 19:12:20,817 [MainThread  ] [INFO ]  Features already cached: D:\Theses\Deep-Neural-Networks-for-Video-Classification\cache/features/resnet50/max/
2020-11-11 19:12:20,817 [MainThread  ] [INFO ]  Loading features sequence data into memory [may take a few minutes]
1   XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
{'architecture': 'video_lrcnn_frozen', 'dropout': 0.2, 'layer_1_size': 256, 'layer_2_size': 512, 'layer_3_size': 256, 'model_id': 1, 'pooling': 'max', 'pretrained_model_name': 'resnet50', 'sequence_length': 20, 'sequence_model': 'LSTM', 'sequence_model_layers': 2}
Done initializing data with #samples: train=247, valid=155, test=266
2020-11-11 19:12:24,753 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:24,754 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:25,004 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:25,005 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:25,281 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:25,282 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:25,557 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:25,558 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:25,833 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:25,834 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:26,070 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:26,071 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:26,288 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:26,289 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:26,497 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:26,497 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:26,699 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:26,700 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:26,903 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:26,904 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:27,102 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:27,103 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:27,318 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:27,318 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:27,535 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:27,535 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:27,752 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:27,753 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:27,951 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:27,952 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:28,155 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:28,156 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:28,399 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:28,403 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:28,667 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:28,668 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:28,926 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:28,927 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
2020-11-11 19:12:29,226 [MainThread  ] [WARNI]  Early stopping conditioned on metric `val_acc` which is not available. Available metrics are: loss,accuracy,val_loss,val_accuracy
2020-11-11 19:12:29,227 [MainThread  ] [WARNI]  Can save best model only with val_acc available, skipping.
H1 {'loss': [1.1680704355239868, 0.6717468500137329, 0.5939275622367859, 0.46642377972602844, 0.36455848813056946, 0.2543645203113556, 0.22103974223136902, 0.1652027815580368, 0.10433871299028397, 0.07396731525659561, 0.07385416328907013, 0.08015889674425125, 0.08559573441743851, 0.14561280608177185, 0.2681539058685303, 0.1664297878742218, 0.08188217878341675, 0.11118330806493759, 0.10880503058433533, 0.07352413237094879], 'accuracy': [0.5668016076087952, 0.6153846383094788, 0.692307710647583, 0.7894737124443054, 0.862348198890686, 0.9149797558784485, 0.9230769276618958, 0.9352226853370667, 0.9554656147956848, 0.9676113128662109, 0.9757084846496582, 0.9635627269744873, 0.9635627269744873, 0.9392712712287903, 0.8906882405281067, 0.9473684430122375, 0.9676113128662109, 0.9716598987579346, 0.9514170289039612, 0.9797570705413818], 'val_loss': [1.0502870082855225, 0.6980264186859131, 1.1717759370803833, 0.5694771409034729, 1.0995045900344849, 0.5141857266426086, 0.5460548400878906, 0.6494520902633667, 0.7877650856971741, 1.367462396621704, 0.9089471697807312, 1.142378330230713, 1.4670153856277466, 2.3518826961517334, 0.9998852014541626, 1.3017445802688599, 1.0129077434539795, 1.4146616458892822, 1.0920008420944214, 1.041746735572815], 'val_accuracy': [0.5032258033752441, 0.5032258033752441, 0.5032258033752441, 0.7225806713104248, 0.625806450843811, 0.7935484051704407, 0.774193525314331, 0.7806451320648193, 0.7612903118133545, 0.774193525314331, 0.7419354915618896, 0.7806451320648193, 0.7096773982048035, 0.6387096643447876, 0.6580645442008972, 0.6387096643447876, 0.7354838848114014, 0.6645161509513855, 0.7161290049552917, 0.7935484051704407]}
stopped_epoch1 19
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-24-eae6ea118bfc> in <module>
     15                             verbose=True)
     16 
---> 17 architecture.train_model()
     18 
     19 gc.collect()

D:\Theses\Deep-Neural-Networks-for-Video-Classification\deepvideoclassification\architectures.py in train_model(self, epochs, patience)
    928         print('H1', history1.history)
    929         print('stopped_epoch1',stopped_epoch1)
--> 930         print(len(history1.history['val_acc']))
    931         print(history1.history['val_acc'][stopped_epoch1])
    932 

KeyError: 'val_acc'

[abhiray92]

I guess it isn't able to get the validation data. I hope this link helps -

https://stackoverflow.com/questions/49035200/keras-early-stopping-callback-error-val-loss-metric-not-available/56490322

[alxcnwy]

The error is that there's no val_acc key on the history object for some reason.

I unfortunately don't have time to fix this for you - my advice would be to edit that line to:

print(history1)

Cheers

[abhiray92]

Hi Alex, I understand. Well, I tried 'print(history1', but it didn't work. However, now I am getting the below error -

2020-11-11 20:36:44,440 [MainThread  ] [INFO ]  Model folder exists but no results found - potential error in previous model training
2020-11-11 20:36:44,440 [MainThread  ] [INFO ]  Loading data
2020-11-11 20:36:44,520 [MainThread  ] [INFO ]  Features already cached: D:\Theses\Deep-Neural-Networks-for-Video-Classification\cache/features/resnet50/max/
2020-11-11 20:36:44,520 [MainThread  ] [INFO ]  Loading features sequence data into memory [may take a few minutes]
2020-11-11 20:36:44,634 [MainThread  ] [INFO ]  NumExpr defaulting to 8 threads.
1   XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
{'architecture': 'video_lrcnn_frozen', 'dropout': 0.2, 'layer_1_size': 256, 'layer_2_size': 512, 'layer_3_size': 256, 'model_id': 1, 'pooling': 'max', 'pretrained_model_name': 'resnet50', 'sequence_length': 20, 'sequence_model': 'LSTM', 'sequence_model_layers': 2}
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-eae6ea118bfc> in <module>
     13                             layer_3_size = experiment['layer_3_size'],
     14                             dropout = experiment['dropout'],
---> 15                             verbose=True)
     16 
     17 architecture.train_model()

D:\Theses\Deep-Neural-Networks-for-Video-Classification\deepvideoclassification\architectures.py in __init__(self, model_id, architecture, sequence_length, frame_size, pretrained_model_name, pooling, sequence_model, sequence_model_layers, layer_1_size, layer_2_size, layer_3_size, dropout, convolution_kernel_size, model_weights_path, batch_size, verbose, logger)
    426                                 return_CNN_features = True,
    427                                 pretrained_model_name = self.pretrained_model_name,
--> 428                                 pooling = self.pooling)
    429 
    430 

D:\Theses\Deep-Neural-Networks-for-Video-Classification\deepvideoclassification\data.py in __init__(self, sequence_length, return_CNN_features, pretrained_model_name, pooling, frame_size, augmentation, oversampling, model_weights_path, custom_model_name, return_generator, batch_size, verbose)
    746             self.x_train = np.concatenate(self.x_train, axis=0)
    747             self.y_train = np.concatenate(self.y_train, axis=0)
--> 748             self.x_valid = np.concatenate(self.x_valid, axis=0)
    749             self.y_valid = np.concatenate(self.y_valid, axis=0)
    750             self.x_test = np.concatenate(self.x_test, axis=0)

<__array_function__ internals> in concatenate(*args, **kwargs)

ValueError: need at least one array to concatenate

A little guidance would be just fine, you don't have to fix it :).

[alxcnwy]

hmm that's a nontrivial thing to debug... I'm really sorry but I unfortunately can't help with this any time soon. Good luck! 😅

[Sebagam]

That's because you are using TensorFlow 2.0, you should replace "val_acc" by "val_accuracy"