hifigan discriminator error

[haiderasad]

while training the vocoder hifigan i encounter a error

Traceback (most recent call last): File "/home/haider/Documents/TTS_files/ai4bharat/Trainer/trainer/trainer.py", line 1500, in fit self._fit() File "/home/haider/Documents/TTS_files/ai4bharat/Trainer/trainer/trainer.py", line 1484, in _fit self.train_epoch(epoch) File "/home/haider/Documents/TTS_files/ai4bharat/Trainer/trainer/trainer.py", line 1261, in train_epoch _, _ = self.train_step(batch, batch_num_steps, cur_step, loader_start_time) File "/home/haider/Documents/TTS_files/ai4bharat/Trainer/trainer/trainer.py", line 1124, in train_step step_optimizer=step_optimizer, File "/home/haider/Documents/TTS_files/ai4bharat/Trainer/trainer/trainer.py", line 979, in _optimize outputs, loss_dict = self._model_train_step(batch, model, criterion, optimizer_idx=optimizer_idx) File "/home/haider/Documents/TTS_files/ai4bharat/Trainer/trainer/trainer.py", line 937, in _model_train_step return model.train_step(*input_args) File "/home/haider/Documents/TTS_files/ai4bharat/TTS/TTS/vocoder/models/gan.py", line 136, in train_step D_out_fake = self.model_d(y_hat.detach()) File "/home/haider/anaconda3/envs/tts-env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/home/haider/Documents/TTS_files/ai4bharat/TTS/TTS/vocoder/models/hifigan_discriminator.py", line 218, in forward scores_, feats_ = self.msd(x) File "/home/haider/anaconda3/envs/tts-env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/home/haider/Documents/TTS_files/ai4bharat/TTS/TTS/vocoder/models/hifigan_discriminator.py", line 194, in forward score, feat = d(x) File "/home/haider/anaconda3/envs/tts-env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/home/haider/Documents/TTS_files/ai4bharat/TTS/TTS/vocoder/models/hifigan_discriminator.py", line 155, in forward x = l(x) File "/home/haider/anaconda3/envs/tts-env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1201, in _call_impl result = hook(self, input) File "/home/haider/anaconda3/envs/tts-env/lib/python3.7/site-packages/torch/nn/utils/spectral_norm.py", line 107, in __call__ setattr(module, self.name, self.compute_weight(module, do_power_iteration=module.training)) File "/home/haider/anaconda3/envs/tts-env/lib/python3.7/site-packages/torch/nn/utils/spectral_norm.py", line 86, in compute_weight v = normalize(torch.mv(weight_mat.t(), u), dim=0, eps=self.eps, out=v) RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)``

this is not a OOM error as i have confirmed that, seems as though input size is not matching the conv1d layer