Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transmission request stuck 0 of 0 bytes for 65 seconds and will be aborted #16

Open
Digital1O1 opened this issue Dec 9, 2024 · 0 comments

Comments

@Digital1O1
Copy link

Digital1O1 commented Dec 9, 2024

For what it's worth, I'm running Windows but I'm 'remoting' into my WSL2 instance through VSCode and I am using a Python virtual environment

I was running the 02_neural_network.ipynb notebook last night and everything seemed fine.

But when I was re-running the same notebook earlier today it doesn't seem like I can get access to the Google bucket due to the following output

Epoch 1/10
2024-12-08 20:46:07.169669: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe4d4001ce0 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Froses%2F483444865_65962cea07_m.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.001255 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1733712369.038676   64376 service.cc:148] XLA service 0x7fe51c0090d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1733712369.038836   64376 service.cc:156]   StreamExecutor device (0): NVIDIA GeForce GTX 1660 Ti, Compute Capability 7.5
2024-12-08 20:46:09.127375: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
I0000 00:00:1733712369.348996   64376 cuda_dnn.cc:529] Loaded cuDNN version 90300
     11/Unknown 65s 16ms/step - accuracy: 0.1980 - loss: 79.0183
I0000 00:00:1733712370.408962   64376 device_compiler.h:188] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
     67/Unknown 80s 234ms/step - accuracy: 0.2495 - loss: 43.8153
2024-12-08 20:47:26.502372: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe4f41dc6c0 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Fdaisy%2F20619292635_9857a12d54.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.000827 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
     95/Unknown 146s 866ms/step - accuracy: 0.2662 - loss: 35.8853
2024-12-08 20:48:32.177232: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe4f01a8200 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Ftulips%2F7448453762_aea8739f1b.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.000959 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
    104/Unknown 209s 1s/step - accuracy: 0.2706 - loss: 33.9945
2024-12-08 20:48:35.154446: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
	 [[{{node IteratorGetNext}}]]
2024-12-08 20:48:35.154492: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
	 [[{{node IteratorGetNext}}]]
	 [[IteratorGetNext/_4]]
2024-12-08 20:48:35.154503: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8449551428342137023
2024-12-08 20:48:35.154533: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8021097489352039294
/home/digital101/Python-3.12-V/lib/python3.12/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
  self._interrupted_warning()
2024-12-08 20:49:37.610611: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe424001670 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Fdandelion%2F6012046444_fd80afb63a_n.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.000673 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
104/104 ━━━━━━━━━━━━━━━━━━━━ 274s 2s/step - accuracy: 0.2710 - loss: 33.8008 - val_accuracy: 0.3946 - val_loss: 1.8215
Epoch 2/10
2024-12-08 20:49:40.030128: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
	 [[{{node IteratorGetNext}}]]
	 [[IteratorGetNext/_4]]
2024-12-08 20:49:40.030206: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8449551428342137023
2024-12-08 20:49:40.030238: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8021097489352039294
  2/104 ━━━━━━━━━━━━━━━━━━━━ 29s 286ms/step - accuracy: 0.3906 - loss: 1.5181 
2024-12-08 20:50:45.359300: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe3e4002d90 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Fsunflowers%2F6606817351_10f6e43a09.jpg) has been stuck at 0 of 0 bytes for 65 seconds and will be aborted. CURL timing information: lookup time: 0.001246 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
 20/104 ━━━━━━━━━━━━━━━━━━━━ 4:57 4s/step - accuracy: 0.3960 - loss: 1.6590
2024-12-08 20:51:48.797824: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe3f41deab0 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Ftulips%2F405035580_94b793e71d.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.000728 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
 42/104 ━━━━━━━━━━━━━━━━━━━━ 3:25 3s/step - accuracy: 0.3999 - loss: 1.6421
2024-12-08 20:52:54.594822: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe40010f490 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Ftulips%2F4571353297_5634177744_n.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.001908 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
 65/104 ━━━━━━━━━━━━━━━━━━━━ 2:01 3s/step - accuracy: 0.4052 - loss: 1.6294
2024-12-08 20:54:00.958634: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe400149a80 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Froses%2F16339359979_6d742660b8_n.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.001205 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
102/104 ━━━━━━━━━━━━━━━━━━━━ 5s 3s/step - accuracy: 0.4036 - loss: 1.6525
2024-12-08 20:55:11.227103: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe3e01eed50 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Fdaisy%2F4697206799_19dd2a3193_m.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.00056 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
103/104 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 0.4036 - loss: 1.6533
2024-12-08 20:55:12.550250: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe3f41735a0 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Fdaisy%2F20329326505_a777c71cc2.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.000813 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
104/104 ━━━━━━━━━━━━━━━━━━━━ 0s 3s/step - accuracy: 0.4035 - loss: 1.6541
2024-12-08 20:55:14.149417: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8449551428342137023
2024-12-08 20:55:14.149480: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8021097489352039294
104/104 ━━━━━━━━━━━━━━━━━━━━ 337s 3s/step - accuracy: 0.4034 - loss: 1.6548 - val_accuracy: 0.3459 - val_loss: 2.0934
Epoch 3/10
2024-12-08 20:55:17.436657: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
	 [[{{node IteratorGetNext}}]]
	 [[IteratorGetNext/_4]]
2024-12-08 20:55:17.436718: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8449551428342137023
2024-12-08 20:55:17.436751: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8021097489352039294
  7/104 ━━━━━━━━━━━━━━━━━━━━ 32s 332ms/step - accuracy: 0.4021 - loss: 1.6853
2024-12-08 20:56:20.995477: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe4d0101ba0 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Froses%2F7502389724_85b4a6c855_n.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.001129 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
102/104 ━━━━━━━━━━━━━━━━━━━━ 1s 906ms/step - accuracy: 0.4143 - loss: 1.6417
2024-12-08 20:56:49.544512: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8449551428342137023
2024-12-08 20:56:49.544611: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8021097489352039294
2024-12-08 20:57:51.164976: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe42c003580 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Ftulips%2F5697471591_200ff951fa_n.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.000939 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
2024-12-08 20:57:51.383284: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe448003e10 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Fsunflowers%2F9535500195_543d0b729b.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.001199 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
2024-12-08 20:57:51.545394: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe450001930 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Ftulips%2F8623170936_83f4152431.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.0012 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
2024-12-08 20:57:51.588392: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe444101bd0 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Fdandelion%2F21523597492_39b6765cd7_m.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.000879 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
104/104 ━━━━━━━━━━━━━━━━━━━━ 156s 2s/step - accuracy: 0.4140 - loss: 1.6457 - val_accuracy: 0.3054 - val_loss: 3.3878
Epoch 4/10
2024-12-08 20:57:53.658359: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8449551428342137023
2024-12-08 20:57:53.658428: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8021097489352039294
 19/104 ━━━━━━━━━━━━━━━━━━━━ 23s 278ms/step - accuracy: 0.3725 - loss: 2.5240
2024-12-08 20:59:00.845295: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe3fc12f9b0 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Fdandelion%2F14278605962_d3cce5522f.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.001334 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
 83/104 ━━━━━━━━━━━━━━━━━━━━ 21s 1s/step - accuracy: 0.3797 - loss: 2.3692
2024-12-08 21:00:18.494972: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe3f8182080 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Fdaisy%2F5874818796_3efbb8769d.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.000924 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
2024-12-08 21:00:18.520498: E external/local_tsl/tsl/platform/cloud/curl_http_request.cc:611] The transmission  of request 0x7fe3e816d840 (URI: https://storage.googleapis.com/practical-ml-vision-book/flowers_5_jpeg%2Fflower_photos%2Fdaisy%2F476857510_d2b30175de_n.jpg) has been stuck at 0 of 0 bytes for 61 seconds and will be aborted. CURL timing information: lookup time: 0.001376 (No error), connect time: 0 (No error), pre-transfer time: 0 (No error), start-transfer time: 0 (No error)
102/104 ━━━━━━━━━━━━━━━━━━━━ 2s 1s/step - accuracy: 0.3830 - loss: 2.3413
2024-12-08 21:00:38.878066: W tensorflow/core/framework/op_kernel.cc:1841] OP_REQUIRES failed at whole_file_read_ops.cc:116 : FAILED_PRECONDITION: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resolve host name', error details: Could not resolve host: storage.googleapis.com
	 when reading gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/sunflowers/4895721788_f10208ab77_n.jpg
2024-12-08 21:00:38.878482: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8449551428342137023
2024-12-08 21:00:38.878601: I tensorflow/core/framework/local_rendezvous.cc:424] Local rendezvous recv item cancelled. Key hash: 8021097489352039294

I know people in the past has had issues getting access to the Google bucket so I ran the following gsutil function and was able to list out the contents in said directory

digital101@Digital101:~/practical-ml-vision-book$ gsutil ls gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/LICENSE.txt
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/README.txt
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/all_data.csv
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy.zip
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion.zip
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dict.txt
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/eval_set.csv
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/flowers_200_csv.zip
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/flowers_200_folders.zip
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/flowers_200_presplit.zip
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/flowers_200_unlabeled.zip
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/flowers_full_with_csv.zip
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/inception_v3_2016_08_28.ckpt
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/open_image_inception_v3.ckpt
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/roses.zip
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/sunflowers.zip
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/train_set.csv
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips.zip
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy/
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/roses/
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/sunflowers/
gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/

Since I don't use Python often, I'm not sure where to start troubleshooting-wise and could use any help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant