Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VAMB crashes after fitting and clustering with tensor error. No clusters.tsv #387

Closed
shiraz-shah opened this issue Jan 15, 2025 · 5 comments
Closed

VAMB crashes after fitting and clustering with tensor error. No clusters.tsv #387

shiraz-shah opened this issue Jan 15, 2025 · 5 comments

Comments

@shiraz-shah
Copy link

Dear VAMB team,
Are you familiar with this error?

It seems both encoding and clustering were successful (since latent.npz and vae_clusters_metadata.tsv are there), but VAMB fails with the error below, and no clusters.tsv file was generated.

2025-01-15 12:22:20.943 | INFO    | 	Encoding to latent representation
2025-01-15 12:22:30.405 | INFO    | 	Trained VAE and encoded in 884.05 seconds.

2025-01-15 12:22:30.406 | INFO    | Clustering
2025-01-15 12:22:30.406 | INFO    | 	Windowsize: 300
2025-01-15 12:22:30.406 | INFO    | 	Min successful thresholds detected: 15
2025-01-15 12:22:30.406 | INFO    | 	Max clusters: None
2025-01-15 12:22:30.406 | INFO    | 	Use CUDA for clustering: True
2025-01-15 12:22:30.406 | INFO    | 	Binsplitter: "@"
2025-01-15 12:24:47.901 | ERROR   | An error has been caught in function 'main', process 'MainProcess' (4135283), thread 'MainThread' (135716811642368):
Traceback (most recent call last):

  File "/data/shiraz/miniconda3/envs/vamb/bin/vamb", line 8, in <module>
    sys.exit(main())
    │   │    └ <function main at 0x7b6dfc63e3e0>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>

> File "/data/shiraz/VAMB/vamb/vamb/__main__.py", line 2267, in main
    runner.run()
    │      └ <function BasicArguments.run at 0x7b6dfc63d620>
    └ <vamb.__main__.VAEArguments object at 0x7b6e009e0860>

  File "/data/shiraz/VAMB/vamb/vamb/__main__.py", line 1469, in run
    self.run_inner()
    │    └ <function VAEArguments.run_inner at 0x7b6dfc63dc60>
    └ <vamb.__main__.VAEArguments object at 0x7b6e009e0860>

  File "/data/shiraz/VAMB/vamb/vamb/__main__.py", line 1594, in run_inner
    run(
    └ <function run at 0x7b6dfc63cd60>

  File "/data/shiraz/VAMB/vamb/vamb/__main__.py", line 995, in run
    cluster_and_write_files(
    └ <function cluster_and_write_files at 0x7b6dfc63cb80>

  File "/data/shiraz/VAMB/vamb/vamb/__main__.py", line 825, in cluster_and_write_files
    for i, cluster in enumerate(clusters):
        │  │                    └ <itertools.islice object at 0x7b6df3d43ec0>
        │  └ <vamb.cluster.Cluster object at 0x7b6df08c3530>
        └ 68479

  File "/data/shiraz/VAMB/vamb/vamb/cluster.py", line 297, in __next__
    cluster, _, points = self.find_cluster()
                         │    └ <function ClusterGenerator.find_cluster at 0x7b6e8cb70c20>
                         └ ClusterGenerator(46 points, 68480 clusters)

  File "/data/shiraz/VAMB/vamb/vamb/cluster.py", line 541, in find_cluster
    threshold = self.find_threshold(distances)
                │    │              └ tensor([0.5936, 0.5207, 0.5416, 0.5254, 0.5320, 0.5004, 0.5448, 0.5621, 0.5671,
                │    │                        0.4736, 0.4296, 0.4629, 0.5061, 0.436...
                │    └ <function ClusterGenerator.find_threshold at 0x7b6e8cb70b80>
                └ ClusterGenerator(46 points, 68480 clusters)

  File "/data/shiraz/VAMB/vamb/vamb/cluster.py", line 455, in find_threshold
    below_xmax = (distances <= _XMAX) & self.kept_mask
                  │            │        │    └ <member 'kept_mask' of 'ClusterGenerator' objects>
                  │            │        └ ClusterGenerator(46 points, 68480 clusters)
                  │            └ 0.3
                  └ tensor([0.5936, 0.5207, 0.5416, 0.5254, 0.5320, 0.5004, 0.5448, 0.5621, 0.5671,
                            0.4736, 0.4296, 0.4629, 0.5061, 0.436...

RuntimeError: The size of tensor a (57) must match the size of tensor b (46) at non-singleton dimension 0
@jakobnissen
Copy link
Member

Dear @shiraz-shah
Can you attach the log file?

@shiraz-shah
Copy link
Author

Of note, this was after performing the BAM parsing part one machine, and then transferring the two npz files to a different machine that has a GPU.

shiraz@ersa:~/data/Omer/bin$ cat log.txt 
2025-01-15 12:07:45.285 | INFO    | Starting Vamb version 4.1.4.dev121+g7dc2f98
2025-01-15 12:07:45.285 | INFO    | Random seed is 32123030429022605
2025-01-15 12:07:45.285 | INFO    | Invoked with CLI args: 'f/data/shiraz/miniconda3/envs/vamb/bin/vamb bin default --composition composition.npz --rpkm abundance.npz --outdir bin --cuda -o @'
2025-01-15 12:07:45.286 | INFO    | Loading TNF
2025-01-15 12:07:45.286 | INFO    | 	Minimum sequence length: 2000
2025-01-15 12:07:45.286 | INFO    | 	Loading composition from npz at: composition.npz
2025-01-15 12:07:46.090 | INFO    | 	Kept 3431728865 bases in 347769 sequences
2025-01-15 12:07:46.090 | INFO    | 	Processed TNF in 0.8 seconds.

2025-01-15 12:07:46.090 | INFO    | Loading depths
2025-01-15 12:07:46.090 | INFO    | 	Reference hash: 954727658f9154f86639d84c068275fe
2025-01-15 12:07:46.090 | INFO    | 	Loading depths from npz at: abundance.npz
2025-01-15 12:07:46.227 | INFO    | 	Processed RPKM in 0.14 seconds.

2025-01-15 12:07:46.359 | INFO    | Creating and training VAE
2025-01-15 12:07:46.423 | INFO    | 	Created VAE
2025-01-15 12:07:47.251 | INFO    | 	Network properties:
2025-01-15 12:07:47.252 | INFO    | 	CUDA: True
2025-01-15 12:07:47.253 | INFO    | 	Alpha: 0.15
2025-01-15 12:07:47.253 | INFO    | 	Beta: 200.0
2025-01-15 12:07:47.254 | INFO    | 	Dropout: 0.2
2025-01-15 12:07:47.254 | INFO    | 	N hidden: 512, 512
2025-01-15 12:07:47.255 | INFO    | 	N latent: 32
2025-01-15 12:07:47.255 | INFO    | 	Training properties:
2025-01-15 12:07:47.256 | INFO    | 	N epochs: 300
2025-01-15 12:07:47.256 | INFO    | 	Starting batch size: 256
2025-01-15 12:07:47.257 | INFO    | 	Batchsteps: 25, 75, 150, 225
2025-01-15 12:07:47.257 | INFO    | 	N sequences: 347769
2025-01-15 12:07:47.258 | INFO    | 	N samples: 39
2025-01-15 12:07:57.839 | INFO    | 		Epoch:   1  Loss: 5.26572e-01  CE: 4.49034e-01  AB: 3.88529e-03  SSE: 5.59068e-02  KLD: 1.76854e-02  Batchsize:  256
2025-01-15 12:08:06.725 | INFO    | 		Epoch:   2  Loss: 4.78818e-01  CE: 4.08563e-01  AB: 2.90130e-03  SSE: 4.97539e-02  KLD: 1.76517e-02  Batchsize:  256
2025-01-15 12:08:15.686 | INFO    | 		Epoch:   3  Loss: 4.70280e-01  CE: 4.01775e-01  AB: 2.80087e-03  SSE: 4.83654e-02  KLD: 1.73743e-02  Batchsize:  256
2025-01-15 12:08:24.435 | INFO    | 		Epoch:   4  Loss: 4.65855e-01  CE: 3.98419e-01  AB: 2.74381e-03  SSE: 4.75059e-02  KLD: 1.72148e-02  Batchsize:  256
2025-01-15 12:08:33.248 | INFO    | 		Epoch:   5  Loss: 4.63132e-01  CE: 3.96434e-01  AB: 2.69822e-03  SSE: 4.69020e-02  KLD: 1.71259e-02  Batchsize:  256
2025-01-15 12:08:42.189 | INFO    | 		Epoch:   6  Loss: 4.61187e-01  CE: 3.95022e-01  AB: 2.64540e-03  SSE: 4.64639e-02  KLD: 1.70850e-02  Batchsize:  256
2025-01-15 12:08:51.188 | INFO    | 		Epoch:   7  Loss: 4.59670e-01  CE: 3.93983e-01  AB: 2.62091e-03  SSE: 4.60893e-02  KLD: 1.70142e-02  Batchsize:  256
2025-01-15 12:09:00.157 | INFO    | 		Epoch:   8  Loss: 4.58630e-01  CE: 3.93229e-01  AB: 2.58553e-03  SSE: 4.58309e-02  KLD: 1.70082e-02  Batchsize:  256
2025-01-15 12:09:08.901 | INFO    | 		Epoch:   9  Loss: 4.57506e-01  CE: 3.92424e-01  AB: 2.56289e-03  SSE: 4.55906e-02  KLD: 1.69584e-02  Batchsize:  256
2025-01-15 12:09:18.011 | INFO    | 		Epoch:  10  Loss: 4.56768e-01  CE: 3.91866e-01  AB: 2.56008e-03  SSE: 4.53880e-02  KLD: 1.69715e-02  Batchsize:  256
2025-01-15 12:09:26.741 | INFO    | 		Epoch:  11  Loss: 4.56215e-01  CE: 3.91432e-01  AB: 2.54720e-03  SSE: 4.53074e-02  KLD: 1.69632e-02  Batchsize:  256
2025-01-15 12:09:35.580 | INFO    | 		Epoch:  12  Loss: 4.55562e-01  CE: 3.91015e-01  AB: 2.52246e-03  SSE: 4.51070e-02  KLD: 1.69424e-02  Batchsize:  256
2025-01-15 12:09:44.399 | INFO    | 		Epoch:  13  Loss: 4.55074e-01  CE: 3.90614e-01  AB: 2.51440e-03  SSE: 4.50007e-02  KLD: 1.69616e-02  Batchsize:  256
2025-01-15 12:09:53.243 | INFO    | 		Epoch:  14  Loss: 4.54505e-01  CE: 3.90225e-01  AB: 2.50617e-03  SSE: 4.48630e-02  KLD: 1.69465e-02  Batchsize:  256
2025-01-15 12:10:01.863 | INFO    | 		Epoch:  15  Loss: 4.54137e-01  CE: 3.89950e-01  AB: 2.49722e-03  SSE: 4.47877e-02  KLD: 1.69364e-02  Batchsize:  256
2025-01-15 12:10:10.662 | INFO    | 		Epoch:  16  Loss: 4.53844e-01  CE: 3.89709e-01  AB: 2.49195e-03  SSE: 4.47175e-02  KLD: 1.69477e-02  Batchsize:  256
2025-01-15 12:10:19.573 | INFO    | 		Epoch:  17  Loss: 4.53592e-01  CE: 3.89482e-01  AB: 2.50317e-03  SSE: 4.46549e-02  KLD: 1.69766e-02  Batchsize:  256
2025-01-15 12:10:28.267 | INFO    | 		Epoch:  18  Loss: 4.53143e-01  CE: 3.89198e-01  AB: 2.46922e-03  SSE: 4.45460e-02  KLD: 1.69534e-02  Batchsize:  256
2025-01-15 12:10:37.009 | INFO    | 		Epoch:  19  Loss: 4.52965e-01  CE: 3.89032e-01  AB: 2.46515e-03  SSE: 4.45013e-02  KLD: 1.69855e-02  Batchsize:  256
2025-01-15 12:10:45.884 | INFO    | 		Epoch:  20  Loss: 4.52731e-01  CE: 3.88846e-01  AB: 2.46612e-03  SSE: 4.44598e-02  KLD: 1.69786e-02  Batchsize:  256
2025-01-15 12:10:54.966 | INFO    | 		Epoch:  21  Loss: 4.52417e-01  CE: 3.88649e-01  AB: 2.44502e-03  SSE: 4.43968e-02  KLD: 1.69665e-02  Batchsize:  256
2025-01-15 12:11:03.791 | INFO    | 		Epoch:  22  Loss: 4.52255e-01  CE: 3.88553e-01  AB: 2.46519e-03  SSE: 4.42905e-02  KLD: 1.69665e-02  Batchsize:  256
2025-01-15 12:11:12.713 | INFO    | 		Epoch:  23  Loss: 4.52029e-01  CE: 3.88387e-01  AB: 2.45019e-03  SSE: 4.42555e-02  KLD: 1.69626e-02  Batchsize:  256
2025-01-15 12:11:21.663 | INFO    | 		Epoch:  24  Loss: 4.51855e-01  CE: 3.88234e-01  AB: 2.43509e-03  SSE: 4.42238e-02  KLD: 1.69769e-02  Batchsize:  256
2025-01-15 12:11:30.648 | INFO    | 		Epoch:  25  Loss: 4.51630e-01  CE: 3.88060e-01  AB: 2.45544e-03  SSE: 4.41699e-02  KLD: 1.69711e-02  Batchsize:  256
2025-01-15 12:11:35.407 | INFO    | 		Epoch:  26  Loss: 4.48925e-01  CE: 3.86446e-01  AB: 2.29842e-03  SSE: 4.33230e-02  KLD: 1.68610e-02  Batchsize:  512
2025-01-15 12:11:40.195 | INFO    | 		Epoch:  27  Loss: 4.48763e-01  CE: 3.86335e-01  AB: 2.28854e-03  SSE: 4.33052e-02  KLD: 1.68763e-02  Batchsize:  512
2025-01-15 12:11:44.800 | INFO    | 		Epoch:  28  Loss: 4.48738e-01  CE: 3.86262e-01  AB: 2.29500e-03  SSE: 4.33032e-02  KLD: 1.69005e-02  Batchsize:  512
2025-01-15 12:11:49.501 | INFO    | 		Epoch:  29  Loss: 4.48759e-01  CE: 3.86291e-01  AB: 2.28766e-03  SSE: 4.32839e-02  KLD: 1.69100e-02  Batchsize:  512
2025-01-15 12:11:54.271 | INFO    | 		Epoch:  30  Loss: 4.48726e-01  CE: 3.86254e-01  AB: 2.29732e-03  SSE: 4.32786e-02  KLD: 1.69309e-02  Batchsize:  512
2025-01-15 12:11:59.081 | INFO    | 		Epoch:  31  Loss: 4.48666e-01  CE: 3.86170e-01  AB: 2.30730e-03  SSE: 4.32805e-02  KLD: 1.69017e-02  Batchsize:  512
2025-01-15 12:12:03.750 | INFO    | 		Epoch:  32  Loss: 4.48598e-01  CE: 3.86157e-01  AB: 2.29797e-03  SSE: 4.32567e-02  KLD: 1.69077e-02  Batchsize:  512
2025-01-15 12:12:08.492 | INFO    | 		Epoch:  33  Loss: 4.48500e-01  CE: 3.86082e-01  AB: 2.30229e-03  SSE: 4.32113e-02  KLD: 1.69125e-02  Batchsize:  512
2025-01-15 12:12:13.191 | INFO    | 		Epoch:  34  Loss: 4.48477e-01  CE: 3.86065e-01  AB: 2.30465e-03  SSE: 4.32114e-02  KLD: 1.69238e-02  Batchsize:  512
2025-01-15 12:12:18.159 | INFO    | 		Epoch:  35  Loss: 4.48361e-01  CE: 3.85960e-01  AB: 2.30863e-03  SSE: 4.31859e-02  KLD: 1.69261e-02  Batchsize:  512
2025-01-15 12:12:22.745 | INFO    | 		Epoch:  36  Loss: 4.48260e-01  CE: 3.85914e-01  AB: 2.28370e-03  SSE: 4.31640e-02  KLD: 1.69050e-02  Batchsize:  512
2025-01-15 12:12:27.328 | INFO    | 		Epoch:  37  Loss: 4.48223e-01  CE: 3.85874e-01  AB: 2.29094e-03  SSE: 4.31286e-02  KLD: 1.69420e-02  Batchsize:  512
2025-01-15 12:12:32.033 | INFO    | 		Epoch:  38  Loss: 4.48200e-01  CE: 3.85838e-01  AB: 2.28885e-03  SSE: 4.31483e-02  KLD: 1.69479e-02  Batchsize:  512
2025-01-15 12:12:36.712 | INFO    | 		Epoch:  39  Loss: 4.48057e-01  CE: 3.85717e-01  AB: 2.29308e-03  SSE: 4.31421e-02  KLD: 1.69147e-02  Batchsize:  512
2025-01-15 12:12:41.319 | INFO    | 		Epoch:  40  Loss: 4.48028e-01  CE: 3.85734e-01  AB: 2.30264e-03  SSE: 4.30651e-02  KLD: 1.69477e-02  Batchsize:  512
2025-01-15 12:12:45.995 | INFO    | 		Epoch:  41  Loss: 4.47978e-01  CE: 3.85678e-01  AB: 2.28668e-03  SSE: 4.30825e-02  KLD: 1.69471e-02  Batchsize:  512
2025-01-15 12:12:50.685 | INFO    | 		Epoch:  42  Loss: 4.47924e-01  CE: 3.85621e-01  AB: 2.28961e-03  SSE: 4.30956e-02  KLD: 1.69362e-02  Batchsize:  512
2025-01-15 12:12:55.334 | INFO    | 		Epoch:  43  Loss: 4.47865e-01  CE: 3.85586e-01  AB: 2.28888e-03  SSE: 4.30698e-02  KLD: 1.69226e-02  Batchsize:  512
2025-01-15 12:13:00.013 | INFO    | 		Epoch:  44  Loss: 4.47839e-01  CE: 3.85581e-01  AB: 2.27652e-03  SSE: 4.30644e-02  KLD: 1.69316e-02  Batchsize:  512
2025-01-15 12:13:04.757 | INFO    | 		Epoch:  45  Loss: 4.47858e-01  CE: 3.85567e-01  AB: 2.28337e-03  SSE: 4.30627e-02  KLD: 1.69451e-02  Batchsize:  512
2025-01-15 12:13:09.521 | INFO    | 		Epoch:  46  Loss: 4.47718e-01  CE: 3.85495e-01  AB: 2.28827e-03  SSE: 4.30260e-02  KLD: 1.69262e-02  Batchsize:  512
2025-01-15 12:13:14.269 | INFO    | 		Epoch:  47  Loss: 4.47698e-01  CE: 3.85469e-01  AB: 2.28374e-03  SSE: 4.30324e-02  KLD: 1.69409e-02  Batchsize:  512
2025-01-15 12:13:18.938 | INFO    | 		Epoch:  48  Loss: 4.47688e-01  CE: 3.85459e-01  AB: 2.28099e-03  SSE: 4.30098e-02  KLD: 1.69535e-02  Batchsize:  512
2025-01-15 12:13:23.525 | INFO    | 		Epoch:  49  Loss: 4.47499e-01  CE: 3.85330e-01  AB: 2.28682e-03  SSE: 4.29702e-02  KLD: 1.69188e-02  Batchsize:  512
2025-01-15 12:13:28.236 | INFO    | 		Epoch:  50  Loss: 4.47615e-01  CE: 3.85400e-01  AB: 2.27406e-03  SSE: 4.30155e-02  KLD: 1.69239e-02  Batchsize:  512
2025-01-15 12:13:33.168 | INFO    | 		Epoch:  51  Loss: 4.47448e-01  CE: 3.85288e-01  AB: 2.26706e-03  SSE: 4.29931e-02  KLD: 1.69345e-02  Batchsize:  512
2025-01-15 12:13:37.773 | INFO    | 		Epoch:  52  Loss: 4.47352e-01  CE: 3.85241e-01  AB: 2.27091e-03  SSE: 4.29334e-02  KLD: 1.69163e-02  Batchsize:  512
2025-01-15 12:13:42.586 | INFO    | 		Epoch:  53  Loss: 4.47324e-01  CE: 3.85215e-01  AB: 2.26378e-03  SSE: 4.29336e-02  KLD: 1.69182e-02  Batchsize:  512
2025-01-15 12:13:47.335 | INFO    | 		Epoch:  54  Loss: 4.47298e-01  CE: 3.85238e-01  AB: 2.25896e-03  SSE: 4.28965e-02  KLD: 1.69301e-02  Batchsize:  512
2025-01-15 12:13:52.069 | INFO    | 		Epoch:  55  Loss: 4.47288e-01  CE: 3.85144e-01  AB: 2.27017e-03  SSE: 4.29555e-02  KLD: 1.69204e-02  Batchsize:  512
2025-01-15 12:13:56.733 | INFO    | 		Epoch:  56  Loss: 4.47298e-01  CE: 3.85216e-01  AB: 2.25568e-03  SSE: 4.29089e-02  KLD: 1.69386e-02  Batchsize:  512
2025-01-15 12:14:01.640 | INFO    | 		Epoch:  57  Loss: 4.47208e-01  CE: 3.85119e-01  AB: 2.26785e-03  SSE: 4.29009e-02  KLD: 1.69344e-02  Batchsize:  512
2025-01-15 12:14:06.416 | INFO    | 		Epoch:  58  Loss: 4.47183e-01  CE: 3.85087e-01  AB: 2.26046e-03  SSE: 4.29198e-02  KLD: 1.69461e-02  Batchsize:  512
2025-01-15 12:14:11.212 | INFO    | 		Epoch:  59  Loss: 4.47153e-01  CE: 3.85087e-01  AB: 2.27137e-03  SSE: 4.28972e-02  KLD: 1.69307e-02  Batchsize:  512
2025-01-15 12:14:15.898 | INFO    | 		Epoch:  60  Loss: 4.47186e-01  CE: 3.85079e-01  AB: 2.25772e-03  SSE: 4.29191e-02  KLD: 1.69452e-02  Batchsize:  512
2025-01-15 12:14:20.469 | INFO    | 		Epoch:  61  Loss: 4.47044e-01  CE: 3.85002e-01  AB: 2.25733e-03  SSE: 4.28774e-02  KLD: 1.69268e-02  Batchsize:  512
2025-01-15 12:14:25.194 | INFO    | 		Epoch:  62  Loss: 4.47054e-01  CE: 3.85005e-01  AB: 2.25549e-03  SSE: 4.28733e-02  KLD: 1.69505e-02  Batchsize:  512
2025-01-15 12:14:30.043 | INFO    | 		Epoch:  63  Loss: 4.46966e-01  CE: 3.84925e-01  AB: 2.25849e-03  SSE: 4.28743e-02  KLD: 1.69311e-02  Batchsize:  512
2025-01-15 12:14:34.673 | INFO    | 		Epoch:  64  Loss: 4.47025e-01  CE: 3.84997e-01  AB: 2.25786e-03  SSE: 4.28733e-02  KLD: 1.69227e-02  Batchsize:  512
2025-01-15 12:14:39.544 | INFO    | 		Epoch:  65  Loss: 4.46969e-01  CE: 3.84939e-01  AB: 2.25627e-03  SSE: 4.28514e-02  KLD: 1.69277e-02  Batchsize:  512
2025-01-15 12:14:44.300 | INFO    | 		Epoch:  66  Loss: 4.46987e-01  CE: 3.84948e-01  AB: 2.27877e-03  SSE: 4.28502e-02  KLD: 1.69259e-02  Batchsize:  512
2025-01-15 12:14:48.977 | INFO    | 		Epoch:  67  Loss: 4.46838e-01  CE: 3.84849e-01  AB: 2.25025e-03  SSE: 4.28585e-02  KLD: 1.69127e-02  Batchsize:  512
2025-01-15 12:14:53.827 | INFO    | 		Epoch:  68  Loss: 4.46849e-01  CE: 3.84842e-01  AB: 2.26051e-03  SSE: 4.28308e-02  KLD: 1.69111e-02  Batchsize:  512
2025-01-15 12:14:58.613 | INFO    | 		Epoch:  69  Loss: 4.46789e-01  CE: 3.84791e-01  AB: 2.25286e-03  SSE: 4.28194e-02  KLD: 1.69341e-02  Batchsize:  512
2025-01-15 12:15:03.306 | INFO    | 		Epoch:  70  Loss: 4.46802e-01  CE: 3.84827e-01  AB: 2.26161e-03  SSE: 4.28035e-02  KLD: 1.69229e-02  Batchsize:  512
2025-01-15 12:15:08.092 | INFO    | 		Epoch:  71  Loss: 4.46763e-01  CE: 3.84799e-01  AB: 2.25100e-03  SSE: 4.28118e-02  KLD: 1.69129e-02  Batchsize:  512
2025-01-15 12:15:12.938 | INFO    | 		Epoch:  72  Loss: 4.46737e-01  CE: 3.84722e-01  AB: 2.25371e-03  SSE: 4.28254e-02  KLD: 1.69321e-02  Batchsize:  512
2025-01-15 12:15:17.590 | INFO    | 		Epoch:  73  Loss: 4.46817e-01  CE: 3.84754e-01  AB: 2.27317e-03  SSE: 4.28562e-02  KLD: 1.69532e-02  Batchsize:  512
2025-01-15 12:15:22.375 | INFO    | 		Epoch:  74  Loss: 4.46747e-01  CE: 3.84755e-01  AB: 2.25323e-03  SSE: 4.28057e-02  KLD: 1.69404e-02  Batchsize:  512
2025-01-15 12:15:27.067 | INFO    | 		Epoch:  75  Loss: 4.46570e-01  CE: 3.84658e-01  AB: 2.25684e-03  SSE: 4.27687e-02  KLD: 1.69006e-02  Batchsize:  512
2025-01-15 12:15:29.557 | INFO    | 		Epoch:  76  Loss: 4.45253e-01  CE: 3.83877e-01  AB: 2.18201e-03  SSE: 4.23211e-02  KLD: 1.68694e-02  Batchsize: 1024
2025-01-15 12:15:32.092 | INFO    | 		Epoch:  77  Loss: 4.45058e-01  CE: 3.83737e-01  AB: 2.15340e-03  SSE: 4.22837e-02  KLD: 1.68782e-02  Batchsize: 1024
2025-01-15 12:15:34.505 | INFO    | 		Epoch:  78  Loss: 4.45059e-01  CE: 3.83728e-01  AB: 2.16629e-03  SSE: 4.22727e-02  KLD: 1.68953e-02  Batchsize: 1024
2025-01-15 12:15:37.172 | INFO    | 		Epoch:  79  Loss: 4.45057e-01  CE: 3.83738e-01  AB: 2.16103e-03  SSE: 4.22987e-02  KLD: 1.68717e-02  Batchsize: 1024
2025-01-15 12:15:39.627 | INFO    | 		Epoch:  80  Loss: 4.45033e-01  CE: 3.83718e-01  AB: 2.16651e-03  SSE: 4.22837e-02  KLD: 1.68725e-02  Batchsize: 1024
2025-01-15 12:15:42.163 | INFO    | 		Epoch:  81  Loss: 4.45082e-01  CE: 3.83760e-01  AB: 2.16073e-03  SSE: 4.22819e-02  KLD: 1.68878e-02  Batchsize: 1024
2025-01-15 12:15:44.716 | INFO    | 		Epoch:  82  Loss: 4.45083e-01  CE: 3.83751e-01  AB: 2.16173e-03  SSE: 4.22906e-02  KLD: 1.68841e-02  Batchsize: 1024
2025-01-15 12:15:47.216 | INFO    | 		Epoch:  83  Loss: 4.45179e-01  CE: 3.83785e-01  AB: 2.17621e-03  SSE: 4.23158e-02  KLD: 1.69069e-02  Batchsize: 1024
2025-01-15 12:15:49.863 | INFO    | 		Epoch:  84  Loss: 4.45174e-01  CE: 3.83807e-01  AB: 2.17110e-03  SSE: 4.22887e-02  KLD: 1.69117e-02  Batchsize: 1024
2025-01-15 12:15:52.503 | INFO    | 		Epoch:  85  Loss: 4.45103e-01  CE: 3.83760e-01  AB: 2.17622e-03  SSE: 4.22807e-02  KLD: 1.68918e-02  Batchsize: 1024
2025-01-15 12:15:54.968 | INFO    | 		Epoch:  86  Loss: 4.45137e-01  CE: 3.83754e-01  AB: 2.16369e-03  SSE: 4.22896e-02  KLD: 1.69372e-02  Batchsize: 1024
2025-01-15 12:15:57.453 | INFO    | 		Epoch:  87  Loss: 4.45157e-01  CE: 3.83765e-01  AB: 2.17109e-03  SSE: 4.23260e-02  KLD: 1.69046e-02  Batchsize: 1024
2025-01-15 12:15:59.915 | INFO    | 		Epoch:  88  Loss: 4.45142e-01  CE: 3.83784e-01  AB: 2.17320e-03  SSE: 4.22800e-02  KLD: 1.69125e-02  Batchsize: 1024
2025-01-15 12:16:02.669 | INFO    | 		Epoch:  89  Loss: 4.45065e-01  CE: 3.83727e-01  AB: 2.17192e-03  SSE: 4.22664e-02  KLD: 1.69088e-02  Batchsize: 1024
2025-01-15 12:16:05.318 | INFO    | 		Epoch:  90  Loss: 4.44992e-01  CE: 3.83679e-01  AB: 2.16636e-03  SSE: 4.22917e-02  KLD: 1.68728e-02  Batchsize: 1024
2025-01-15 12:16:07.747 | INFO    | 		Epoch:  91  Loss: 4.45042e-01  CE: 3.83720e-01  AB: 2.16935e-03  SSE: 4.22637e-02  KLD: 1.68984e-02  Batchsize: 1024
2025-01-15 12:16:10.197 | INFO    | 		Epoch:  92  Loss: 4.45046e-01  CE: 3.83703e-01  AB: 2.17870e-03  SSE: 4.22584e-02  KLD: 1.69133e-02  Batchsize: 1024
2025-01-15 12:16:12.902 | INFO    | 		Epoch:  93  Loss: 4.45047e-01  CE: 3.83691e-01  AB: 2.16946e-03  SSE: 4.22910e-02  KLD: 1.68991e-02  Batchsize: 1024
2025-01-15 12:16:15.311 | INFO    | 		Epoch:  94  Loss: 4.45015e-01  CE: 3.83691e-01  AB: 2.15157e-03  SSE: 4.22772e-02  KLD: 1.69089e-02  Batchsize: 1024
2025-01-15 12:16:17.809 | INFO    | 		Epoch:  95  Loss: 4.44965e-01  CE: 3.83653e-01  AB: 2.17272e-03  SSE: 4.22675e-02  KLD: 1.68844e-02  Batchsize: 1024
2025-01-15 12:16:20.263 | INFO    | 		Epoch:  96  Loss: 4.44933e-01  CE: 3.83627e-01  AB: 2.16381e-03  SSE: 4.22349e-02  KLD: 1.69083e-02  Batchsize: 1024
2025-01-15 12:16:22.713 | INFO    | 		Epoch:  97  Loss: 4.44962e-01  CE: 3.83652e-01  AB: 2.16584e-03  SSE: 4.22499e-02  KLD: 1.69005e-02  Batchsize: 1024
2025-01-15 12:16:25.245 | INFO    | 		Epoch:  98  Loss: 4.44976e-01  CE: 3.83637e-01  AB: 2.17026e-03  SSE: 4.22656e-02  KLD: 1.69000e-02  Batchsize: 1024
2025-01-15 12:16:27.636 | INFO    | 		Epoch:  99  Loss: 4.44949e-01  CE: 3.83610e-01  AB: 2.17862e-03  SSE: 4.22465e-02  KLD: 1.69111e-02  Batchsize: 1024
2025-01-15 12:16:30.177 | INFO    | 		Epoch: 100  Loss: 4.44919e-01  CE: 3.83610e-01  AB: 2.17149e-03  SSE: 4.22391e-02  KLD: 1.69014e-02  Batchsize: 1024
2025-01-15 12:16:32.643 | INFO    | 		Epoch: 101  Loss: 4.44797e-01  CE: 3.83566e-01  AB: 2.14790e-03  SSE: 4.22092e-02  KLD: 1.68843e-02  Batchsize: 1024
2025-01-15 12:16:35.094 | INFO    | 		Epoch: 102  Loss: 4.44840e-01  CE: 3.83583e-01  AB: 2.14712e-03  SSE: 4.22263e-02  KLD: 1.68870e-02  Batchsize: 1024
2025-01-15 12:16:37.514 | INFO    | 		Epoch: 103  Loss: 4.44832e-01  CE: 3.83541e-01  AB: 2.16729e-03  SSE: 4.22414e-02  KLD: 1.68805e-02  Batchsize: 1024
2025-01-15 12:16:40.010 | INFO    | 		Epoch: 104  Loss: 4.44781e-01  CE: 3.83534e-01  AB: 2.14720e-03  SSE: 4.22189e-02  KLD: 1.68886e-02  Batchsize: 1024
2025-01-15 12:16:42.485 | INFO    | 		Epoch: 105  Loss: 4.44829e-01  CE: 3.83548e-01  AB: 2.15735e-03  SSE: 4.22315e-02  KLD: 1.68975e-02  Batchsize: 1024
2025-01-15 12:16:44.966 | INFO    | 		Epoch: 106  Loss: 4.44865e-01  CE: 3.83589e-01  AB: 2.15759e-03  SSE: 4.22237e-02  KLD: 1.69077e-02  Batchsize: 1024
2025-01-15 12:16:47.439 | INFO    | 		Epoch: 107  Loss: 4.44850e-01  CE: 3.83547e-01  AB: 2.15314e-03  SSE: 4.22412e-02  KLD: 1.69110e-02  Batchsize: 1024
2025-01-15 12:16:49.865 | INFO    | 		Epoch: 108  Loss: 4.44768e-01  CE: 3.83496e-01  AB: 2.16220e-03  SSE: 4.22163e-02  KLD: 1.68989e-02  Batchsize: 1024
2025-01-15 12:16:52.378 | INFO    | 		Epoch: 109  Loss: 4.44754e-01  CE: 3.83496e-01  AB: 2.16256e-03  SSE: 4.22027e-02  KLD: 1.69045e-02  Batchsize: 1024
2025-01-15 12:16:54.880 | INFO    | 		Epoch: 110  Loss: 4.44750e-01  CE: 3.83480e-01  AB: 2.15472e-03  SSE: 4.22024e-02  KLD: 1.69136e-02  Batchsize: 1024
2025-01-15 12:16:57.377 | INFO    | 		Epoch: 111  Loss: 4.44778e-01  CE: 3.83509e-01  AB: 2.15523e-03  SSE: 4.22066e-02  KLD: 1.69194e-02  Batchsize: 1024
2025-01-15 12:17:00.107 | INFO    | 		Epoch: 112  Loss: 4.44730e-01  CE: 3.83461e-01  AB: 2.15262e-03  SSE: 4.22383e-02  KLD: 1.68910e-02  Batchsize: 1024
2025-01-15 12:17:02.568 | INFO    | 		Epoch: 113  Loss: 4.44710e-01  CE: 3.83467e-01  AB: 2.16305e-03  SSE: 4.21939e-02  KLD: 1.68997e-02  Batchsize: 1024
2025-01-15 12:17:05.126 | INFO    | 		Epoch: 114  Loss: 4.44692e-01  CE: 3.83454e-01  AB: 2.15894e-03  SSE: 4.22008e-02  KLD: 1.68848e-02  Batchsize: 1024
2025-01-15 12:17:07.626 | INFO    | 		Epoch: 115  Loss: 4.44773e-01  CE: 3.83512e-01  AB: 2.15947e-03  SSE: 4.22062e-02  KLD: 1.69031e-02  Batchsize: 1024
2025-01-15 12:17:10.220 | INFO    | 		Epoch: 116  Loss: 4.44783e-01  CE: 3.83500e-01  AB: 2.17040e-03  SSE: 4.22027e-02  KLD: 1.69092e-02  Batchsize: 1024
2025-01-15 12:17:12.699 | INFO    | 		Epoch: 117  Loss: 4.44697e-01  CE: 3.83452e-01  AB: 2.15574e-03  SSE: 4.21872e-02  KLD: 1.69067e-02  Batchsize: 1024
2025-01-15 12:17:15.180 | INFO    | 		Epoch: 118  Loss: 4.44687e-01  CE: 3.83444e-01  AB: 2.16363e-03  SSE: 4.21772e-02  KLD: 1.69056e-02  Batchsize: 1024
2025-01-15 12:17:17.841 | INFO    | 		Epoch: 119  Loss: 4.44636e-01  CE: 3.83421e-01  AB: 2.15680e-03  SSE: 4.21517e-02  KLD: 1.69027e-02  Batchsize: 1024
2025-01-15 12:17:20.307 | INFO    | 		Epoch: 120  Loss: 4.44626e-01  CE: 3.83411e-01  AB: 2.15694e-03  SSE: 4.21471e-02  KLD: 1.69070e-02  Batchsize: 1024
2025-01-15 12:17:22.744 | INFO    | 		Epoch: 121  Loss: 4.44641e-01  CE: 3.83419e-01  AB: 2.16795e-03  SSE: 4.21709e-02  KLD: 1.68979e-02  Batchsize: 1024
2025-01-15 12:17:25.163 | INFO    | 		Epoch: 122  Loss: 4.44635e-01  CE: 3.83410e-01  AB: 2.15464e-03  SSE: 4.21902e-02  KLD: 1.68875e-02  Batchsize: 1024
2025-01-15 12:17:27.542 | INFO    | 		Epoch: 123  Loss: 4.44722e-01  CE: 3.83450e-01  AB: 2.14959e-03  SSE: 4.21967e-02  KLD: 1.69297e-02  Batchsize: 1024
2025-01-15 12:17:30.098 | INFO    | 		Epoch: 124  Loss: 4.44590e-01  CE: 3.83388e-01  AB: 2.14699e-03  SSE: 4.21737e-02  KLD: 1.68914e-02  Batchsize: 1024
2025-01-15 12:17:32.506 | INFO    | 		Epoch: 125  Loss: 4.44588e-01  CE: 3.83373e-01  AB: 2.16250e-03  SSE: 4.21469e-02  KLD: 1.69102e-02  Batchsize: 1024
2025-01-15 12:17:34.891 | INFO    | 		Epoch: 126  Loss: 4.44551e-01  CE: 3.83356e-01  AB: 2.15549e-03  SSE: 4.21559e-02  KLD: 1.68861e-02  Batchsize: 1024
2025-01-15 12:17:37.311 | INFO    | 		Epoch: 127  Loss: 4.44529e-01  CE: 3.83370e-01  AB: 2.15036e-03  SSE: 4.21323e-02  KLD: 1.68859e-02  Batchsize: 1024
2025-01-15 12:17:39.738 | INFO    | 		Epoch: 128  Loss: 4.44512e-01  CE: 3.83329e-01  AB: 2.16228e-03  SSE: 4.21479e-02  KLD: 1.68782e-02  Batchsize: 1024
2025-01-15 12:17:42.261 | INFO    | 		Epoch: 129  Loss: 4.44547e-01  CE: 3.83327e-01  AB: 2.16696e-03  SSE: 4.21475e-02  KLD: 1.69104e-02  Batchsize: 1024
2025-01-15 12:17:45.027 | INFO    | 		Epoch: 130  Loss: 4.44507e-01  CE: 3.83315e-01  AB: 2.15993e-03  SSE: 4.21464e-02  KLD: 1.68876e-02  Batchsize: 1024
2025-01-15 12:17:47.496 | INFO    | 		Epoch: 131  Loss: 4.44504e-01  CE: 3.83328e-01  AB: 2.14059e-03  SSE: 4.21190e-02  KLD: 1.69224e-02  Batchsize: 1024
2025-01-15 12:17:49.987 | INFO    | 		Epoch: 132  Loss: 4.44476e-01  CE: 3.83275e-01  AB: 2.15022e-03  SSE: 4.21613e-02  KLD: 1.69015e-02  Batchsize: 1024
2025-01-15 12:17:52.617 | INFO    | 		Epoch: 133  Loss: 4.44517e-01  CE: 3.83343e-01  AB: 2.16025e-03  SSE: 4.21384e-02  KLD: 1.68749e-02  Batchsize: 1024
2025-01-15 12:17:55.141 | INFO    | 		Epoch: 134  Loss: 4.44438e-01  CE: 3.83246e-01  AB: 2.16017e-03  SSE: 4.21324e-02  KLD: 1.69022e-02  Batchsize: 1024
2025-01-15 12:17:57.863 | INFO    | 		Epoch: 135  Loss: 4.44484e-01  CE: 3.83302e-01  AB: 2.14472e-03  SSE: 4.21507e-02  KLD: 1.69040e-02  Batchsize: 1024
2025-01-15 12:18:00.453 | INFO    | 		Epoch: 136  Loss: 4.44510e-01  CE: 3.83316e-01  AB: 2.15655e-03  SSE: 4.21231e-02  KLD: 1.69143e-02  Batchsize: 1024
2025-01-15 12:18:02.965 | INFO    | 		Epoch: 137  Loss: 4.44509e-01  CE: 3.83314e-01  AB: 2.16455e-03  SSE: 4.21258e-02  KLD: 1.69196e-02  Batchsize: 1024
2025-01-15 12:18:05.610 | INFO    | 		Epoch: 138  Loss: 4.44497e-01  CE: 3.83313e-01  AB: 2.14365e-03  SSE: 4.21349e-02  KLD: 1.69117e-02  Batchsize: 1024
2025-01-15 12:18:08.070 | INFO    | 		Epoch: 139  Loss: 4.44489e-01  CE: 3.83302e-01  AB: 2.14615e-03  SSE: 4.21447e-02  KLD: 1.69059e-02  Batchsize: 1024
2025-01-15 12:18:10.512 | INFO    | 		Epoch: 140  Loss: 4.44467e-01  CE: 3.83275e-01  AB: 2.15268e-03  SSE: 4.21299e-02  KLD: 1.69114e-02  Batchsize: 1024
2025-01-15 12:18:13.017 | INFO    | 		Epoch: 141  Loss: 4.44408e-01  CE: 3.83259e-01  AB: 2.14603e-03  SSE: 4.21172e-02  KLD: 1.68849e-02  Batchsize: 1024
2025-01-15 12:18:15.609 | INFO    | 		Epoch: 142  Loss: 4.44437e-01  CE: 3.83278e-01  AB: 2.15389e-03  SSE: 4.21027e-02  KLD: 1.69115e-02  Batchsize: 1024
2025-01-15 12:18:18.015 | INFO    | 		Epoch: 143  Loss: 4.44338e-01  CE: 3.83242e-01  AB: 2.13246e-03  SSE: 4.20766e-02  KLD: 1.68953e-02  Batchsize: 1024
2025-01-15 12:18:20.568 | INFO    | 		Epoch: 144  Loss: 4.44380e-01  CE: 3.83228e-01  AB: 2.14880e-03  SSE: 4.21198e-02  KLD: 1.68938e-02  Batchsize: 1024
2025-01-15 12:18:23.011 | INFO    | 		Epoch: 145  Loss: 4.44387e-01  CE: 3.83270e-01  AB: 2.14121e-03  SSE: 4.21023e-02  KLD: 1.68833e-02  Batchsize: 1024
2025-01-15 12:18:25.802 | INFO    | 		Epoch: 146  Loss: 4.44394e-01  CE: 3.83240e-01  AB: 2.14812e-03  SSE: 4.21189e-02  KLD: 1.68965e-02  Batchsize: 1024
2025-01-15 12:18:28.390 | INFO    | 		Epoch: 147  Loss: 4.44350e-01  CE: 3.83197e-01  AB: 2.15277e-03  SSE: 4.21025e-02  KLD: 1.69051e-02  Batchsize: 1024
2025-01-15 12:18:30.847 | INFO    | 		Epoch: 148  Loss: 4.44341e-01  CE: 3.83182e-01  AB: 2.13828e-03  SSE: 4.21014e-02  KLD: 1.69192e-02  Batchsize: 1024
2025-01-15 12:18:33.291 | INFO    | 		Epoch: 149  Loss: 4.44377e-01  CE: 3.83242e-01  AB: 2.14528e-03  SSE: 4.20894e-02  KLD: 1.69037e-02  Batchsize: 1024
2025-01-15 12:18:35.903 | INFO    | 		Epoch: 150  Loss: 4.44377e-01  CE: 3.83225e-01  AB: 2.14188e-03  SSE: 4.20926e-02  KLD: 1.69222e-02  Batchsize: 1024
2025-01-15 12:18:37.507 | INFO    | 		Epoch: 151  Loss: 4.43511e-01  CE: 3.82737e-01  AB: 2.08931e-03  SSE: 4.18353e-02  KLD: 1.68537e-02  Batchsize: 2048
2025-01-15 12:18:39.103 | INFO    | 		Epoch: 152  Loss: 4.43397e-01  CE: 3.82653e-01  AB: 2.09444e-03  SSE: 4.17740e-02  KLD: 1.68737e-02  Batchsize: 2048
2025-01-15 12:18:40.718 | INFO    | 		Epoch: 153  Loss: 4.43309e-01  CE: 3.82623e-01  AB: 2.09482e-03  SSE: 4.17683e-02  KLD: 1.68318e-02  Batchsize: 2048
2025-01-15 12:18:42.290 | INFO    | 		Epoch: 154  Loss: 4.43349e-01  CE: 3.82637e-01  AB: 2.09874e-03  SSE: 4.17606e-02  KLD: 1.68483e-02  Batchsize: 2048
2025-01-15 12:18:43.863 | INFO    | 		Epoch: 155  Loss: 4.43354e-01  CE: 3.82616e-01  AB: 2.09387e-03  SSE: 4.17853e-02  KLD: 1.68666e-02  Batchsize: 2048
2025-01-15 12:18:45.415 | INFO    | 		Epoch: 156  Loss: 4.43316e-01  CE: 3.82616e-01  AB: 2.08694e-03  SSE: 4.17548e-02  KLD: 1.68581e-02  Batchsize: 2048
2025-01-15 12:18:47.056 | INFO    | 		Epoch: 157  Loss: 4.43345e-01  CE: 3.82635e-01  AB: 2.09300e-03  SSE: 4.17635e-02  KLD: 1.68666e-02  Batchsize: 2048
2025-01-15 12:18:48.742 | INFO    | 		Epoch: 158  Loss: 4.43397e-01  CE: 3.82638e-01  AB: 2.08467e-03  SSE: 4.17844e-02  KLD: 1.68921e-02  Batchsize: 2048
2025-01-15 12:18:50.449 | INFO    | 		Epoch: 159  Loss: 4.43375e-01  CE: 3.82636e-01  AB: 2.09655e-03  SSE: 4.17593e-02  KLD: 1.68924e-02  Batchsize: 2048
2025-01-15 12:18:52.005 | INFO    | 		Epoch: 160  Loss: 4.43371e-01  CE: 3.82623e-01  AB: 2.09096e-03  SSE: 4.17871e-02  KLD: 1.68675e-02  Batchsize: 2048
2025-01-15 12:18:53.715 | INFO    | 		Epoch: 161  Loss: 4.43379e-01  CE: 3.82616e-01  AB: 2.10467e-03  SSE: 4.17755e-02  KLD: 1.68903e-02  Batchsize: 2048
2025-01-15 12:18:55.373 | INFO    | 		Epoch: 162  Loss: 4.43347e-01  CE: 3.82627e-01  AB: 2.09154e-03  SSE: 4.17623e-02  KLD: 1.68665e-02  Batchsize: 2048
2025-01-15 12:18:56.987 | INFO    | 		Epoch: 163  Loss: 4.43396e-01  CE: 3.82657e-01  AB: 2.09300e-03  SSE: 4.17689e-02  KLD: 1.68783e-02  Batchsize: 2048
2025-01-15 12:18:58.618 | INFO    | 		Epoch: 164  Loss: 4.43434e-01  CE: 3.82662e-01  AB: 2.10286e-03  SSE: 4.17768e-02  KLD: 1.68880e-02  Batchsize: 2048
2025-01-15 12:19:00.345 | INFO    | 		Epoch: 165  Loss: 4.43343e-01  CE: 3.82611e-01  AB: 2.08887e-03  SSE: 4.17737e-02  KLD: 1.68735e-02  Batchsize: 2048
2025-01-15 12:19:02.061 | INFO    | 		Epoch: 166  Loss: 4.43443e-01  CE: 3.82669e-01  AB: 2.09846e-03  SSE: 4.17920e-02  KLD: 1.68881e-02  Batchsize: 2048
2025-01-15 12:19:03.656 | INFO    | 		Epoch: 167  Loss: 4.43365e-01  CE: 3.82620e-01  AB: 2.10144e-03  SSE: 4.17591e-02  KLD: 1.68845e-02  Batchsize: 2048
2025-01-15 12:19:05.238 | INFO    | 		Epoch: 168  Loss: 4.43406e-01  CE: 3.82662e-01  AB: 2.09739e-03  SSE: 4.17716e-02  KLD: 1.68801e-02  Batchsize: 2048
2025-01-15 12:19:06.809 | INFO    | 		Epoch: 169  Loss: 4.43357e-01  CE: 3.82623e-01  AB: 2.09292e-03  SSE: 4.17580e-02  KLD: 1.68813e-02  Batchsize: 2048
2025-01-15 12:19:08.425 | INFO    | 		Epoch: 170  Loss: 4.43422e-01  CE: 3.82667e-01  AB: 2.09131e-03  SSE: 4.17750e-02  KLD: 1.68943e-02  Batchsize: 2048
2025-01-15 12:19:10.020 | INFO    | 		Epoch: 171  Loss: 4.43295e-01  CE: 3.82608e-01  AB: 2.08536e-03  SSE: 4.17448e-02  KLD: 1.68657e-02  Batchsize: 2048
2025-01-15 12:19:11.624 | INFO    | 		Epoch: 172  Loss: 4.43345e-01  CE: 3.82599e-01  AB: 2.09639e-03  SSE: 4.17757e-02  KLD: 1.68804e-02  Batchsize: 2048
2025-01-15 12:19:13.187 | INFO    | 		Epoch: 173  Loss: 4.43298e-01  CE: 3.82565e-01  AB: 2.09784e-03  SSE: 4.17537e-02  KLD: 1.68817e-02  Batchsize: 2048
2025-01-15 12:19:14.866 | INFO    | 		Epoch: 174  Loss: 4.43328e-01  CE: 3.82610e-01  AB: 2.10426e-03  SSE: 4.17423e-02  KLD: 1.68726e-02  Batchsize: 2048
2025-01-15 12:19:16.449 | INFO    | 		Epoch: 175  Loss: 4.43358e-01  CE: 3.82592e-01  AB: 2.09323e-03  SSE: 4.17745e-02  KLD: 1.68973e-02  Batchsize: 2048
2025-01-15 12:19:18.096 | INFO    | 		Epoch: 176  Loss: 4.43312e-01  CE: 3.82608e-01  AB: 2.10795e-03  SSE: 4.17318e-02  KLD: 1.68743e-02  Batchsize: 2048
2025-01-15 12:19:19.734 | INFO    | 		Epoch: 177  Loss: 4.43332e-01  CE: 3.82594e-01  AB: 2.09632e-03  SSE: 4.17741e-02  KLD: 1.68736e-02  Batchsize: 2048
2025-01-15 12:19:21.371 | INFO    | 		Epoch: 178  Loss: 4.43328e-01  CE: 3.82565e-01  AB: 2.09129e-03  SSE: 4.17691e-02  KLD: 1.69109e-02  Batchsize: 2048
2025-01-15 12:19:22.937 | INFO    | 		Epoch: 179  Loss: 4.43327e-01  CE: 3.82607e-01  AB: 2.08975e-03  SSE: 4.17585e-02  KLD: 1.68818e-02  Batchsize: 2048
2025-01-15 12:19:24.562 | INFO    | 		Epoch: 180  Loss: 4.43299e-01  CE: 3.82560e-01  AB: 2.08810e-03  SSE: 4.17798e-02  KLD: 1.68750e-02  Batchsize: 2048
2025-01-15 12:19:26.126 | INFO    | 		Epoch: 181  Loss: 4.43273e-01  CE: 3.82572e-01  AB: 2.09944e-03  SSE: 4.17307e-02  KLD: 1.68782e-02  Batchsize: 2048
2025-01-15 12:19:27.703 | INFO    | 		Epoch: 182  Loss: 4.43302e-01  CE: 3.82604e-01  AB: 2.09380e-03  SSE: 4.17435e-02  KLD: 1.68683e-02  Batchsize: 2048
2025-01-15 12:19:29.268 | INFO    | 		Epoch: 183  Loss: 4.43269e-01  CE: 3.82522e-01  AB: 2.09545e-03  SSE: 4.17649e-02  KLD: 1.68846e-02  Batchsize: 2048
2025-01-15 12:19:30.866 | INFO    | 		Epoch: 184  Loss: 4.43352e-01  CE: 3.82626e-01  AB: 2.09016e-03  SSE: 4.17511e-02  KLD: 1.68887e-02  Batchsize: 2048
2025-01-15 12:19:32.474 | INFO    | 		Epoch: 185  Loss: 4.43304e-01  CE: 3.82556e-01  AB: 2.09116e-03  SSE: 4.17726e-02  KLD: 1.68877e-02  Batchsize: 2048
2025-01-15 12:19:34.034 | INFO    | 		Epoch: 186  Loss: 4.43338e-01  CE: 3.82588e-01  AB: 2.10494e-03  SSE: 4.17568e-02  KLD: 1.68924e-02  Batchsize: 2048
2025-01-15 12:19:35.767 | INFO    | 		Epoch: 187  Loss: 4.43285e-01  CE: 3.82572e-01  AB: 2.09813e-03  SSE: 4.17338e-02  KLD: 1.68827e-02  Batchsize: 2048
2025-01-15 12:19:37.461 | INFO    | 		Epoch: 188  Loss: 4.43265e-01  CE: 3.82571e-01  AB: 2.08312e-03  SSE: 4.17316e-02  KLD: 1.68844e-02  Batchsize: 2048
2025-01-15 12:19:39.120 | INFO    | 		Epoch: 189  Loss: 4.43262e-01  CE: 3.82573e-01  AB: 2.09296e-03  SSE: 4.17611e-02  KLD: 1.68423e-02  Batchsize: 2048
2025-01-15 12:19:40.846 | INFO    | 		Epoch: 190  Loss: 4.43218e-01  CE: 3.82524e-01  AB: 2.09565e-03  SSE: 4.17266e-02  KLD: 1.68782e-02  Batchsize: 2048
2025-01-15 12:19:42.418 | INFO    | 		Epoch: 191  Loss: 4.43288e-01  CE: 3.82552e-01  AB: 2.10209e-03  SSE: 4.17518e-02  KLD: 1.68881e-02  Batchsize: 2048
2025-01-15 12:19:43.988 | INFO    | 		Epoch: 192  Loss: 4.43302e-01  CE: 3.82555e-01  AB: 2.10329e-03  SSE: 4.17597e-02  KLD: 1.68987e-02  Batchsize: 2048
2025-01-15 12:19:45.703 | INFO    | 		Epoch: 193  Loss: 4.43269e-01  CE: 3.82571e-01  AB: 2.09337e-03  SSE: 4.17305e-02  KLD: 1.68829e-02  Batchsize: 2048
2025-01-15 12:19:47.266 | INFO    | 		Epoch: 194  Loss: 4.43203e-01  CE: 3.82517e-01  AB: 2.08903e-03  SSE: 4.17210e-02  KLD: 1.68785e-02  Batchsize: 2048
2025-01-15 12:19:48.869 | INFO    | 		Epoch: 195  Loss: 4.43243e-01  CE: 3.82550e-01  AB: 2.09094e-03  SSE: 4.17249e-02  KLD: 1.68815e-02  Batchsize: 2048
2025-01-15 12:19:50.525 | INFO    | 		Epoch: 196  Loss: 4.43230e-01  CE: 3.82525e-01  AB: 2.09510e-03  SSE: 4.17357e-02  KLD: 1.68810e-02  Batchsize: 2048
2025-01-15 12:19:52.143 | INFO    | 		Epoch: 197  Loss: 4.43236e-01  CE: 3.82520e-01  AB: 2.09627e-03  SSE: 4.17316e-02  KLD: 1.68905e-02  Batchsize: 2048
2025-01-15 12:19:53.707 | INFO    | 		Epoch: 198  Loss: 4.43182e-01  CE: 3.82484e-01  AB: 2.08335e-03  SSE: 4.17177e-02  KLD: 1.68949e-02  Batchsize: 2048
2025-01-15 12:19:55.367 | INFO    | 		Epoch: 199  Loss: 4.43210e-01  CE: 3.82536e-01  AB: 2.09322e-03  SSE: 4.17026e-02  KLD: 1.68813e-02  Batchsize: 2048
2025-01-15 12:19:56.937 | INFO    | 		Epoch: 200  Loss: 4.43191e-01  CE: 3.82518e-01  AB: 2.09203e-03  SSE: 4.17130e-02  KLD: 1.68686e-02  Batchsize: 2048
2025-01-15 12:19:58.504 | INFO    | 		Epoch: 201  Loss: 4.43216e-01  CE: 3.82489e-01  AB: 2.10457e-03  SSE: 4.17252e-02  KLD: 1.68916e-02  Batchsize: 2048
2025-01-15 12:20:00.169 | INFO    | 		Epoch: 202  Loss: 4.43160e-01  CE: 3.82495e-01  AB: 2.08839e-03  SSE: 4.17139e-02  KLD: 1.68649e-02  Batchsize: 2048
2025-01-15 12:20:01.768 | INFO    | 		Epoch: 203  Loss: 4.43159e-01  CE: 3.82479e-01  AB: 2.09322e-03  SSE: 4.17081e-02  KLD: 1.68812e-02  Batchsize: 2048
2025-01-15 12:20:03.339 | INFO    | 		Epoch: 204  Loss: 4.43205e-01  CE: 3.82533e-01  AB: 2.09256e-03  SSE: 4.17109e-02  KLD: 1.68761e-02  Batchsize: 2048
2025-01-15 12:20:04.914 | INFO    | 		Epoch: 205  Loss: 4.43154e-01  CE: 3.82490e-01  AB: 2.10151e-03  SSE: 4.16844e-02  KLD: 1.68818e-02  Batchsize: 2048
2025-01-15 12:20:06.508 | INFO    | 		Epoch: 206  Loss: 4.43135e-01  CE: 3.82506e-01  AB: 2.07500e-03  SSE: 4.16785e-02  KLD: 1.68792e-02  Batchsize: 2048
2025-01-15 12:20:08.111 | INFO    | 		Epoch: 207  Loss: 4.43162e-01  CE: 3.82494e-01  AB: 2.08833e-03  SSE: 4.16929e-02  KLD: 1.68849e-02  Batchsize: 2048
2025-01-15 12:20:09.719 | INFO    | 		Epoch: 208  Loss: 4.43106e-01  CE: 3.82484e-01  AB: 2.08136e-03  SSE: 4.16884e-02  KLD: 1.68577e-02  Batchsize: 2048
2025-01-15 12:20:11.316 | INFO    | 		Epoch: 209  Loss: 4.43119e-01  CE: 3.82461e-01  AB: 2.08386e-03  SSE: 4.16920e-02  KLD: 1.68884e-02  Batchsize: 2048
2025-01-15 12:20:12.941 | INFO    | 		Epoch: 210  Loss: 4.43132e-01  CE: 3.82463e-01  AB: 2.08810e-03  SSE: 4.16871e-02  KLD: 1.68991e-02  Batchsize: 2048
2025-01-15 12:20:14.541 | INFO    | 		Epoch: 211  Loss: 4.43120e-01  CE: 3.82452e-01  AB: 2.09262e-03  SSE: 4.16933e-02  KLD: 1.68835e-02  Batchsize: 2048
2025-01-15 12:20:16.150 | INFO    | 		Epoch: 212  Loss: 4.43213e-01  CE: 3.82548e-01  AB: 2.10010e-03  SSE: 4.16926e-02  KLD: 1.68787e-02  Batchsize: 2048
2025-01-15 12:20:17.775 | INFO    | 		Epoch: 213  Loss: 4.43180e-01  CE: 3.82473e-01  AB: 2.10074e-03  SSE: 4.17007e-02  KLD: 1.69117e-02  Batchsize: 2048
2025-01-15 12:20:19.422 | INFO    | 		Epoch: 214  Loss: 4.43192e-01  CE: 3.82475e-01  AB: 2.09843e-03  SSE: 4.17028e-02  KLD: 1.69171e-02  Batchsize: 2048
2025-01-15 12:20:21.055 | INFO    | 		Epoch: 215  Loss: 4.43098e-01  CE: 3.82439e-01  AB: 2.08528e-03  SSE: 4.16781e-02  KLD: 1.68912e-02  Batchsize: 2048
2025-01-15 12:20:22.640 | INFO    | 		Epoch: 216  Loss: 4.43155e-01  CE: 3.82498e-01  AB: 2.09398e-03  SSE: 4.16767e-02  KLD: 1.68853e-02  Batchsize: 2048
2025-01-15 12:20:24.241 | INFO    | 		Epoch: 217  Loss: 4.43090e-01  CE: 3.82431e-01  AB: 2.08800e-03  SSE: 4.16878e-02  KLD: 1.68802e-02  Batchsize: 2048
2025-01-15 12:20:25.870 | INFO    | 		Epoch: 218  Loss: 4.43170e-01  CE: 3.82487e-01  AB: 2.09514e-03  SSE: 4.17029e-02  KLD: 1.68897e-02  Batchsize: 2048
2025-01-15 12:20:27.513 | INFO    | 		Epoch: 219  Loss: 4.43117e-01  CE: 3.82438e-01  AB: 2.07924e-03  SSE: 4.16964e-02  KLD: 1.69035e-02  Batchsize: 2048
2025-01-15 12:20:29.130 | INFO    | 		Epoch: 220  Loss: 4.43013e-01  CE: 3.82381e-01  AB: 2.08999e-03  SSE: 4.16672e-02  KLD: 1.68833e-02  Batchsize: 2048
2025-01-15 12:20:30.674 | INFO    | 		Epoch: 221  Loss: 4.43120e-01  CE: 3.82448e-01  AB: 2.07809e-03  SSE: 4.16975e-02  KLD: 1.69054e-02  Batchsize: 2048
2025-01-15 12:20:32.298 | INFO    | 		Epoch: 222  Loss: 4.43122e-01  CE: 3.82477e-01  AB: 2.09369e-03  SSE: 4.16722e-02  KLD: 1.68803e-02  Batchsize: 2048
2025-01-15 12:20:33.899 | INFO    | 		Epoch: 223  Loss: 4.43062e-01  CE: 3.82413e-01  AB: 2.08942e-03  SSE: 4.16806e-02  KLD: 1.68851e-02  Batchsize: 2048
2025-01-15 12:20:35.479 | INFO    | 		Epoch: 224  Loss: 4.43044e-01  CE: 3.82433e-01  AB: 2.07263e-03  SSE: 4.16586e-02  KLD: 1.68771e-02  Batchsize: 2048
2025-01-15 12:20:37.056 | INFO    | 		Epoch: 225  Loss: 4.43098e-01  CE: 3.82424e-01  AB: 2.09086e-03  SSE: 4.16941e-02  KLD: 1.68932e-02  Batchsize: 2048
2025-01-15 12:20:38.396 | INFO    | 		Epoch: 226  Loss: 4.42608e-01  CE: 3.82169e-01  AB: 2.05136e-03  SSE: 4.15316e-02  KLD: 1.68556e-02  Batchsize: 4096
2025-01-15 12:20:39.720 | INFO    | 		Epoch: 227  Loss: 4.42427e-01  CE: 3.82070e-01  AB: 2.04548e-03  SSE: 4.14775e-02  KLD: 1.68368e-02  Batchsize: 4096
2025-01-15 12:20:41.130 | INFO    | 		Epoch: 228  Loss: 4.42414e-01  CE: 3.82045e-01  AB: 2.04300e-03  SSE: 4.14597e-02  KLD: 1.68627e-02  Batchsize: 4096
2025-01-15 12:20:42.485 | INFO    | 		Epoch: 229  Loss: 4.42372e-01  CE: 3.82073e-01  AB: 2.04426e-03  SSE: 4.14379e-02  KLD: 1.68192e-02  Batchsize: 4096
2025-01-15 12:20:43.904 | INFO    | 		Epoch: 230  Loss: 4.42419e-01  CE: 3.82056e-01  AB: 2.06696e-03  SSE: 4.14429e-02  KLD: 1.68533e-02  Batchsize: 4096
2025-01-15 12:20:45.220 | INFO    | 		Epoch: 231  Loss: 4.42426e-01  CE: 3.82023e-01  AB: 2.05732e-03  SSE: 4.14548e-02  KLD: 1.68960e-02  Batchsize: 4096
2025-01-15 12:20:46.607 | INFO    | 		Epoch: 232  Loss: 4.42440e-01  CE: 3.82056e-01  AB: 2.05144e-03  SSE: 4.14541e-02  KLD: 1.68776e-02  Batchsize: 4096
2025-01-15 12:20:47.996 | INFO    | 		Epoch: 233  Loss: 4.42372e-01  CE: 3.82018e-01  AB: 2.05211e-03  SSE: 4.14317e-02  KLD: 1.68688e-02  Batchsize: 4096
2025-01-15 12:20:49.407 | INFO    | 		Epoch: 234  Loss: 4.42429e-01  CE: 3.82033e-01  AB: 2.06059e-03  SSE: 4.14690e-02  KLD: 1.68649e-02  Batchsize: 4096
2025-01-15 12:20:50.801 | INFO    | 		Epoch: 235  Loss: 4.42372e-01  CE: 3.82041e-01  AB: 2.03702e-03  SSE: 4.14548e-02  KLD: 1.68381e-02  Batchsize: 4096
2025-01-15 12:20:52.179 | INFO    | 		Epoch: 236  Loss: 4.42422e-01  CE: 3.82051e-01  AB: 2.05900e-03  SSE: 4.14566e-02  KLD: 1.68564e-02  Batchsize: 4096
2025-01-15 12:20:53.665 | INFO    | 		Epoch: 237  Loss: 4.42432e-01  CE: 3.82052e-01  AB: 2.04325e-03  SSE: 4.14637e-02  KLD: 1.68718e-02  Batchsize: 4096
2025-01-15 12:20:55.172 | INFO    | 		Epoch: 238  Loss: 4.42466e-01  CE: 3.82057e-01  AB: 2.05600e-03  SSE: 4.14672e-02  KLD: 1.68870e-02  Batchsize: 4096
2025-01-15 12:20:56.498 | INFO    | 		Epoch: 239  Loss: 4.42409e-01  CE: 3.82006e-01  AB: 2.04831e-03  SSE: 4.14749e-02  KLD: 1.68842e-02  Batchsize: 4096
2025-01-15 12:20:57.839 | INFO    | 		Epoch: 240  Loss: 4.42426e-01  CE: 3.82057e-01  AB: 2.05173e-03  SSE: 4.14478e-02  KLD: 1.68739e-02  Batchsize: 4096
2025-01-15 12:20:59.196 | INFO    | 		Epoch: 241  Loss: 4.42458e-01  CE: 3.82063e-01  AB: 2.05128e-03  SSE: 4.14425e-02  KLD: 1.69037e-02  Batchsize: 4096
2025-01-15 12:21:00.669 | INFO    | 		Epoch: 242  Loss: 4.42447e-01  CE: 3.82043e-01  AB: 2.04925e-03  SSE: 4.14811e-02  KLD: 1.68763e-02  Batchsize: 4096
2025-01-15 12:21:01.953 | INFO    | 		Epoch: 243  Loss: 4.42439e-01  CE: 3.82076e-01  AB: 2.05464e-03  SSE: 4.14370e-02  KLD: 1.68794e-02  Batchsize: 4096
2025-01-15 12:21:03.349 | INFO    | 		Epoch: 244  Loss: 4.42465e-01  CE: 3.82059e-01  AB: 2.05382e-03  SSE: 4.14803e-02  KLD: 1.68734e-02  Batchsize: 4096
2025-01-15 12:21:04.797 | INFO    | 		Epoch: 245  Loss: 4.42478e-01  CE: 3.82077e-01  AB: 2.05078e-03  SSE: 4.14804e-02  KLD: 1.68717e-02  Batchsize: 4096
2025-01-15 12:21:06.104 | INFO    | 		Epoch: 246  Loss: 4.42429e-01  CE: 3.82028e-01  AB: 2.05821e-03  SSE: 4.14663e-02  KLD: 1.68791e-02  Batchsize: 4096
2025-01-15 12:21:07.471 | INFO    | 		Epoch: 247  Loss: 4.42456e-01  CE: 3.82038e-01  AB: 2.05456e-03  SSE: 4.14686e-02  KLD: 1.68973e-02  Batchsize: 4096
2025-01-15 12:21:08.921 | INFO    | 		Epoch: 248  Loss: 4.42423e-01  CE: 3.82040e-01  AB: 2.06411e-03  SSE: 4.14504e-02  KLD: 1.68683e-02  Batchsize: 4096
2025-01-15 12:21:10.351 | INFO    | 		Epoch: 249  Loss: 4.42412e-01  CE: 3.82035e-01  AB: 2.05199e-03  SSE: 4.14584e-02  KLD: 1.68653e-02  Batchsize: 4096
2025-01-15 12:21:11.761 | INFO    | 		Epoch: 250  Loss: 4.42384e-01  CE: 3.82015e-01  AB: 2.04022e-03  SSE: 4.14638e-02  KLD: 1.68674e-02  Batchsize: 4096
2025-01-15 12:21:13.105 | INFO    | 		Epoch: 251  Loss: 4.42450e-01  CE: 3.82053e-01  AB: 2.04859e-03  SSE: 4.14626e-02  KLD: 1.68863e-02  Batchsize: 4096
2025-01-15 12:21:14.509 | INFO    | 		Epoch: 252  Loss: 4.42377e-01  CE: 3.82050e-01  AB: 2.05216e-03  SSE: 4.14185e-02  KLD: 1.68569e-02  Batchsize: 4096
2025-01-15 12:21:15.962 | INFO    | 		Epoch: 253  Loss: 4.42413e-01  CE: 3.82020e-01  AB: 2.06107e-03  SSE: 4.14607e-02  KLD: 1.68752e-02  Batchsize: 4096
2025-01-15 12:21:17.436 | INFO    | 		Epoch: 254  Loss: 4.42477e-01  CE: 3.82033e-01  AB: 2.05171e-03  SSE: 4.14808e-02  KLD: 1.69157e-02  Batchsize: 4096
2025-01-15 12:21:18.907 | INFO    | 		Epoch: 255  Loss: 4.42423e-01  CE: 3.82022e-01  AB: 2.04250e-03  SSE: 4.14656e-02  KLD: 1.68901e-02  Batchsize: 4096
2025-01-15 12:21:20.336 | INFO    | 		Epoch: 256  Loss: 4.42422e-01  CE: 3.82036e-01  AB: 2.04182e-03  SSE: 4.14472e-02  KLD: 1.68986e-02  Batchsize: 4096
2025-01-15 12:21:21.654 | INFO    | 		Epoch: 257  Loss: 4.42392e-01  CE: 3.82053e-01  AB: 2.05024e-03  SSE: 4.14206e-02  KLD: 1.68639e-02  Batchsize: 4096
2025-01-15 12:21:23.029 | INFO    | 		Epoch: 258  Loss: 4.42343e-01  CE: 3.82031e-01  AB: 2.02749e-03  SSE: 4.14284e-02  KLD: 1.68613e-02  Batchsize: 4096
2025-01-15 12:21:24.362 | INFO    | 		Epoch: 259  Loss: 4.42431e-01  CE: 3.82035e-01  AB: 2.05373e-03  SSE: 4.14637e-02  KLD: 1.68847e-02  Batchsize: 4096
2025-01-15 12:21:25.770 | INFO    | 		Epoch: 260  Loss: 4.42440e-01  CE: 3.82019e-01  AB: 2.04881e-03  SSE: 4.14885e-02  KLD: 1.68802e-02  Batchsize: 4096
2025-01-15 12:21:27.086 | INFO    | 		Epoch: 261  Loss: 4.42362e-01  CE: 3.81991e-01  AB: 2.04578e-03  SSE: 4.14333e-02  KLD: 1.68914e-02  Batchsize: 4096
2025-01-15 12:21:28.414 | INFO    | 		Epoch: 262  Loss: 4.42394e-01  CE: 3.82007e-01  AB: 2.04870e-03  SSE: 4.14663e-02  KLD: 1.68729e-02  Batchsize: 4096
2025-01-15 12:21:29.719 | INFO    | 		Epoch: 263  Loss: 4.42357e-01  CE: 3.82023e-01  AB: 2.04072e-03  SSE: 4.14255e-02  KLD: 1.68732e-02  Batchsize: 4096
2025-01-15 12:21:31.004 | INFO    | 		Epoch: 264  Loss: 4.42317e-01  CE: 3.82000e-01  AB: 2.04096e-03  SSE: 4.14316e-02  KLD: 1.68503e-02  Batchsize: 4096
2025-01-15 12:21:32.532 | INFO    | 		Epoch: 265  Loss: 4.42398e-01  CE: 3.82002e-01  AB: 2.04121e-03  SSE: 4.14536e-02  KLD: 1.69028e-02  Batchsize: 4096
2025-01-15 12:21:33.911 | INFO    | 		Epoch: 266  Loss: 4.42436e-01  CE: 3.82033e-01  AB: 2.05233e-03  SSE: 4.14507e-02  KLD: 1.69033e-02  Batchsize: 4096
2025-01-15 12:21:35.355 | INFO    | 		Epoch: 267  Loss: 4.42414e-01  CE: 3.82024e-01  AB: 2.05289e-03  SSE: 4.14332e-02  KLD: 1.69046e-02  Batchsize: 4096
2025-01-15 12:21:36.835 | INFO    | 		Epoch: 268  Loss: 4.42404e-01  CE: 3.82058e-01  AB: 2.04661e-03  SSE: 4.14284e-02  KLD: 1.68742e-02  Batchsize: 4096
2025-01-15 12:21:38.189 | INFO    | 		Epoch: 269  Loss: 4.42367e-01  CE: 3.82001e-01  AB: 2.05023e-03  SSE: 4.14495e-02  KLD: 1.68685e-02  Batchsize: 4096
2025-01-15 12:21:39.594 | INFO    | 		Epoch: 270  Loss: 4.42340e-01  CE: 3.81985e-01  AB: 2.05078e-03  SSE: 4.14281e-02  KLD: 1.68758e-02  Batchsize: 4096
2025-01-15 12:21:41.028 | INFO    | 		Epoch: 271  Loss: 4.42404e-01  CE: 3.82021e-01  AB: 2.05144e-03  SSE: 4.14516e-02  KLD: 1.68818e-02  Batchsize: 4096
2025-01-15 12:21:42.380 | INFO    | 		Epoch: 272  Loss: 4.42327e-01  CE: 3.82005e-01  AB: 2.05170e-03  SSE: 4.14133e-02  KLD: 1.68576e-02  Batchsize: 4096
2025-01-15 12:21:43.910 | INFO    | 		Epoch: 273  Loss: 4.42351e-01  CE: 3.81999e-01  AB: 2.05656e-03  SSE: 4.14354e-02  KLD: 1.68586e-02  Batchsize: 4096
2025-01-15 12:21:45.254 | INFO    | 		Epoch: 274  Loss: 4.42407e-01  CE: 3.81996e-01  AB: 2.04484e-03  SSE: 4.14774e-02  KLD: 1.68878e-02  Batchsize: 4096
2025-01-15 12:21:46.643 | INFO    | 		Epoch: 275  Loss: 4.42310e-01  CE: 3.81978e-01  AB: 2.04576e-03  SSE: 4.14310e-02  KLD: 1.68552e-02  Batchsize: 4096
2025-01-15 12:21:47.982 | INFO    | 		Epoch: 276  Loss: 4.42315e-01  CE: 3.81981e-01  AB: 2.04110e-03  SSE: 4.14317e-02  KLD: 1.68673e-02  Batchsize: 4096
2025-01-15 12:21:49.362 | INFO    | 		Epoch: 277  Loss: 4.42391e-01  CE: 3.82015e-01  AB: 2.04280e-03  SSE: 4.14247e-02  KLD: 1.69129e-02  Batchsize: 4096
2025-01-15 12:21:50.749 | INFO    | 		Epoch: 278  Loss: 4.42294e-01  CE: 3.81958e-01  AB: 2.05041e-03  SSE: 4.14012e-02  KLD: 1.68836e-02  Batchsize: 4096
2025-01-15 12:21:52.173 | INFO    | 		Epoch: 279  Loss: 4.42345e-01  CE: 3.81984e-01  AB: 2.04631e-03  SSE: 4.14221e-02  KLD: 1.68937e-02  Batchsize: 4096
2025-01-15 12:21:53.608 | INFO    | 		Epoch: 280  Loss: 4.42347e-01  CE: 3.82015e-01  AB: 2.04545e-03  SSE: 4.14122e-02  KLD: 1.68686e-02  Batchsize: 4096
2025-01-15 12:21:54.964 | INFO    | 		Epoch: 281  Loss: 4.42381e-01  CE: 3.82004e-01  AB: 2.03802e-03  SSE: 4.14310e-02  KLD: 1.69068e-02  Batchsize: 4096
2025-01-15 12:21:56.381 | INFO    | 		Epoch: 282  Loss: 4.42342e-01  CE: 3.81976e-01  AB: 2.04576e-03  SSE: 4.14085e-02  KLD: 1.69134e-02  Batchsize: 4096
2025-01-15 12:21:57.716 | INFO    | 		Epoch: 283  Loss: 4.42336e-01  CE: 3.81994e-01  AB: 2.05010e-03  SSE: 4.14108e-02  KLD: 1.68835e-02  Batchsize: 4096
2025-01-15 12:21:59.107 | INFO    | 		Epoch: 284  Loss: 4.42308e-01  CE: 3.81956e-01  AB: 2.04665e-03  SSE: 4.14484e-02  KLD: 1.68577e-02  Batchsize: 4096
2025-01-15 12:22:00.521 | INFO    | 		Epoch: 285  Loss: 4.42341e-01  CE: 3.82025e-01  AB: 2.04548e-03  SSE: 4.14206e-02  KLD: 1.68508e-02  Batchsize: 4096
2025-01-15 12:22:01.845 | INFO    | 		Epoch: 286  Loss: 4.42337e-01  CE: 3.81977e-01  AB: 2.04200e-03  SSE: 4.14201e-02  KLD: 1.68956e-02  Batchsize: 4096
2025-01-15 12:22:03.126 | INFO    | 		Epoch: 287  Loss: 4.42351e-01  CE: 3.81990e-01  AB: 2.04538e-03  SSE: 4.14206e-02  KLD: 1.68944e-02  Batchsize: 4096
2025-01-15 12:22:04.566 | INFO    | 		Epoch: 288  Loss: 4.42304e-01  CE: 3.81967e-01  AB: 2.05373e-03  SSE: 4.14061e-02  KLD: 1.68776e-02  Batchsize: 4096
2025-01-15 12:22:05.850 | INFO    | 		Epoch: 289  Loss: 4.42325e-01  CE: 3.81972e-01  AB: 2.05550e-03  SSE: 4.14177e-02  KLD: 1.68805e-02  Batchsize: 4096
2025-01-15 12:22:07.227 | INFO    | 		Epoch: 290  Loss: 4.42308e-01  CE: 3.81979e-01  AB: 2.04293e-03  SSE: 4.13872e-02  KLD: 1.69014e-02  Batchsize: 4096
2025-01-15 12:22:08.551 | INFO    | 		Epoch: 291  Loss: 4.42322e-01  CE: 3.81989e-01  AB: 2.02308e-03  SSE: 4.14332e-02  KLD: 1.68796e-02  Batchsize: 4096
2025-01-15 12:22:09.948 | INFO    | 		Epoch: 292  Loss: 4.42314e-01  CE: 3.81953e-01  AB: 2.05578e-03  SSE: 4.14231e-02  KLD: 1.68818e-02  Batchsize: 4096
2025-01-15 12:22:11.319 | INFO    | 		Epoch: 293  Loss: 4.42320e-01  CE: 3.81964e-01  AB: 2.05814e-03  SSE: 4.14231e-02  KLD: 1.68749e-02  Batchsize: 4096
2025-01-15 12:22:12.680 | INFO    | 		Epoch: 294  Loss: 4.42302e-01  CE: 3.81949e-01  AB: 2.05330e-03  SSE: 4.14144e-02  KLD: 1.68865e-02  Batchsize: 4096
2025-01-15 12:22:14.064 | INFO    | 		Epoch: 295  Loss: 4.42338e-01  CE: 3.81978e-01  AB: 2.04981e-03  SSE: 4.14150e-02  KLD: 1.68913e-02  Batchsize: 4096
2025-01-15 12:22:15.408 | INFO    | 		Epoch: 296  Loss: 4.42272e-01  CE: 3.81929e-01  AB: 2.03861e-03  SSE: 4.14173e-02  KLD: 1.68909e-02  Batchsize: 4096
2025-01-15 12:22:16.776 | INFO    | 		Epoch: 297  Loss: 4.42303e-01  CE: 3.81976e-01  AB: 2.04276e-03  SSE: 4.14049e-02  KLD: 1.68824e-02  Batchsize: 4096
2025-01-15 12:22:18.153 | INFO    | 		Epoch: 298  Loss: 4.42300e-01  CE: 3.81978e-01  AB: 2.04807e-03  SSE: 4.14377e-02  KLD: 1.68416e-02  Batchsize: 4096
2025-01-15 12:22:19.539 | INFO    | 		Epoch: 299  Loss: 4.42260e-01  CE: 3.81924e-01  AB: 2.03906e-03  SSE: 4.14227e-02  KLD: 1.68773e-02  Batchsize: 4096
2025-01-15 12:22:20.925 | INFO    | 		Epoch: 300  Loss: 4.42271e-01  CE: 3.81958e-01  AB: 2.04135e-03  SSE: 4.13958e-02  KLD: 1.68797e-02  Batchsize: 4096
2025-01-15 12:22:20.943 | INFO    | 	Encoding to latent representation
2025-01-15 12:22:30.405 | INFO    | 	Trained VAE and encoded in 884.05 seconds.

2025-01-15 12:22:30.406 | INFO    | Clustering
2025-01-15 12:22:30.406 | INFO    | 	Windowsize: 300
2025-01-15 12:22:30.406 | INFO    | 	Min successful thresholds detected: 15
2025-01-15 12:22:30.406 | INFO    | 	Max clusters: None
2025-01-15 12:22:30.406 | INFO    | 	Use CUDA for clustering: True
2025-01-15 12:22:30.406 | INFO    | 	Binsplitter: "@"
2025-01-15 12:24:47.901 | ERROR   | An error has been caught in function 'main', process 'MainProcess' (4135283), thread 'MainThread' (135716811642368):
Traceback (most recent call last):

  File "/data/shiraz/miniconda3/envs/vamb/bin/vamb", line 8, in <module>
    sys.exit(main())
    │   │    └ <function main at 0x7b6dfc63e3e0>
    │   └ <built-in function exit>
    └ <module 'sys' (built-in)>

> File "/data/shiraz/VAMB/vamb/vamb/__main__.py", line 2267, in main
    runner.run()
    │      └ <function BasicArguments.run at 0x7b6dfc63d620>
    └ <vamb.__main__.VAEArguments object at 0x7b6e009e0860>

  File "/data/shiraz/VAMB/vamb/vamb/__main__.py", line 1469, in run
    self.run_inner()
    │    └ <function VAEArguments.run_inner at 0x7b6dfc63dc60>
    └ <vamb.__main__.VAEArguments object at 0x7b6e009e0860>

  File "/data/shiraz/VAMB/vamb/vamb/__main__.py", line 1594, in run_inner
    run(
    └ <function run at 0x7b6dfc63cd60>

  File "/data/shiraz/VAMB/vamb/vamb/__main__.py", line 995, in run
    cluster_and_write_files(
    └ <function cluster_and_write_files at 0x7b6dfc63cb80>

  File "/data/shiraz/VAMB/vamb/vamb/__main__.py", line 825, in cluster_and_write_files
    for i, cluster in enumerate(clusters):
        │  │                    └ <itertools.islice object at 0x7b6df3d43ec0>
        │  └ <vamb.cluster.Cluster object at 0x7b6df08c3530>
        └ 68479

  File "/data/shiraz/VAMB/vamb/vamb/cluster.py", line 297, in __next__
    cluster, _, points = self.find_cluster()
                         │    └ <function ClusterGenerator.find_cluster at 0x7b6e8cb70c20>
                         └ ClusterGenerator(46 points, 68480 clusters)

  File "/data/shiraz/VAMB/vamb/vamb/cluster.py", line 541, in find_cluster
    threshold = self.find_threshold(distances)
                │    │              └ tensor([0.5936, 0.5207, 0.5416, 0.5254, 0.5320, 0.5004, 0.5448, 0.5621, 0.5671,
                │    │                        0.4736, 0.4296, 0.4629, 0.5061, 0.436...
                │    └ <function ClusterGenerator.find_threshold at 0x7b6e8cb70b80>
                └ ClusterGenerator(46 points, 68480 clusters)

  File "/data/shiraz/VAMB/vamb/vamb/cluster.py", line 455, in find_threshold
    below_xmax = (distances <= _XMAX) & self.kept_mask
                  │            │        │    └ <member 'kept_mask' of 'ClusterGenerator' objects>
                  │            │        └ ClusterGenerator(46 points, 68480 clusters)
                  │            └ 0.3
                  └ tensor([0.5936, 0.5207, 0.5416, 0.5254, 0.5320, 0.5004, 0.5448, 0.5621, 0.5671,
                            0.4736, 0.4296, 0.4629, 0.5061, 0.436...

RuntimeError: The size of tensor a (57) must match the size of tensor b (46) at non-singleton dimension 0
shiraz@ersa:~/data/Omer/bin$

@shiraz-shah
Copy link
Author

The log file for the preceding (BAM-file-parsing) run is below, and it was interrupted (Ctrl-C) after epoch 1:

2025-01-15 11:42:37.318 | INFO    | Starting Vamb version 4.1.4.dev121+g7dc2f98
2025-01-15 11:42:37.319 | INFO    | Random seed is 53925372418564247
2025-01-15 11:42:37.319 | INFO    | Invoked with CLI args: 'f/data/shiraz/miniconda3/envs/vamb/bin/vamb bin default --outdir bin --fasta ass.fna --bamdir map -o @'
2025-01-15 11:42:37.320 | INFO    | Loading TNF
2025-01-15 11:42:37.320 | INFO    | 	Minimum sequence length: 2000
2025-01-15 11:42:37.320 | INFO    | 	Loading data from FASTA file ass.fna
2025-01-15 11:44:01.447 | INFO    | 	Kept 3431728865 bases in 347769 sequences
2025-01-15 11:44:01.448 | INFO    | 	Processed TNF in 84.13 seconds.

2025-01-15 11:44:01.448 | INFO    | Loading depths
2025-01-15 11:44:01.448 | INFO    | 	Reference hash: 954727658f9154f86639d84c068275fe
2025-01-15 11:44:01.448 | INFO    | 	Parsing 39 BAM files with 8 threads
2025-01-15 11:44:01.449 | INFO    | 	Min identity: 0.0
2025-01-15 12:02:45.923 | INFO    | 	Order of columns is:
2025-01-15 12:02:45.924 | INFO    | 	     0: map/F350014176_L01_UDB-121.bam
2025-01-15 12:02:45.924 | INFO    | 	     1: map/F350027910_L01_UDB-97.bam
2025-01-15 12:02:45.924 | INFO    | 	     2: map/F350027910_L01_UDB-107.bam
2025-01-15 12:02:45.924 | INFO    | 	     3: map/F350014176_L01_UDB-124.bam
2025-01-15 12:02:45.924 | INFO    | 	     4: map/F350027910_L02_UDB-115.bam
2025-01-15 12:02:45.925 | INFO    | 	     5: map/F350027910_L02_UDB-118.bam
2025-01-15 12:02:45.925 | INFO    | 	     6: map/F350027910_L02_UDB-108.bam
2025-01-15 12:02:45.925 | INFO    | 	     7: map/F350027910_L02_UDB-116.bam
2025-01-15 12:02:45.925 | INFO    | 	     8: map/F350014176_L01_UDB-127.bam
2025-01-15 12:02:45.925 | INFO    | 	     9: map/F350014176_L01_UDB-119.bam
2025-01-15 12:02:45.925 | INFO    | 	    10: map/F350027910_L01_UDB-102.bam
2025-01-15 12:02:45.926 | INFO    | 	    11: map/F350014176_L01_UDB-138.bam
2025-01-15 12:02:45.926 | INFO    | 	    12: map/F350014176_L01_UDB-122.bam
2025-01-15 12:02:45.926 | INFO    | 	    13: map/F350027910_L01_UDB-100.bam
2025-01-15 12:02:45.926 | INFO    | 	    14: map/F350027910_L01_UDB-105.bam
2025-01-15 12:02:45.926 | INFO    | 	    15: map/F350014176_L01_UDB-126.bam
2025-01-15 12:02:45.926 | INFO    | 	    16: map/F350014176_L01_UDB-136.bam
2025-01-15 12:02:45.926 | INFO    | 	    17: map/F350027910_L02_UDB-112.bam
2025-01-15 12:02:45.927 | INFO    | 	    18: map/F350027910_L01_UDB-104.bam
2025-01-15 12:02:45.927 | INFO    | 	    19: map/F350014176_L01_UDB-123.bam
2025-01-15 12:02:45.927 | INFO    | 	    20: map/F350014176_L01_UDB-135.bam
2025-01-15 12:02:45.927 | INFO    | 	    21: map/F350027946_L02_UDB-111.bam
2025-01-15 12:02:45.927 | INFO    | 	    22: map/F350014176_L01_UDB-125.bam
2025-01-15 12:02:45.927 | INFO    | 	    23: map/F350027910_L01_UDB-103.bam
2025-01-15 12:02:45.927 | INFO    | 	    24: map/F350027910_L01_UDB-106.bam
2025-01-15 12:02:45.928 | INFO    | 	    25: map/F350027910_L02_UDB-109.bam
2025-01-15 12:02:45.928 | INFO    | 	    26: map/F350027910_L01_UDB-99.bam
2025-01-15 12:02:45.928 | INFO    | 	    27: map/F350027946_L02_UDB-114.bam
2025-01-15 12:02:45.928 | INFO    | 	    28: map/F350014176_L01_UDB-120.bam
2025-01-15 12:02:45.928 | INFO    | 	    29: map/F350027910_L02_UDB-110.bam
2025-01-15 12:02:45.928 | INFO    | 	    30: map/F350027910_L02_UDB-114.bam
2025-01-15 12:02:45.929 | INFO    | 	    31: map/F350027910_L02_UDB-111.bam
2025-01-15 12:02:45.929 | INFO    | 	    32: map/F350027946_L02_UDB-115.bam
2025-01-15 12:02:45.929 | INFO    | 	    33: map/F350027946_L02_UDB-113.bam
2025-01-15 12:02:45.929 | INFO    | 	    34: map/F350027946_L02_UDB-112.bam
2025-01-15 12:02:45.929 | INFO    | 	    35: map/F350027910_L01_UDB-142.bam
2025-01-15 12:02:45.929 | INFO    | 	    36: map/F350027910_L01_UDB-101.bam
2025-01-15 12:02:45.929 | INFO    | 	    37: map/F350027946_L02_UDB-110.bam
2025-01-15 12:02:45.930 | INFO    | 	    38: map/F350027910_L02_UDB-113.bam
2025-01-15 12:02:45.930 | INFO    | 	Processed RPKM in 1124.48 seconds.

2025-01-15 12:02:46.320 | INFO    | Creating and training VAE
2025-01-15 12:02:46.645 | INFO    | 	Created VAE
2025-01-15 12:02:50.725 | INFO    | 	Network properties:
2025-01-15 12:02:50.725 | INFO    | 	CUDA: False
2025-01-15 12:02:50.726 | INFO    | 	Alpha: 0.15
2025-01-15 12:02:50.726 | INFO    | 	Beta: 200.0
2025-01-15 12:02:50.727 | INFO    | 	Dropout: 0.2
2025-01-15 12:02:50.727 | INFO    | 	N hidden: 512, 512
2025-01-15 12:02:50.727 | INFO    | 	N latent: 32
2025-01-15 12:02:50.728 | INFO    | 	Training properties:
2025-01-15 12:02:50.728 | INFO    | 	N epochs: 300
2025-01-15 12:02:50.728 | INFO    | 	Starting batch size: 256
2025-01-15 12:02:50.729 | INFO    | 	Batchsteps: 25, 75, 150, 225
2025-01-15 12:02:50.729 | INFO    | 	N sequences: 347769
2025-01-15 12:02:50.729 | INFO    | 	N samples: 39
2025-01-15 12:03:13.809 | INFO    | 		Epoch:   1  Loss: 5.26089e-01  CE: 4.48494e-01  AB: 3.80155e-03  SSE: 5.61425e-02  KLD: 1.76222e-02  Batchsize:  256

@jakobnissen
Copy link
Member

jakobnissen commented Jan 15, 2025 •
edited
Loading

Yes - it looks like the bug that I fixed a while back with #373
We're currently (at a snail's pace, unfortunately) doing a bunch of testing before the release of v5, since we expect that the development of Vamb the last year or so have introduced a whole bunch of bugs.
So until a v5 is out, there might be issues like that.
For now, until the release of v5, I recommend using the latest version on the master branch, which has accumulated the most bugfixes.

Try it out with the current master branch and please let me know if the bug persists!

@shiraz-shah
Copy link
Author

Got it. Thanks, Jakob!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jakobnissen @shiraz-shah