psnr drop gradually

[menglongyue]

Thanks for your excellent work. I meet with a problem. As training progresses, PSNR decreases，from 20.82 to 20.00. how to solve it ?

[ashawkey]

@menglongyue Hi, could you provide more information? e.g., the dataset you use, the command line, the full output log.

[menglongyue]

Thanks for your reply. the training images is 3000x 2000, I set both bound and bg_radius to 16. the command line is:python main_nerf.py mydata --workspace trial_nerf -O --bound 16 --bg_radius 16，and use llff2nerf.py to get the json file, the aabb_scale not changed, default to 2. here is my training log:
[INFO] Trainer: ngp | 2022-07-07_15-48-19 | cuda | fp16 | /hy-tmp/trail_nerf2
[INFO] #parameters: 15081712
[INFO] Loading latest checkpoint ...
[WARN] No checkpoint found, model randomly initialized.
==> Start Training Epoch 1, lr=0.010000 ...
==> Finished Epoch 1.
==> Start Training Epoch 2, lr=0.009994 ...
==> Finished Epoch 2.
==> Start Training Epoch 3, lr=0.009988 ...
==> Finished Epoch 3.
==> Start Training Epoch 4, lr=0.009983 ...
==> Finished Epoch 4.
==> Start Training Epoch 5, lr=0.009977 ...
==> Finished Epoch 5.
==> Start Training Epoch 6, lr=0.009971 ...
==> Finished Epoch 6.
==> Start Training Epoch 7, lr=0.009966 ...
==> Finished Epoch 7.
==> Start Training Epoch 8, lr=0.009960 ...
==> Finished Epoch 8.
==> Start Training Epoch 9, lr=0.009954 ...
==> Finished Epoch 9.
==> Start Training Epoch 10, lr=0.009948 ...
==> Finished Epoch 10.
==> Start Training Epoch 11, lr=0.009943 ...
==> Finished Epoch 11.
==> Start Training Epoch 12, lr=0.009937 ...
==> Finished Epoch 12.
==> Start Training Epoch 13, lr=0.009931 ...
==> Finished Epoch 13.
==> Start Training Epoch 14, lr=0.009925 ...
==> Finished Epoch 14.
==> Start Training Epoch 15, lr=0.009920 ...
==> Finished Epoch 15.
==> Start Training Epoch 16, lr=0.009914 ...
==> Finished Epoch 16.
==> Start Training Epoch 17, lr=0.009908 ...
==> Finished Epoch 17.
==> Start Training Epoch 18, lr=0.009903 ...
==> Finished Epoch 18.
==> Start Training Epoch 19, lr=0.009897 ...
==> Finished Epoch 19.
==> Start Training Epoch 20, lr=0.009891 ...
==> Finished Epoch 20.
==> Start Training Epoch 21, lr=0.009886 ...
==> Finished Epoch 21.
==> Start Training Epoch 22, lr=0.009880 ...
==> Finished Epoch 22.
==> Start Training Epoch 23, lr=0.009874 ...
==> Finished Epoch 23.
==> Start Training Epoch 24, lr=0.009868 ...
==> Finished Epoch 24.
==> Start Training Epoch 25, lr=0.009863 ...
==> Finished Epoch 25.
==> Start Training Epoch 26, lr=0.009857 ...
==> Finished Epoch 26.
==> Start Training Epoch 27, lr=0.009851 ...
==> Finished Epoch 27.
==> Start Training Epoch 28, lr=0.009846 ...
==> Finished Epoch 28.
==> Start Training Epoch 29, lr=0.009840 ...
==> Finished Epoch 29.
==> Start Training Epoch 30, lr=0.009834 ...
==> Finished Epoch 30.
==> Start Training Epoch 31, lr=0.009829 ...
==> Finished Epoch 31.
==> Start Training Epoch 32, lr=0.009823 ...
==> Finished Epoch 32.
==> Start Training Epoch 33, lr=0.009817 ...
==> Finished Epoch 33.
==> Start Training Epoch 34, lr=0.009812 ...
==> Finished Epoch 34.
==> Start Training Epoch 35, lr=0.009806 ...
==> Finished Epoch 35.
==> Start Training Epoch 36, lr=0.009801 ...
==> Finished Epoch 36.
==> Start Training Epoch 37, lr=0.009795 ...
==> Finished Epoch 37.
==> Start Training Epoch 38, lr=0.009789 ...
==> Finished Epoch 38.
==> Start Training Epoch 39, lr=0.009784 ...
==> Finished Epoch 39.
==> Start Training Epoch 40, lr=0.009778 ...
==> Finished Epoch 40.
==> Start Training Epoch 41, lr=0.009772 ...
==> Finished Epoch 41.
==> Start Training Epoch 42, lr=0.009767 ...
==> Finished Epoch 42.
==> Start Training Epoch 43, lr=0.009761 ...
==> Finished Epoch 43.
==> Start Training Epoch 44, lr=0.009756 ...
==> Finished Epoch 44.
==> Start Training Epoch 45, lr=0.009750 ...
==> Finished Epoch 45.
==> Start Training Epoch 46, lr=0.009744 ...
==> Finished Epoch 46.
==> Start Training Epoch 47, lr=0.009739 ...
==> Finished Epoch 47.
==> Start Training Epoch 48, lr=0.009733 ...
==> Finished Epoch 48.
==> Start Training Epoch 49, lr=0.009727 ...
==> Finished Epoch 49.
==> Start Training Epoch 50, lr=0.009722 ...
==> Finished Epoch 50.
++> Evaluate at epoch 50 ...
PSNR = 20.739291
++> Evaluate epoch 50 Finished.
[INFO] New best result: None --> 0.008512702537700534
==> Start Training Epoch 51, lr=0.009716 ...
==> Finished Epoch 51.
==> Start Training Epoch 52, lr=0.009711 ...
==> Finished Epoch 52.
==> Start Training Epoch 53, lr=0.009705 ...
==> Finished Epoch 53.
==> Start Training Epoch 54, lr=0.009700 ...
==> Finished Epoch 54.
==> Start Training Epoch 55, lr=0.009694 ...
==> Finished Epoch 55.
==> Start Training Epoch 56, lr=0.009688 ...
==> Finished Epoch 56.
==> Start Training Epoch 57, lr=0.009683 ...
==> Finished Epoch 57.
==> Start Training Epoch 58, lr=0.009677 ...
==> Finished Epoch 58.
==> Start Training Epoch 59, lr=0.009672 ...
==> Finished Epoch 59.
==> Start Training Epoch 60, lr=0.009666 ...
==> Finished Epoch 60.
==> Start Training Epoch 61, lr=0.009661 ...
==> Finished Epoch 61.
==> Start Training Epoch 62, lr=0.009655 ...
==> Finished Epoch 62.
==> Start Training Epoch 63, lr=0.009649 ...
==> Finished Epoch 63.
==> Start Training Epoch 64, lr=0.009644 ...
==> Finished Epoch 64.
==> Start Training Epoch 65, lr=0.009638 ...
==> Finished Epoch 65.
==> Start Training Epoch 66, lr=0.009633 ...
==> Finished Epoch 66.
==> Start Training Epoch 67, lr=0.009627 ...
==> Finished Epoch 67.
==> Start Training Epoch 68, lr=0.009622 ...
==> Finished Epoch 68.
==> Start Training Epoch 69, lr=0.009616 ...
==> Finished Epoch 69.
==> Start Training Epoch 70, lr=0.009611 ...
==> Finished Epoch 70.
==> Start Training Epoch 71, lr=0.009605 ...
==> Finished Epoch 71.
==> Start Training Epoch 72, lr=0.009600 ...
==> Finished Epoch 72.
==> Start Training Epoch 73, lr=0.009594 ...
==> Finished Epoch 73.
==> Start Training Epoch 74, lr=0.009588 ...
==> Finished Epoch 74.
==> Start Training Epoch 75, lr=0.009583 ...
==> Finished Epoch 75.
==> Start Training Epoch 76, lr=0.009577 ...
==> Finished Epoch 76.
==> Start Training Epoch 77, lr=0.009572 ...
==> Finished Epoch 77.
==> Start Training Epoch 78, lr=0.009566 ...
==> Finished Epoch 78.
==> Start Training Epoch 79, lr=0.009561 ...
==> Finished Epoch 79.
==> Start Training Epoch 80, lr=0.009555 ...
==> Finished Epoch 80.
==> Start Training Epoch 81, lr=0.009550 ...
==> Finished Epoch 81.
==> Start Training Epoch 82, lr=0.009544 ...
==> Finished Epoch 82.
==> Start Training Epoch 83, lr=0.009539 ...
==> Finished Epoch 83.
==> Start Training Epoch 84, lr=0.009533 ...
==> Finished Epoch 84.
==> Start Training Epoch 85, lr=0.009528 ...
==> Finished Epoch 85.
==> Start Training Epoch 86, lr=0.009522 ...
==> Finished Epoch 86.
==> Start Training Epoch 87, lr=0.009517 ...
==> Finished Epoch 87.
==> Start Training Epoch 88, lr=0.009512 ...
==> Finished Epoch 88.
==> Start Training Epoch 89, lr=0.009506 ...
==> Finished Epoch 89.
==> Start Training Epoch 90, lr=0.009501 ...
==> Finished Epoch 90.
==> Start Training Epoch 91, lr=0.009495 ...
==> Finished Epoch 91.
==> Start Training Epoch 92, lr=0.009490 ...
==> Finished Epoch 92.
==> Start Training Epoch 93, lr=0.009484 ...
==> Finished Epoch 93.
==> Start Training Epoch 94, lr=0.009479 ...
==> Finished Epoch 94.
==> Start Training Epoch 95, lr=0.009473 ...
==> Finished Epoch 95.
==> Start Training Epoch 96, lr=0.009468 ...
==> Finished Epoch 96.
==> Start Training Epoch 97, lr=0.009462 ...
==> Finished Epoch 97.
==> Start Training Epoch 98, lr=0.009457 ...
==> Finished Epoch 98.
==> Start Training Epoch 99, lr=0.009451 ...
==> Finished Epoch 99.
==> Start Training Epoch 100, lr=0.009446 ...
==> Finished Epoch 100.
++> Evaluate at epoch 100 ...
PSNR = 20.808737
++> Evaluate epoch 100 Finished.
[INFO] New best result: 0.008512702537700534 --> 0.008385548368096352
==> Start Training Epoch 101, lr=0.009441 ...
==> Finished Epoch 101.
==> Start Training Epoch 102, lr=0.009435 ...
==> Finished Epoch 102.
==> Start Training Epoch 103, lr=0.009430 ...
==> Finished Epoch 103.
==> Start Training Epoch 104, lr=0.009424 ...
==> Finished Epoch 104.
==> Start Training Epoch 105, lr=0.009419 ...
==> Finished Epoch 105.
==> Start Training Epoch 106, lr=0.009413 ...
==> Finished Epoch 106.
==> Start Training Epoch 107, lr=0.009408 ...
==> Finished Epoch 107.
==> Start Training Epoch 108, lr=0.009403 ...
==> Finished Epoch 108.
==> Start Training Epoch 109, lr=0.009397 ...
==> Finished Epoch 109.
==> Start Training Epoch 110, lr=0.009392 ...
==> Finished Epoch 110.
==> Start Training Epoch 111, lr=0.009386 ...
==> Finished Epoch 111.
==> Start Training Epoch 112, lr=0.009381 ...
==> Finished Epoch 112.
==> Start Training Epoch 113, lr=0.009376 ...
==> Finished Epoch 113.
==> Start Training Epoch 114, lr=0.009370 ...
==> Finished Epoch 114.
==> Start Training Epoch 115, lr=0.009365 ...
==> Finished Epoch 115.
==> Start Training Epoch 116, lr=0.009359 ...
==> Finished Epoch 116.
==> Start Training Epoch 117, lr=0.009354 ...
==> Finished Epoch 117.
==> Start Training Epoch 118, lr=0.009349 ...
==> Finished Epoch 118.
==> Start Training Epoch 119, lr=0.009343 ...
==> Finished Epoch 119.
==> Start Training Epoch 120, lr=0.009338 ...
==> Finished Epoch 120.
==> Start Training Epoch 121, lr=0.009333 ...
==> Finished Epoch 121.
==> Start Training Epoch 122, lr=0.009327 ...
==> Finished Epoch 122.
==> Start Training Epoch 123, lr=0.009322 ...
==> Finished Epoch 123.
==> Start Training Epoch 124, lr=0.009316 ...
==> Finished Epoch 124.
==> Start Training Epoch 125, lr=0.009311 ...
==> Finished Epoch 125.
==> Start Training Epoch 126, lr=0.009306 ...
==> Finished Epoch 126.
==> Start Training Epoch 127, lr=0.009300 ...
==> Finished Epoch 127.
==> Start Training Epoch 128, lr=0.009295 ...
==> Finished Epoch 128.
==> Start Training Epoch 129, lr=0.009290 ...
==> Finished Epoch 129.
==> Start Training Epoch 130, lr=0.009284 ...
==> Finished Epoch 130.
==> Start Training Epoch 131, lr=0.009279 ...
==> Finished Epoch 131.
==> Start Training Epoch 132, lr=0.009274 ...
==> Finished Epoch 132.
==> Start Training Epoch 133, lr=0.009268 ...
==> Finished Epoch 133.
==> Start Training Epoch 134, lr=0.009263 ...
==> Finished Epoch 134.
==> Start Training Epoch 135, lr=0.009258 ...
==> Finished Epoch 135.
==> Start Training Epoch 136, lr=0.009252 ...
==> Finished Epoch 136.
==> Start Training Epoch 137, lr=0.009247 ...
==> Finished Epoch 137.
==> Start Training Epoch 138, lr=0.009242 ...
==> Finished Epoch 138.
==> Start Training Epoch 139, lr=0.009236 ...
==> Finished Epoch 139.
==> Start Training Epoch 140, lr=0.009231 ...
==> Finished Epoch 140.
==> Start Training Epoch 141, lr=0.009226 ...
==> Finished Epoch 141.
==> Start Training Epoch 142, lr=0.009220 ...
==> Finished Epoch 142.
==> Start Training Epoch 143, lr=0.009215 ...
==> Finished Epoch 143.
==> Start Training Epoch 144, lr=0.009210 ...
==> Finished Epoch 144.
==> Start Training Epoch 145, lr=0.009204 ...
==> Finished Epoch 145.
==> Start Training Epoch 146, lr=0.009199 ...
==> Finished Epoch 146.
==> Start Training Epoch 147, lr=0.009194 ...
==> Finished Epoch 147.
==> Start Training Epoch 148, lr=0.009189 ...
==> Finished Epoch 148.
==> Start Training Epoch 149, lr=0.009183 ...
==> Finished Epoch 149.
==> Start Training Epoch 150, lr=0.009178 ...
==> Finished Epoch 150.
++> Evaluate at epoch 150 ...
PSNR = 20.734601
++> Evaluate epoch 150 Finished.
==> Start Training Epoch 151, lr=0.009173 ...
==> Finished Epoch 151.
==> Start Training Epoch 152, lr=0.009167 ...
==> Finished Epoch 152.
==> Start Training Epoch 153, lr=0.009162 ...
==> Finished Epoch 153.
==> Start Training Epoch 154, lr=0.009157 ...
==> Finished Epoch 154.
==> Start Training Epoch 155, lr=0.009152 ...
==> Finished Epoch 155.
==> Start Training Epoch 156, lr=0.009146 ...
==> Finished Epoch 156.
==> Start Training Epoch 157, lr=0.009141 ...
==> Finished Epoch 157.
==> Start Training Epoch 158, lr=0.009136 ...
==> Finished Epoch 158.
==> Start Training Epoch 159, lr=0.009131 ...
==> Finished Epoch 159.
==> Start Training Epoch 160, lr=0.009125 ...
==> Finished Epoch 160.
==> Start Training Epoch 161, lr=0.009120 ...
==> Finished Epoch 161.
==> Start Training Epoch 162, lr=0.009115 ...
==> Finished Epoch 162.
==> Start Training Epoch 163, lr=0.009110 ...
==> Finished Epoch 163.
==> Start Training Epoch 164, lr=0.009104 ...
==> Finished Epoch 164.
==> Start Training Epoch 165, lr=0.009099 ...
==> Finished Epoch 165.
==> Start Training Epoch 166, lr=0.009094 ...
==> Finished Epoch 166.
==> Start Training Epoch 167, lr=0.009089 ...
==> Finished Epoch 167.
==> Start Training Epoch 168, lr=0.009083 ...
==> Finished Epoch 168.
==> Start Training Epoch 169, lr=0.009078 ...
==> Finished Epoch 169.
==> Start Training Epoch 170, lr=0.009073 ...
==> Finished Epoch 170.
==> Start Training Epoch 171, lr=0.009068 ...
==> Finished Epoch 171.
==> Start Training Epoch 172, lr=0.009063 ...
==> Finished Epoch 172.
==> Start Training Epoch 173, lr=0.009057 ...
==> Finished Epoch 173.
==> Start Training Epoch 174, lr=0.009052 ...
==> Finished Epoch 174.
==> Start Training Epoch 175, lr=0.009047 ...
==> Finished Epoch 175.
==> Start Training Epoch 176, lr=0.009042 ...
==> Finished Epoch 176.
==> Start Training Epoch 177, lr=0.009036 ...
==> Finished Epoch 177.
==> Start Training Epoch 178, lr=0.009031 ...
==> Finished Epoch 178.
==> Start Training Epoch 179, lr=0.009026 ...
==> Finished Epoch 179.
==> Start Training Epoch 180, lr=0.009021 ...
==> Finished Epoch 180.
==> Start Training Epoch 181, lr=0.009016 ...
==> Finished Epoch 181.
==> Start Training Epoch 182, lr=0.009011 ...
==> Finished Epoch 182.
==> Start Training Epoch 183, lr=0.009005 ...
==> Finished Epoch 183.
==> Start Training Epoch 184, lr=0.009000 ...
==> Finished Epoch 184.
==> Start Training Epoch 185, lr=0.008995 ...
==> Finished Epoch 185.
==> Start Training Epoch 186, lr=0.008990 ...
==> Finished Epoch 186.
==> Start Training Epoch 187, lr=0.008985 ...
==> Finished Epoch 187.
==> Start Training Epoch 188, lr=0.008979 ...
==> Finished Epoch 188.
==> Start Training Epoch 189, lr=0.008974 ...
==> Finished Epoch 189.
==> Start Training Epoch 190, lr=0.008969 ...
==> Finished Epoch 190.
==> Start Training Epoch 191, lr=0.008964 ...
==> Finished Epoch 191.
==> Start Training Epoch 192, lr=0.008959 ...
==> Finished Epoch 192.
==> Start Training Epoch 193, lr=0.008954 ...
==> Finished Epoch 193.
==> Start Training Epoch 194, lr=0.008948 ...
==> Finished Epoch 194.
==> Start Training Epoch 195, lr=0.008943 ...
==> Finished Epoch 195.
==> Start Training Epoch 196, lr=0.008938 ...
==> Finished Epoch 196.
==> Start Training Epoch 197, lr=0.008933 ...
==> Finished Epoch 197.
==> Start Training Epoch 198, lr=0.008928 ...
==> Finished Epoch 198.
==> Start Training Epoch 199, lr=0.008923 ...
==> Finished Epoch 199.
==> Start Training Epoch 200, lr=0.008918 ...
==> Finished Epoch 200.
++> Evaluate at epoch 200 ...
PSNR = 20.714772
++> Evaluate epoch 200 Finished.
==> Start Training Epoch 201, lr=0.008913 ...
==> Finished Epoch 201.
==> Start Training Epoch 202, lr=0.008907 ...
==> Finished Epoch 202.
==> Start Training Epoch 203, lr=0.008902 ...
==> Finished Epoch 203.
==> Start Training Epoch 204, lr=0.008897 ...
==> Finished Epoch 204.
==> Start Training Epoch 205, lr=0.008892 ...
==> Finished Epoch 205.
==> Start Training Epoch 206, lr=0.008887 ...
==> Finished Epoch 206.
==> Start Training Epoch 207, lr=0.008882 ...
==> Finished Epoch 207.
==> Start Training Epoch 208, lr=0.008877 ...
==> Finished Epoch 208.
==> Start Training Epoch 209, lr=0.008872 ...
==> Finished Epoch 209.
==> Start Training Epoch 210, lr=0.008866 ...
==> Finished Epoch 210.
==> Start Training Epoch 211, lr=0.008861 ...
==> Finished Epoch 211.
==> Start Training Epoch 212, lr=0.008856 ...
==> Finished Epoch 212.
==> Start Training Epoch 213, lr=0.008851 ...
==> Finished Epoch 213.
==> Start Training Epoch 214, lr=0.008846 ...
==> Finished Epoch 214.
==> Start Training Epoch 215, lr=0.008841 ...
==> Finished Epoch 215.
==> Start Training Epoch 216, lr=0.008836 ...
==> Finished Epoch 216.
==> Start Training Epoch 217, lr=0.008831 ...
==> Finished Epoch 217.
==> Start Training Epoch 218, lr=0.008826 ...
==> Finished Epoch 218.
==> Start Training Epoch 219, lr=0.008821 ...
==> Finished Epoch 219.
==> Start Training Epoch 220, lr=0.008816 ...
==> Finished Epoch 220.
==> Start Training Epoch 221, lr=0.008810 ...
==> Finished Epoch 221.
==> Start Training Epoch 222, lr=0.008805 ...
==> Finished Epoch 222.
==> Start Training Epoch 223, lr=0.008800 ...
==> Finished Epoch 223.
==> Start Training Epoch 224, lr=0.008795 ...
==> Finished Epoch 224.
==> Start Training Epoch 225, lr=0.008790 ...
==> Finished Epoch 225.
==> Start Training Epoch 226, lr=0.008785 ...
==> Finished Epoch 226.
==> Start Training Epoch 227, lr=0.008780 ...
==> Finished Epoch 227.
==> Start Training Epoch 228, lr=0.008775 ...
==> Finished Epoch 228.
==> Start Training Epoch 229, lr=0.008770 ...
==> Finished Epoch 229.
==> Start Training Epoch 230, lr=0.008765 ...
==> Finished Epoch 230.
==> Start Training Epoch 231, lr=0.008760 ...
==> Finished Epoch 231.
==> Start Training Epoch 232, lr=0.008755 ...
==> Finished Epoch 232.
==> Start Training Epoch 233, lr=0.008750 ...
==> Finished Epoch 233.
==> Start Training Epoch 234, lr=0.008745 ...
==> Finished Epoch 234.
==> Start Training Epoch 235, lr=0.008740 ...
==> Finished Epoch 235.
==> Start Training Epoch 236, lr=0.008735 ...
==> Finished Epoch 236.
==> Start Training Epoch 237, lr=0.008730 ...
==> Finished Epoch 237.
==> Start Training Epoch 238, lr=0.008725 ...
==> Finished Epoch 238.
==> Start Training Epoch 239, lr=0.008720 ...
==> Finished Epoch 239.
==> Start Training Epoch 240, lr=0.008715 ...
==> Finished Epoch 240.
==> Start Training Epoch 241, lr=0.008710 ...
==> Finished Epoch 241.
==> Start Training Epoch 242, lr=0.008705 ...
==> Finished Epoch 242.
==> Start Training Epoch 243, lr=0.008700 ...
==> Finished Epoch 243.
==> Start Training Epoch 244, lr=0.008695 ...
==> Finished Epoch 244.
==> Start Training Epoch 245, lr=0.008690 ...
==> Finished Epoch 245.
==> Start Training Epoch 246, lr=0.008685 ...
==> Finished Epoch 246.
==> Start Training Epoch 247, lr=0.008680 ...
==> Finished Epoch 247.
==> Start Training Epoch 248, lr=0.008675 ...
==> Finished Epoch 248.
==> Start Training Epoch 249, lr=0.008670 ...
==> Finished Epoch 249.
==> Start Training Epoch 250, lr=0.008665 ...
==> Finished Epoch 250.
++> Evaluate at epoch 250 ...
PSNR = 20.682181
++> Evaluate epoch 250 Finished.
==> Start Training Epoch 251, lr=0.008660 ...
==> Finished Epoch 251.
==> Start Training Epoch 252, lr=0.008655 ...
==> Finished Epoch 252.
==> Start Training Epoch 253, lr=0.008650 ...
==> Finished Epoch 253.
==> Start Training Epoch 254, lr=0.008645 ...
==> Finished Epoch 254.
==> Start Training Epoch 255, lr=0.008640 ...
==> Finished Epoch 255.
==> Start Training Epoch 256, lr=0.008635 ...
==> Finished Epoch 256.
==> Start Training Epoch 257, lr=0.008630 ...
==> Finished Epoch 257.
==> Start Training Epoch 258, lr=0.008625 ...
==> Finished Epoch 258.
==> Start Training Epoch 259, lr=0.008620 ...
==> Finished Epoch 259.
==> Start Training Epoch 260, lr=0.008615 ...
==> Finished Epoch 260.
==> Start Training Epoch 261, lr=0.008610 ...
==> Finished Epoch 261.
==> Start Training Epoch 262, lr=0.008605 ...
==> Finished Epoch 262.
==> Start Training Epoch 263, lr=0.008600 ...
==> Finished Epoch 263.
==> Start Training Epoch 264, lr=0.008595 ...
==> Finished Epoch 264.
==> Start Training Epoch 265, lr=0.008590 ...
==> Finished Epoch 265.
==> Start Training Epoch 266, lr=0.008585 ...
==> Finished Epoch 266.
==> Start Training Epoch 267, lr=0.008580 ...
==> Finished Epoch 267.
==> Start Training Epoch 268, lr=0.008575 ...
==> Finished Epoch 268.
==> Start Training Epoch 269, lr=0.008570 ...
==> Finished Epoch 269.
==> Start Training Epoch 270, lr=0.008565 ...
==> Finished Epoch 270.
==> Start Training Epoch 271, lr=0.008561 ...
==> Finished Epoch 271.
==> Start Training Epoch 272, lr=0.008556 ...
==> Finished Epoch 272.
==> Start Training Epoch 273, lr=0.008551 ...
==> Finished Epoch 273.
==> Start Training Epoch 274, lr=0.008546 ...
==> Finished Epoch 274.
==> Start Training Epoch 275, lr=0.008541 ...
==> Finished Epoch 275.
==> Start Training Epoch 276, lr=0.008536 ...
==> Finished Epoch 276.
==> Start Training Epoch 277, lr=0.008531 ...
==> Finished Epoch 277.
==> Start Training Epoch 278, lr=0.008526 ...
==> Finished Epoch 278.
==> Start Training Epoch 279, lr=0.008521 ...
==> Finished Epoch 279.
==> Start Training Epoch 280, lr=0.008516 ...
==> Finished Epoch 280.
==> Start Training Epoch 281, lr=0.008511 ...
==> Finished Epoch 281.
==> Start Training Epoch 282, lr=0.008506 ...
==> Finished Epoch 282.
==> Start Training Epoch 283, lr=0.008502 ...
==> Finished Epoch 283.
==> Start Training Epoch 284, lr=0.008497 ...
==> Finished Epoch 284.
==> Start Training Epoch 285, lr=0.008492 ...
==> Finished Epoch 285.
==> Start Training Epoch 286, lr=0.008487 ...
==> Finished Epoch 286.
==> Start Training Epoch 287, lr=0.008482 ...
==> Finished Epoch 287.
==> Start Training Epoch 288, lr=0.008477 ...
==> Finished Epoch 288.
==> Start Training Epoch 289, lr=0.008472 ...
==> Finished Epoch 289.
==> Start Training Epoch 290, lr=0.008467 ...
==> Finished Epoch 290.
==> Start Training Epoch 291, lr=0.008463 ...
==> Finished Epoch 291.
==> Start Training Epoch 292, lr=0.008458 ...
==> Finished Epoch 292.
==> Start Training Epoch 293, lr=0.008453 ...
==> Finished Epoch 293.
==> Start Training Epoch 294, lr=0.008448 ...
==> Finished Epoch 294.
==> Start Training Epoch 295, lr=0.008443 ...
==> Finished Epoch 295.
==> Start Training Epoch 296, lr=0.008438 ...
==> Finished Epoch 296.
==> Start Training Epoch 297, lr=0.008433 ...
==> Finished Epoch 297.
==> Start Training Epoch 298, lr=0.008428 ...
==> Finished Epoch 298.
==> Start Training Epoch 299, lr=0.008424 ...
==> Finished Epoch 299.
==> Start Training Epoch 300, lr=0.008419 ...
==> Finished Epoch 300.
++> Evaluate at epoch 300 ...
PSNR = 20.659329
++> Evaluate epoch 300 Finished.
==> Start Training Epoch 301, lr=0.008414 ...
==> Finished Epoch 301.
==> Start Training Epoch 302, lr=0.008409 ...
==> Finished Epoch 302.
==> Start Training Epoch 303, lr=0.008404 ...
==> Finished Epoch 303.
==> Start Training Epoch 304, lr=0.008399 ...
==> Finished Epoch 304.
==> Start Training Epoch 305, lr=0.008395 ...
==> Finished Epoch 305.
==> Start Training Epoch 306, lr=0.008390 ...
==> Finished Epoch 306.
==> Start Training Epoch 307, lr=0.008385 ...
==> Finished Epoch 307.
==> Start Training Epoch 308, lr=0.008380 ...
==> Finished Epoch 308.
==> Start Training Epoch 309, lr=0.008375 ...
==> Finished Epoch 309.
==> Start Training Epoch 310, lr=0.008370 ...
==> Finished Epoch 310.
==> Start Training Epoch 311, lr=0.008366 ...
==> Finished Epoch 311.
==> Start Training Epoch 312, lr=0.008361 ...
==> Finished Epoch 312.
==> Start Training Epoch 313, lr=0.008356 ...
==> Finished Epoch 313.
==> Start Training Epoch 314, lr=0.008351 ...
==> Finished Epoch 314.
==> Start Training Epoch 315, lr=0.008346 ...
==> Finished Epoch 315.
==> Start Training Epoch 316, lr=0.008342 ...
==> Finished Epoch 316.
==> Start Training Epoch 317, lr=0.008337 ...
==> Finished Epoch 317.
==> Start Training Epoch 318, lr=0.008332 ...
==> Finished Epoch 318.
==> Start Training Epoch 319, lr=0.008327 ...
==> Finished Epoch 319.
==> Start Training Epoch 320, lr=0.008322 ...
==> Finished Epoch 320.
==> Start Training Epoch 321, lr=0.008318 ...
==> Finished Epoch 321.
==> Start Training Epoch 322, lr=0.008313 ...
==> Finished Epoch 322.
==> Start Training Epoch 323, lr=0.008308 ...
==> Finished Epoch 323.
==> Start Training Epoch 324, lr=0.008303 ...
==> Finished Epoch 324.
==> Start Training Epoch 325, lr=0.008299 ...
==> Finished Epoch 325.
==> Start Training Epoch 326, lr=0.008294 ...
==> Finished Epoch 326.
==> Start Training Epoch 327, lr=0.008289 ...
==> Finished Epoch 327.
==> Start Training Epoch 328, lr=0.008284 ...
==> Finished Epoch 328.
==> Start Training Epoch 329, lr=0.008279 ...
==> Finished Epoch 329.
==> Start Training Epoch 330, lr=0.008275 ...
==> Finished Epoch 330.
==> Start Training Epoch 331, lr=0.008270 ...
==> Finished Epoch 331.
==> Start Training Epoch 332, lr=0.008265 ...
==> Finished Epoch 332.
==> Start Training Epoch 333, lr=0.008260 ...
==> Finished Epoch 333.
==> Start Training Epoch 334, lr=0.008256 ...
==> Finished Epoch 334.
==> Start Training Epoch 335, lr=0.008251 ...
==> Finished Epoch 335.
==> Start Training Epoch 336, lr=0.008246 ...
==> Finished Epoch 336.
==> Start Training Epoch 337, lr=0.008241 ...
==> Finished Epoch 337.
==> Start Training Epoch 338, lr=0.008237 ...
==> Finished Epoch 338.
==> Start Training Epoch 339, lr=0.008232 ...
==> Finished Epoch 339.
==> Start Training Epoch 340, lr=0.008227 ...
==> Finished Epoch 340.
==> Start Training Epoch 341, lr=0.008222 ...
==> Finished Epoch 341.
==> Start Training Epoch 342, lr=0.008218 ...
==> Finished Epoch 342.
==> Start Training Epoch 343, lr=0.008213 ...
==> Finished Epoch 343.
==> Start Training Epoch 344, lr=0.008208 ...
==> Finished Epoch 344.
==> Start Training Epoch 345, lr=0.008204 ...
==> Finished Epoch 345.
==> Start Training Epoch 346, lr=0.008199 ...
==> Finished Epoch 346.
==> Start Training Epoch 347, lr=0.008194 ...
==> Finished Epoch 347.
==> Start Training Epoch 348, lr=0.008189 ...
==> Finished Epoch 348.
==> Start Training Epoch 349, lr=0.008185 ...
==> Finished Epoch 349.
==> Start Training Epoch 350, lr=0.008180 ...
==> Finished Epoch 350.
++> Evaluate at epoch 350 ...
PSNR = 20.634949
++> Evaluate epoch 350 Finished.
==> Start Training Epoch 351, lr=0.008175 ...
==> Finished Epoch 351.
==> Start Training Epoch 352, lr=0.008171 ...
==> Finished Epoch 352.
==> Start Training Epoch 353, lr=0.008166 ...
==> Finished Epoch 353.
==> Start Training Epoch 354, lr=0.008161 ...
==> Finished Epoch 354.
==> Start Training Epoch 355, lr=0.008156 ...
==> Finished Epoch 355.
==> Start Training Epoch 356, lr=0.008152 ...
==> Finished Epoch 356.
==> Start Training Epoch 357, lr=0.008147 ...
==> Finished Epoch 357.
==> Start Training Epoch 358, lr=0.008142 ...
==> Finished Epoch 358.
==> Start Training Epoch 359, lr=0.008138 ...
==> Finished Epoch 359.
==> Start Training Epoch 360, lr=0.008133 ...
==> Finished Epoch 360.
==> Start Training Epoch 361, lr=0.008128 ...
==> Finished Epoch 361.
==> Start Training Epoch 362, lr=0.008124 ...
==> Finished Epoch 362.
==> Start Training Epoch 363, lr=0.008119 ...
==> Finished Epoch 363.
==> Start Training Epoch 364, lr=0.008114 ...
==> Finished Epoch 364.
==> Start Training Epoch 365, lr=0.008110 ...
==> Finished Epoch 365.
==> Start Training Epoch 366, lr=0.008105 ...
==> Finished Epoch 366.
==> Start Training Epoch 367, lr=0.008100 ...
==> Finished Epoch 367.
==> Start Training Epoch 368, lr=0.008096 ...
==> Finished Epoch 368.
==> Start Training Epoch 369, lr=0.008091 ...
==> Finished Epoch 369.
==> Start Training Epoch 370, lr=0.008086 ...
==> Finished Epoch 370.
==> Start Training Epoch 371, lr=0.008082 ...
==> Finished Epoch 371.
==> Start Training Epoch 372, lr=0.008077 ...
==> Finished Epoch 372.
==> Start Training Epoch 373, lr=0.008072 ...
==> Finished Epoch 373.
==> Start Training Epoch 374, lr=0.008068 ...
==> Finished Epoch 374.
==> Start Training Epoch 375, lr=0.008063 ...
==> Finished Epoch 375.
==> Start Training Epoch 376, lr=0.008058 ...
==> Finished Epoch 376.
==> Start Training Epoch 377, lr=0.008054 ...
==> Finished Epoch 377.
==> Start Training Epoch 378, lr=0.008049 ...
==> Finished Epoch 378.
==> Start Training Epoch 379, lr=0.008045 ...
==> Finished Epoch 379.
==> Start Training Epoch 380, lr=0.008040 ...
==> Finished Epoch 380.
==> Start Training Epoch 381, lr=0.008035 ...
==> Finished Epoch 381.
==> Start Training Epoch 382, lr=0.008031 ...
==> Finished Epoch 382.
==> Start Training Epoch 383, lr=0.008026 ...
==> Finished Epoch 383.
==> Start Training Epoch 384, lr=0.008021 ...
==> Finished Epoch 384.
==> Start Training Epoch 385, lr=0.008017 ...
==> Finished Epoch 385.
==> Start Training Epoch 386, lr=0.008012 ...
==> Finished Epoch 386.
==> Start Training Epoch 387, lr=0.008008 ...
==> Finished Epoch 387.
==> Start Training Epoch 388, lr=0.008003 ...
==> Finished Epoch 388.
==> Start Training Epoch 389, lr=0.007998 ...
==> Finished Epoch 389.
==> Start Training Epoch 390, lr=0.007994 ...
==> Finished Epoch 390.
==> Start Training Epoch 391, lr=0.007989 ...
==> Finished Epoch 391.
==> Start Training Epoch 392, lr=0.007985 ...
==> Finished Epoch 392.
==> Start Training Epoch 393, lr=0.007980 ...
==> Finished Epoch 393.
==> Start Training Epoch 394, lr=0.007975 ...
==> Finished Epoch 394.
==> Start Training Epoch 395, lr=0.007971 ...
==> Finished Epoch 395.
==> Start Training Epoch 396, lr=0.007966 ...
==> Finished Epoch 396.
==> Start Training Epoch 397, lr=0.007962 ...
==> Finished Epoch 397.
==> Start Training Epoch 398, lr=0.007957 ...
==> Finished Epoch 398.
==> Start Training Epoch 399, lr=0.007952 ...
==> Finished Epoch 399.
==> Start Training Epoch 400, lr=0.007948 ...
==> Finished Epoch 400.
++> Evaluate at epoch 400 ...
PSNR = 20.621731
++> Evaluate epoch 400 Finished.
==> Start Training Epoch 401, lr=0.007943 ...
==> Finished Epoch 401.
==> Start Training Epoch 402, lr=0.007939 ...
==> Finished Epoch 402.
==> Start Training Epoch 403, lr=0.007934 ...
==> Finished Epoch 403.
==> Start Training Epoch 404, lr=0.007930 ...
==> Finished Epoch 404.
==> Start Training Epoch 405, lr=0.007925 ...
==> Finished Epoch 405.
==> Start Training Epoch 406, lr=0.007920 ...
==> Finished Epoch 406.
==> Start Training Epoch 407, lr=0.007916 ...
==> Finished Epoch 407.
==> Start Training Epoch 408, lr=0.007911 ...
==> Finished Epoch 408.
==> Start Training Epoch 409, lr=0.007907 ...
==> Finished Epoch 409.
==> Start Training Epoch 410, lr=0.007902 ...
==> Finished Epoch 410.
==> Start Training Epoch 411, lr=0.007898 ...
==> Finished Epoch 411.
==> Start Training Epoch 412, lr=0.007893 ...
==> Finished Epoch 412.
==> Start Training Epoch 413, lr=0.007889 ...
==> Finished Epoch 413.
==> Start Training Epoch 414, lr=0.007884 ...
==> Finished Epoch 414.
==> Start Training Epoch 415, lr=0.007880 ...
==> Finished Epoch 415.
==> Start Training Epoch 416, lr=0.007875 ...
==> Finished Epoch 416.
==> Start Training Epoch 417, lr=0.007870 ...
==> Finished Epoch 417.
==> Start Training Epoch 418, lr=0.007866 ...
==> Finished Epoch 418.
==> Start Training Epoch 419, lr=0.007861 ...
==> Finished Epoch 419.

[ashawkey]

Myabe you could try to use a much larger --bg_radius like 64 or 128? It is basically an (experimental) approximation of the infinite-far background sphere, so the camera move should be neglectable compared to the bg radius.
Or you could simply turn it off, since the --bound is already large enough to model far objects.
Finally, you could try to do the same thing with the original instant-ngp to get an expectation of how well the scene can be modelled.

[menglongyue]

ok, I will try it. I still have a question. I change the aabb_scale parameter to 4 in llff2nerf.py(default value is 2) . how to set this parameter?

[ashawkey]

In fact aabb_scale has no use in this codebase, it's replaced by --bound as stated in the readme. You may use the same aabb_scale as bound for the original instant-ngp codebase.

[yyyue502]

I met similar issuse when training object with white background, some artifacts are projected to the bounding box, you may try to clip near and far or consider using the corse-to-fine sampling strategy like NeRF.