Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Round compressed grid locations #2509

Merged
merged 22 commits into from
Jun 18, 2024
Merged

Round compressed grid locations #2509

merged 22 commits into from
Jun 18, 2024

Conversation

soheilshahrouz
Copy link
Contributor

When translating a grid location to a compressed grid location, we call std::lower_bound() to find the corresponding compressed location. Assume that we want to translate the grid lcoation (66, 67) for a DSP block in a grid with DSP columns at x=50 and x=70. The current grid_loc_to_compressed_loc_approx() would select the column at x=50, while the one at x=70 was closer. This PR changes grid_loc_to_compressed_loc_approx() to choose the closest compressed location instead.

In calculate_centroid_loc(), we cast float variables to int, which is equivalent to taking their floor. This means that the centroid layer is always zero unless all connected pins are on layer 1.

How Has This Been Tested?

I ran experiments on Titan benchmarks. I'll post QoR summary in a comment in this PR. If the results are promising, we can experiment with other benchmarks and architectures.

@github-actions github-actions bot added VPR VPR FPGA Placement & Routing Tool lang-cpp C/C++ code labels Mar 18, 2024
@soheilshahrouz
Copy link
Contributor Author

Titan benchmarks

vtr_flow_elapsed_time pack_time placed_wl total_swap place_time placed_cpd routed_wl heap_pop route_cpd
master 2858.225484 398.634554 2572680.235 18027807.27 1153.033348 16.20830923 3213503.781 153895598.5 17.3042333
PR 2757.462696 393.088044 2528556.546 18024886.98 1122.98716 16.28941875 3161054.637 139388529.9 17.39327478
Ratio 0.9647463824 0.9860862291 0.9828491361 0.9998380121 0.973941614 1.005004194 0.983678518 0.9057343496 1.005145647

Link to QoR spreadsheet

@vaughnbetz
Copy link
Contributor

QoR looks promising ... will need to go through and check the various test failures to make sure they are small circuit QoR changes that are neutral or irrelevant.
I suggest also running Koios for QoR.

@vaughnbetz
Copy link
Contributor

(Your idea): could also run a 3D arch regtest (your call).

@soheilshahrouz
Copy link
Contributor Author

vtr_reg_nightly_test1 and vtr_reg_nightly_test1_odin failures are fixed.

vtr_reg_nightly_test3_odin hits the 7 hours executin time limit, which is a little strange because vtr_reg_nightly_test3 finishes in ~1 hour.

@vaughnbetz
Copy link
Contributor

If this looks like a random failure we could move some test out of vtr_reg_nightly_test3_odin (and put it in another suite, or tweak the designs used).

@soheilshahrouz
Copy link
Contributor Author

I ran nightly3_odin tasks on wintermute. There is no significant execution time difference between this branch and the master branch on wintermute.

@vaughnbetz
Copy link
Contributor

OK, if you resolve the conflicts we can merge. They look like a bunch of golden result conflicts, so hopefully not too hard to resolve.

Maybe the odin II run is at the edge of 6 hours and we need to split it?

@soheilshahrouz
Copy link
Contributor Author

Maybe the odin II run is at the edge of 6 hours and we need to split it?

I am running tasks in nightly3_odin as different CI tests to see how long each of them takes. On master, the entire test takes ~3.5 hours. If we are at the edge of the 7 hours time limit, it means that the execution time has doubled.

@soheilshahrouz
Copy link
Contributor Author

The mcml circuit in vtr_reg_qor_chain_predictor_off has an execution time about 5 times longer than the master branch. Finding the minimum channel width takes 38 times longer than on the master branch.

I missed this unusual increase in runtime because I was using the geomean to compare the execution time of two branches. Since only a single circuit's runtime increased significantly, the geomean value didn't change much.

@soheilshahrouz
Copy link
Contributor Author

soheilshahrouz commented May 31, 2024

vpr.txt

It seems that the router tries to route the design at a channel width much lower than the minimum channel width. Since routing_failure_predictor is off, it does not terminate the router even when each iteration takes ~500 seconds. Is this behaviour expected?

@vaughnbetz
Copy link
Contributor

Yes, that can happen. With the routing predictor off, a minimum channel width search could take a long time. We could just take mcml out of the test -- probably doing a minimum channel width search with larger circuits and the predictor off is not a great idea.

…est3_odin

With routing failure predictor turned off, finding the minimum channel width takes for this circuit takes so long that the CI test failed.
@vaughnbetz
Copy link
Contributor

Update: @soheilshahrouz has removed mcml from the problematic (slow) test. Need to remove it from the golden results:

2024-06-03T19:56:18.0542628Z �[32;1m19:56:18�[0m | regression_tests/vtr_reg_nightly_test3_odin/vtr_reg_qor_chain_depop...[Pass]
2024-06-03T19:56:18.0545553Z �[32;1m19:56:18�[0m | Error: Required case k6_frac_N10_frac_chain_mem32K_40nm.xml/mcml.v missing from task results: /root/vtr-verilog-to-routing/vtr-verilog-to-routing/vtr_flow/tasks/regression_tests/vtr_reg_nightly_test3_odin/vtr_reg_qor_chain_predictor_off/run001/parse_results.txt
2024-06-03T19:56:18.0547610Z �[32;1m19:56:18�[0m |
2024-06-03T19:56:18.0548391Z �[32;1m19:56:18�[0m | Test 'vtr_reg_nightly_test3_odin' had -1 qor test failures
2024-06-03T19:56:18.0549325Z �[32;1m19:56:18�[0m |
2024-06-03T19:56:18.0550053Z �[32;1m19:56:18�[0m | Test 'vtr_reg_nightly_test3_odin' had 0 run failures

Also some minimum channel width increases (out of the allowed bounds) on a few small circuits, which is probably due to overwriting new golden results with the results from this PR. Checking that all those failures are in that class and updating golden should fix that.

@vaughnbetz
Copy link
Contributor

Whoops, it looks like another merge I did nuked this. The error is that an enum was changed to an enum class.

/root/vtr-verilog-to-routing/vtr-verilog-to-routing/vpr/src/pack/cluster_util.cpp:1248:20: error: 'BLK_FAILED_FEASIBLE' was not declared in this scope; did you mean 'e_block_pack_status::BLK_FAILED_FEASIBLE

If you change that line to return e_block_pack_status::BLK_FAILED_FEASIBLE this PR should work again Kimia.

@vaughnbetz
Copy link
Contributor

Looks like we're getting disk quota issues in CI. @AlexandreSinger : I think you were about to change the artifact collection settings? That might be getting time critical.

2024-06-06T17:14:03.2100495Z Requested labels: self-hosted, Linux, X64
2024-06-06T17:14:03.2100777Z Job defined at: verilog-to-routing/vtr-verilog-to-routing/.github/workflows/test.yml@refs/heads/round_comp_loc
2024-06-06T17:14:03.2100881Z Waiting for a runner to pick up this job...
2024-06-06T17:14:04.1713331Z Job is about to start running on the runner: gh-actions-runner-vtr_199 (repository)
2024-06-06T17:14:08.1524125Z �[32;1m17:14:08�[0m | Attempting to spawn a machine... (PID: 11569)
2024-06-06T17:14:08.1534275Z �[32;1m17:14:08�[0m | Instance name: gh-actions-runner-vtr-auto-spawned199
2024-06-06T17:14:08.1537539Z �[32;1m17:14:08�[0m | Instance type: n2-highmem-16
2024-06-06T17:14:08.1539870Z �[32;1m17:14:08�[0m | Disk type: pd-ssd
2024-06-06T17:14:08.1543555Z �[32;1m17:14:08�[0m | LABELS: [{'github_job_full': 'run_tests_vtr_reg_nightly_test3_16_', 'github_sha': 'efe606d49f7b7bbb45b42f6c829fdca63370eafb', 'github_run_id': '9405078419'}]
2024-06-06T17:14:08.1548843Z �[32;1m17:14:08�[0m | Preemptible: False
2024-06-06T17:14:10.9444418Z �[32;1m17:14:10�[0m | Attempting to spawn a machine in us-central1-a
2024-06-06T17:14:30.9285699Z �[32;1m17:14:30�[0m | Error occured while spawning the instance: {'code': 'QUOTA_EXCEEDED', 'message': "Quota 'DISKS_TOTAL_GB' exceeded. Limit: 102400.0 in region us-central1.", 'errorDetails': [{'quotaInfo': {'metricName': 'compute.googleapis.com/disks_total_storage', 'limitName': 'DISKS-TOTAL-GB-per-project-region', 'dimensions': {'region': 'us-central1'}, 'limit': 102400}}]}
2024-06-06T17:14:30.9609730Z �[32;1m17:14:30�[0m |
2024-06-06T17:14:32.0701444Z �[32;1m17:14:32�[0m |
2024-06-06T17:14:33.5684742Z �[32;1m17:14:33�[0m | Attempting to spawn a machine... (PID: 12463)
2024-06-06T17:14:33.5709456Z �[32;1m17:14:33�[0m | Instance name: gh-actions-runner-vtr-auto-spawned199
2024-06-06T17:14:33.5711162Z �[32;1m17:14:33�[0m | Instance type: n2-highmem-16
2024-06-06T17:14:33.5712307Z �[32;1m17:14:33�[0m | Disk type: pd-ssd
2024-06-06T17:14:33.5714142Z �[32;1m17:14:33�[0m | LABELS: [{'github_job_full': 'run_tests_vtr_reg_nightly_test3_16_', 'github_sha': 'efe606d49f7b7bbb45b42f6c829fdca63370eafb', 'github_run_id': '9405078419'}]
2024-06-06T17:14:33.5716472Z �[32;1m17:14:33�[0m | Preemptible: False
2024-06-06T17:14:36.1309915Z �[32;1m17:14:36�[0m | Attempting to spawn a machine in us-central1-a
2024-06-06T17:15:00.0817421Z �[32;1m17:15:00�[0m | Error occured while spawning the instance: {'code': 'QUOTA_EXCEEDED', 'message': "Quota 'DISKS_TOTAL_GB' exceeded. Limit: 102400.0 in region us-central1.", 'errorDetails': [{'quotaInfo': {'metricName': 'compute.googleapis.com/disks_total_storage', 'limitName': 'DISKS-TOTAL-GB-per-project-region', 'dimensions': {'region': 'us-central1'}, 'limit': 102400}}]}
2024-06-06T17:15:00.1133073Z �[32;1m17:15:00�[0m |
2024-06-06T17:15:01.3026225Z �[32;1m17:15:01�[0m |
2024-06-06T17:15:02.7894178Z �[32;1m17:15:02�[0m | Attempting to spawn a machine... (PID: 13558)
2024-06-06T17:15:02.7922919Z �[32;1m17:15:02�[0m | Instance name: gh-actions-runner-vtr-auto-spawned199
2024-06-06T17:15:02.7924375Z �[32;1m17:15:02�[0m | Instance type: n2-highmem-16
2024-06-06T17:15:02.7925335Z �[32;1m17:15:02�[0m | Disk type: pd-ssd
2024-06-06T17:15:02.7926921Z �[32;1m17:15:02�[0m | LABELS: [{'github_job_full': 'run_tests_vtr_reg_nightly_test3_16_', 'github_sha': 'efe606d49f7b7bbb45b42f6c829fdca63370eafb', 'github_run_id': '9405078419'}]
2024-06-06T17:15:02.7929127Z �[32;1m17:15:02�[0m | Preemptible: False
2024-06-06T17:15:05.2320704Z �[32;1m17:15:05�[0m | Attempting to spawn a machine in us-central1-a
2024-06-06T17:15:23.9091269Z �[32;1m17:15:23�[0m | Error occured while spawning the instance: {'code': 'QUOTA_EXCEEDED', 'message': "Quota 'DISKS_TOTAL_GB' exceeded. Limit: 102400.0 in region us-central1.", 'errorDetails': [{'quotaInfo': {'metricName': 'compute.googleapis.com/disks_total_storage', 'limitName': 'DISKS-TOTAL-GB-per-project-region', 'dimensions': {'region': 'us-central1'}, 'limit': 102400}}]}
2024-06-06T17:15:23.9413133Z �[32;1m17:15:23�[0m |
2024-06-06T17:15:24.9601691Z �[32;1m17:15:24�[0m |
2024-06-06T17:15:26.2297039Z �[32;1m17:15:26�[0m | Attempting to spawn a machine... (PID: 14339)
2024-06-06T17:15:26.2320467Z �[32;1m17:15:26�[0m | Instance name: gh-actions-runner-vtr-auto-spawned199
2024-06-06T17:15:26.2321782Z �[32;1m17:15:26�[0m | Instance type: n2-highmem-16
2024-06-06T17:15:26.2322904Z �[32;1m17:15:26�[0m | Disk type: pd-ssd
2024-06-06T17:15:26.2324603Z �[32;1m17:15:26�[0m | LABELS: [{'github_job_full': 'run_tests_vtr_reg_nightly_test3_16_', 'github_sha': 'efe606d49f7b7bbb45b42f6c829fdca63370eafb', 'github_run_id': '9405078419'}]
2024-06-06T17:15:26.2328944Z �[32;1m17:15:26�[0m | Preemptible: False
2024-06-06T17:15:29.1885332Z �[32;1m17:15:29�[0m | Attempting to spawn a machine in us-central1-a
2024-06-06T17:15:49.8489609Z �[32;1m17:15:49�[0m | Error occured while spawning the instance: {'code': 'QUOTA_EXCEEDED', 'message': "Quota 'DISKS_TOTAL_GB' exceeded. Limit: 102400.0 in region us-central1.", 'errorDetails': [{'quotaInfo': {'metricName': 'compute.googleapis.com/disks_total_storage', 'limitName': 'DISKS-TOTAL-GB-per-project-region', 'dimensions': {'region': 'us-central1'}, 'limit': 102400}}]}
2024-06-06T17:15:49.8850793Z �[32;1m17:15:49�[0m |
2024-06-06T17:15:51.0529951Z �[32;1m17:15:51�[0m |
2024-06-06T17:15:52.3631606Z �[32;1m17:15:52�[0m | Attempting to spawn a machine... (PID: 16229)
2024-06-06T17:15:52.3658954Z �[32;1m17:15:52�[0m | Instance name: gh-actions-runner-vtr-auto-spawned199
2024-06-06T17:15:52.3660375Z �[32;1m17:15:52�[0m | Instance type: n2-highmem-16
2024-06-06T17:15:52.3661452Z �[32;1m17:15:52�[0m | Disk type: pd-ssd
2024-06-06T17:15:52.3663496Z �[32;1m17:15:52�[0m | LABELS: [{'github_job_full': 'run_tests_vtr_reg_nightly_test3_16_', 'github_sha': 'efe606d49f7b7bbb45b42f6c829fdca63370eafb', 'github_run_id': '9405078419'}]
2024-06-06T17:15:52.3665604Z �[32;1m17:15:52�[0m | Preemptible: False
2024-06-06T17:15:54.6993528Z �[32;1m17:15:54�[0m | Attempting to spawn a machine in us-central1-a
2024-06-06T17:16:14.8833117Z �[32;1m17:16:14�[0m | Error occured while spawning the instance: {'code': 'QUOTA_EXCEEDED', 'message': "Quota 'DISKS_TOTAL_GB' exceeded. Limit: 102400.0 in region us-central1.", 'errorDetails': [{'quotaInfo': {'metricName': 'compute.googleapis.com/disks_total_storage', 'limitName': 'DISKS-TOTAL-GB-per-project-region', 'dimensions': {'region': 'us-central1'}, 'limit': 102400}}]}
2024-06-06T17:16:14.9192247Z �[32;1m17:16:14�[0m |
2024-06-06T17:16:16.0565190Z �[32;1m17:16:16�[0m |
2024-06-06T17:16:17.0661708Z ##[error]VM starter exited with non-zero exit code: 1

@AlexandreSinger
Copy link
Contributor

AlexandreSinger commented Jun 6, 2024

@vaughnbetz Perhaps. However, I think the artifacts are stored on the GitHub servers not the self-hosted runners.

My theory is that we have too many CI runs in flight right now. We currently have 12 CI runs in flight, each one running 17 independent tests (running on different self-hosted runners), and we have 100 self-hosted runners. I do not think its a coincidence that we are starting to hit issues when the number of runners required goes above 100.

I think once we change the CI triggers it should help reduce the load.

…rea_per_tile for vtr_reg_nightly_test1_odin/arithmetic_tasks/multless_consts/fixed_k6_frac_2ripple_N8_22nm.xml/mult_007.v
@soheilshahrouz
Copy link
Contributor Author

@vaughnbetz
I updated the golden results for CI tests that were failing for out of bound QoR values. Now, all CI tests pass.

echo_compressed_grids() only printed the first element in each compressed grid axis
3D support added a new dimension to each compressed grid axis. This new dimension was not indexed properly in echo_compressed_grids()
@github-actions github-actions bot added the libarchfpga Library for handling FPGA Architecture descriptions label Jun 14, 2024
@vaughnbetz vaughnbetz merged commit 871b28a into master Jun 18, 2024
102 checks passed
@vaughnbetz vaughnbetz deleted the round_comp_loc branch June 18, 2024 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lang-cpp C/C++ code libarchfpga Library for handling FPGA Architecture descriptions VPR VPR FPGA Placement & Routing Tool
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants