Skip to content

Latest commit

 

History

History
76 lines (74 loc) · 25.6 KB

Report.md

File metadata and controls

76 lines (74 loc) · 25.6 KB
Raw

Results for "Guiding Language Models of Code with Global Context using Monitors"

Summary of Results (Table 1 in the paper)

configuration ('Compilation Rate (CR)', 'score@1') ('Compilation Rate (CR)', 'score@2') ('Compilation Rate (CR)', 'score@3') ('Compilation Rate (CR)', 'score@4') ('Compilation Rate (CR)', 'score@5') ('Compilation Rate (CR)', 'score@6') ('Next Identifier Match (NIM)', 'score@1') ('Next Identifier Match (NIM)', 'score@2') ('Next Identifier Match (NIM)', 'score@3') ('Next Identifier Match (NIM)', 'score@4') ('Next Identifier Match (NIM)', 'score@5') ('Next Identifier Match (NIM)', 'score@6') ('Identifier Sequence Match (ISM)', 'score@1') ('Identifier Sequence Match (ISM)', 'score@2') ('Identifier Sequence Match (ISM)', 'score@3') ('Identifier Sequence Match (ISM)', 'score@4') ('Identifier Sequence Match (ISM)', 'score@5') ('Identifier Sequence Match (ISM)', 'score@6') ('Prefix Match (PM)', 'score@1') ('Prefix Match (PM)', 'score@2') ('Prefix Match (PM)', 'score@3') ('Prefix Match (PM)', 'score@4') ('Prefix Match (PM)', 'score@5') ('Prefix Match (PM)', 'score@6')
CG-350M 35.72 43.04 46.82 49.27 51.04 52.43 68.89 72.68 74.41 75.50 76.30 76.94 26.10 28.74 29.99 30.79 31.38 31.86 21.81 24.12 25.23 25.93 26.45 26.86
CG-350M-MGD 44.58 53.70 58.41 61.47 63.69 65.37 76.22 79.89 81.53 82.55 83.26 83.80 28.15 30.93 32.27 33.14 33.79 34.31 23.38 25.78 26.94 27.69 28.25 28.69
CG-2B 42.05 48.75 52.07 54.21 55.78 57.01 73.81 77.26 78.85 79.86 80.57 81.11 30.20 32.94 34.31 35.21 35.86 36.38 25.52 27.93 29.12 29.91 30.49 30.95
CG-2B-MGD 52.88 61.23 65.23 67.74 69.52 70.91 80.52 83.75 85.21 86.13 86.80 87.32 32.36 35.31 36.78 37.75 38.46 39.03 27.28 29.84 31.11 31.95 32.57 33.06
CG-6B 44.19 50.59 53.77 55.87 57.42 58.64 74.53 77.83 79.33 80.28 80.98 81.55 31.01 33.77 35.12 36.00 36.66 37.17 26.35 28.79 29.95 30.71 31.26 31.69
CG-6B-MGD 55.18 62.92 66.66 69.10 70.88 72.28 81.09 84.14 85.48 86.31 86.90 87.35 33.17 35.99 37.39 38.30 38.99 39.55 28.12 30.58 31.76 32.53 33.10 33.56
SC 44.70 51.32 54.69 56.93 58.63 59.97 75.74 78.91 80.32 81.21 81.87 82.40 31.84 34.59 35.97 36.89 37.58 38.14 26.75 29.10 30.27 31.05 31.63 32.10
SC-MGD 55.34 63.56 67.49 69.94 71.69 73.03 82.30 85.24 86.53 87.34 87.94 88.42 34.14 37.02 38.46 39.41 40.12 40.69 28.60 31.10 32.35 33.17 33.77 34.25
SC-classExprTypes 48.06 55.40 59.04 61.41 63.17 64.57 78.57 81.61 82.96 83.80 84.42 84.91 33.21 36.06 37.48 38.41 39.11 39.67 27.93 30.41 31.64 32.45 33.06 33.55
SC-classExprTypes-MGD 56.80 65.20 69.23 71.77 73.59 75.01 83.55 86.39 87.62 88.39 88.94 89.37 34.92 37.86 39.31 40.27 40.99 41.56 29.30 31.82 33.06 33.88 34.49 34.98
SC-RLPG 50.59 57.76 61.24 63.48 65.11 66.39 79.65 82.51 83.75 84.50 85.03 85.42 35.57 38.52 40.03 41.03 41.77 42.35 30.34 32.93 34.23 35.08 35.71 36.21
SC-RLPG-MGD 60.50 68.78 72.66 75.08 76.80 78.14 84.71 87.28 88.37 89.05 89.53 89.89 37.48 40.56 42.11 43.13 43.88 44.47 31.81 34.53 35.88 36.77 37.44 37.97
SC-FIM 51.72 59.31 62.94 65.25 66.92 68.23 79.58 82.42 83.68 84.49 85.08 85.56 35.53 38.49 39.95 40.91 41.64 42.22 30.34 32.93 34.19 35.02 35.63 36.12
SC-FIM-MGD 62.16 70.73 74.71 77.17 78.89 80.19 84.68 87.24 88.33 89.01 89.50 89.89 37.55 40.61 42.13 43.13 43.89 44.50 31.94 34.60 35.90 36.76 37.40 37.91
SC-FIM-classExprTypes 53.30 61.43 65.35 67.82 69.60 70.97 81.36 84.13 85.32 86.06 86.58 86.99 36.10 39.04 40.48 41.42 42.12 42.67 30.62 33.19 34.44 35.26 35.87 36.36
SC-FIM-classExprTypes-MGD 61.71 70.61 74.70 77.20 78.97 80.33 85.32 87.81 88.89 89.56 90.04 90.42 37.68 40.61 42.02 42.94 43.63 44.18 31.88 34.51 35.79 36.63 37.25 37.75
TD-3 51.71 56.73 59.11 60.64 61.77 62.66 80.76 83.42 84.56 85.27 85.78 86.18 38.57 41.49 42.87 43.77 44.44 44.97 33.13 35.69 36.90 37.70 38.29 38.77
TD-3-MGD 61.23 67.27 70.11 71.93 73.24 74.26 86.32 88.71 89.74 90.37 90.83 91.19 40.69 43.67 45.12 46.07 46.77 47.33 34.29 36.84 38.07 38.87 39.46 39.94

Effect of MGD on Models across Parameter Scale and Architectures (Ref. section 4.1)

Compilation Rate (CR) score@k Relative Change in Compilation Rate (CR) (score@6)
Next Identifier Match (NIM) score@k Relative Change in Next Identifier Match (NIM) (score@6)
:-------------------------: :-------------------------:
Identifier Sequence Match (ISM) score@k Relative Change in Identifier Sequence Match (ISM) (score@6)
:-------------------------: :-------------------------:
Prefix Match (PM) score@k Relative Change in Prefix Match (PM) (score@6)
:-------------------------: :-------------------------:

Effect of MGD and Prompt Augmentation Strategies (Ref. section 4.2)

Compilation Rate (CR) score@k Relative Change in Compilation Rate (CR) (score@6)
Next Identifier Match (NIM) score@k Relative Change in Next Identifier Match (NIM) (score@6)
:-------------------------: :-------------------------:
Identifier Sequence Match (ISM) score@k Relative Change in Identifier Sequence Match (ISM) (score@6)
:-------------------------: :-------------------------:
Prefix Match (PM) score@k Relative Change in Prefix Match (PM) (score@6)
:-------------------------: :-------------------------:

Effect of MGD on Fill-in-the-middle (FIM) Decoding (Ref. section 4.3 and appendix E)

Compilation Rate (CR) score@k Relative Change in Compilation Rate (CR) (score@6)
Next Identifier Match (NIM) score@k Relative Change in Next Identifier Match (NIM) (score@6)
:-------------------------: :-------------------------:
Identifier Sequence Match (ISM) score@k Relative Change in Identifier Sequence Match (ISM) (score@6)
:-------------------------: :-------------------------:
Prefix Match (PM) score@k Relative Change in Prefix Match (PM) (score@6)
:-------------------------: :-------------------------:

Effect of Identifier Complexity on Next Identifier Match (Ref. section 4.4 and appendix F)

Distribution of methods by most complex identifier in DotPrompts

Next Identifier Match (NIM) score@6 by identifier complexity

NIM score@6 Relative Change in NIM (score@6)
NIM score@6 Relative Change in NIM (score@6)
:-------------------------: :-------------------------:
NIM score@6 Relative Change in NIM (score@6)
:-------------------------: :-------------------------: