Skip to content

Commit 92743cb

Browse files
VerviousVervious
committed
update
1 parent a91aa2f commit 92743cb

24 files changed

+4819
-660
lines changed

‎README.md

+16
Original file line numberDiff line numberDiff line change
@@ -567,3 +567,19 @@ Position embeddings are definitely additive. If we switch from `x = tok_emb + po
567567

568568
![loss plot](img/11-axm-multposi.png)
569569

570+
If we do `y = self.attn(x), z=self.mlp(x*y)*y`, the capability is slightly weakened, but not fully destroyed (it probably also converges?)
571+
572+
![loss plot](img/11-axm-mlpinputmodi.png)
573+
574+
Adjusting the input again, `y = self.attn(x), z=self.mlp(x+y)*y`
575+
576+
![loss plot](img/11-axm-mlpinputmod2i.png)
577+
578+
So really, we want to feed in just x into the mlp. (Why?)
579+
580+
Now, let's see what happens if we don't feed in the residual to future layers, but we do add the positional embedding back in:
581+
582+
![loss plot](img/11-axm-addposembagain-noresi.png)
583+
584+
Nope! The residual is clearly important for reasons other than the initial positional embedding.
585+

‎img/11-axm-addposembagain-noresi.png

27.1 KB
Loading

‎img/11-axm-addposembagaina.mp4

371 KB
Binary file not shown.

‎img/11-axm-addposembagaini.png

26.8 KB
Loading

‎img/11-axm-mlpinputmod-norms.png

116 KB
Loading

‎img/11-axm-mlpinputmod2i.png

27.1 KB
Loading

‎img/11-axm-mlpinputmoda.mp4

404 KB
Binary file not shown.

‎img/11-axm-mlpinputmodi.png

28.5 KB
Loading

‎img/11-axm-multposi.png

-3 Bytes
Loading

‎log-ben/11-axm-addposembagain-log.txt

+772
Large diffs are not rendered by default.

‎log-ben/11-axm-addposembagain-nores-log.txt

+371
Large diffs are not rendered by default.

‎log-ben/11-axm-mlpinputmod-log.txt

+1,483
Large diffs are not rendered by default.

‎log-ben/11-axm-mlpinputmod2-log.txt

+534
Large diffs are not rendered by default.

‎log/screen.txt

+360-648
Large diffs are not rendered by default.

‎play.ipynb

+9-9
Large diffs are not rendered by default.

‎samples-ben/11-axm-addposembagain-main.txt

+113
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
1: sample 0: A Poem for you! Roses are red, Potatoes are
2+
------
3+
( competing:0.0002 out:0.0002 diced:0.0003 67:0.0002 Italian:0.0002 Carnegie:0.0002 Klingon:0.0001ock:0.0001)
4+
( Highland:0.0002 Fork:0.0002 mysteries:0.0001 Cannabis:0.0002A:0.0002 discussing:0.0002 Apprentice:0.0002 moratorium:0.0002)
5+
(A:0.0002 Marcos:0.0002umed:0.0002 Marcos:0.0001A:0.0002,:0.0002 enqu:0.0002 accommod:0.0001)
6+
(A:0.0002 Marcos:0.0002A:0.0002 Guy:0.0002A:0.0002 reminds:0.0002 egalitarian:0.0002 neurons:0.0001)
7+
(A:0.0002?":0.0002A:0.0002 Guy:0.0002A:0.0002riger:0.0001 egalitarian:0.0002A:0.0001)
8+
(�:0.0002?":0.0002A:0.0002 Guy:0.0002A:0.0002imei:0.0001ogeneous:0.0002 Lv:0.0001)
9+
(�:0.0002�:0.0002 am:0.0002imei:0.0002A:0.0002imei:0.0002ogeneous:0.0002 Lv:0.0002)
10+
(�:0.0002�:0.0002 am:0.0002imei:0.0002�:0.0002imei:0.0002ogeneous:0.0002 Lv:0.0002)
11+
(�:0.0002�:0.0002 am:0.0002imei:0.0002�:0.0002ogeneous:0.0002ogeneous:0.0002�:0.0002)
12+
(�:0.0002�:0.0001 am:0.0002imei:0.0002�:0.0002ogeneous:0.0002ogeneous:0.0002�:0.0002)
13+
(�:0.0002 Holt:0.0001 am:0.0002ogeneous:0.0002�:0.0002ogeneous:0.0002ogeneous:0.0002�:0.0002)
14+
(�:0.0002 Holt:0.0001 am:0.0002ogeneous:0.0002�:0.0002ogeneous:0.0002ogeneous:0.0002�:0.0002)
15+
(�:0.0002 Holt:0.0001 am:0.0002ogeneous:0.0002�:0.0002ogeneous:0.0002ogeneous:0.0002�:0.0002)
16+
�
17+
------
18+
( out:0.0002 diced:0.0003 67:0.0002 Italian:0.0002 Carnegie:0.0002 Klingon:0.0001ock:0.0001 aired:0.0002)
19+
( Fork:0.0002 mysteries:0.0001 Cannabis:0.0002A:0.0002 discussing:0.0002 Apprentice:0.0002 moratorium:0.0002reements:0.0001)
20+
( Marcos:0.0002umed:0.0002 Marcos:0.0001A:0.0002,:0.0002 enqu:0.0002 accommod:0.0001�:0.0003)
21+
( Marcos:0.0002A:0.0002 Guy:0.0002A:0.0002 reminds:0.0002 egalitarian:0.0002 neurons:0.0001�:0.0003)
22+
(?":0.0002A:0.0002 Guy:0.0002A:0.0002riger:0.0001 egalitarian:0.0002A:0.0001�:0.0003)
23+
(?":0.0002A:0.0002 Guy:0.0002A:0.0002imei:0.0001ogeneous:0.0002 Lv:0.0001�:0.0003)
24+
(�:0.0002 am:0.0002imei:0.0002A:0.0002imei:0.0002ogeneous:0.0002 Lv:0.0002�:0.0003)
25+
(�:0.0002 am:0.0002imei:0.0002�:0.0002imei:0.0002ogeneous:0.0002 Lv:0.0002�:0.0003)
26+
(�:0.0002 am:0.0002imei:0.0002�:0.0002ogeneous:0.0002ogeneous:0.0002�:0.0002�:0.0003)
27+
(�:0.0001 am:0.0002imei:0.0002�:0.0002ogeneous:0.0002ogeneous:0.0002�:0.0002�:0.0003)
28+
( Holt:0.0001 am:0.0002ogeneous:0.0002�:0.0002ogeneous:0.0002ogeneous:0.0002�:0.0002�:0.0003)
29+
( Holt:0.0001 am:0.0002ogeneous:0.0002�:0.0002ogeneous:0.0002ogeneous:0.0002�:0.0002�:0.0003)
30+
( Holt:0.0001 am:0.0002ogeneous:0.0002�:0.0002ogeneous:0.0002ogeneous:0.0002�:0.0002�:0.0003)
31+
���������������
32+
50: sample 0: A Poem for you! Roses are red, Potatoes are
33+
------
34+
( It:0.0013The:0.0071 It:0.0048The:0.0093 hardcore:0.0008 When:0.0011The:0.0103 the:0.0405)
35+
( for:0.0308 you:0.0452.:0.0330 you:0.0608,:0.0181,:0.0359 you:0.0562 you:0.0315)
36+
( for:0.0325 you:0.0508.:0.0374 you:0.0645,:0.0217,:0.0439 you:0.0649
37+
:0.0337)
38+
(.:0.0342 you:0.0549.:0.0425 you:0.0698,:0.0254,:0.0532 you:0.0732
39+
:0.0430)
40+
(.:0.0374 you:0.0564.:0.0466 you:0.0737,:0.0287,:0.0608 you:0.0762
41+
:0.0481)
42+
(.:0.0396 you:0.0569.:0.0498 you:0.0757,:0.0315,:0.0693 you:0.0767
43+
:0.0500)
44+
(.:0.0430 you:0.0532.:0.0535 you:0.0786,:0.0337,:0.0771 you:0.0718
45+
:0.0518)
46+
(.:0.0442 you:0.0486.:0.0542 you:0.0776,:0.0347,:0.0830 you:0.0664
47+
:0.0535)
48+
(.:0.0469 you:0.0432.:0.0566 you:0.0752,:0.0359,:0.0874 you:0.0601
49+
:0.0537)
50+
(.:0.0481 you:0.0386.:0.0562 you:0.0728,:0.0359,:0.0894 you:0.0530
51+
:0.0554)
52+
(.:0.0496 you:0.0347.:0.0557 you:0.0693,:0.0361,:0.0913 you:0.0466
53+
:0.0557)
54+
(.:0.0496 the:0.0330.:0.0542 you:0.0645,:0.0364,:0.0913 you:0.0410
55+
:0.0557)
56+
(.:0.0496 the:0.0330.:0.0542 you:0.0645,:0.0364,:0.0913 you:0.0410
57+
:0.0557)
58+
59+
60+
------
61+
(The:0.0071 It:0.0048The:0.0093 hardcore:0.0008 When:0.0011The:0.0103 the:0.0405
62+
:0.0586)
63+
( you:0.0452.:0.0330 you:0.0608,:0.0181,:0.0359 you:0.0562 you:0.0315
64+
:0.0190)
65+
( you:0.0508.:0.0374 you:0.0645,:0.0217,:0.0439 you:0.0649
66+
:0.0337
67+
:0.0625)
68+
( you:0.0549.:0.0425 you:0.0698,:0.0254,:0.0532 you:0.0732
69+
:0.0430
70+
:0.0645)
71+
( you:0.0564.:0.0466 you:0.0737,:0.0287,:0.0608 you:0.0762
72+
:0.0481
73+
:0.0640)
74+
( you:0.0569.:0.0498 you:0.0757,:0.0315,:0.0693 you:0.0767
75+
:0.0500
76+
:0.0645)
77+
( you:0.0532.:0.0535 you:0.0786,:0.0337,:0.0771 you:0.0718
78+
:0.0518
79+
:0.0630)
80+
( you:0.0486.:0.0542 you:0.0776,:0.0347,:0.0830 you:0.0664
81+
:0.0535
82+
:0.0635)
83+
( you:0.0432.:0.0566 you:0.0752,:0.0359,:0.0874 you:0.0601
84+
:0.0537
85+
:0.0635)
86+
( you:0.0386.:0.0562 you:0.0728,:0.0359,:0.0894 you:0.0530
87+
:0.0554
88+
:0.0635)
89+
( you:0.0347.:0.0557 you:0.0693,:0.0361,:0.0913 you:0.0466
90+
:0.0557
91+
:0.0620)
92+
( the:0.0330.:0.0542 you:0.0645,:0.0364,:0.0913 you:0.0410
93+
:0.0557
94+
:0.0620)
95+
( the:0.0330.:0.0542 you:0.0645,:0.0364,:0.0913 you:0.0410
96+
:0.0557
97+
:0.0620)
98+
99+
100+
101+
102+
103+
104+
105+
106+
107+
108+
109+
110+
111+
112+
113+

‎samples-ben/11-axm-addposembagain-nores-main.txt

+64
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
1: sample 0: A Poem for you! Roses are red, Potatoes are
2+
------
3+
(Source:0.0002 slashing:0.0002 abst:0.0002form:0.0002 ballistic:0.0002 gamer:0.0002 Cornel:0.0002phans:0.0002)
4+
(code:0.0002 Insider:0.0002 Noise:0.0002 storage:0.0002ected:0.0002also:0.0002 Across:0.0002phas:0.0002)
5+
(:0.0002 ADC:0.0003Design:0.0002 Robb:0.0002ura:0.0002marriage:0.0001886:0.0001eth:0.0001)
6+
(A:0.0002 Ful:0.0002 times:0.0002ties:0.0002 Euro:0.0002Log:0.0003 Zach:0.0002Log:0.0002)
7+
( ideally:0.0002meric:0.0001 Lew:0.0002 wander:0.0001 magic:0.0001 Treaty:0.0001 appalled:0.0001 knowledgeable:0.0002)
8+
(raved:0.0002Log:0.0002Log:0.0002raved:0.0002Log:0.0002raved:0.0002Log:0.0002Log:0.0002)
9+
( Suite:0.0001 ter:0.0002scene:0.0002scene:0.0001scene:0.0001scene:0.0001Builder:0.0001scene:0.0001)
10+
(Log:0.0002Log:0.0002Log:0.0002Log:0.0002Log:0.0002Log:0.0002Log:0.0002Log:0.0002)
11+
( stimuli:0.0002process:0.0002scene:0.0002Builder:0.0002Builder:0.0002Builder:0.0002Builder:0.0002Builder:0.0001)
12+
( dancing:0.0002 dancing:0.0001 tant:0.0002ishable:0.0001ishable:0.0001ishable:0.0001 Dietary:0.0001 Dietary:0.0001)
13+
( dancing:0.0002 dancing:0.0002 rem:0.0002remote:0.0002remote:0.0002remote:0.0002remote:0.0001 Cody:0.0001)
14+
(fair:0.0002fair:0.0002 Fried:0.0002 Fried:0.0002 Fried:0.0002UTH:0.0002 Pork:0.0002 Pork:0.0002)
15+
(fair:0.0002fair:0.0002 Fried:0.0002 Fried:0.0002 Fried:0.0002UTH:0.0002 Pork:0.0002 Pork:0.0002)
16+
Pork
17+
------
18+
( slashing:0.0002 abst:0.0002form:0.0002 ballistic:0.0002 gamer:0.0002 Cornel:0.0002phans:0.0002 intens:0.0002)
19+
( Insider:0.0002 Noise:0.0002 storage:0.0002ected:0.0002also:0.0002 Across:0.0002phas:0.0002 ****:0.0002)
20+
( ADC:0.0003Design:0.0002 Robb:0.0002ura:0.0002marriage:0.0001886:0.0001eth:0.0001letters:0.0002)
21+
( Ful:0.0002 times:0.0002ties:0.0002 Euro:0.0002Log:0.0003 Zach:0.0002Log:0.0002Log:0.0002)
22+
(meric:0.0001 Lew:0.0002 wander:0.0001 magic:0.0001 Treaty:0.0001 appalled:0.0001 knowledgeable:0.0002 momentarily:0.0001)
23+
(Log:0.0002Log:0.0002raved:0.0002Log:0.0002raved:0.0002Log:0.0002Log:0.0002Log:0.0002)
24+
( ter:0.0002scene:0.0002scene:0.0001scene:0.0001scene:0.0001Builder:0.0001scene:0.0001scene:0.0001)
25+
(Log:0.0002Log:0.0002Log:0.0002Log:0.0002Log:0.0002Log:0.0002Log:0.0002Log:0.0002)
26+
(process:0.0002scene:0.0002Builder:0.0002Builder:0.0002Builder:0.0002Builder:0.0002Builder:0.0001Builder:0.0001)
27+
( dancing:0.0001 tant:0.0002ishable:0.0001ishable:0.0001ishable:0.0001 Dietary:0.0001 Dietary:0.0001 Dietary:0.0001)
28+
( dancing:0.0002 rem:0.0002remote:0.0002remote:0.0002remote:0.0002remote:0.0001 Cody:0.0001 Cody:0.0002)
29+
(fair:0.0002 Fried:0.0002 Fried:0.0002 Fried:0.0002UTH:0.0002 Pork:0.0002 Pork:0.0002 Pork:0.0002)
30+
(fair:0.0002 Fried:0.0002 Fried:0.0002 Fried:0.0002UTH:0.0002 Pork:0.0002 Pork:0.0002 Pork:0.0002)
31+
Pork Pork Pork Pork Pork Pork Pork Pork Pork Pork Pork Pork Pork 1978 Pork
32+
50: sample 0: A Poem for you! Roses are red, Potatoes are
33+
------
34+
( 2004:0.0003
35+
:0.0011 of:0.0011 of:0.0030 of:0.0026 of:0.0023 of:0.0020 of:0.0035)
36+
( the:0.0276 the:0.0630 the:0.0698 the:0.0771 the:0.0820 the:0.0801 the:0.0840 the:0.0859)
37+
( 2004:0.0002 2004:0.0002 2004:0.0002 of:0.0002 of:0.0003 of:0.0003 of:0.0003 of:0.0003)
38+
( the:0.0256 the:0.0281 the:0.0281 the:0.0298 the:0.0315 the:0.0325 the:0.0344 the:0.0334)
39+
(rael:0.0002rael:0.0002 Rivals:0.0002 Rivals:0.0002 Rivals:0.0002 Rivals:0.0002 obscure:0.0002 obscure:0.0002)
40+
( the:0.0032 the:0.0046 the:0.0055 the:0.0068 the:0.0078 the:0.0087 the:0.0098 the:0.0102)
41+
(UC:0.0002UC:0.0002UC:0.0002UC:0.0002UC:0.0002UC:0.0002 1978:0.0002 1978:0.0002)
42+
( the:0.0192 the:0.0190 the:0.0190 the:0.0189 the:0.0189 the:0.0194 the:0.0200 the:0.0195)
43+
( dismay:0.0002 dismay:0.0002 dismay:0.0002 dismay:0.0002 dismay:0.0002 dismay:0.0002 dismay:0.0002 dismay:0.0002)
44+
( of:0.0020 of:0.0018 of:0.0015 of:0.0014 of:0.0013 of:0.0012 of:0.0011 of:0.0010)
45+
(UC:0.0002UC:0.0002UC:0.0002UC:0.0002UC:0.0002UC:0.0002UC:0.0002UC:0.0002)
46+
( the:0.0115 the:0.0115 the:0.0115 the:0.0119 the:0.0123 the:0.0131 the:0.0139 the:0.0135)
47+
( the:0.0115 the:0.0115 the:0.0115 the:0.0119 the:0.0123 the:0.0131 the:0.0139 the:0.0135)
48+
the
49+
------
50+
(
51+
:0.0011 of:0.0011 of:0.0030 of:0.0026 of:0.0023 of:0.0020 of:0.0035 of:0.0012)
52+
( the:0.0630 the:0.0698 the:0.0771 the:0.0820 the:0.0801 the:0.0840 the:0.0859 the:0.1040)
53+
( 2004:0.0002 2004:0.0002 of:0.0002 of:0.0003 of:0.0003 of:0.0003 of:0.0003 of:0.0003)
54+
( the:0.0281 the:0.0281 the:0.0298 the:0.0315 the:0.0325 the:0.0344 the:0.0334 the:0.0344)
55+
(rael:0.0002 Rivals:0.0002 Rivals:0.0002 Rivals:0.0002 Rivals:0.0002 obscure:0.0002 obscure:0.0002 obscure:0.0002)
56+
( the:0.0046 the:0.0055 the:0.0068 the:0.0078 the:0.0087 the:0.0098 the:0.0102 the:0.0109)
57+
(UC:0.0002UC:0.0002UC:0.0002UC:0.0002UC:0.0002 1978:0.0002 1978:0.0002 1978:0.0002)
58+
( the:0.0190 the:0.0190 the:0.0189 the:0.0189 the:0.0194 the:0.0200 the:0.0195 the:0.0194)
59+
( dismay:0.0002 dismay:0.0002 dismay:0.0002 dismay:0.0002 dismay:0.0002 dismay:0.0002 dismay:0.0002 dismay:0.0002)
60+
( of:0.0018 of:0.0015 of:0.0014 of:0.0013 of:0.0012 of:0.0011 of:0.0010 of:0.0009)
61+
(UC:0.0002UC:0.0002UC:0.0002UC:0.0002UC:0.0002UC:0.0002UC:0.0002UC:0.0002)
62+
( the:0.0115 the:0.0115 the:0.0119 the:0.0123 the:0.0131 the:0.0139 the:0.0135 the:0.0143)
63+
( the:0.0115 the:0.0115 the:0.0119 the:0.0123 the:0.0131 the:0.0139 the:0.0135 the:0.0143)
64+
the the the the the the the the the the the the the the the

‎samples-ben/11-axm-mlpinputmod-main.txt

+180
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
1: sample 0: A Poem for you! Roses are red, Potatoes are
2+
------
3+
(2011:0.0002 Boo:0.0002 Panama:0.0002bang:0.0002Kick:0.0003oys:0.0002 Archie:0.0002 anth:0.0001)
4+
(em:0.0005em:0.0005em:0.0007em:0.0006em:0.0005em:0.0005em:0.0006em:0.0004)
5+
( Hav:0.0002 Hav:0.0002 Hav:0.0002 Hav:0.0002 Hav:0.0002 Hav:0.0002 Hav:0.0002 Hav:0.0002)
6+
(A:0.0002A:0.0002A:0.0002 repe:0.0001 repe:0.0001 repe:0.0001em:0.0001em:0.0001)
7+
(A:0.0002A:0.0002A:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001)
8+
(A:0.0002A:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001)
9+
(A:0.0002 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001)
10+
(A:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001)
11+
(ful:0.0001ful:0.0001ful:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001)
12+
(ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001 ―:0.0001 ―:0.0001)
13+
(ful:0.0002ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001)
14+
(ful:0.0002ful:0.0002ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001)
15+
(ful:0.0002ful:0.0002ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001)
16+
ful
17+
------
18+
( Boo:0.0002 Panama:0.0002bang:0.0002Kick:0.0003oys:0.0002 Archie:0.0002 anth:0.0001 carts:0.0002)
19+
(em:0.0005em:0.0007em:0.0006em:0.0005em:0.0005em:0.0006em:0.0004em:0.0005)
20+
( Hav:0.0002 Hav:0.0002 Hav:0.0002 Hav:0.0002 Hav:0.0002 Hav:0.0002 Hav:0.0002 Dec:0.0001)
21+
(A:0.0002A:0.0002 repe:0.0001 repe:0.0001 repe:0.0001em:0.0001em:0.0001em:0.0001)
22+
(A:0.0002A:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001)
23+
(A:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001)
24+
( ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001)
25+
( ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001)
26+
(ful:0.0001ful:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001 ―:0.0001)
27+
(ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001 ―:0.0001 ―:0.0001 ―:0.0001)
28+
(ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001 ―:0.0001)
29+
(ful:0.0002ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001)
30+
(ful:0.0002ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001ful:0.0001)
31+
fulfulfulfulfulfulfulfulfulfulfulfulfulfulful
32+
50: sample 0: A Poem for you! Roses are red, Potatoes are
33+
------
34+
(.:0.0157 the:0.0723 the:0.0908 the:0.1226 the:0.1172 the:0.0532 the:0.0835 the:0.1045)
35+
(.:0.0557.:0.0598.:0.0635.:0.0737,:0.0640,:0.0747,:0.0728,:0.0815)
36+
(.:0.0776.:0.0762.:0.0796.:0.0815.:0.0781.:0.0811.:0.0801.:0.0811)
37+
(.:0.0645.:0.0645.:0.0659.:0.0684.:0.0649.:0.0654.:0.0664.:0.0674)
38+
(.:0.0581.:0.0583.:0.0581.:0.0596.:0.0574.:0.0596.:0.0588.:0.0601)
39+
(.:0.0532.:0.0522.:0.0537.:0.0552.:0.0532.:0.0532.:0.0544.:0.0540)
40+
(.:0.0486.:0.0493.:0.0505.:0.0508.:0.0500.:0.0503.:0.0513.:0.0510)
41+
(.:0.0469.:0.0461.:0.0474.:0.0479.:0.0469.:0.0471.:0.0481.:0.0479)
42+
(.:0.0435.:0.0442.:0.0442.:0.0459.:0.0452.:0.0454.:0.0449.:0.0461)
43+
(.:0.0417.:0.0425.:0.0422.:0.0442.:0.0432.:0.0435.:0.0430.:0.0442)
44+
(.:0.0398.:0.0405.:0.0415.:0.0422.:0.0413.:0.0415.:0.0413.:0.0422)
45+
(.:0.0391.:0.0386.:0.0396.:0.0403.:0.0396.:0.0396.:0.0405.:0.0403)
46+
(.:0.0391.:0.0386.:0.0396.:0.0403.:0.0396.:0.0396.:0.0405.:0.0403)
47+
.
48+
------
49+
( the:0.0723 the:0.0908 the:0.1226 the:0.1172 the:0.0532 the:0.0835 the:0.1045.:0.0986)
50+
(.:0.0598.:0.0635.:0.0737,:0.0640,:0.0747,:0.0728,:0.0815,:0.1035)
51+
(.:0.0762.:0.0796.:0.0815.:0.0781.:0.0811.:0.0801.:0.0811.:0.0938)
52+
(.:0.0645.:0.0659.:0.0684.:0.0649.:0.0654.:0.0664.:0.0674.:0.0752)
53+
(.:0.0583.:0.0581.:0.0596.:0.0574.:0.0596.:0.0588.:0.0601.:0.0654)
54+
(.:0.0522.:0.0537.:0.0552.:0.0532.:0.0532.:0.0544.:0.0540.:0.0593)
55+
(.:0.0493.:0.0505.:0.0508.:0.0500.:0.0503.:0.0513.:0.0510.:0.0547)
56+
(.:0.0461.:0.0474.:0.0479.:0.0469.:0.0471.:0.0481.:0.0479.:0.0515)
57+
(.:0.0442.:0.0442.:0.0459.:0.0452.:0.0454.:0.0449.:0.0461.:0.0483)
58+
(.:0.0425.:0.0422.:0.0442.:0.0432.:0.0435.:0.0430.:0.0442.:0.0464)
59+
(.:0.0405.:0.0415.:0.0422.:0.0413.:0.0415.:0.0413.:0.0422.:0.0444)
60+
(.:0.0386.:0.0396.:0.0403.:0.0396.:0.0396.:0.0405.:0.0403.:0.0425)
61+
(.:0.0386.:0.0396.:0.0403.:0.0396.:0.0396.:0.0405.:0.0403.:0.0425)
62+
...............
63+
550: sample 0: A Poem for you! Roses are red, Potatoes are
64+
------
65+
(for:0.0018 help:0.0129-:0.0178 such:0.0098
66+
:0.0261In:0.0026 most:0.0136
67+
:0.1650)
68+
(.:0.2520,:0.0688,:0.1729 and:0.1201,:0.1602,:0.2070,:0.0918
69+
:0.1963)
70+
(.:0.2480 the:0.0996,:0.2354 and:0.1592,:0.1758,:0.2676,:0.0923
71+
:0.2773)
72+
(.:0.2754 the:0.1245,:0.2812 and:0.2080,:0.1455,:0.3262 the:0.1221
73+
:0.3184)
74+
(.:0.2871 the:0.1279,:0.3008 and:0.2383,:0.0879,:0.3594 the:0.1328
75+
:0.3301)
76+
(.:0.2773 the:0.1216,:0.2852 and:0.2461,:0.0383,:0.3984 the:0.1270
77+
:0.3125)
78+
(.:0.2734 the:0.1123,:0.2480 and:0.2236om:0.0439,:0.4199 the:0.1113
79+
:0.3008)
80+
(.:0.2480 the:0.0977,:0.2061 and:0.1895om:0.0503,:0.4180 the:0.0923
81+
:0.2637)
82+
(.:0.2197 a:0.0781,:0.1582 and:0.1572om:0.0530,:0.3926 the:0.0713
83+
:0.2236)
84+
(.:0.1934 a:0.0625,:0.1123 and:0.1240om:0.0535,:0.3301 the:0.0530
85+
:0.1943)
86+
(.:0.1504 a:0.0483,:0.0820 you:0.1187om:0.0530,:0.2578 the:0.0408
87+
:0.1719)
88+
(.:0.1216 a:0.0400,:0.0630 you:0.1138om:0.0535,:0.2031 the:0.0344
89+
:0.1650)
90+
(.:0.1216 a:0.0400,:0.0630 you:0.1138om:0.0535,:0.2031 the:0.0344
91+
:0.1650)
92+
93+
94+
------
95+
( help:0.0129-:0.0178 such:0.0098
96+
:0.0261In:0.0026 most:0.0136
97+
:0.1650The:0.0791)
98+
(,:0.0688,:0.1729 and:0.1201,:0.1602,:0.2070,:0.0918
99+
:0.1963.:0.2539)
100+
( the:0.0996,:0.2354 and:0.1592,:0.1758,:0.2676,:0.0923
101+
:0.2773.:0.2354)
102+
( the:0.1245,:0.2812 and:0.2080,:0.1455,:0.3262 the:0.1221
103+
:0.3184.:0.2891)
104+
( the:0.1279,:0.3008 and:0.2383,:0.0879,:0.3594 the:0.1328
105+
:0.3301.:0.2617)
106+
( the:0.1216,:0.2852 and:0.2461,:0.0383,:0.3984 the:0.1270
107+
:0.3125-:0.2295)
108+
( the:0.1123,:0.2480 and:0.2236om:0.0439,:0.4199 the:0.1113
109+
:0.3008-:0.2334)
110+
( the:0.0977,:0.2061 and:0.1895om:0.0503,:0.4180 the:0.0923
111+
:0.2637-:0.1982)
112+
( a:0.0781,:0.1582 and:0.1572om:0.0530,:0.3926 the:0.0713
113+
:0.2236-:0.1650)
114+
( a:0.0625,:0.1123 and:0.1240om:0.0535,:0.3301 the:0.0530
115+
:0.1943-:0.1328)
116+
( a:0.0483,:0.0820 you:0.1187om:0.0530,:0.2578 the:0.0408
117+
:0.1719-:0.1128)
118+
( a:0.0400,:0.0630 you:0.1138om:0.0535,:0.2031 the:0.0344
119+
:0.1650-:0.0957)
120+
( a:0.0400,:0.0630 you:0.1138om:0.0535,:0.2031 the:0.0344
121+
:0.1650-:0.0957)
122+
- The most popular in the most popular.
123+
- The popular is popular
124+
1050: sample 0: A Poem for you! Roses are red, Potatoes are
125+
------
126+
(http:0.0017 help:0.0061st:0.0056 be:0.0064 the:0.0125 avoid:0.0023 be:0.0060
127+
:0.0625)
128+
(,:0.2451,:0.2246,:0.2969 and:0.2432,:0.3301,:0.3809,:0.3164,:0.2695)
129+
(,:0.2334,:0.1895,:0.2891 and:0.2402,:0.3242,:0.4023,:0.2617
130+
:0.2266)
131+
(,:0.2422,:0.1865,:0.3086 and:0.3203,:0.3281,:0.4180,:0.2598
132+
:0.2617)
133+
(,:0.2520,:0.1748,:0.3223 and:0.3828,:0.3262,:0.4453,:0.2256
134+
:0.3633)
135+
(,:0.2734,:0.1631,:0.3301 and:0.4727,:0.3105,:0.4746,:0.1943
136+
:0.4766)
137+
(,:0.2891 the:0.1553 and:0.3379 and:0.5117,:0.2676,:0.5078 the:0.1738
138+
:0.5469)
139+
(,:0.2930 the:0.1631 and:0.3184 and:0.4805,:0.1807,:0.5820 the:0.1904
140+
:0.5938)
141+
(,:0.2773 the:0.1455 and:0.2832 and:0.3555om:0.2344,:0.6445 the:0.1445
142+
:0.5312)
143+
(,:0.2393 the:0.0908 and:0.2051 and:0.1699om:0.2969,:0.7148 the:0.0884
144+
:0.3438)
145+
(,:0.1855 a:0.0459,:0.1191 you:0.1299om:0.2910,:0.6992 a:0.0530
146+
:0.2012)
147+
(,:0.1328 a:0.0271,:0.0640 you:0.1074om:0.2598,:0.6055 a:0.0354
148+
:0.1455)
149+
(,:0.1328 a:0.0271,:0.0640 you:0.1074om:0.2598,:0.6055 a:0.0354
150+
:0.1455)
151+
152+
153+
------
154+
( help:0.0061st:0.0056 be:0.0064 the:0.0125 avoid:0.0023 be:0.0060
155+
:0.0625The:0.0481)
156+
(,:0.2246,:0.2969 and:0.2432,:0.3301,:0.3809,:0.3164,:0.2695.:0.2871)
157+
(,:0.1895,:0.2891 and:0.2402,:0.3242,:0.4023,:0.2617
158+
:0.2266
159+
:0.2363)
160+
(,:0.1865,:0.3086 and:0.3203,:0.3281,:0.4180,:0.2598
161+
:0.2617.:0.3262)
162+
(,:0.1748,:0.3223 and:0.3828,:0.3262,:0.4453,:0.2256
163+
:0.3633.:0.3965)
164+
(,:0.1631,:0.3301 and:0.4727,:0.3105,:0.4746,:0.1943
165+
:0.4766.:0.4707)
166+
( the:0.1553 and:0.3379 and:0.5117,:0.2676,:0.5078 the:0.1738
167+
:0.5469.:0.3906)
168+
( the:0.1631 and:0.3184 and:0.4805,:0.1807,:0.5820 the:0.1904
169+
:0.5938-:0.4453)
170+
( the:0.1455 and:0.2832 and:0.3555om:0.2344,:0.6445 the:0.1445
171+
:0.5312-:0.4551)
172+
( the:0.0908 and:0.2051 and:0.1699om:0.2969,:0.7148 the:0.0884
173+
:0.3438-:0.3320)
174+
( a:0.0459,:0.1191 you:0.1299om:0.2910,:0.6992 a:0.0530
175+
:0.2012-:0.2471)
176+
( a:0.0271,:0.0640 you:0.1074om:0.2598,:0.6055 a:0.0354
177+
:0.1455-:0.1670)
178+
( a:0.0271,:0.0640 you:0.1074om:0.2598,:0.6055 a:0.0354
179+
:0.1455-:0.1670)
180+
- The Poem Poem Poem Poem Poem Poem Po

0 commit comments

Comments
 (0)