"They say that a good thing never lasts
 And that it has to fall
 Those are the people that did not
 Amount too much at all"
/Madonna - 'Give it 2 me'/

"- Do you believe in it?
 - I know it Fay."
/"After Dark, My Sweet", movie 1990/

Welcome to Sanmayce "downloads" page.

Table Note1: Corpus 'Gamera' statistics obtained with Leprechaun_x-leton revision 15.
Table Note2: Corpus 'Gamera' ripped as one 34,273,505,280 bytes TAR file consisted of 562,504 TXT files.
Table Note3: In Leprechaun_x-leton revision 15FIXFIX two bugs were fixed, downloadable at bottom of section 2, the entire table must be remade - the x-grams number is significantly bigger. The remake will take a few weeks maybe - for example (on laptop equipped with HDD and Merom 2166MHz and [1758MB RAM used]) it takes 64passes*1800s=~32hours (the exact stats: Total time: 120382 second(s) or Total performance: 23,365P/s i.e. phrases per second) to rip all the 879,557,846 distinct 4-grams.

x-grams Order     x-grams Total Number   x-grams Distinct Number   x-grams Distinct Size   Total memory needed for one pass 1-grams 4,589,933,215 9,181,275 207,307,606 bytes 720,042KB 2-grams 3,835,376,293 124,669,942 3,267,897,913 bytes 10,746,363KB 3-grams 2,674,920,107 360,981,708 10,990,406,362 bytes 30,903,872KB 4-grams 2,812,845,037 879,557,846 30,931,053,072 bytes 87,861,963KB 5-grams 1,587,143,109 726,496,853 28,676,806,441 bytes 82,888,636KB 6-grams 1,197,407,768 652,099,162 28,994,002,731 bytes 83,523,982KB 7-grams 889,495,472 524,279,771 26,052,174,976 bytes 74,669,323KB 8-grams 1,439,078,375 812,576,024 45,963,360,220 bytes 128,842,779KB 9-grams 465,990,400 296,741,735 17,911,340,011 bytes 51,404,184KB

Rip Note1: All these x-grams, what is their purpose? Simply to make phrase-checking (with ranking) possible. The first English 4-gram phrase-checker named Graphein (revision 2, powered by 800,000,000+ Gamera corpus 4-grams) is about to emerge anytime soon. This 14~GB package will allow (in two steps: copying the TXT file(s) into a specific folder and executing a desktop shortcut) auto-loading into NOTEPAD the resultant TXT file containing familiar and unfamiliar (to Gamera corpus) 4-grams. An example follows in form of a PDF booklet (26 A4 pages):
'Crisis' Does NOT Equal 'Danger' Plus 'Opportunity' by Victor H. Mair 2+4-grammed.pdf.
Also a quick look at results for *having_been* pattern being searched among the 812,576,024 8-grams:
Kazuya_Gamera_8-grams_requests_a_small_part_'HAVING_BEEN'.log
Rip Note2: In next Leprechaun_x-leton revision 16 (respectively Graphein revision 3) dump of all slots (including b-trees) will be added thus enabling fast linear phrase-checking (against a given corpus) which is IOPS ONLY bound i.e. latency of the drive will be the only bottleneck (not CPU and physical RAM dependent).
Rip Note3: The following b-tree heights show (roughly) how many IOPS (seeks i.e. repositionings) are needed (when virtual memory 'Z/z' options is used) in order to find an x-gram:
           Order 1:
           Highest Tree not counting ROOT Level i.e. CORONA levels(littler THE BETTER): 2
           Order 2:
           Highest Tree not counting ROOT Level i.e. CORONA levels(littler THE BETTER): 3
           Order 3:
           Highest Tree not counting ROOT Level i.e. CORONA levels(littler THE BETTER): 4
           Order 4:
           Highest Tree not counting ROOT Level i.e. CORONA levels(littler THE BETTER): 5
           Order 5:
           Highest Tree not counting ROOT Level i.e. CORONA levels(littler THE BETTER): 5
           Order 6:
           Highest Tree not counting ROOT Level i.e. CORONA levels(littler THE BETTER): 4
           Order 7:
           Highest Tree not counting ROOT Level i.e. CORONA levels(littler THE BETTER): 4
           Order 8:
           Highest Tree not counting ROOT Level i.e. CORONA levels(littler THE BETTER): 5
           Order 9:
           Highest Tree not counting ROOT Level i.e. CORONA levels(littler THE BETTER): 4
When using slowest drives (that is HDDs) it takes maximum 5 IOPS (packet 512-bytes) to find any x-gram i.e. some 5x10milliseconds which gives 20+ x-grams per second rate for all orders.
Rip Note4: Most suitable (virtual memory 'Z/z' options) drive for heavy loads (RANDOM accesses mostly): the Fusion-io ioDrive SSD Series SLC/MLC PCIe: ioDrive Duo 320GB allowing 238,000 Mixed (75/25 r/w) IOPS (512 Byte), ioDrive Octal 5.12TB offers 729,000 75/25 Mix IOPS (512 B)!!! Not impressed? Compare these IOPS with the super-poor 100 IOPS which fastest HDD can deliver.



Rip Note5: Currently corpus 'Gamera' holds 1- billion x-grams for each order - in order an x-gram wordlist to get sorted it takes roughly 1,000,000,000*8bytes (one 64bit pointer for each x-gram) physical RAM or in short: 8GB.
Rip Note6: Console log showing how 7-grams were being ripped from corpus 'Gamera': gamera_7-grams_log.txt shows 64 passes executed - for one pass needed memory is 74,669,323KB, pshaw!





Leprechaun_x-leton:
A free open-source and demonically fast phrase ripper.


The long-awaited (by me, he-he) Super-Leprechaun, pure 64bit addressing both as 32bit and 64bit code with adjustable hash table and MULTI-PASS MODE.
Powerful and fast: unlimited (in practice) vocabulary size, rips at 1+ million phrases per second rate. 
Creates x-gram unique phrases, allowing a vocabulary consisted of x (1..10) COLLOCATIVE words to be analyzed.

Slow-Rip (in one pass) for BILLIONS (on SSD) of x-grams. An external B-trees technique used, currently 16,777,216 B-Trees of order 3.
For 5 billion 4-grams it needs less than 633GB (136bytes*5,000,000,000) 30microseconds seek-time external memory, though!
The advent of 640GB physical/internal RAM is commenced, having such a computer (30nanoseconds seek-time internal memory) will unleash the beast.
Yes, we are in limbo (that is, nowadays 640MB main memory) i.e. in-between the oldadays 640KB and the newadays 640GB.
The nifty thing about MPM (Multi-Pass Mode): Allows to rip the whole electronic English on a simple PC (with HDD and 4GB RAM).

...
0,000,003 worried_about_whether_the
0,000,001 worried_about_their_small
0,000,001 worried_about_the_rest
0,000,001 worried_about_their_antipathy
0,000,001 worried_about_thoughts_circulating
0,000,001 worried_about_the_destination
0,000,004 worried_about_the_man
0,000,001 worried_about_god_s
0,000,001 worried_about_others_if
0,000,001 worried_about_rolls_royces
0,000,001 worried_about_anything_the
...
0,000,013 which_has_become_a
0,000,001 which_gives_life_the
0,000,001 which_for_centuries_they
0,000,002 which_formally_he_is
0,000,001 which_extends_both_to
0,000,001 which_everything_would_be
0,000,001 which_everything_false_is
0,000,001 which_everything_was_destroyed
0,000,001 which_exploited_the_whole
0,000,001 which_fell_on_the
0,000,001 which_dogo_is_missing
0,000,001 which_emperor_was_born
0,000,003 which_do_you_want
0,000,001 which_does_contain_the
0,000,002 which_does_not_grow
0,000,003 which_develops_into_awareness
0,000,001 which_dissolves_all_paradoxes
...
0,000,001 continuously_brag_that_you
0,000,001 continuously_cripple_the_research
0,000,001 continuously_ask_for_attention
0,000,001 continuously_available_to_the
0,000,001 continuous_showering_of_blessings
0,000,001 continuous_trouble_during_my
0,000,001 continuously_and_you_will
0,000,001 continuously_being_pulled_downwards
0,000,002 continuously_escaping_from_yourself
0,000,001 continuous_motion_won_t
0,000,001 continuous_oxygen_is_needed
0,000,001 continuous_fear_that_i
0,000,001 continuous_hunger_for_war
0,000,001 continuous_conflict_within_you
...
0,000,001 is_as_strongly_convinced
0,000,001 is_as_mindless_as
0,000,002 is_as_nonexistential_as
0,000,001 is_as_powerless_as
0,000,001 is_as_imperfect_as
0,000,002 is_as_inconceivable_as
0,000,003 is_as_if_for
0,000,001 is_as_distant_as
0,000,002 is_as_enlightened_as
0,000,002 is_as_formless_as
...
0,000,001 indignantly_at_this_suggestion
0,000,001 indira_gandhi_both_were
0,000,001 indira_s_congress_party
0,000,001 indirect_way_of_indicating
0,000,002 indistinguishable_from_the_patriarchs
0,000,004 individual_is_a_reality
0,000,001 individuality_of_an_enlightened
0,000,002 infinite_energy_within_you
0,000,001 indifferent_where_ordinarily_you
0,000,002 indigestion_was_your_real
0,000,001 indicative_of_a_fear
0,000,001 indicative_of_a_healthy
...
All-in-all: IT is capable of ripping the whole electronic English at once while counting the repetitions of each x-gram.
For reference: http://www.sanmayce.com/Downloads/index.html#Leprechaun



http://soundcloud.com/search?q[fulltext]=sam+obernik

              

              

              

              

              

              

              

              

              

Copyleft Sanmayce 2012-Feb-12