Overlapper-Blender: a 32bit console tool revision 1+ designed to sort, to remove duplicated lines and to clash two wordlists onto one another. A short overview follows: The package (Overlapper-Blender_r1+.zip) is 31,842,792 bytes bytes and is downloadable at http://www.sanmayce.com/Downloads/Overlapper-Blender_r1+.zip The console log below shows what is its contents and how to use. Enjoy! D:\_KAZE_new-stuff\Overlapper-Blender_r1+>dir Volume in drive D is H320_Vol5 Volume Serial Number is 0CB3-C881 Directory of D:\_KAZE_new-stuff\Overlapper-Blender_r1+ 03/02/2011 09:48 PM . 03/02/2011 09:48 PM .. 03/02/2011 09:44 PM 30 1.txt 03/02/2011 09:44 PM 14 2.txt 03/02/2011 09:44 PM 43,930 Overlapper-Blender_r1+.c 03/02/2011 09:44 PM 66,048 Overlapper-Blender_r1+.exe 03/02/2011 09:44 PM 57,389,250 _Agatha Christie_Texts.txt 03/02/2011 09:44 PM 27,024,497 _Sherlock Holmes_Texts.txt 03/02/2011 09:44 PM 20,326,151 _Sunnah and Hadith and Qur'an.txt 03/02/2011 09:44 PM 17,183,313 _The_Holy_Bible_4-versions.txt 8 File(s) 122,033,233 bytes 2 Dir(s) 1,211,654,144 bytes free D:\_KAZE_new-stuff\Overlapper-Blender_r1+>"Overlapper-Blender_r1+.exe" Overlapper-Blender r.1+, written by Kaze. Usage: Overlapper-Blender wordlistfile1 wordlistfile2 Note1: wordlistfile1's lines encountered in wordlistfile2's lines go to 'Overlapped.txt' file. Note2: wordlistfile1's lines blended (no repetitions allowed) with wordlistfile2's lines go to 'Blended.txt' file. Note3: wordlistfile1's lines not encountered in wordlistfile2 go to 'Unfamiliar.txt' file. Note1a: (wordlistfile1 logical_AND wordlistfile2) = 'Overlapped.txt' file. Note2a: (wordlistfile1 logical_OR wordlistfile2) = 'Blended.txt' file. Note3a: wordlistfile1 - (wordlistfile1 logical_AND wordlistfile2) = 'Unfamiliar.txt' file. Note4: If you need only one file to be sorted-and-deduplicated then use this: D:\_KAZE_new-stuff\Overlapper-Blender_r1+>copy con Empty ^Z 1 file(s) copied. D:\_KAZE_new-stuff\Overlapper-Blender_r1+>dir Empty 03/02/2011 08:59 PM 0 Empty D:\_KAZE_new-stuff\Overlapper-Blender_r1+>Overlapper-Blender wordlistfile1 Empty Note5: Current pool(due to 32bit address limitation) for incoming strings is 1GB. D:\_KAZE_new-stuff\Overlapper-Blender_r1+>copy con wordlist1 dumbo jumbo jumbo Dumbo ^Z 1 file(s) copied. D:\_KAZE_new-stuff\Overlapper-Blender_r1+>copy con Empty ^Z 1 file(s) copied. D:\_KAZE_new-stuff\Overlapper-Blender_r1+>"Overlapper-Blender_r1+.exe" wordlist1 Empty Overlapper-Blender r.1+, written by Kaze. Size of 1st input file: 28 Size of 2nd input file: 0 Allocating 1024MB ... Lines in 1st input file: 4 Lines in 2nd input file: 0 Allocated memory for pointers-to-words in MB: 1 Allocated memory for pointers-to-words in MB: 1 Sorting 4 Pointers ... Deduplicating duplicates and dumping all into 'Blended.txt' ... Dumping deduplicated duplicates into 'Overlapped.txt' ... Dumping all-from-first-file except deduplicated duplicates into 'Unfamiliar.txt' ... Blended lines, i.e. combined lines from both files: 3 Overlapped lines, i.e. lines common for both files: 0 Unfamiliar lines, i.e. lines from 1st file not encountered in 2nd file: 3 D:\_KAZE_new-stuff\Overlapper-Blender_r1+>dir Volume in drive D is H320_Vol5 Volume Serial Number is 0CB3-C881 Directory of D:\_KAZE_new-stuff\Overlapper-Blender_r1+ 03/02/2011 09:49 PM . 03/02/2011 09:49 PM .. 03/02/2011 09:44 PM 30 1.txt 03/02/2011 09:44 PM 14 2.txt 03/02/2011 09:49 PM 21 Blended.txt 03/02/2011 09:49 PM 0 Empty 03/02/2011 09:49 PM 0 Overlapped.txt 03/02/2011 09:44 PM 43,930 Overlapper-Blender_r1+.c 03/02/2011 09:44 PM 66,048 Overlapper-Blender_r1+.exe 03/02/2011 09:49 PM 21 Unfamiliar.txt 03/02/2011 09:49 PM 28 wordlist1 03/02/2011 09:44 PM 57,389,250 _Agatha Christie_Texts.txt 03/02/2011 09:44 PM 27,024,497 _Sherlock Holmes_Texts.txt 03/02/2011 09:44 PM 20,326,151 _Sunnah and Hadith and Qur'an.txt 03/02/2011 09:44 PM 17,183,313 _The_Holy_Bible_4-versions.txt 13 File(s) 122,033,303 bytes 2 Dir(s) 1,211,654,144 bytes free D:\_KAZE_new-stuff\Overlapper-Blender_r1+>type Blended.txt Dumbo dumbo jumbo D:\_KAZE_new-stuff\Overlapper-Blender_r1+>del *./p D:\_KAZE_new-stuff\Overlapper-Blender_r1+\Empty, Delete (Y/N)? y D:\_KAZE_new-stuff\Overlapper-Blender_r1+\wordlist1, Delete (Y/N)? y D:\_KAZE_new-stuff\Overlapper-Blender_r1+>del ?.txt/p D:\_KAZE_new-stuff\Overlapper-Blender_r1+\1.txt, Delete (Y/N)? y D:\_KAZE_new-stuff\Overlapper-Blender_r1+\2.txt, Delete (Y/N)? y D:\_KAZE_new-stuff\Overlapper-Blender_r1+>dir Volume in drive D is H320_Vol5 Volume Serial Number is 0CB3-C881 Directory of D:\_KAZE_new-stuff\Overlapper-Blender_r1+ 03/02/2011 09:51 PM . 03/02/2011 09:51 PM .. 03/02/2011 09:49 PM 21 Blended.txt 03/02/2011 09:49 PM 0 Overlapped.txt 03/02/2011 09:44 PM 43,930 Overlapper-Blender_r1+.c 03/02/2011 09:44 PM 66,048 Overlapper-Blender_r1+.exe 03/02/2011 09:49 PM 21 Unfamiliar.txt 03/02/2011 09:44 PM 57,389,250 _Agatha Christie_Texts.txt 03/02/2011 09:44 PM 27,024,497 _Sherlock Holmes_Texts.txt 03/02/2011 09:44 PM 20,326,151 _Sunnah and Hadith and Qur'an.txt 03/02/2011 09:44 PM 17,183,313 _The_Holy_Bible_4-versions.txt 9 File(s) 122,033,231 bytes 2 Dir(s) 1,211,654,144 bytes free D:\_KAZE_new-stuff\Overlapper-Blender_r1+>"Overlapper-Blender_r1+.exe" "_Agatha Christie_Texts.txt" "_Sherlock Holmes_Texts.txt" Overlapper-Blender r.1+, written by Kaze. Size of 1st input file: 57389250 Size of 2nd input file: 27024497 Allocating 1024MB ... Lines in 1st input file: 2615513 Lines in 2nd input file: 1233227 Allocated memory for pointers-to-words in MB: 15 Allocated memory for pointers-to-words in MB: 10 Sorting 3848740 Pointers ... Deduplicating duplicates and dumping all into 'Blended.txt' ... Dumping deduplicated duplicates into 'Overlapped.txt' ... Dumping all-from-first-file except deduplicated duplicates into 'Unfamiliar.txt' ... Blended lines, i.e. combined lines from both files: 3746539 Overlapped lines, i.e. lines common for both files: 102201 Unfamiliar lines, i.e. lines from 1st file not encountered in 2nd file: 2513312 D:\_KAZE_new-stuff\Overlapper-Blender_r1+>dir Volume in drive D is H320_Vol5 Volume Serial Number is 0CB3-C881 Directory of D:\_KAZE_new-stuff\Overlapper-Blender_r1+ 03/02/2011 09:51 PM . 03/02/2011 09:51 PM .. 03/02/2011 09:51 PM 82,470,333 Blended.txt 03/02/2011 09:51 PM 1,943,414 Overlapped.txt 03/02/2011 09:44 PM 43,930 Overlapper-Blender_r1+.c 03/02/2011 09:44 PM 66,048 Overlapper-Blender_r1+.exe 03/02/2011 09:51 PM 55,445,836 Unfamiliar.txt 03/02/2011 09:44 PM 57,389,250 _Agatha Christie_Texts.txt 03/02/2011 09:44 PM 27,024,497 _Sherlock Holmes_Texts.txt 03/02/2011 09:44 PM 20,326,151 _Sunnah and Hadith and Qur'an.txt 03/02/2011 09:44 PM 17,183,313 _The_Holy_Bible_4-versions.txt 9 File(s) 261,892,772 bytes 2 Dir(s) 1,071,788,032 bytes free D:\_KAZE_new-stuff\Overlapper-Blender_r1+>ren Blended.txt Blended_Agatha_VS_Sherlock.txt D:\_KAZE_new-stuff\Overlapper-Blender_r1+>ren Unfamiliar.txt Unfamiliar_Agatha_VS_Sherlock.txt D:\_KAZE_new-stuff\Overlapper-Blender_r1+>ren Overlapped.txt Overlapped_Agatha_VS_Sherlock.txt D:\_KAZE_new-stuff\Overlapper-Blender_r1+>"Overlapper-Blender_r1+.exe" "_Sunnah and Hadith and Qur'an.txt" _The_Holy_Bible_4-versions.txt Overlapper-Blender r.1+, written by Kaze. Size of 1st input file: 20326151 Size of 2nd input file: 17183313 Allocating 1024MB ... Lines in 1st input file: 936195 Lines in 2nd input file: 795822 Allocated memory for pointers-to-words in MB: 7 Allocated memory for pointers-to-words in MB: 4 Sorting 1732017 Pointers ... Deduplicating duplicates and dumping all into 'Blended.txt' ... Dumping deduplicated duplicates into 'Overlapped.txt' ... Dumping all-from-first-file except deduplicated duplicates into 'Unfamiliar.txt' ... Blended lines, i.e. combined lines from both files: 1702979 Overlapped lines, i.e. lines common for both files: 29038 Unfamiliar lines, i.e. lines from 1st file not encountered in 2nd file: 907157 D:\_KAZE_new-stuff\Overlapper-Blender_r1+>dir Volume in drive D is H320_Vol5 Volume Serial Number is 0CB3-C881 Directory of D:\_KAZE_new-stuff\Overlapper-Blender_r1+ 03/02/2011 09:55 PM . 03/02/2011 09:55 PM .. 03/02/2011 09:55 PM 36,965,004 Blended.txt 03/02/2011 09:51 PM 82,470,333 Blended_Agatha_VS_Sherlock.txt 03/02/2011 09:55 PM 544,460 Overlapped.txt 03/02/2011 09:51 PM 1,943,414 Overlapped_Agatha_VS_Sherlock.txt 03/02/2011 09:44 PM 43,930 Overlapper-Blender_r1+.c 03/02/2011 09:44 PM 66,048 Overlapper-Blender_r1+.exe 03/02/2011 09:55 PM 19,781,691 Unfamiliar.txt 03/02/2011 09:51 PM 55,445,836 Unfamiliar_Agatha_VS_Sherlock.txt 03/02/2011 09:44 PM 57,389,250 _Agatha Christie_Texts.txt 03/02/2011 09:44 PM 27,024,497 _Sherlock Holmes_Texts.txt 03/02/2011 09:44 PM 20,326,151 _Sunnah and Hadith and Qur'an.txt 03/02/2011 09:44 PM 17,183,313 _The_Holy_Bible_4-versions.txt 12 File(s) 319,183,927 bytes 2 Dir(s) 1,014,493,184 bytes free D:\_KAZE_new-stuff\Overlapper-Blender_r1+>ren Blended.txt Blended_Islam_VS_Bible.txt D:\_KAZE_new-stuff\Overlapper-Blender_r1+>ren Unfamiliar.txt Unfamiliar_Islam_VS_Bible.txt D:\_KAZE_new-stuff\Overlapper-Blender_r1+>ren Overlapped.txt Overlapped_Islam_VS_Bible.txt D:\_KAZE_new-stuff\Overlapper-Blender_r1+>dir Volume in drive D is H320_Vol5 Volume Serial Number is 0CB3-C881 Directory of D:\_KAZE_new-stuff\Overlapper-Blender_r1+ 03/02/2011 09:57 PM . 03/02/2011 09:57 PM .. 03/02/2011 09:51 PM 82,470,333 Blended_Agatha_VS_Sherlock.txt 03/02/2011 09:55 PM 36,965,004 Blended_Islam_VS_Bible.txt 03/02/2011 09:51 PM 1,943,414 Overlapped_Agatha_VS_Sherlock.txt 03/02/2011 09:55 PM 544,460 Overlapped_Islam_VS_Bible.txt 03/02/2011 09:44 PM 43,930 Overlapper-Blender_r1+.c 03/02/2011 09:44 PM 66,048 Overlapper-Blender_r1+.exe 03/02/2011 09:51 PM 55,445,836 Unfamiliar_Agatha_VS_Sherlock.txt 03/02/2011 09:55 PM 19,781,691 Unfamiliar_Islam_VS_Bible.txt 03/02/2011 09:44 PM 57,389,250 _Agatha Christie_Texts.txt 03/02/2011 09:44 PM 27,024,497 _Sherlock Holmes_Texts.txt 03/02/2011 09:44 PM 20,326,151 _Sunnah and Hadith and Qur'an.txt 03/02/2011 09:44 PM 17,183,313 _The_Holy_Bible_4-versions.txt 12 File(s) 319,183,927 bytes 2 Dir(s) 1,014,493,184 bytes free D:\_KAZE_new-stuff\Overlapper-Blender_r1+>"Overlapper-Blender_r1+.exe" Overlapped_Agatha_VS_Sherlock.txt Overlapped_Islam_VS_Bible.txt Overlapper-Blender r.1+, written by Kaze. Size of 1st input file: 1943414 Size of 2nd input file: 544460 Allocating 1024MB ... Lines in 1st input file: 102201 Lines in 2nd input file: 29038 Allocated memory for pointers-to-words in MB: 1 Allocated memory for pointers-to-words in MB: 1 Sorting 131239 Pointers ... Deduplicating duplicates and dumping all into 'Blended.txt' ... Dumping deduplicated duplicates into 'Overlapped.txt' ... Dumping all-from-first-file except deduplicated duplicates into 'Unfamiliar.txt' ... Blended lines, i.e. combined lines from both files: 125299 Overlapped lines, i.e. lines common for both files: 5940 Unfamiliar lines, i.e. lines from 1st file not encountered in 2nd file: 96261 D:\_KAZE_new-stuff\Overlapper-Blender_r1+>dir Volume in drive D is H320_Vol5 Volume Serial Number is 0CB3-C881 Directory of D:\_KAZE_new-stuff\Overlapper-Blender_r1+ 03/02/2011 09:58 PM . 03/02/2011 09:58 PM .. 03/02/2011 09:58 PM 2,383,290 Blended.txt 03/02/2011 09:51 PM 82,470,333 Blended_Agatha_VS_Sherlock.txt 03/02/2011 09:55 PM 36,965,004 Blended_Islam_VS_Bible.txt 03/02/2011 09:58 PM 104,584 Overlapped.txt 03/02/2011 09:51 PM 1,943,414 Overlapped_Agatha_VS_Sherlock.txt 03/02/2011 09:55 PM 544,460 Overlapped_Islam_VS_Bible.txt 03/02/2011 09:44 PM 43,930 Overlapper-Blender_r1+.c 03/02/2011 09:44 PM 66,048 Overlapper-Blender_r1+.exe 03/02/2011 09:58 PM 1,838,830 Unfamiliar.txt 03/02/2011 09:51 PM 55,445,836 Unfamiliar_Agatha_VS_Sherlock.txt 03/02/2011 09:55 PM 19,781,691 Unfamiliar_Islam_VS_Bible.txt 03/02/2011 09:44 PM 57,389,250 _Agatha Christie_Texts.txt 03/02/2011 09:44 PM 27,024,497 _Sherlock Holmes_Texts.txt 03/02/2011 09:44 PM 20,326,151 _Sunnah and Hadith and Qur'an.txt 03/02/2011 09:44 PM 17,183,313 _The_Holy_Bible_4-versions.txt 15 File(s) 323,510,631 bytes 2 Dir(s) 1,010,163,712 bytes free D:\_KAZE_new-stuff\Overlapper-Blender_r1+>ren Blended.txt Blended_Overlapped_Agatha_VS_Sherlock_Overlapped_Islam_VS_Bible.txt D:\_KAZE_new-stuff\Overlapper-Blender_r1+>ren Unfamiliar.txt Unfamiliar_Overlapped_Agatha_VS_Sherlock_Overlapped_Islam_VS_Bible.txt D:\_KAZE_new-stuff\Overlapper-Blender_r1+>ren Overlapped.txt Overlapped_Overlapped_Agatha_VS_Sherlock_Overlapped_Islam_VS_Bible.txt D:\_KAZE_new-stuff\Overlapper-Blender_r1+>dir *overlapped*overlapped* Volume in drive D is H320_Vol5 Volume Serial Number is 0CB3-C881 Directory of D:\_KAZE_new-stuff\Overlapper-Blender_r1+ 03/02/2011 09:58 PM 2,383,290 Blended_Overlapped_Agatha_VS_Sherlock_Overlapped_Islam_VS_Bible.txt 03/02/2011 09:58 PM 104,584 Overlapped_Overlapped_Agatha_VS_Sherlock_Overlapped_Islam_VS_Bible.txt 03/02/2011 09:58 PM 1,838,830 Unfamiliar_Overlapped_Agatha_VS_Sherlock_Overlapped_Islam_VS_Bible.txt 3 File(s) 4,326,704 bytes 0 Dir(s) 1,010,159,616 bytes free D:\_KAZE_new-stuff\Overlapper-Blender_r1+>type Overlapped_Overlapped_Agatha_VS_Sherlock_Overlapped_Islam_VS_Bible.txt a_breath_of_the a_change_in_the a_child_in_the a_child_of_the a_copy_of_the a_corner_of_the a_day_or_two a_day_when_the a_description_of_the a_drop_of_water a_fair_amount_of a_few_of_the a_few_of_them a_fire_in_the a_flock_of_sheep a_friend_of_his a_friend_of_mine a_friend_of_the a_good_piece_of a_great_amount_of a_great_number_of a_hole_in_the a_horse_in_the a_house_in_the a_hundred_and_fifty a_knowledge_of_the a_large_part_of a_letter_from_the a_letter_to_him a_letter_to_the a_letter_which_he a_little_from_the a_loaf_of_bread a_long_time_after a_long_time_in a_long_time_to a_loud_voice_and a_man_and_his a_man_comes_to a_man_does_not a_man_has_no a_man_in_his a_man_in_the a_man_of_evil a_man_of_high a_man_of_his a_man_of_the a_man_of_very a_man_of_your a_man_on_the a_man_or_woman a_man_s_mind a_man_to_whom a_man_who_can a_man_who_did a_man_who_had a_man_who_has a_man_who_is a_man_who_was a_man_whom_he a_man_whom_i a_man_whose_name a_man_with_a a_meeting_of_the a_member_of_the a_part_of_his a_part_of_it a_part_of_the a_part_of_them a_piece_of_the a_piece_of_wood a_place_for_the a_place_in_the a_place_where_there a_portion_of_the a_quarter_of_a a_rich_man_and a_share_in_the a_short_distance_from a_sign_from_the a_sign_of_the a_stone_s_throw a_sum_of_money a_thing_as_this a_thing_of_which a_thing_that_is a_thing_to_be a_third_of_the a_time_when_they a_time_when_you a_tree_in_the a_very_beautiful_woman a_very_long_time ... D:\_KAZE_new-stuff\Overlapper-Blender_r1+>