Adding changes

comments.txt

@@ -0,0 +1,53 @@
+buildIndex
+"""
+#function to read documents from collection, tokenize and build the index with tokens
+		#index should also contain positional information of the terms in the document --- term: [(ID1,[pos1,pos2,..]), (ID2, [pos1,pos2,�]),�.]
+		#use unique document IDs
+		"""
+		1) This method gets invoked as soon as user inputs dictionary path
+		2) Iterates through all the files in the dictionary
+		3) For each file, read every line and stores it in lines list.
+		4) Then iterates through lines list and split each line using '\W+' de limiter
+		5) Removes all the empty words and iterates through wordsList
+		6) Convert each word to lower case and search for word in dictionary keys.
+		7) If word is present in dictionary, search for document Id.
+		8) If document id is same as current doc id, append the position of the word to posting list else, create a new map with
+			docId as key and position as value.
+		9) If word is not present in dictionary, create a new entry in map with word as key, it's docId and position as value
+		10) Example map ->
+				"word" : {docId : [position_List], docId : [positionList]}
+		11) Each file is given a unique doc id
+		12) position variable is used calculate position of all words and is initialized to 1 everytime a new file is opened to read
+		"""
+
+and_query 
+"""
+#function for identifying relevant docs using the index
+		""" This method takes list of query terms as input
+		1) If the length of query_terms is equal to 1, It will get the posting list for the only term. If no result is found,
+		   appropriate message will be printed. Else, print the respective file name by traversing docIdToFileName map
+		   for all the posting lists.
+		2) If there are more than 1 item in query_terms, it will get posting list for first and second term. Then calls mergePostingList()
+		 	by passing posting lists of term0 and term1. The merge result will be used to get intersection of posting list of term3.
+		 	This will be repeated for all the subsequent query terms.
+		3) Result of step 2 will have merged list of all the query terms
+		4) Uses the final merged list to traverse through DocIdToFileName map and prints respective file names
+		"""
+
+getPostingList
+""" Takes term as input parameter, It will search for key in the index, if any of the key matches search term,
+		posting list of that key will be returned as a list. If key is not found returns None"""
+
+mergePostingList
+""" This function takes 2 lists as input parameters and returns intersection of both in a sorted form """
+
+main
+"""
+	1) main method prompts for directory path containing input text files and path of the query file where all the query
+	terms are mentioned
+	2) creates an object of index class and calls build Index
+	3) Once index is built, it will print dictonory
+	4) Then, prints document id fileName map
+	5) Read each line from query file, tokenize query terms and perform search on newly built index
+	6) print time taken to perform these actions
+	"""

docs/Text-2.txt

@@ -0,0 +1,39 @@
+
+RUSSIA WHO'S IN CHARGE HERE ? IT WAS IN 1954 THAT NIKITA
+
+KHRUSHCHEV LAUNCHED HIS GRANDIOSE " VIRGIN LANDS " GAMBLE . PART OF THE
+
+PLAN WAS TO PLOW UP 32 MILLION ACRES OF MARGINAL LAND IN KAZAKHSTAN,
+
+AND SETTLE IT WITH COMMUNIST " PIONEERS, " WHO WERE TO PLANT AND
+
+PRODUCE HUGE QUANTITIES OF DESPERATELY NEEDED GRAIN WITHIN TWO YEARS .
+
+NIKITA'S SCHEME FLOPPED . THERE WAS NOT ENOUGH RAINFALL, AND THE
+
+PIONEERS DID NOT TAKE TO TRACTOR LIFE ON THE BLEAK FRONTIER . EXCEPT
+
+FOR 1958, EACH HARVEST HAS BEEN LOWER THAN THE PREVIOUS YEAR'S . WORST
+
+YEAR OF ALL WAS 1962, WHEN THE VIRGIN LANDS DELIVERED ONLY HALF THEIR
+
+QUOTAS . NATURALLY, KHRUSHCHEV TAKES NONE OF THE BLAME FOR THE FIASCO .
+
+THREE YEARS AGO HE FOUND A SCAPEGOAT IN KAZAKHSTAN PARTY BOSS NIKOLAI
+
+BELYAEV, FIRED HIM FOR HIS " ERRORS . " LAST WEEK BELYAEV'S SUCCESSOR,
+
+DINMUKHAMED KUNAEV, WAS SIMILARLY BOUNCED FOR " LAPSES " IN HIS WORK .
+
+FOR GOOD MEASURE, MOSCOW ALSO PURGED THE FORMER PREMIER OF THE
+
+TERRITORY FROM THE LOCAL PARTY'S CENTRAL COMMITTEE . IT WAS PERHAPS NO
+
+COINCIDENCE THAT NIKOLAI IGNATOV, 61, A ONETIME KHRUSHCHEV CRONY, LAST
+
+WEEK ABRUPTLY LEFT HIS POST AS A SOVIET DEPUTY PREMIER AFTER ONLY NINE
+
+MONTHS ON THE JOB . FARM EXPERT IGNATOV HAD THE MISFORTUNE TO BE BOSS
+
+OF A SPECIAL COMMITTEE TO BOOST FOOD PRODUCTION .
+

docs/Text-3.txt

@@ -0,0 +1,85 @@
+
+BERLIN ONE LAST RUN HANS WEIDNER HAD BEEN HOPING FOR MONTHS TO
+
+ESCAPE DRAB EAST GERMANY AND MAKE HIS WAY TO THE WEST . THE ODDS WERE
+
+AGAINST HIM, FOR WEIDNER, 40, WAS A CRIPPLE ON CRUTCHES WHO LIVED IN
+
+THE VILLAGE OF NEUGERSDORF, 115 MILES SOUTHEAST OF THE FRONTIER OF
+
+FREEDOM BUT HANS WEIDNER DID HAVE ONE MAJOR ASSET, THE BUS THAT
+
+HE OPERATED FOR THE LOCAL COMMUNIST REGIME . IT WAS AN UGLY THING, AND
+
+ANCIENT . ITS CHASSIS CREAKED, AND THE ENGINE COUGHED ; A
+
+CREAM-COLORED COAT OF PAINT COULD NOT DISGUISE THE WELTS AND BRUISES OF
+
+TWO DECADES OF CHUGGING SERVICE . IN FACT, THE BUS WAS READY FOR THE
+
+JUNK PILE WHEN WEIDNER DECIDED TO PRESS IT INTO SERVICE FOR ONE LAST
+
+RUN . SHARP BLADES . THE HAZARDS WOULD BE GREAT ON THE JOURNEY TO THE
+
+BORDER ; SO WEIDNER SIGNED UP A FELLOW VILLAGER, JURGEN WAGNER, 22, TO
+
+TAKE THE WHEEL . EIGHT DAYS BEFORE CHRISTMAS, THE PAIR BEGAN THE
+
+FEVERISH PREPARATIONS IN WEIDNER'S GARAGE . FIRST WEIDNER AND WAGNER
+
+ATTACHED A HEAVY SNOWPLOW TO THE FRONT OF THE BUS, NOT TO PLOW SNOW,
+
+BUT TO SCOOP AWAY THE HEAVY OBSTACLES THEY KNEW AWAITED THEM AT
+
+ROADBLOCKS AHEAD . TO ALL SIX LUGS ON EACH FRONT WHEEL THEY BOLTED
+
+SHARP BLADES OF THE TOUGHEST STEEL, AFFIXED SO THAT THE WHIRLING EDGES
+
+WOULD CHOP BARBED WIRE TO BITS . THEN THEY WEDGED ONE-QUARTER-INCH
+
+SECTIONS OF STEEL PLATE INSIDE THE BUS TO STOP BULLETS . AT LAST ALL
+
+WAS READY . ON CHRISTMAS EVE, WEIDNER AND WAGNER PILED THEIR WIVES AND
+
+FOUR CHILDREN ABOARD, NOT FORGETTING THREE TONS OF HOUSEHOLD BELONGINGS
+
+. FOR ADDED PROTECTION THE PLOTTERS SHOVELED A TON OF COAL AND POTATOES
+
+INTO THE BACK OF THE BUS . THEN THEY CHUGGED OFF NORTH TOWARD BERLIN
+
+ALONG BACK ROADS TO ESCAPE COMMUNIST PATROLS . JUST BEFORE THEY REACHED
+
+THE WALL, THEY PLANNED TO SWING WEST IN ORDER TO ENTER THE EAST-WEST
+
+AUTOBAHN LEADING TO THE U.S . SECTOR OF THE CITY . EN ROUTE, THE
+
+RADIATOR FROZE IN THE SUBZERO WEATHER . THAT FIXED, THEY WERE ONLY A
+
+FEW MILES FARTHER WHEN A TIRE BLEW OUT . THE KIDS WERE CRYING AND THE
+
+WIVES SHIVERING WITH COLD AND PANIC WHEN, AT LAST, THEY ARRIVED AT
+
+DREWITZ, THE MOST HEAVILY GUARDED CHECKPOINT ON THE ENTIRE AUTOBAHN TO
+
+BERLIN . IT WAS NO TIME TO STOP AND RECONSIDER . FLYING POTATOES . "
+
+WAH-AH, WAH-AH, " SHRIEKED THE POLICE-TYPE KLAXONS THAT WEIDNER HAD
+
+THOUGHTFULLY INSTALLED IN ADVANCE . THE COMMUNIST GUARDS OBEDIENTLY
+
+RAISED THE FIRST OF THREE BARRIERS . BUT WHAT WAS A BUS DOING ON
+
+EMERGENCY DUTY ? SUDDENLY THE SHOOTING BEGAN TOO LATE . WAGNER, AT 40
+
+M.P.H., WAS ALREADY CRASHING THROUGH THE SECOND BARRIER 100 YARDS
+
+AHEAD, THEN THE THIRD, ONLY 20 YARDS AWAY . ITS WINDSHIELD SMASHED, ITS
+
+PASSENGERS SHAKEN, ITS CARGO OF COAL AND POTATOES IN EVERY CORNER OF
+
+THE CAB, THE OLD BUS FINALLY LURCHED TO A STOP A FEW MILES DOWN THE
+
+ROAD WHERE THE COMMUNISTS NO LONGER MATTERED AT THE U.S . CHECKPOINT, A
+
+FOOT OR TWO INSIDE WEST BERLIN .
+

docs/Text-35.txt

@@ -0,0 +1,59 @@
+
+SYRIA FOILED AGAIN GENERALLY SPEAKING, SYRIA'S SOLDIERS ARE NO
+
+GREAT SHAKES AT FIGHTING . BUT THEY'RE A PERSISTENT LOT WHEN IT COMES
+
+TO OVERTHROWING THE GOVERNMENT AT HOME . NO FEWER THAN TEN ATTEMPTED
+
+COUPS HAVE TAKEN PLACE IN 13 YEARS . ONE GROUP OF OFFICERS HAS PLOTTED
+
+STEADILY FOR MORE THAN TWO YEARS . LED BY COLONEL ABDEL KARIM NAHLAWI,
+
+39, THE MILITARY MALCONTENTS WERE IN ON THE COUP THAT SEVERED SYRIA'S
+
+UNION WITH EGYPT . LAST MARCH, THEY TRIED TO TAKE OVER THE GOVERNMENT,
+
+BUT WERE PACKED OFF AS MILITARY ATTACHES IN SYRIAN EMBASSIES ABROAD .
+
+LAST WEEK THE OFFICERS WERE UP TO THEIR OLD TRICKS AGAIN . INCENSED BY
+
+A DAMASCUS DECREE STRIPPING THEM OF THEIR ARMY STATUS, THE GROUP
+
+ARRANGED A DRAMATIC RENDEZVOUS IN TURKEY, THEN QUIETLY CROSSED THE
+
+FRONTIER AND MADE FOR THEIR OLD BARRACKS INSIDE SYRIA . GREETED
+
+JOYOUSLY BY SOME OF THEIR FORMER COMRADES IN ARMS, NAHLAWI'S MEN ISSUED
+
+A PUBLIC DEMAND THAT THEIR DISCHARGES BE CANCELED, AND THAT A NEW
+
+GENERAL STAFF TO THEIR LIKING BE PUT IN POWER . TO CONFUSE THINGS, THE
+
+REBELLIOUS SOLDIERS INSISTED ON A PLEBISCITE TO DECIDE ON CLOSER
+
+RELATIONS WITH NASSER'S EGYPT . OTHERWISE, NAHLAWI THREATENED, TROOPS
+
+UNDER HIS CONTROL WOULD MARCH ON DAMASCUS . FOR PRESIDENT NAZEM
+
+EL-KOUDSI, IT WAS AN OLD FAMILIAR TUNE . COOLLY, HE PLAYED FOR TIME,
+
+AGREED TO " CONSIDER " THE DEMANDS IF NAHLAWI WOULD NEGOTIATE AT ARMY
+
+HEADQUARTERS . THE TALKS DRAGGED ON FOR THREE CRISIS-FILLED DAYS .
+
+THEN, KOUDSI MOBILIZED HIS OWN FORCES, ONE NIGHT SUDDENLY SURROUNDED
+
+THE ARMY GHQ WITH ARMORED CARS . COLONEL NAHLAWI GOT THE POINT . IN
+
+ANOTHER COUNTRY, HE AND HIS MEN MIGHT HAVE BEEN JAILED, OR EVEN
+
+EXECUTED FOR TREASON . BUT KOUDSI, WHO KEEPS A PREPARED RESIGNATION IN
+
+HIS DESK JUST IN CASE THE SOLDIERS SHOULD SOME DAY WIN, CHOSE NOT TO
+
+PUSH HIS LUCK . ESCORTED ABOARD AN AIRLINER AND GIVEN $1,000 APIECE TO
+
+COVER THEIR EXPENSES, THE REBELS WERE SENT BACK TO DIPLOMATIC POSTS IN
+
+CIVVIES .
+

docs/Text-36.txt

@@ -0,0 +1,39 @@
+
+SOUTH ARABIA
+
+AND ADEN MAKES TWELVE WITH ITS STUPEFYING TEMPERATURES AND BLEAK
+
+TERRAIN, BRITAIN'S ADEN COLONY ON THE SOUTHWESTERN FRINGE OF THE
+
+ARABIAN PENINSULA IS ONE OF THE WORLD'S MOST UNATTRACTIVE PIECES OF
+
+REAL ESTATE . BUT IT HAS ITS VALUE NONETHELESS . BETWEEN SUEZ AND
+
+SINGAPORE, IT IS THE ONLY SUITABLE FUELING AND VICTUALING STATION FOR
+
+THE BRITISH NAVY, AND 8,000 TROOPS OF HER MAJESTY'S MIDDLE EAST COMMAND
+
+ARE STATIONED ON ITS 75 SQUARE MILES OF OVERHEATED ROCK . TO STRENGTHEN
+
+THE PRECIOUS LITTLE COLONY AGAINST THE COVETOUS DESIRES OF NEARBY
+
+YEMEN, THE BRITISH SINCE 1959 HAVE BEEN LINKING NEIGHBORING SULTANATES
+
+AND EMIRATES IN A NEW FEDERATION OF SOUTH ARABIA . ADEN WAS TO JOIN IN
+
+MARCH, BUT AN OUTBURST OF RIOTS SPARKED BY PRO-YEMEN LABOR LEADERS AND
+
+THE EMERGENCE OF AN EGYPTIAN BACKED NATIONALIST REGIME IN YEMEN ITSELF
+
+PERSUADED BRITAIN TO SPEED THINGS UP . LAST WEEK THE BRITISH MADE ADEN
+
+THE TWELFTH MEMBER OF THE FEDERATION, RETAINING CONTROL BUT VAGUELY
+
+PROMISING EVENTUAL INDEPENDENCE . ARAB NATIONALISTS WERE UNHAPPY .
+
+WHILE THE FEDERATION'S BLACK, YELLOW, GREEN AND BLUE FLAG WENT UP OVER
+
+GOVERNMENT BUILDINGS IN PLACE OF THE UNION JACK, ANTI-FEDERATION
+
+RESIDENTS RAISED MOURNING FLAGS INSTEAD .
+

docs/Text-37.txt

@@ -0,0 +1,47 @@
+
+TUNISIA PALS NO MORE ONE OF THE MOST PERISHABLE COMMODITIES IN THE
+
+ARAB WORLD SEEMS TO BE GRATITUDE . WHEN ALGERIA WON INDEPENDENCE FROM
+
+FRANCE AFTER SEVEN YEARS OF BLOODY WAR, ITS PEOPLE WERE DEEPLY INDEBTED
+
+TO NEIGHBORING TUNISIA, WHOSE PRESIDENT HABIB BOURGUIBA, 59, HAD GIVEN
+
+SHELTER TO 200,000 ALGERIAN REFUGEES AND PROVIDED A REFUGE FOR THE
+
+TRAINING AND EQUIPMENT OF 18,000 ALGERIAN FIGHTING MEN . BUT LAST WEEK,
+
+IN A SHOUTING, GESTICULATING SPEECH IN THE VAST CASBAH SQUARE OF TUNIS,
+
+BOURGUIBA SAID IT WAS " SURPRISING " TO DISCOVER THAT ALGERIA HAS "
+
+BECOME A RESIDENCE FOR CRIMINALS AND PLOTTERS AGAINST THE GOVERNMENT OF
+
+TUNISIA . " HE SEEMED MOSTLY UPSET BY THE FACT THAT ALGERIA HAS REFUSED
+
+TO EXTRADITE ONE BOUBEKER MUSTAFI, A TUNISIAN ACCUSED OF BEING PARTY TO
+
+THE CHRISTMAS ASSASSINATION PLOT AGAINST BOURGUIBA (TIME, JAN . 4), FOR
+
+WHICH 13 TUNISIANS HAVE BEEN CONDEMNED TO DEATH . BOURGUIBA RIDICULED
+
+46-YEAR-OLD ALGERIAN PREMIER AHMED BEN BELLA AS " AN INEXPERIENCED,
+
+EXCITED YOUTH, " AND " A SIMPLE-MINDED PEASANT, " BUT CONCEDED THAT "
+
+NOT ALL MEMBERS OF THE ALGERIAN GOVERNMENT WERE INVOLVED . " IN ANY
+
+CASE, BOASTED BOURGUIBA, " WE HAVE ENOUGH POWER AND STRENGTH TO FACE
+
+ANY ADVERSARY . I DON'T INTEND TO KEEP UP HYPOCRITICAL RELATIONS WITH
+
+ANY GOVERNMENT . " AT WEEK'S END BOURGUIBA MADE HIS DISPLEASURE
+
+OFFICIAL BY SUMMONING HOME TUNISIA'S AMBASSADOR TO ALGERIA . BUT THERE
+
+WAS NO BREAK IN DIPLOMATIC RELATIONS, NOR ANY SIGN THAT TUNISIA'S
+
+10,000-MAN ARMY WAS PREPARING TO TAKE ON ALGERIA'S 120,000 VETERAN
+
+TROOPS .
+

docs/Text-4.txt

@@ -0,0 +1,35 @@
+
+THE ROAD TO JAIL IS PAVED WITH NONOBJECTIVE ART SINCE THE
+
+KREMLIN'S SHARPEST BARBS THESE DAYS ARE AIMED AT MODERN ART AND "
+
+WESTERN ESPIONAGE, " IT WAS JUST A MATTER OF TIME BEFORE THE KGB'S COPS
+
+WOULD TURN UP A VICTIM WHOSE WRONGDOINGS COMBINED BOTH EVILS . HE
+
+TURNED OUT TO BE A LENINGRAD PHYSICS TEACHER WHOSE TASTE FOR ABSTRACT
+
+PAINTING ALLEGEDLY LED HIM TO JOIN THE U.S . SPY SERVICE . POLICE SAID
+
+THEY FIRST SPOTTED THE TEACHER, ONE RUDOLF FRIEDMAN, AS HE MUTTERED
+
+UNCOMPLIMENTARY REMARKS ABOUT SOCIALIST REALISM WHILE STROLLING THROUGH
+
+LENINGRAD'S RUSSIAN MUSEUM . A WELL-DRESSED U.S . TOURIST APPROACHED
+
+HIM, ENTHUSIASTICALLY SHARED HIS SENTIMENTS, AND PROMISED TO SEND
+
+FRIEDMAN REPRODUCTIONS OF AVANT-GARDE PAINTINGS FROM AMERICA . THE
+
+PICTURE FRIEDMAN LIKED BEST, SAID THE COPS INDIGNANTLY, WAS A " CHAOS
+
+OF BLACK, RED AND BLUE SPLOTCHES CAPTIONED I NEED YOU TONIGHT . " SOON,
+
+THEY SAID, THE TEACHER WAS GETTING MESSAGES FROM THE U.S . WRITTEN IN
+
+INVISIBLE INK . JUST AS FRIEDMAN PREPARED TO DELIVER INFORMATION " VERY
+
+REMOTE FROM THEORETICAL ARGUMENTS ABOUT ABSTRACT ART, " POLICEMOVED IN
+
+AND HUSTLED HIM OFF TO JAIL .
+

docs/Text-8.txt

@@ -0,0 +1,31 @@
+
+RED CHINA FIXING FRONTIERS RED CHINA PRODUCED A SECOND SURPRISE
+
+LAST WEEK . AT PEKING AIRPORT, PREMIER CHOU EN-LAI WELCOMED OUTER
+
+MONGOLIA'S PREMIER YUMZHAGIIN TSEDENBAL, 46, WHO IS NORMALLY REGARDED
+
+AS A RUSSIAN PUPPET . WHISKED OFF IN A BLACK, SOVIET-MADE LIMOUSINE
+
+AMONG CROWDS DUTIFULLY WAVING CHINESE AND MONGOLIAN FLAGS, TSEDENBAL
+
+WAS PUT THROUGH THE USUAL ROUTINE OF TOASTS, BANQUETS AND FULSOME
+
+SPEECHES . THEN, ON THE SAME DAY THAT RED CHINA ANNOUNCED PLANS TO
+
+DEFINE ITS BORDERS WITH PAKISTAN, TSEDENBAL AND CHOU EN-LAI SIGNED A
+
+TREATY FIXING THE 2,500-MILE FRONTIER BETWEEN THEIR TWO COUNTRIES . IT
+
+WAS ANOTHER LITTLE LESSON AIMED AT DELHI . AT THE FAREWELL BANQUET FOR
+
+TSEDENBAL, PREMIER CHOU EN-LAI SMOOTHLY NOTED THAT RED CHINA HAD NOW
+
+SOLVED ITS BORDER PROBLEMS ON THE BASIS OF " PEACEFUL COEXISTENCE "
+
+WITH BURMA, NEPAL, PAKISTAN AND OUTER MONGOLIA, MAKING THE POINT THAT
+
+ONLY TWO NEIGHBORS NOW REMAIN WITH WHOM CHINA HAS NOT MADE A BORDER
+
+ADJUSTMENT : INDIA AND THE SOVIET UNION .
+