Added main script that allows for Calendar integration, and misc. cha…

…nges * HackerEarth scraper has similarly been reworked to be functional; Bugfixes for CodeChef scraper * Created a main file; It performs all the functionalities covered by executor.sh. Updated README and requirements
pes-alcoding-club · Jul 21, 2020 · c09cd5e · c09cd5e
1 parent 0fca134
commit c09cd5e
Show file tree

Hide file tree

Showing 11 changed files with 270 additions and 94 deletions.
diff --git a/.gitignore b/.gitignore
@@ -43,4 +43,8 @@ pip-delete-this-directory.txt
 .pytest_cache/
 
 # Mypy (static type checking)
-.mypy_cache/
+.mypy_cache/
+
+# Credentials for calendar
+credentials.json
+token.pickle
diff --git a/README.md b/README.md
@@ -3,28 +3,28 @@
 
 [![Build Status](https://travis-ci.com/varunvora/alcoding.svg?branch=master)](https://travis-ci.com/varunvora/alcoding)
 
-Alcoding Club of [PES University](https://pes.edu/) maintains ratings of its students who are active in [competitive programming](https://en.wikipedia.org/wiki/Competitive_programming). This repository contains the ratings and the code which generates it.
+Alcoding Club of [PES University](https://pes.edu/) maintains ratings of its students who are active in [competitive programming](https://en.wikipedia.org/wiki/Competitive_programming). This repository contains the ratings and the code that generates it.
 
 ## Purpose
-An intra-college rating is maintained so that the club can identify good coders.  The club will group these students and help them improve at competitive programming by organizing meet-ups, providing resources, arranging contests and develop a coding community in the University.
+An intra-college rating is maintained to aid the club in identifying good coders. The club aims to help these students improve their competitive programming skills by organizing meet-ups, providing resources, arranging contests and developing a coding community in the University.
 
 
 ## Ratings
-The ratings are calculated by students' performances in [specified contests](database/README.md).
+The ratings are calculated using students' performances in [specified contests](database/README.md).
 
 ### Mechanism
-A [rank list](database/contest_ranks) of registered students is generated at the end of each contest. A rating is computed from the rank list, which indicates their relative performance. The implementation is almost the same as [Codechef's Rating Mechanism](https://www.codechef.com/ratings) which is a modified version of [Elo rating system](https://en.wikipedia.org/wiki/Elo_rating_system). To avoid students from [protecting their ratings](https://en.wikipedia.org/wiki/Elo_rating_system#Game_activity_versus_protecting_one's_rating) and encourage participation, a decay rule is also added which decrements a student's rating by 1% if she does not take part in 5 consecutive contests.
+A [rank list](database/contest_ranks) of registered students is generated at the end of each contest. A rating is computed from the rank list, which indicates their relative performance. The implementation is very similar to [Codechef's Rating Mechanism](https://www.codechef.com/ratings) which is a modified version of the [Elo rating system](https://en.wikipedia.org/wiki/Elo_rating_system). To prevent students from [protecting their ratings](https://en.wikipedia.org/wiki/Elo_rating_system#Game_activity_versus_protecting_one's_rating) and encourage participation, a decay rule, which decrements a student's rating by 1% if they do not take part in 5 consecutive rated contests, is also added.
 
 
 ### Verification
-The [code that generates the rating](ratings/processor.py) is open. Along with that we have provided [a script with which you can verify](executor.sh) that the displayed ratings are correct. This script resets all students' ratings, and computes the ratings after all the contest ranks are considered. You may [report an issue](https://github.com/varunvora/alcoding/issues) if you find any discrepancy.
+The [code that generates the rating](ratings/processor.py) is open. Further, we also provide [a method with which you can verify](run.py) the displayed ratings. This method resets all students' ratings, and recomputes the ratings of every student after considering all contest ranks. Please do [report an issue](https://github.com/pes-alcoding-club/student-ratings/issues) if you find any discrepancy.
 
 ## Calendar
-Alcoding Club maintains a [Google calendar for competitive programming](https://calendar.google.com/calendar?cid=N3RsZGt1dXEwcW1mOW9ub2Jxb3ByZ2Z1cDRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ). Contests that are marked as "Rated" will be considered for these ratings. 
+Alcoding Club maintains a [Google Calendar for competitive programming](https://calendar.google.com/calendar?cid=N3RsZGt1dXEwcW1mOW9ub2Jxb3ByZ2Z1cDRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ). Contests that are marked "Rated" will be considered for these ratings. 
 ## Contribute
-This project is still very small so there are no strict guidelines for contribution. For now we are following [PEP 8 -- Style Guide for Python Code](https://www.python.org/dev/peps/pep-0008/).
+At the moment, there are no strict guidelines for contribution. As a standard, we follow the [PEP 8 -- Style Guide for Python Code](https://www.python.org/dev/peps/pep-0008/).
 
-You can [report an issue](https://github.com/varunvora/alcoding/issues) if you find a bug or any other change you would like to make. You may also make a [pull request](https://github.com/varunvora/alcoding/pulls). It would be helpful if you use [our Github labels](https://github.com/varunvora/alcoding/labels) for all issues and pull requests. Be sure to clearly document and describe any issues or changes.
+Feel free to [report an issue](https://github.com/pes-alcoding-club/student-ratings/issues) if you find a bug, or have any other change you would like to see. You may also create a [pull request](https://github.com/pes-alcoding-club/student-ratings/pulls). It would be helpful if you use [our Github labels](https://github.com/pes-alcoding-club/student-ratings/labels) for all issues and pull requests. Be sure to clearly document and describe any issues or changes.
 
 ## FAQ
 
@@ -37,11 +37,11 @@ You can [report an issue](https://github.com/varunvora/alcoding/issues) if you f
 1. Which contests are taken into account for rating?
 
     Contests in ['Competitive Programming PESU' Calendar](https://calendar.google.com/calendar?cid=N3RsZGt1dXEwcW1mOW9ub2Jxb3ByZ2Z1cDRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ) are considered for ratings.
-1. How can I tell if these ratings are legitimate?
+1. How can I tell whether these ratings are legitimate?
 
-    You can verify the ratings yourself by running [this script](executor.sh). It resets all students' ratings to default values and recomputes it for all contests so far in chronological order. 
-
-1. How can I get the scoreboard only for some particular contest(s)?
-
-    Clone this repository, open [executor.sh](executor.sh) and remove the contests you do not want the scoreboard for. Run this script and check [scoreboard.csv](scoreboard.csv).
+    You can verify the ratings yourself by calling the [make_scoreboard] function in [run.py](run.py). It resets all students' ratings to default values and recomputes it for all contests so far in chronological order. 
 
+1. How can I make a scoreboard for a few particular contests?
+    Firstly, clone this repository.
+    Create your own [contest_names_file.in](database/contest_names_file.in) and add the contest names in the format [platform]-[month]-[contest_code]. In [run.py](run.py), change the [contest_names_file_path] variable's value to your file's path.
+    Now call the [make_scoreboard] function in [run.py](run.py) with the required parameters and check [scoreboard.csv](scoreboard.csv).
diff --git a/database/contestsToProcess.in → database/contest_names_file.in b/database/contestsToProcess.in → database/contest_names_file.in
diff --git a/database/db_tools.py b/database/db_tools.py
@@ -1,14 +1,14 @@
 import re
 import sys
 import csv
-import logging
 from os import listdir
 from os.path import join
 from collections import Counter
 from functools import lru_cache
 from typing import List, Set, Tuple, Dict, Callable, Any
 from tinydb import TinyDB, where
 from ratings import elo
+from utils import log
 
 DB_FILE: str = 'database/db.json'
 CONTEST_RANKS_DIR: str = 'database/contest_ranks'
@@ -57,7 +57,7 @@ def reset_database(db_file: str = DB_FILE) -> None:
             BEST: elo.DEFAULT_RATING,
             TIMES_PLAYED: 0,
             LAST_FIVE: 5})
-    logging.info(f'Successfully reset database and stored in {db_file}')
+    log.info(f'Successfully reset database and stored in {db_file}')
 
 
 def get_site_name_from_file_name(file_name: str) -> str:
@@ -68,8 +68,8 @@ def get_site_name_from_file_name(file_name: str) -> str:
     """
     file_name_parts = file_name.split("-")
     if len(file_name_parts) < 2 or file_name_parts[0] not in SITES:
-        logging.error(f"Invalid filename '{file_name}' in contest ranks. File name convention is"
-                      f"'site-contest-details.in'")
+        log.error(f"Invalid filename '{file_name}' in contest ranks. File name convention is"
+                      f"'site-month-contestCode.in'")
         quit()
     return file_name_parts[0]
 
@@ -144,18 +144,18 @@ def log_unmapped_handles(site_username_tuple_list: List[Tuple[str, str]]) -> Non
 
         log_unmapped_handles(site_handle_tuple_list)
 
-    logging.info('Mapped ')
+    log.info('Mapped usernames to SRNs')
 
 
 def remove_unmapped_handles_from_rank_file(file_name: str) -> None:
     """
     Removes unmapped handles from outdated rank files
     to reduce space and time it takes for the script to run
     """
-    with open(join(CONTEST_RANKS_DIR, file_name), 'r') as rank_file:
+    with open(file_name, 'r') as rank_file:
         input_data: str = rank_file.read()
-
-    with open(join(CONTEST_RANKS_DIR, file_name), 'w') as rank_file:
+    count = 0
+    with open(file_name, 'w') as rank_file:
         for user_name_line in input_data.split("\n"):
             check_occurrence_in_line: bool = False
             for user_name in user_name_line.split():
@@ -164,7 +164,9 @@ def remove_unmapped_handles_from_rank_file(file_name: str) -> None:
                     rank_file.write(user_name + " ")
             if check_occurrence_in_line:
                 rank_file.write("\n")
-    logging.info(f'Cleaned {file_name}')
+                count+=1
+    loginfo = file_name.split('/')[2]
+    log.info(f'Cleaned {loginfo}')
 
 
 def export_to_csv(db_file: str = DB_FILE, scoreboard_file: str = SCOREBOARD_FILE) -> None:
@@ -190,7 +192,7 @@ def export_to_csv(db_file: str = DB_FILE, scoreboard_file: str = SCOREBOARD_FILE
         wr = csv.writer(fp)
         wr.writerows(csv_table)
 
-    logging.info(f'Successfully exported database from {db_file} to {scoreboard_file}')
+    log.info(f'Successfully exported database from {db_file} to {scoreboard_file}')
 
 
 def prettify(db_file: str = DB_FILE) -> None:
@@ -201,11 +203,11 @@ def prettify(db_file: str = DB_FILE) -> None:
         fp.write_back(fp.all())
 
 
-if __name__ == "__main__":
+'''if __name__ == "__main__":
     # While executing this script, you can specify which function to execute
     func_str: str = sys.argv[1]
     try:
         func_obj: Callable = globals()[func_str]
         func_obj(*sys.argv[2:])  # Arguments to specified function can be passed
-    except KeyError:
-        logging.error(f'Provided invalid argument. No function {func_str}')
+    except KeyError:'
+        log.error(f'Provided invalid argument. No function {func_str}')'''
diff --git a/ratings/processor.py b/ratings/processor.py
@@ -1,22 +1,23 @@
 import sys
-import logging
 from time import time
 from ratings import elo
 from database import db_tools as db
 from tinydb import TinyDB, where
-
+from utils import log
 
 class RatingProcessor:
 
-    def __init__(self, database: TinyDB, rank_file):
+    def __init__(self, database: TinyDB, rank_file_path):
         self.database: TinyDB = database
 
         self.N: int = 0
         self.Cf: float = 0.0
         self.Rb_Vb_list: list = []
         self.usn_rank_dict: dict = {}
+        self.rank_file_path = rank_file_path
+        self.rank_file = open(rank_file_path)
 
-        self.read_contest_ranks(rank_file)  # sets usn_rank_dict
+        self.read_contest_ranks(self.rank_file)  # sets usn_rank_dict
         self.set_contest_details()  # sets N, Cf and Rb_Vb_list
         self.process_competition()  # uses the set attributes to compute new ratings
 
@@ -35,9 +36,9 @@ def read_contest_ranks(self, rank_file) -> None:
                     self.usn_rank_dict[usn] = current_rank
                     same_rank_count += 1
                 else:
-                    logging.info(f'Ignoring usn {usn}')
+                    log.info(f'Ignoring SRN {usn}')
             current_rank += same_rank_count  # ranks are not 1, 1, 1, 2 but 1, 1, 1, 4
-        logging.debug(self.usn_rank_dict)
+        log.debug(self.usn_rank_dict)
 
     def set_contest_details(self) -> None:
         """
@@ -54,12 +55,12 @@ def set_contest_details(self) -> None:
         self.N = len(self.usn_rank_dict)
         self.Cf = elo.Cf(rating_list, vol_list, self.N)
         self.Rb_Vb_list = list(zip(rating_list, vol_list))
-        logging.debug(f'Contest: {rank_file_path}\nPlayers: {self.N}\nCompetition Factor: {self.Cf}')
+        log.debug(f'Contest: {self.rank_file_path}\nPlayers: {self.N}\nCompetition Factor: {self.Cf}')
 
     @staticmethod
     def _decay_player(player_dict: dict) -> None:
         """
-        Reduces ratings by 10% for those who have competed at least once
+        Reduces ratings by 1% for those who have competed at least once
         but have not taken part in the past 5 contests
         :param player_dict: dict with all details of a player
         """
@@ -76,7 +77,7 @@ def _decay_player(player_dict: dict) -> None:
         player_dict[db.RATING] = rating
         player_dict[db.LAST_FIVE] = max(1, last_five)
 
-        logging.debug('Successfully decayed ratings')
+        log.debug('Successfully decayed ratings')
 
     def _update_player(self, player_dict: dict, actual_rank: int) -> None:
         """
@@ -99,27 +100,27 @@ def _update_player(self, player_dict: dict, actual_rank: int) -> None:
         player_dict[db.BEST] = max(old_best, new_rating)
         player_dict[db.LAST_FIVE] = 5
 
-        logging.debug('Successfully updated ratings')
+        log.debug('Successfully updated ratings')
 
     def process_competition(self) -> None:
 
         rows = self.database.all()
         for row in rows:
-            logging.debug(f'Before: {row}')
+            log.debug(f'Before: {row}')
             if row[db.USN] in self.usn_rank_dict:
                 actual_rank = self.usn_rank_dict[row[db.USN]]
                 self._update_player(row, actual_rank)
             else:
                 self._decay_player(row)
-            logging.debug(f'After: {row}')
+            log.debug(f'After: {row}')
         self.database.write_back(rows)
 
 
-def read_argv(argv_format_alert: str):
-    """
+"""def read_argv(argv_format_alert: str):
+    '''
     :param argv_format_alert: An error message on what the command line arguments should be
     :return: rank file if argv is valid
-    """
+    '''
     try:
         assert len(sys.argv) == 2
         rank_file = sys.argv[1]
@@ -128,15 +129,28 @@ def read_argv(argv_format_alert: str):
             return rank_file
 
         except IOError or FileNotFoundError:
-            logging.error(f'Invalid file path for rank file: {rank_file}\n{argv_format_alert}')
+            error(f'Invalid file path for rank file: {rank_file}\n{argv_format_alert}')
             quit()
 
     except AssertionError:
-        logging.error(f'Invalid command line arguments.\n{argv_format_alert}')
-        quit()
+        error(f'Invalid command line arguments.\n{argv_format_alert}')
+        quit()"""
 
+def process(rank_file_path):
+    start_time = time()
+    # Main logic starts here
+    database_obj = TinyDB(db.DB_FILE)
+    RatingProcessor(database_obj, rank_file_path)
+    database_obj.close()
 
-if __name__ == "__main__":
+    duration = time()-start_time
+    log.debug(f'Updated ratings for {rank_file_path}')
+    if duration > 10:
+        log.critical(f'Ratings update for {rank_file_path} took {duration} seconds.\n'
+                         f'Consider removing unnecessary handles or optimize ratings algorithm')
+
+
+'''if __name__ == "__main__":
     start_time = time()
 
     argv_format = 'processor.py rank_file_path'
@@ -149,7 +163,7 @@ def read_argv(argv_format_alert: str):
     database_obj.close()
 
     duration = time()-start_time
-    logging.debug(f'Updated ratings for {rank_file_path}')
+    log.debug(f'Updated ratings for {rank_file_path}')
     if duration > 10:
-        logging.critical(f'Ratings update for {rank_file_path} took {duration} seconds.\n'
-                         f'Consider removing unnecessary handles or optimize ratings algorithm')
+        logging.log.critical(f'Ratings update for {rank_file_path} took {duration} seconds.\n'
+                         f'Consider removing unnecessary handles or optimize ratings algorithm')'''
diff --git a/requirements.txt b/requirements.txt
@@ -1,4 +1,6 @@
-tinydb
-requests
-bs4
-selenium
+requests==2.22.0
+beautifulsoup4==4.9.1
+google_api_python_client==1.10.0
+google_auth_oauthlib==0.4.1
+selenium==3.141.0
+tinydb==3.15.2