Skip to content

dostonhamrakulov/Datenbaken-RSS-reader-project

BranchesTags

Repository files navigation

Database and Web Technologies

Programming languages and frameworks

Backend:
	- Java
	- Java Spring Boot Framework

Database:
	- MySQL

Frontend:
	- PHP
	- Javascript
	- HTML5, CSS and Bootstrap

IDE and tools

	- Intellij Idea, 
	- Postman
	- Eclipse

Authors

 - Doston Hamrakulov 
 - Sagar Kafle

Positions

Software Engineers, Web Developers, Freelancers

Recorded video as demo:

Demo

Backend

2.2.1 Backend

  • - 1.The backend aggregates and processes the data of the various web feeds.
  • - It can handle multiple standards of web feeds, supporting at least one version of RSS and the Atom format.
  • - It prevents the updating of web feeds in a too small time interval, so that it waits at least 10 minutes between two requests for a web feed (and blocks other requests to do so).
  • - It ensures a consistent storage of the data in a database.
  • - Existing records can also be updated, but no duplicates are created.
  • - Records that have reached a certain age are deleted and no longer inserted. By default, this age should be 30 days.

2.2.2 Database

  • - The database stores the aggregated and processed data of the web feeds.
  • - It can also be used to store the configuration.
  • - The frontend must never communicate directly with the database.

2.2.3 REST based interface

  • - The REST based interface provides communication between the frontend and the database.
  • - It delivers the data from the database to the frontend.
  • - It can also be used to store configuration settings in the database.
  • - Furthermore, it is possible to use the REST based interface to initiate an update of the web feeds by the backend. Therefore, it is not necessary to implement a proactive service which automatically performs the update.

2.2.4 Frontend

  • - The information of all web feeds can be displayed. The following details must at least be available:
    • - a)title of the content
    • - b)clickable link to the webpage with the actual content
    • - c)date when the content was published according to the web feed
    • - d)date when the entry in the web feed first got detected by the program
    • - e)provider where the content is available
  • - The display of the contents of individual providers, i.e. web feed links, can be enabled or disabled.
  • - Usability aspects are taken into account.
  • - The design is appealing.

2.2.4 Frontend

  • - The following information can be displayed for each provider:
    • - a)address (URL) of the web feed
    • - b)date of the last successful update of the web feed
    • - c)Date of the latest record according to information from the web feed (the date should also be available if all records are older than the threshold for deleting old records and therefore no record of the web feed is in the database anymore)
    • - d)date of the last attempt to update with the number of records found or an indication of an error that occurred during the update attempt
  • - At least five different working web feeds and a malfunctioning web feed are available for demonstration purposes.

2.2.5 Additional task for groups

  • - The backend can also handle "broken" web feeds that do not fully adhere to the labelled standard.

  • - For each record, there is a detail view that shows the contents included in the web feed.

  • - The time interval for the web feed update blockade and the age for deleting old records can be set via the frontend.

  • - The complete web feed management takes place via the frontend, whereby at least the following options exist beyond the previous display options:

    • - Insert new web feeds with address (UsRL) and display name, which is also used in the overview
    • - Modify the entry of a web feed (change the URL or display name) while preserving existing records in the database
    • - Delete the entry of a web feed, deleting all associated data from the database

  • - It is possible to export the data of the web feeds selected in the frontend view as a new web feed in RSS or Atom format. This export should also be reachable via a normal URL with a GET request and contain all stored data. Ultimately, this should make it possible to transform a web feed from RSS to Atom format or vice versa.

  • - It can handle multiple standards of web feeds, supporting at least one version of RSS and the Atom format.

    • - supporting RSS format
    • - supporting Atom format

  • - It prevents the updating of web feeds in a too small time interval, so that it waits at least 10 minutes between two requests for a web feed (and blocks other requests to do so).

  • - Records that have reached a certain age are deleted and no longer inserted. By default, this age should be 30 days.

  • - Connecting to Database

  • - Queries over DB with Spring

  • - APIs for web_feed_providers

  • - APIs for web_feed

  • - APIs for User

  • - APIs for User Login

  • - RSS Reading

    • - Validating RSS data
    • - Reading whole RSS file from url
    • - Getting individual RSS items
    • - Validating RSS items before inserting into DB

  • - Writing individual RSS items into DB in time interval

    • - checking whether a link is present or not
    • - Consitancy of data like whether title is there or published date

  • - Getting all new web feeds from web feed provider when program starts

  • - sdds

  • - Update web_feed_provider

  • - Delete all feeds of web_feed_provider after provider is deleted

  • - Rewrite delete method of web_feed_providers properly with userid

    • - Check if feed_provider exists or not
    • - Check if a user exists or not
    • - return Status Code, OK, NOT FOUND
    • - Make DELETE request

  • - Inserting web feeds when a provider is added

    • - checking whether a link is present or not
    • - Consistancy of data like whether title is there or published date

  • - getting number of all feeds -- GET request

  • - getting number of feeds for a specific provider -- GET request

Some Tasks

  • - It prevents the updating of web feeds in a too small time interval, so that it waits at least 10 minutes between two requests for a web feed (and blocks other requests to do so). Sagar

  • - It ensures a consistent storage of the data in a database.

  • - Existing records can also be updated, but no duplicates are created

    • - based on title and published date

  • - Records that have reached a certain age are deleted and no longer inserted. By default, this age should be 30 days.

  • - Furthermore, it is possible to use the REST based interface to initiate an update of the web feeds by the backend. Therefore, it is not necessary to implement a proactive service which automatically performs the update.

  • - The display of the contents of individual providers, i.e. web feed links, can be enabled or disabled.

  • - Date of the latest record according to information from the web feed (the date should also be available if all records are older than the threshold for deleting old records and therefore no record of the web feed is in the database anymore)

  • - date of the last attempt to update with the number of records found or an indication of an error that occurred during the update attempt

  • -

  • - The time interval for the web feed update blockade and the age for deleting old records can be set via the frontend.

  • - Fixing userid is always zero while adding provider

  • - Deleting all feeds when their provider is deleted by id

16 June 2019

  • - updating number of Feeds of a provider
  • - updating Latest_record_date of a provider(it means taking a publishedDate of latest web_feed and inserting into the provider)
  • - update providers of a user based on his/her update-period

Additional Tasks:

  • - The backend can also handle "broken" web feeds that do not fully adhere to the labelled standard.
  • - For each record, there is a detail view that shows the contents included in the web feed.
  • - The time interval for the web feed update blockade and the age for deleting old records can be set via the frontend.
  • - The complete web feed management takes place via the frontend, whereby at least the following options exist beyond the previous display options:
    • - Insert new web feeds with address (URL) and display name, which is also used in the overview
    • - Modify the entry of a web feed (change the URL or display name) while preserving existing records in the database
    • - Delete the entry of a web feed, deleting all associated data from the database
  • - It is possible to export the data of the web feeds selected in the frontend view as a new web feed in RSS or Atom format. This export should also be reachable via a normal URL with a GET request and contain all stored data. Ultimately, this should make it possible to transform a web feed from RSS to Atom format or vice versa.
    • - Exporting as JSON to Frontend is done
    • - Converting JSON into RSS or ATOM Sagar

APIs for User:

{
        
        "email": "user1@gmail.com",
        "name": "user1",
        "password":"user1@"
}

Response:

Created - CREATED 301

or bad quest
{
        
        "email": "user1@gmail.com",
        "password":"user1@"
}

Response:

{
    "id": 1084,
    "email": "user1@gmail.com",
    "name": "user1",
    "password": "user1@",
    "status": 0,
    "feedage": 0,
    "updateperiod": 0
}

with Status code: FOUND 302
or 
HTTP Status Code: NOT_FOUND
{
    "id": 101,
    "feedage": 22
}

Response

{
    "id": 101,
    "name": "user1",
    "email": "user1@gmail.com",
    "status": 1,
    "feedage": 22,
    "updateperiod": 55
}

with Http Status: OK (200)

if not found, it will return Status: BAD_REQUEST(400)
{
    "id": 101,
    "updateperiod": 66
}

Response

{
    "id": 101,
    "name": "user1",
    "email": "user1@gmail.com",
    "status": 1,
    "feedage": 22,
    "updateperiod": 66
}

with Http Status: OK (200)

if not found, it will return Status: BAD_REQUEST(400)

Response:

[
    {
        "id": 101,
        "email": "user1@gmail.com",
        "name": "user1",
        "password": "user1_pass",
        "status": 1,
        "feedage": 30,
        "updateperiod": 1
    }
]

with Http Status: FOUND (302)

if not found, it will return Status: NOT_FOUND (404)
{
    "id": 101,
    "email": "user1@gmail.com",
    "name": "user1",
    "password": "user1_pass",
    "status": 1,
    "feedage": 30,
    "updateperiod": 1
}

with Http Status: FOUND (302)

if not found, it will return Status: NOT_FOUND (404)
{
    "last_updated_date": "Wed, 03 Jul 2019 15:54:01 CEST",
    "current_date": "Wed, 03 Jul 2019 16:02:11 CEST",
    "updated_ago": "8",
    "updated_willbe": "Wed, 03 Jul 2019 16:04:01 CEST",
    "updated": "False"
}

with Http Status: OK

APIs for Web_feed_providers:

[
    {
        "id": 1,
        "name": "Name_1",
        "link": "https://www.espn.com/espn/rss/news",
        "updateddate": "Sat, 22 Jun 2019 19:57:38 CEST",
        "latestrecorddate": "",
        "lastattempt": "Sat, 22 Jun 2019 19:57:38 CEST",
        "numfeeds": 32,
        "error": 3,
        "userid": 101
    },
...
]

with Http Status: OK
if not found, it will return Status: 404 Not Found

Response will be :

{
    "id": 1,
    "name": "Name_1",
    "link": "https://www.espn.com/espn/rss/news",
    "updateddate": "Sat, 22 Jun 2019 19:57:38 CEST",
    "latestrecorddate": "",
    "lastattempt": "Sat, 22 Jun 2019 19:57:38 CEST",
    "numfeeds": 32,
    "error": 3,
    "userid": 101
}

with Http Status: OK
if not found, it will return Status: 404 Not Found
Response HTTP OK
{
	
	"name":"Name_1",
	"link" : "https://rss.nytimes.com/services/xml/rss/nyt/Technology.xml",
	"userid":11
}

[
    {
	    "id": 1,
	    "name": "Name_1",
	    "link": "https://www.espn.com/espn/rss/news",
	    "updateddate": "Sat, 22 Jun 2019 19:57:38 CEST",
	    "latestrecorddate": "",
	    "lastattempt": "Sat, 22 Jun 2019 19:57:38 CEST",
	    "numfeeds": 32,
	    "error": 3,
	    "userid": 101
	},
...
]
Deleted with Status code: OK 200
or
cannot delete with Status code: NOT_FOUND 404
{
        "id": 1173,
        "updateddate": "Tue, 11 Jun 2019 00:30:19 CEST",
        "latestrecorddate": "Tue, 30 Jun 2019 00:30:21 CEST",
        "lastattempt": "Tue, 30 Jun 2019 00:30:21 CEST",
        "numfeeds": 130
}

Reponse:
1 (Updated) with Status code: OK 200
or
0(cannot update) with Status code: NOT_FOUND 404
{
   
	"id": 1173,
	"name": "New name",
	"link": "new url"

}

Reponse:
Updated with Status code: OK 200
or
cannot update with Status code: NOT_FOUND 404
{
        "id": 1173,
        "error": 111
}

Reponse:
Updated with Status code: OK 200
or
cannot update with Status code: NOT_FOUND 404

APIs for Web_feed:

Response:

{
	"id": 2,
	"title": "Round-by-round picks from the NBA draft",
	"link": "https://www.espn.com/nba/draft/rounds",
	"description": "Check out every pick from 1-60, including Zion Williamson and Ja Morant as the top two selections.",
	"publisheddate": "Fri, 21 Jun 2019 04:28:35 CEST",
	"importeddate": "Fri, 21 Jun 2019 14:27:11 CEST",
	"providerid": 1,
	"deleted": "False",
	"error": "True"
}

with Http Status: OK

if not found, it will return Status: 404 Not Found
[
    {
        "id": 2,
        "title": "Round-by-round picks from the NBA draft",
        "link": "https://www.espn.com/nba/draft/rounds",
        "description": "Check out every pick from 1-60, including Zion Williamson and Ja Morant as the top two selections.",
        "publisheddate": "Fri, 21 Jun 2019 04:28:35 CEST",
        "importeddate": "Fri, 21 Jun 2019 14:27:11 CEST",
        "providerid": 1,
        "deleted": "False",
        "error": "True"
    },
...
]

with Http Status: OK
if not found, it will return Status: 404 Not Found
{
    "link": "https://www.nytimes.com/2019/06/08/world/americas/mexico-tariffs-migration.html?emc=rss&partner=rss"
}

ResponseBody:

[
    {
        "id": 2,
        "title": "Round-by-round picks from the NBA draft",
        "link": "https://www.espn.com/nba/draft/rounds",
        "description": "Check out every pick from 1-60, including Zion Williamson and Ja Morant as the top two selections.",
        "publisheddate": "Fri, 21 Jun 2019 04:28:35 CEST",
        "importeddate": "Fri, 21 Jun 2019 14:27:11 CEST",
        "providerid": 1,
        "deleted": "False",
        "error": "True"
    },
...
]

with Http Status: FOUND

if not found, it will return Status: NO_CONTENT
{
		
        "title": "Title_3",
        "link": "Link4",
        "description": "Description4",
        "publisheddate": "cfbfgb1",
        "importeddate": "bfgbfgb fg",
        "providerid": 1,
        "deleted": "False",
        "error":"False"
}

Reponse:

with Http Status: Created

if not found, it will return Status: BAD_GATEWAY
88

with Http Status: OK

if not found, it will return Status: BAD_GATEWAY

[
    {
        "id": 2,
        "title": "Round-by-round picks from the NBA draft",
        "link": "https://www.espn.com/nba/draft/rounds",
        "description": "Check out every pick from 1-60, including Zion Williamson and Ja Morant as the top two selections.",
        "publisheddate": "Fri, 21 Jun 2019 04:28:35 CEST",
        "importeddate": "Fri, 21 Jun 2019 14:27:11 CEST",
        "providerid": 1,
        "deleted": "False",
        "error": "True"
    },

...
]
Deleted with Status code: OK 200
or
cannot delete with Status code: NOT_FOUND 404
{
        "id": 1003,
        "userid": 102
}

Response:

Deleted with Status code: OK 200
or
cannot delete with Status code: NOT_FOUND 404
{
        "title": "Updated again",
        "link": "https://www.nytimes.com/2019/06/10/world/asia/sewol-ferry-accident.html?emc=rss&partner=rss",
        "publisheddate": "Mon, 10 Jun 2019 23:54:20 CEST"
	
}

Response:

Updated with Status code: OK 200
or
cannot update with Status code: BAD_REQUEST
[
    {
        "id": 2,
        "title": "Round-by-round picks from the NBA draft",
        "link": "https://www.espn.com/nba/draft/rounds",
        "description": "Check out every pick from 1-60, including Zion Williamson and Ja Morant as the top two selections.",
        "publisheddate": "Fri, 21 Jun 2019 04:28:35 CEST",
        "importeddate": "Fri, 21 Jun 2019 14:27:11 CEST",
        "providerid": 1,
        "deleted": "False",
        "error": "True"
    },
...
with Status Code: OK
]
88

with Http Status: OK

if not found, it will return Status: NOT_FOUND
Deleted

with Http Status: OK

or
"Error occured" Status Code: INTERNAL_SERVER_ERROR
{
        "id": 3918,
        "title": "New name",
        "link": "htvdfvdfvdfvdftps://www.nytimes.com/2019/06/18/worlssdvsdfvsfvfd/africa/congo-ethnic-violence-ebola.html?emc=rss&partner=rss"
        
}

Updated

with Http Status: OK

or
"Error occured" Status Code: INTERNAL_SERVER_ERROR

Setup: - Installing Rome in Eclipse

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •  

Languages