Web‐Scrybe Wiki

Overview

Web-Scrybe is a web-based tool designed to scrape content from online platforms.

Features

• Web Search Engine Support: Enables web searches on Google and Bing using Selenium, bypassing rate limits and eliminating the need for costly API usage.

• Reddit Hot Topic Support: API that can read hot topics from any subreddit group offers a powerful capability to tap into real-time discussions and trending topics across different communities on Reddit.

• Code Quality and Stability

  1. SonarQube Integration: The project uses SonarQube for code analysis to ensure high code quality and maintainability.
  2. Automated Testing: Comprehensive unit and integration testing setup using JUnit and MockMVC to verify stability before deployment.

• Docker and Docker Compose Support: web-scrybe is designed to be easily deployable and scalable using Docker and Docker Compose. This makes it an ideal choice for quick prototyping, development, and production deployments.

Current Status

Project Phase: Alpha

Web-Scrybe is currently in the Alpha phase, focusing on core features and stability improvements. Here are the current milestones:

• Backend Development:

  1. The core backend services, built on Spring Boot, are functional.
  2. Integration with Maven for dependency management and continuous integration is complete.

• CI/CD Pipeline:

  1. A GitHub Actions pipeline is set up for automated builds and tests. The pipeline includes:
     1. Maven Build Verification: Ensures the code compiles without errors.
     2. Test Execution: Runs all unit and integration tests to check stability.

• Upcoming Features:

  1. Selenium Based Integration with Medium Platform
  2. Integration with Twitter API
  3. Selenium Based Integration with LinkedIn Platform