In order to legally create a search, Bing Search API under the Bing Cognitive Services was used. This takes use of Bing's search engine to return the results. Currently, the only way of using Google as the search engine is to scrape it manually which won't be sustainable in the long run because of Google's detection to bots.
Firebase was used as a means of data persistence. While a relational database is an option, Firebase was a primary choice to reduce costs of hosting the app.
To retrieve pages returned from the search, axios was used. Initially, puppeteer should be used but it takes more than 512 MB of memory. Check next/puppeteer branch for the implementation. Puppeteer was replaced with axios in order to reduce server cost.
- Setup your environment with the following variables:
ACCESS_TOKEN- facebook app access tokenVERIFY_TOKEN- random string for verification purposesAPP_SECRET- facebook app secretBING_ACCESS_KEY- Bing cognitive services API keyFIREBASE_API_KEY- Firebase API keyFIREBASE_DATABASE_URL- Firebase database URL
- Install node dependencies.
- Start the development server with
yarn start.
The bot cannot open or has limited support with:
- pages that are against Facebook's Community Standards
- client-side rendered pages
- pages with Shadow DOM
- sites not using semantic HTML
If you have any questions/suggestions, feel free to contact me at [email protected].