forked from FIRSTMap/scraper
-
Notifications
You must be signed in to change notification settings - Fork 0
GeeII/FIRSTmap-scraper
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Readme file for FIRSTmap scraper #2 This scraper is likely to be a high-maintenance item, as there are a number of hacks to work around incomplete team info from thebluealliance, and differences between place names in different databases. To use this, copy/extract/clone the files to a directory, then run "./get_all_data" Last updated by Gus "GeeTwo" Michel II, 21 Oct 2017 README This file ask_google script to get a lat/lon from a "place" file. Usage: ./ask_google >> geo_cache This file must be run manually; it is not run from get_all_data geo_cache pipe delimited file <location string>|<latitude>|<longitude> this file is manually built from previous calls to ask_google and other sources attribs A list of the team attributes on V3 of thebluealliance API Used by build_teamInfo build_teamInfo Builds teams.js from merged.js and georef data Automatically called directly from get_all_data get_all_data The main script which runs the others (except ask_google) Gets postal data, event data, and team data from the web and builds comps.js and teamInfo.js. Stores intermediate results in data subdirectory, which it creates get_events gets the events list from thebluealliance and builds events.js. Uses lat and lng fields from TBA. Automatically called directly from get_all_data get_lists gets team lists from thebluealliance.com. Automatically called directly from get_all_data get_postal Gets geocoding data from the web and formats some of it. These data files go in the data subdirectory and are used by build_teamInfo Automatically called directly from get_all_data make_latlng extracts the lat/lon from a google place file. called from ask_google merge_lists merges the team "page" lists into one file Automatically called directly from get_all_data TBA-auth A file containing a TBA authorization string. This is not in the repository, as each user is expected to get his/her own. Used by get_lists and get_events. unicodes.ascii Conversion table from unicodes found in team place names to ascii characters; drops diacritical marks and such. Used by build_teamInfo YEAR The current year for purposes of extracting team lists and event list from thebluealliance.com Used by get_lists and merge_lists.
About
Shell and gawk based scraper to build teams.js and competitions.js for FIRSTmap.github.io
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published
Languages
- Shell 100.0%