Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Discourse #45

Open
wants to merge 27 commits into
base: working
Choose a base branch
from
Open

Discourse #45

wants to merge 27 commits into from

Conversation

jbaicoianu
Copy link

Discourse crawler

@jbaicoianu
Initial commit of Discourse crawler and sample data
1309c5e
@jbaicoianu
Removed debug prints
6a76f5b
@jbaicoianu
Updated crawl site list
79feea3
@jbaicoianu
- Use target_dir for crawl output
229f19c 
- Improved crawler output, and summary data format
- Log crawl errors (dns_error, http_error, json_error)
@jbaicoianu
Added "summary" and "topics" crawl types
ea40840
@jbaicoianu
Crawl list cleanup
c5a247c
@jbaicoianu
Profiler support, set retry option based on crawl type
b9f6b8a
@jbaicoianu
Improved crawl data structure, initial support for writing direct to …
3aab776 
…tarballs, code parallelism and optimizations
@jbaicoianu
Improved caching to make resuming a crawl more reliable and faster, i…
19dc7d0 
…mproved stats generation

Fixed crawl problems with certain site structures
Tweaked priorities to try to avoid drowning out the scheduler queue with a single domain
@jbaicoianu
Tweaked spider defaults, add quick summarize step for stats
959229c
@jbaicoianu
Fix for problematic URL structures, improved category caching, first-…
5f098ed 
…pass attempt at verify function
@jbaicoianu
Reverted change to main codepile.py file
ddecfc5
@jbaicoianu
Whitespace
3c6dd02
@jbaicoianu
First pass at discourse preprocessor to convert raw json into lm_data…
9d414fa 
…format jsonl
@jbaicoianu
Added processor support, added command line options
dc026ce
@jbaicoianu
Updated dependencies
6072e64
@jbaicoianu
Big improvements to pipeline flow, with examples
5ef703a
@jbaicoianu
Improved usage examples
73083fe
@jbaicoianu
Use temp sitepath for processing
50c77c1
@jbaicoianu
Improved topic processor output
c123b55
@jbaicoianu
Improved subdir handling during processing
aeca218
@jbaicoianu
Improved path handling, log missing posts and exceptions
344d5c5
@jbaicoianu
Handle sites with missing topic slugs
10de5f1
@jbaicoianu
Improved metadata, support for paginated topics
419e48a
@jbaicoianu
Disable telnet server
1dbbca3
@jbaicoianu
Pass site parameter to get_additional_posts function
438e11d
@jbaicoianu
Handle licenses, use jsonl files instead of loose files
145e77e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jbaicoianu