Skip to content

Commit

Permalink
Reorganized folders
Browse files Browse the repository at this point in the history
Signed-off-by: Lukasz Gryglicki <[email protected]>
  • Loading branch information
lukaszgryglicki committed Jan 23, 2019
1 parent d79c4d8 commit 76053e5
Show file tree
Hide file tree
Showing 1,038 changed files with 24 additions and 23 deletions.
22 changes: 11 additions & 11 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@
*~
.venv
*.swp
clearbit_tools/all_clearbit_queries.csv
clearbit_tools/cncf_enriched.csv
clearbit_tools/input_enriched.csv
clearbit_tools/new_round_enriched.csv
clearbit_tools/unknown_emails_enriched.csv
src/clearbit_tools/all_clearbit_queries.csv
src/clearbit_tools/cncf_enriched.csv
src/clearbit_tools/input_enriched.csv
src/clearbit_tools/new_round_enriched.csv
src/clearbit_tools/unknown_emails_enriched.csv
all.log
git.log
all.log.xz
Expand All @@ -16,7 +16,7 @@ database.dump
datelc.csv
header
x
ghusers/*
src/ghusers/*
# Data files
# *.txt
# *.csv
Expand All @@ -35,8 +35,8 @@ geodata.tsv.xz
partial.json
backup.json
*.htm*
/check_spell
git_logs/*.log
git_logs/*.1
git_logs/*.2
/flist.txt
src/check_spell
src/git_logs/*.log
src/git_logs/*.1
src/git_logs/*.2
src/flist.txt
8 changes: 4 additions & 4 deletions ADD_PROJECT.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Add a non-cncf project/org ( project must be opensource ) to generate affiliations for it.
1. Add the developers of your organization/project to be get affiliated in `./developers_affiliations.txt` in the proper format. Now generate new email-map using `./import_affs.sh`, then: `mv email-map cncf-config/email-map`.
1. Add the developers of your organization/project to be get affiliated in `./developers_affiliations.txt` in the proper format. `cd src/`. Now generate new email-map using `./import_affs.sh`, then: `mv email-map cncf-config/email-map`.
For e.g.
```
developer1: email1@xyz, email2@abc, ...
Expand All @@ -12,7 +12,7 @@
2. Clone all repositories of the project at `~/dev/project_name/`. For cloning either you can use `cncf/velocity` project and writing sql query in BigQuery folder or you can create a new shellscript file in `~/dev/cncf/gitdm/` location with name `clone_project_name.sh`.
And just copy paste this code in that file
```
#!/bin/sh
#!/bin/bash
mkdir ~/dev/project_name/ 2>/dev/null
cd ~/dev/project_name || exit 1
git clone github_repo_clone_url_for_your_project1 || exit 1
Expand All @@ -28,7 +28,7 @@
3. To generate `git.log` file, use this command `./all_repos_log.sh ~/dev/project_name/*`. Make it `uniq`.
4. To run `cncf/gitdm` on a generated `git.log` file do: `~/dev/cncf/gitdm/cncfdm.py -i git.log -r "^vendor/|/vendor/|^Godeps/" -R -n -b ./ -t -z -d -D -U -u -o all.txt -x all.csv -a all_affs.csv > all.out`
4. To run `cncf/gitdm` on a generated `git.log` file do: `~/dev/cncf/gitdm/cncfdm.py -i git.log -r "^vendor/|/vendor/|^Godeps/" -R -n -b ./src/ -t -z -d -D -U -u -o all.txt -x all.csv -a all_affs.csv > all.out`
5. To generate human readable text affiliation files: `SKIP_COMPANIES="(Unknown)" ./gen_aff_files.sh`
Expand All @@ -38,7 +38,7 @@
8. `ghusers_partially_cached.sh` will refetch repos metadata and commits and get users data from `github_users.json` so you can save a lot of API points.
9. To update (enchance) github_users.json with new affiliations `./enchance_json.sh`.
9. To update (enchance) `github_users.json` with new affiliations `./enchance_json.sh`.
10. To merge multiple GitHub logins data (for example propagate known affiliation to unknown or not found on the same GitHub login) run: `./merge_github_logins.sh`.
11. Because this can find new affiliations you can now use `./import_from_github_users.sh` to import back from `github_users.json` and then restart from step 3.
Expand Down
1 change: 1 addition & 0 deletions FORBIDDEN_DATA.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
If you do not want your personal data like names and/or emails to be listed you can do the following.

- Clone cncf/gitdm locally
- `cd src/`
- Run `./add_forbidden_data.rb '[email protected]'` or `./add_forbidden_data.rb 'Your Name' '[email protected]' 'anything you want to remove from repo'`
- Program will generate SHA256 hashes of data provided from command line arguments and add them to `cncf-config/forbidden.csv` file.
- Create PR with updated `cncf-config/forbidden.csv` file. That way your sensitive data won't be visible in a PR.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Only the [Developers affiliations list](https://github.com/cncf/gitdm/blob/maste

The [Company Developers list](https://github.com/cncf/gitdm/blob/master/company_developers.txt) is a computed derivative of the first list.

Other files used for affiliations are [email map file](https://github.com/cncf/gitdm/blob/master/cncf-config/email-map) and [github users](https://github.com/cncf/gitdm/blob/master/github_users.json) file.
Other files used for affiliations are [email map file](https://github.com/cncf/gitdm/blob/master/src/cncf-config/email-map) and [github users](https://github.com/cncf/gitdm/blob/master/src/github_users.json) file.

# Removing affiliations

Expand Down
4 changes: 2 additions & 2 deletions SYNC.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Syncing new affiliations

Make sure that you don't have different case email duplicates in `cncf-config/email-map`: `./lower_unique.sh cncf-config/email-map`.
Make sure that you don't have different case email duplicates in `src/cncf-config/email-map`: `cd src`, `./lower_unique.sh cncf-config/email-map`.

1. If you generated new email-map using `./import_affs.sh`, then: `mv email-map cncf-config/email-map`
2. To generate `git.log` file and make sure it includes all orgs used by `devstats` use cncf/devstats\'s `GHA2DB_PROJECTS_OVERRIDE="+cncf,+opencontainers,+istio,+spinnaker,+knative,+linux" PG_PASS=... GHA2DB_EXTERNAL_INFO=1 GHA2DB_PROCESS_REPOS=1 ./get_repos` and then final command line it generates. Make it `uniq`.
3. Update `repos.txt` to contain all repositories returned by the above command.
4. To run `cncf/gitdm` on a generated `git.log` file do: `~/dev/alt/gitdm/cncfdm.py -i git.log -r "^vendor/|/vendor/|^Godeps/" -R -n -b ./ -t -z -d -D -A -U -u -o all.txt -x all.csv -a all_affs.csv > all.out`
4. To run `cncf/gitdm` on a generated `git.log` file do: `cd src/; ~/dev/alt/gitdm/src/cncfdm.py -i git.log -r "^vendor/|/vendor/|^Godeps/" -R -n -b ./ -t -z -d -D -A -U -u -o all.txt -x all.csv -a all_affs.csv > all.out`
5. To generate human readable text affiliation files: first run: `./enchance_all_affs.sh` then: `SKIP_COMPANIES="(Unknown)" ./gen_aff_files.sh`.
6. If updating via `ghusers.sh` or `ghusers_cached.sh` (step 6) - run `generate_actors.sh` too.
7. Consider `./ghusers_cached.sh` or `./ghusers.sh` (if you run this, then copy result json somewhere and get 0-committers from previous version to save GH API points). Sometimes you should just run `./ghusers.sh` without cache.
Expand Down
2 changes: 0 additions & 2 deletions import_affs.sh

This file was deleted.

File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion final_analysis.sh β†’ src/final_analysis.sh
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,4 @@ echo 'Stacked charts'

echo 'Done'

echo "vim all.txt all_affs.csv correlations.txt github_users.json aliaser.txt progress_report.txt developers_affiliations.txt company_developers.txt per_dirs/all_stats.csv"
echo "vim all.txt all_affs.csv correlations.txt github_users.json aliaser.txt progress_report.txt ../developers_affiliations.txt ../company_developers.txt per_dirs/all_stats.csv"
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
4 changes: 2 additions & 2 deletions gen_aff_files.rb β†’ src/gen_aff_files.rb
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ def gen_aff_files(csv_file)
hdr = "# This file is derived from developers_affiliations.txt and so should not be edited directly.\n"
hdr += "# If you see an error, please update developers_affiliations.txt and this file will be fixed\n"
hdr += "# when regenerated.\n"
File.write 'company_developers.txt', hdr + t
File.write '../company_developers.txt', hdr + t

t = ''
names.keys.sort.each do |dev_name|
Expand Down Expand Up @@ -161,7 +161,7 @@ def gen_aff_files(csv_file)
hdr += "# If you see your name with asterisk '*' sign - it means that\n"
hdr += "# multiple affiliations were found for you with different email addresses.\n"
hdr += "# Please merge all of them into one then.\n"
File.write 'developers_affiliations.txt', hdr + t
File.write '../developers_affiliations.txt', hdr + t

if wrongs.count > 0
wrongs = wrongs.uniq
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 2 additions & 0 deletions src/import_affs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
#!/bin/sh
ruby import_affs.rb ../developers_affiliations.txt ../company_developers.txt
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loading

0 comments on commit 76053e5

Please sign in to comment.