Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve processing efficiency in check_parked.sh #409

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Improve processing efficiency in check_parked.sh #409

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e2c8eab
check_parked: avoid subshell in loop
friendly-bits Jan 12, 2025
0877484
check_parked: strip subdomains with gawk
friendly-bits Jan 12, 2025
40bfa64
check_parked.sh: only count lines of the first split file
jarelllama Jan 13, 2025
3244539
check_parked: remove_parked: improve awk command
friendly-bits Jan 13, 2025
9d40d28
Merge branch 'main' of https://github.com/friendly-bits/Scam-Blocklist
friendly-bits Jan 13, 2025
9978bda
check_parked: remove_parked: improve awk arr name
friendly-bits Jan 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 28 additions & 5 deletions scripts/check_parked.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,14 +148,16 @@ find_parked() {
if [[ "$1" == 'x00' ]]; then
local track=true
local count=1
local lines
lines="$(wc -l < "$1")"
fi

# Loop through domains
while read -r domain; do
if [[ "$track" == true ]]; then
if (( count % 100 == 0 )); then
printf "[progress] Analyzed %s%% of domains\n" \
"$(( count * 100 / $(wc -l < "$1") ))"
"$(( count * 100 / lines ))"
fi

(( count++ ))
Expand Down Expand Up @@ -200,10 +202,26 @@ remove_parked() {
mv temp "$SUBDOMAINS"

# Strip subdomains from parked domains
while read -r subdomain; do
sed -i "s/^${subdomain}\.//" parked.tmp
done < "$SUBDOMAINS_TO_REMOVE"
sort -u parked.tmp -o parked.tmp
gawk '
# store lines from subdomains_to_remove as keys in array "subdom"
NR==FNR { subdom[$0]; next }
# process parked.tmp
{
# split current line by "." and store strings in array "arr"
n=split($0,arr,".")
# if "arr" has more than 1 element,
# and string in "subdom" matches 1st element of array "arr", remove subdomain from the line
if (n>1 && arr[1] in subdom) {
regex="^" arr[1] "."
sub(regex,"")
}
# print out the line
print $0
}
' "$SUBDOMAINS_TO_REMOVE" parked.tmp |
sort -u > parked-removed-subdomains.tmp

mv parked-removed-subdomains.tmp parked.tmp

# Remove parked domains from the various files
for file in "$RAW" "$RAW_LIGHT" "$ROOT_DOMAINS"; do
Expand Down Expand Up @@ -232,6 +250,11 @@ cleanup() {

# Entry point

command -v "gawk" 1>/dev/null || {
echo "Error: gawk not found." >&2
exit 1
}

trap cleanup EXIT

$FUNCTION --format-all
Expand Down