Skip to content

Commit b77dafa

Browse files
thewebscrapingthewebscraping
authored
Merge pull request #22 from thewebscraping/bugs/url-params
Bugs: fix url params
2 parents 42090ac + fef6ab4 commit b77dafa

23 files changed

+886
-273
lines changed

‎.github/ISSUE_TEMPLATE/bug_report.yml

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
name: Bug report
2+
description: File a bug report for the TLS Requests
3+
title: "[Bug]: "
4+
labels: ["bug"]
5+
body:
6+
7+
- type: markdown
8+
attributes:
9+
value: |
10+
- Write an issue title above.
11+
- Search [open](https://github.com/thewebscraping/tls-requests/issues?q=is%3Aopen) and [closed](https://github.com/thewebscraping/tls-requests/issues?q=is%3Aclosed) issues to ensure it has not already been reported.
12+
13+
- type: input
14+
attributes:
15+
label: TLS Requests version
16+
description: >
17+
Specify the TLS Requests version
18+
placeholder: v1.0.8
19+
validations:
20+
required: true
21+
22+
- type: textarea
23+
attributes:
24+
label: Issue description
25+
description: |
26+
Describe your issue briefly. What doesn't work, and how do you expect it to work instead?
27+
validations:
28+
required: true
29+
30+
- type: textarea
31+
attributes:
32+
label: Steps to reproduce / Code Sample
33+
description: |
34+
List of steps or sample code that reproduces the issue.
35+
validations:
36+
required: true

‎.github/ISSUE_TEMPLATE/config.yml

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
blank_issues_enabled: false

‎.github/ISSUE_TEMPLATE/feature_request.yml

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
name: Feature Request
2+
description: Propose a feature for the TLS Requests
3+
title: "[Feature Request]: "
4+
labels: ["enhancement"]
5+
body:
6+
7+
- type: markdown
8+
attributes:
9+
value: |
10+
- Write a proposal title above.
11+
- Search [open](https://github.com/thewebscraping/tls-requests/issues?q=is%3Aopen) and [closed](https://github.com/thewebscraping/tls-requests/issues?q=is%3Aclosed) proposals to ensure the feature has not already been suggested.
12+
13+
- type: textarea
14+
attributes:
15+
label: Describe the feature / enhancement and how it would improve things
16+
description: |
17+
- Dont overcomplicate
18+
validations:
19+
required: true
20+
21+
- type: textarea
22+
attributes:
23+
label: Describe how your proposal will work, with code and/or pseudo-code
24+
validations:
25+
required: true

‎.github/workflows/ci.yml

+43-30
Original file line numberDiff line numberDiff line change
@@ -1,58 +1,71 @@
1-
name: CI/CD
1+
name: CI
22

33
on:
44
push:
5-
branches: ['master', 'main', 'dev', 'develop']
5+
branches: ['master', 'main']
66

77
pull_request:
88
branches: ['master', 'main', 'dev', 'develop']
99

1010
jobs:
11-
build:
11+
build-on-ubuntu:
1212
runs-on: ubuntu-latest
1313
strategy:
1414
fail-fast: false
1515
max-parallel: 3
1616
matrix:
17-
python-version: ['3.9', '3.10', '3.11', '3.12', '3.13']
17+
python-version: ['3.9', '3.10', '3.11']
1818
steps:
1919
- uses: actions/checkout@v4
2020
- name: Set up Python ${{ matrix.python-version }}
2121
uses: actions/setup-python@v4
2222
with:
2323
python-version: ${{ matrix.python-version }}
2424

25-
- name: Install Dependencies
25+
- name: Install Dependencies, Lint and Tests
2626
run: |
27-
make init
27+
make init-actions
2828
29-
- name: Lint
29+
build-on-mac:
30+
runs-on: macos-latest
31+
strategy:
32+
fail-fast: false
33+
max-parallel: 1
34+
matrix:
35+
python-version: ['3.9']
36+
steps:
37+
- uses: actions/checkout@v4
38+
- name: Set up Python ${{ matrix.python-version }}
39+
uses: actions/setup-python@v4
40+
with:
41+
python-version: ${{ matrix.python-version }}
42+
43+
- name: Install Dependencies, Lint and Tests
3044
run: |
31-
make lint
45+
make init-actions
3246
33-
- name: Tests
47+
- name: Pytest
3448
run: |
3549
make pytest
3650
37-
deploy:
38-
needs: build
39-
runs-on: ubuntu-latest
40-
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
51+
build-on-windows:
52+
runs-on: windows-latest
53+
strategy:
54+
fail-fast: false
55+
max-parallel: 1
56+
matrix:
57+
python-version: ['3.9']
4158
steps:
42-
- uses: actions/checkout@v4
43-
- name: Configure Git Credentials
44-
run: |
45-
git config user.name github-actions[bot]
46-
git config user.email 41898282+github-actions[bot]@users.noreply.github.com
47-
- uses: actions/setup-python@v5
48-
with:
49-
python-version: '3.10'
50-
- run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
51-
- uses: actions/cache@v4
52-
with:
53-
key: mkdocs-material-${{ env.cache_id }}
54-
path: .cache
55-
restore-keys: |
56-
mkdocs-material-
57-
- run: pip install -r requirements-dev.txt
58-
- run: mkdocs gh-deploy --force
59+
- uses: actions/checkout@v4
60+
- name: Set up Python ${{ matrix.python-version }}
61+
uses: actions/setup-python@v4
62+
with:
63+
python-version: ${{ matrix.python-version }}
64+
65+
- name: Install Dependencies, Lint
66+
run: |
67+
make init-actions
68+
69+
- name: Pytest
70+
run: |
71+
make pytest

‎.github/workflows/documentation.yml

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
name: Build Documentation
2+
on:
3+
release:
4+
types: [created]
5+
jobs:
6+
build:
7+
runs-on: ubuntu-latest
8+
strategy:
9+
max-parallel: 1
10+
matrix:
11+
python-version: ['3.9']
12+
steps:
13+
- uses: actions/checkout@v4
14+
- name: Set up Python ${{ matrix.python-version }}
15+
uses: actions/setup-python@v5
16+
with:
17+
python-version: ${{ matrix.python-version }}
18+
- name: Install Dependencies
19+
run: |
20+
python -m pip install --upgrade pip
21+
pip install -r requirements-dev.txt
22+
- name: Configure Git Credentials
23+
run: |
24+
git config user.name github-actions[bot]
25+
git config user.email 41898282+github-actions[bot]@users.noreply.github.com
26+
- uses: actions/cache@v4
27+
with:
28+
key: mkdocs-material-${{ env.cache_id }}
29+
path: .cache
30+
restore-keys: |
31+
mkdocs-material-
32+
- name: Publish Documentation
33+
run: |
34+
echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
35+
mkdocs gh-deploy --force

‎.github/workflows/publish.yml

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
name: Upload Python Package
2+
3+
on:
4+
release:
5+
types: [created]
6+
7+
jobs:
8+
deploy:
9+
name: Publish PyPI
10+
runs-on: ubuntu-latest
11+
steps:
12+
- uses: actions/checkout@main
13+
- name: Set up Python 3.9
14+
uses: actions/setup-python@v4
15+
with:
16+
python-version: '3.9'
17+
- name: Build package
18+
run: |
19+
python -m pip install -U pip build setuptools twine
20+
- name: Publish
21+
env:
22+
TWINE_USERNAME: __token__
23+
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
24+
run: |
25+
python setup.py sdist bdist_wheel
26+
twine upload --skip-existing dist/*

‎CHANGELOG.md

+13
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,19 @@
11
Release History
22
===============
33

4+
1.0.7 (2024-12-14)
5+
-------------------
6+
**Bugfixes:**
7+
8+
- Fix URL.
9+
- Fix Proxy.
10+
11+
1.0.6 (2024-12-12)
12+
-------------------
13+
**Bugfixes:**
14+
15+
- Fix request file (image file, etc).
16+
417
1.0.5 (2024-12-11)
518
-------------------
619
**Bugfixes:**

‎MANIFEST.in

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
include LICENSE
22
include README.md
33
include CHANGELOG.md
4-
recursive-include tls_requests docs Makefile *.md *.rst
4+
recursive-include tls_requests docs Makefile *.md

‎Makefile

+7-9
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,17 @@
11
.PHONY: docs
2-
init:
2+
init-actions:
33
python -m pip install --upgrade pip
44
python -m pip install -r requirements-dev.txt
5+
python -m black tls_requests
6+
python -m isort tls_requests
7+
python -m flake8 tls_requests
58

69
test:
710
tox -p
811
rm -rf *.egg-info
912

1013
test-readme:
11-
python setup.py check --restructuredtext --strict && ([ $$? -eq 0 ] && echo "README.rst and CHANGELOG.md ok") || echo "Invalid markup in README.md or CHANGELOG.md!"
12-
13-
lint:
14-
python -m black tls_requests
15-
python -m isort tls_requests
16-
python -m flake8 tls_requests
14+
python setup.py check --restructuredtext --strict && ([ $$? -eq 0 ] && echo "README.md and CHANGELOG.md ok") || echo "Invalid markup in README.md or CHANGELOG.md!"
1715

1816
pytest:
1917
python -m pytest tests
@@ -28,12 +26,12 @@ publish-test-pypi:
2826
python -m pip install -r requirements-dev.txt
2927
python -m pip install 'twine>=6.0.1'
3028
python setup.py sdist bdist_wheel
31-
twine upload --repository testpypi dist/*
29+
twine upload --repository testpypi --skip-existing dist/*
3230
rm -rf build dist .egg wrapper_tls_requests.egg-info
3331

3432
publish-pypi:
3533
python -m pip install -r requirements-dev.txt
3634
python -m pip install 'twine>=6.0.1'
3735
python setup.py sdist bdist_wheel
38-
twine upload dist/*
36+
twine upload --skip-existing dist/*
3937
rm -rf build dist .egg wrapper_tls_requests.egg-info

‎README.md

+23-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,15 @@
11
# TLS Requests
2-
TLS Requests is a powerful Python library for secure HTTP requests, offering browser-like TLS fingerprinting, anti-bot bypassing, and high performance.
2+
3+
[![GitHub License](https://img.shields.io/github/license/thewebscraping/tls-requests)](https://github.com/thewebscraping/tls-requests/blob/main/LICENSE)
4+
[![CI](https://github.com/thewebscraping/tls-requests/actions/workflows/ci.yml/badge.svg)](https://github.com/thewebscraping/tls-requests/actions/workflows/ci.yml)
5+
[![PyPI - Version](https://img.shields.io/pypi/v/wrapper-tls-requests)](https://pypi.org/project/wrapper-tls-requests/)
6+
![Python Version](https://img.shields.io/badge/Python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue?style=flat)
7+
![Pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)
8+
9+
[![](https://img.shields.io/badge/Pytest-Linux%20%7C%20MacOS%20%7C%20Windows-blue?style=flat&logo=pytest&logoColor=white)](https://github.com/thewebscraping/tls-requests)
10+
[![Documentation](https://img.shields.io/badge/Mkdocs-Documentation-blue?style=flat&logo=MaterialForMkDocs&logoColor=white)](https://thewebscraping.github.io/tls-requests/)
11+
12+
TLS Requests is a powerful Python library for secure HTTP requests, offering browser-like TLS fingerprinting, anti-bot page bypass, and high performance.
313

414
* * *
515

@@ -110,3 +120,15 @@ Explore the full capabilities of TLS Requests in the documentation:
110120

111121

112122
Read the documentation: [**thewebscraping.github.io/tls-requests/**](https://thewebscraping.github.io/tls-requests/)
123+
124+
**Report Issues**
125+
-----------------
126+
127+
Found a bug? Please [open an issue](https://github.com/thewebscraping/tls-requests/issues/).
128+
129+
By reporting an issue you help improve the project.
130+
131+
**Credits**
132+
-----------------
133+
134+
Special thanks to [bogdanfinn](https://github.com/bogdanfinn/) for creating the awesome [tls-client](https://github.com/bogdanfinn/tls-client).

‎docs/advanced/async_client.md

+29-6
Original file line numberDiff line numberDiff line change
@@ -23,14 +23,37 @@ To send asynchronous HTTP requests, use the `AsyncClient`:
2323

2424
```pycon
2525
>>> import asyncio
26-
>>> async def fetch(url):
27-
async with tls_requests.AsyncClient() as client:
28-
r = await client.get(url)
29-
return r
26+
>>> import random
27+
>>> import time
28+
>>> import tls_requests
29+
>>> async def fetch(idx, url):
30+
async with tls_requests.AsyncClient() as client:
31+
rand = random.uniform(0.1, 1.5)
32+
start_time = time.perf_counter()
33+
print("%s: Sleep for %.2f seconds." % (idx, rand))
34+
await asyncio.sleep(rand)
35+
response = await client.get(url)
36+
end_time = time.perf_counter()
37+
print("%s: Took: %.2f" % (idx, (end_time - start_time)))
38+
return response
39+
>>> async def run(urls):
40+
tasks = [asyncio.create_task(fetch(idx, url)) for idx, url in enumerate(urls)]
41+
responses = await asyncio.gather(*tasks)
42+
return responses
3043

31-
>>> r = asyncio.run(fetch("https://httpbin.org/get"))
44+
>>> start_urls = [
45+
'https://httpbin.org/absolute-redirect/1',
46+
'https://httpbin.org/absolute-redirect/2',
47+
'https://httpbin.org/absolute-redirect/3',
48+
'https://httpbin.org/absolute-redirect/4',
49+
'https://httpbin.org/absolute-redirect/5',
50+
]
51+
52+
53+
>>> r = asyncio.run(run(start_urls))
3254
>>> r
33-
<Response [200 OK]>
55+
[<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>]
56+
3457
```
3558

3659
!!! tip

0 commit comments

Comments
 (0)