Ultra-Fast File Transfer System

A high-performance, asynchronous file transfer system optimised for extensive data with compression, checksums, and robust error handling. Suitable for system-to-system file transfer.

Features

• High Performance: Multi-threaded architecture with configurable worker threads
• Compression: Optional zlib compression to reduce transfer times
• Data Integrity: MD5 checksums for file verification
• Robust Error Handling: Comprehensive retry logic and error recovery
• Progress Monitoring: Real-time transfer statistics and progress updates
• Large File Support: Chunked transfer for files of any size
• Network Optimization: Configurable timeouts and buffer sizes
• Cross-Platform: Works on Windows, Linux, and macOS

Requirements

• Python 3.6 or higher
• No external dependencies (uses only Python standard library)

Installation

1. Clone or download the project files
2. Ensure Python 3.6+ is installed
3. No additional installation required - uses only standard library modules

Usage

Server Mode

Start a server to share files from a directory:

python t.py server --folder /path/to/share --host 0.0.0.0 --port 9999

Server Options:

• --folder: Directory to share (required)
• --host: IP address to bind to (default: 0.0.0.0)
• --port: Port number (default: 9999)
• --workers: Number of worker threads (default: 8)
• --no-compress: Disable compression
• --chunk-size: Chunk size in MB (default: 1)
• --timeout: Socket timeout in seconds (default: 1800)

Client Mode

Connect to a server and download files:

python t.py client --host server_ip --port 9999 --output ./received

Client Options:

• --host: Server IP address (default: localhost)
• --port: Server port number (default: 9999)
• --output: Output directory (default: ./received)
• --workers: Number of worker threads (default: 8)
• --no-compress: Disable compression
• --chunk-size: Chunk size in MB (default: 1)
• --timeout: Socket timeout in seconds (default: 1800)

Connection Test

Test connectivity to a server:

python t.py test --host server_ip --port 9999

Examples

Basic File Transfer

Server (sharing /home/user/documents):

python t.py server --folder /home/user/documents

Client (downloading to ./received):

python t.py client --host 192.168.1.100

High-Performance Transfer

Server with 16 workers and no compression:

python t.py server --folder /data/large_dataset --workers 16 --no-compress --chunk-size 4

Client with matching settings:

python t.py client --host 192.168.1.100 --workers 16 --no-compress --chunk-size 4

Remote Transfer with Custom Settings

Server on specific interface:

python t.py server --folder /backup/files --host 192.168.1.100 --port 8888 --timeout 3600

Client connecting remotely:

python t.py client --host 192.168.1.100 --port 8888 --output /local/backup --timeout 3600

Performance Tuning

Worker Threads

• Default: 8 workers
• CPU-bound: Use number of CPU cores
• I/O-bound: Use 2-4x CPU cores
• Network-limited: Reduce to 4-8 workers

Chunk Size

• Default: 1MB chunks
• Large files: Increase to 4-8MB
• Small files: Keep at 1MB
• Slow networks: Reduce to 512KB

Compression

• Enabled by default for files > 100 bytes
• Disable for pre-compressed files (videos, archives)
• Keep enabled for text, code, and documents

Timeouts

• Default: 30 minutes (1800 seconds)
• Local network: 5-10 minutes
• Internet transfers: 30-60 minutes
• Very large files: Increase as needed

Network Configuration

Firewall Settings

Ensure the specified port is open on both server and client machines:

Linux (iptables):

sudo iptables -A INPUT -p tcp --dport 9999 -j ACCEPT

Windows Firewall:

netsh advfirewall firewall add rule name="File Transfer" dir=in action=allow protocol=TCP localport=9999

Router Configuration

For transfers across networks, configure port forwarding on the server's router to forward the specified port to the server machine.

Troubleshooting

Connection Issues

"Connection refused"

• Server is not running
• Wrong IP address or port
• Firewall blocking connection

"Connection timeout"

• Network connectivity issues
• Server overloaded
• Wrong IP address

"Cannot resolve hostname"

• Invalid hostname/IP address
• DNS resolution issues

Transfer Issues

"Send/Receive timeout"

• Network too slow for large files
• Increase timeout value
• Reduce worker count
• Use smaller chunk sizes

"Checksum mismatch"

• Network corruption
• File changed during transfer
• Storage issues

"Permission denied"

• Insufficient read permissions (server)
• Insufficient write permissions (client)
• Directory doesn't exist

Performance Issues

Slow transfer speeds:

1. Check network bandwidth
2. Adjust worker count
3. Modify chunk size
4. Consider disabling compression
5. Increase socket buffer sizes

High CPU usage:

1. Reduce worker count
2. Disable compression
3. Use larger chunks

High memory usage:

1. Reduce worker count
2. Use smaller chunks
3. Monitor system resources

Technical Details

Architecture

• Multi-threaded: Parallel file processing using ThreadPoolExecutor
• Chunked Transfer: Files split into manageable chunks
• Stateful Protocol: Custom protocol with headers and data sections

Data Flow

1. Server scans directory and builds file manifest
2. Client connects and receives manifest
3. Files are processed in parallel by worker threads
4. Each file is sent with header (metadata) + chunks (data)
5. Client verifies data integrity using checksums
6. Completion signal sent when transfer finishes

File Structure

transfer.py/
├── t.py              # Main transfer system (latest version)
├── t2.py             # Alternative implementation
├── readme.md         # This documentation
└── received/         # Default client download directory