Skip to content

TROUBLESHOOTING_AND_MAINTENANCE.md

Latest commit

 

History

History
executable file
·
762 lines (601 loc) · 19 KB

TROUBLESHOOTING_AND_MAINTENANCE.md

File metadata and controls

executable file
·
762 lines (601 loc) · 19 KB
Raw

LuminariMUD Troubleshooting and Maintenance Guide

Quick Deployment Troubleshooting

If you're having issues with a fresh install, try:

# Full redeploy (rebuilds, resets config, reapplies database schema)
./scripts/deploy.sh --auto --init-world

# If you need to bypass MariaDB completely:
./scripts/deploy.sh --skip-db --init-world

Common quick fixes:

  • Build fails: Make sure you ran the full ./scripts/deploy.sh --auto --init-world
  • MUD won't start: Use --init-world to regenerate minimal world data
  • Missing text files: Deploy script recreates them automatically
  • MySQL errors: Temporarily use --skip-db, but remember the default run triggers the in-engine database initializer (database init)

Overview

This guide provides comprehensive troubleshooting procedures, maintenance tasks, and optimization strategies for LuminariMUD server administrators and developers. It covers common issues, diagnostic procedures, and preventive maintenance practices.

PHP Tools Security Audit Summary

Last Audit: January 24, 2025 | Status: COMPLETE | Risk Level: LOW

Security Audit Results

A comprehensive security audit was performed on all PHP tools in the codebase. 18 security vulnerabilities were identified and successfully remediated across 5 PHP files.

Vulnerabilities Addressed

  • Critical (5 Fixed): Code injection, SQL injection, credential exposure
  • High (8 Fixed): XSS, missing authentication, CSRF, input validation
  • Medium (3 Fixed): Information disclosure, missing security headers
  • Low (2 Fixed): HTML structure issues

Files Secured

File Purpose Risk Level Issues Status
bonus_breakdown.php Item bonus analysis HIGH 5 ✅ FIXED
bonuses.php Bonus cross-reference MEDIUM 3 ✅ FIXED
enter_encounter.php Encounter generator CRITICAL 6 ✅ FIXED
enter_hunt.php Hunt creator CRITICAL 5 ✅ FIXED
enter_spell_help.php Help generator HIGH 4 ✅ FIXED

Security Improvements Implemented

  • Authentication & Authorization: Role-based access control
  • Input Validation: Comprehensive validation framework
  • CSRF Protection: Token-based form protection
  • XSS Prevention: Proper HTML escaping throughout
  • Database Security: Environment-based credentials, parameterized queries
  • Security Headers: Complete security header suite

PHP Tools Maintenance Checklist

Monthly Security Tasks

  • Review access logs for suspicious activity
  • Update PHP and dependencies
  • Verify backup procedures
  • Check error logs for anomalies

Quarterly Security Assessment

  • Penetration testing of PHP tools
  • Review and update access controls
  • Security awareness training
  • Update security policies

Annual Security Audit

  • Comprehensive security review
  • Update security controls
  • Review compliance status

Next Security Review Due: July 24, 2025

Common Issues and Solutions

Fresh Installation Issues (New!)

Script Not Found

Problem: ./deploy.sh: No such file or directory Solution: The script has been moved to the scripts directory:

./scripts/deploy.sh --quick --skip-db --init-world

MUD Won't Start - Missing World Data

Problem: SYSERR: opening index file 'world/zon/index': No such file or directory Solution: Initialize minimal world data:

./scripts/deploy.sh --skip-deps --skip-db --init-world

Missing Text Files

Problem: Multiple SYSERR messages about missing text files (news, motd, credits, etc.) Solution: The deploy script now creates all required text files automatically. Just run:

./scripts/deploy.sh --skip-deps --skip-db --init-world

MySQL Configuration Issues

Problem: SYSERR: Malformed line in MySQL configuration Solution: The MySQL config template now has actual default values. Either:

  1. Skip MySQL: ./scripts/deploy.sh --skip-db
  2. Configure MySQL properly:
cp lib/mysql_config_example lib/mysql_config
# Edit lib/mysql_config and change CHANGE_THIS_PASSWORD to your password

Build and Compilation Problems

Missing Dependencies

Symptoms:

  • fatal error: mysql.h: No such file or directory
  • fatal error: gd.h: No such file or directory
  • undefined reference to mysql_* functions

Solutions:

# Ubuntu/Debian (including WSL2)
sudo apt-get update
sudo apt-get install -y build-essential libcrypt-dev libgd-dev libmariadb-dev \
                        libcurl4-openssl-dev libssl-dev mariadb-server \
                        git make cmake autoconf automake libtool pkg-config

# CentOS/RHEL/Fedora
sudo dnf install gcc make mariadb-server mariadb-devel gd-devel \
                 libcrypt-devel openssl-devel libcurl-devel \
                 autoconf automake libtool

# Verify installation
pkg-config --cflags --libs mariadb
pkg-config --cflags --libs gdlib

Compilation Errors

Symptoms:

  • Multiple undefined symbols
  • Conflicting type declarations
  • Syntax errors in header files

Diagnostic Steps:

# Clean build environment
make clean
rm -f depend

# Regenerate dependencies
make depend

# Verbose compilation to see full errors
make VERBOSE=1

# Check for specific error patterns
make 2>&1 | grep -E "(error|undefined|conflicting)"

Common Fixes:

# Fix missing configuration files
cp campaign.example.h campaign.h
cp mud_options.example.h mud_options.h
cp vnums.example.h vnums.h

# Ensure proper permissions
chmod 644 *.h
chmod 755 configure

# Update system headers
sudo apt-get update && sudo apt-get upgrade

Runtime Issues

Server Startup Failures

Port Already in Use:

# Check what's using the port
netstat -tulpn | grep :4000
lsof -i :4000

# Kill existing process
pkill -f circle
# Or kill specific PID
kill -9 <PID>

# Start server on different port
../bin/circle -p 4001

Database Connection Failures:

# Check MySQL service status
sudo systemctl status mysql
sudo systemctl start mysql

# Test database connection manually
mysql -u luminari -p luminari

# Verify database configuration
grep -E "(MYSQL_|DB_)" campaign.h

# Check MySQL error logs
sudo tail -f /var/log/mysql/error.log

Permission Issues:

# Fix executable permissions
chmod +x ../bin/circle

# Fix data directory permissions
chmod -R 755 lib/
chmod -R 644 lib/world/*
chmod 755 lib/world/

# Fix log file permissions
touch ../log/syslog
chmod 666 ../log/syslog

Memory and Performance Issues

Memory Leaks:

# Run with Valgrind
valgrind --leak-check=full --show-leak-kinds=all \
         --log-file=valgrind.log ../bin/circle

# Monitor memory usage
top -p $(pgrep circle)
ps aux | grep circle

# Check for memory growth over time
while true; do
  ps -o pid,vsz,rss,comm -p $(pgrep circle)
  sleep 300
done

Performance Degradation:

# Profile server performance
make PROFILE=-pg
gprof ../bin/circle gmon.out > profile.txt

# Monitor system resources
iostat -x 1
vmstat 1
sar -u 1

# Check database performance
mysql -u luminari -p -e "SHOW PROCESSLIST;"
mysql -u luminari -p -e "SHOW STATUS LIKE 'Slow_queries';"

Database Issues

Connection Problems

Symptoms:

  • "MySQL server has gone away" errors
  • Connection timeouts
  • Authentication failures

Diagnostic Commands:

# Test basic connectivity
telnet localhost 3306

# Check MySQL configuration
mysql -u root -p -e "SHOW VARIABLES LIKE 'max_connections';"
mysql -u root -p -e "SHOW VARIABLES LIKE 'wait_timeout';"

# Monitor connection status
mysql -u root -p -e "SHOW STATUS LIKE 'Connections';"
mysql -u root -p -e "SHOW STATUS LIKE 'Threads_connected';"

Solutions:

-- Increase connection limits
SET GLOBAL max_connections = 500;
SET GLOBAL wait_timeout = 28800;

-- Check user privileges
SHOW GRANTS FOR 'luminari'@'localhost';

-- Reset user password if needed
ALTER USER 'luminari'@'localhost' IDENTIFIED BY 'new_password';
FLUSH PRIVILEGES;

Data Corruption

Detection:

# Check table integrity
mysql -u luminari -p luminari -e "CHECK TABLE player_data;"
mysql -u luminari -p luminari -e "CHECK TABLE room_data;"

# Repair corrupted tables
mysql -u luminari -p luminari -e "REPAIR TABLE player_data;"

# Backup before repairs
mysqldump -u luminari -p luminari > backup_$(date +%Y%m%d).sql

Network and Connectivity Issues

Connection Drops

Symptoms:

  • Players getting disconnected frequently
  • "Connection reset by peer" errors
  • Timeout issues

Diagnostic Steps:

# Check network interface
ifconfig
netstat -i

# Monitor connection quality
ping -c 10 localhost
traceroute localhost

# Check firewall settings
sudo iptables -L
sudo ufw status

# Monitor socket states
ss -tuln | grep :4000
netstat -an | grep :4000

Solutions:

# Adjust TCP settings
echo 'net.ipv4.tcp_keepalive_time = 600' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_keepalive_intvl = 60' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_keepalive_probes = 3' >> /etc/sysctl.conf
sysctl -p

# Configure firewall
sudo ufw allow 4000/tcp
sudo iptables -A INPUT -p tcp --dport 4000 -j ACCEPT

Diagnostic Tools and Procedures

Log Analysis

System Logs

# Monitor server logs in real-time
tail -f ../log/syslog

# Search for specific errors
grep -i "error\|syserr\|crash" ../log/syslog
grep -i "mysql\|database" ../log/syslog

# Analyze log patterns
awk '/SYSERR/ {print $0}' ../log/syslog | sort | uniq -c | sort -nr

# Check system logs
sudo journalctl -u mysql
sudo tail -f /var/log/syslog

Performance Monitoring

# Create monitoring script
#!/bin/bash
# monitor_server.sh
while true; do
  echo "$(date): $(ps -o pid,pcpu,pmem,vsz,rss,comm -p $(pgrep circle))"
  echo "$(date): $(netstat -an | grep :4000 | wc -l) connections"
  sleep 60
done >> server_monitor.log

Debug Mode Operation

Enable Debug Logging

// In campaign.h, enable debug options
#define DEBUG_MODE 1
#define EXTENDED_LOGGING 1

// Recompile with debug symbols
make clean
make CFLAGS="-g -DDEBUG -O0"

GDB Debugging

# Run server under GDB
gdb ../bin/circle
(gdb) set args -p 4000
(gdb) run

# When crash occurs
(gdb) bt
(gdb) info registers
(gdb) print variable_name
(gdb) list

Database Maintenance

Regular Maintenance Tasks

-- Optimize tables monthly
OPTIMIZE TABLE player_data;
OPTIMIZE TABLE room_data;
OPTIMIZE TABLE object_instances;

-- Update table statistics
ANALYZE TABLE player_data;
ANALYZE TABLE room_data;

-- Check for unused space
SELECT table_name,
       ROUND(((data_length + index_length) / 1024 / 1024), 2) AS "Size (MB)",
       ROUND((data_free / 1024 / 1024), 2) AS "Free (MB)"
FROM information_schema.tables
WHERE table_schema = 'luminari';

Backup Procedures

#!/bin/bash
# backup_database.sh
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="../backup"
mkdir -p $BACKUP_DIR

# Full database backup
mysqldump -u luminari -p luminari > $BACKUP_DIR/full_backup_$DATE.sql

# Player data only
mysqldump -u luminari -p luminari player_data > $BACKUP_DIR/players_$DATE.sql

# Compress old backups
find $BACKUP_DIR -name "*.sql" -mtime +7 -exec gzip {} \;

# Remove backups older than 30 days
find $BACKUP_DIR -name "*.sql.gz" -mtime +30 -delete

Performance Optimization

Server Optimization

Memory Management

// In mud_options.h, adjust memory settings
#define MAX_PLAYERS 300
#define MAX_DESCRIPTORS 500
#define LARGE_BUFFERS 1

// Monitor memory pools
void check_memory_pools() {
  log("Character pool: %d allocated, %d free",
      char_pool_allocated, char_pool_free);
  log("Object pool: %d allocated, %d free",
      obj_pool_allocated, obj_pool_free);
}

CPU Optimization

# Set CPU affinity for server process
taskset -c 0,1 ../bin/circle

# Adjust process priority
nice -n -10 ../bin/circle

# Monitor CPU usage
top -p $(pgrep circle)
perf top -p $(pgrep circle)

Database Optimization

Query Optimization

-- Add indexes for frequently queried columns
CREATE INDEX idx_player_name ON player_data(name);
CREATE INDEX idx_player_level ON player_data(level);
CREATE INDEX idx_last_logon ON player_data(last_logon);

-- Analyze slow queries
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 2;
SHOW VARIABLES LIKE 'slow_query_log_file';

Connection Pool Tuning

-- Optimize connection settings
SET GLOBAL max_connections = 200;
SET GLOBAL thread_cache_size = 50;
SET GLOBAL query_cache_size = 64M;
SET GLOBAL innodb_buffer_pool_size = 512M;

Preventive Maintenance

Daily Tasks

#!/bin/bash
# daily_maintenance.sh

# Check disk space
df -h | grep -E "(/$|/var|/tmp)"

# Rotate logs
if [ -f ../log/syslog ]; then
  mv ../log/syslog ../log/syslog.$(date +%Y%m%d)
  touch ../log/syslog
  chmod 666 ../log/syslog
fi

# Check server process
if ! pgrep circle > /dev/null; then
  echo "$(date): Server not running!" >> ../log/maintenance.log
fi

# Database health check
mysql -u luminari -p luminari -e "SELECT COUNT(*) FROM player_data;" > /dev/null
if [ $? -ne 0 ]; then
  echo "$(date): Database connection failed!" >> ../log/maintenance.log
fi

Weekly Tasks

#!/bin/bash
# weekly_maintenance.sh

# Backup database
./backup_database.sh

# Analyze database performance
mysql -u luminari -p luminari -e "
  SELECT table_name, table_rows,
         ROUND((data_length + index_length) / 1024 / 1024, 2) AS size_mb
  FROM information_schema.tables
  WHERE table_schema = 'luminari'
  ORDER BY size_mb DESC;" > ../log/db_analysis_$(date +%Y%m%d).txt

# Check for memory leaks
if command -v valgrind &> /dev/null; then
  echo "Running memory leak check..."
  timeout 300 valgrind --leak-check=summary ../bin/circle -s 2>&1 | \
    grep -E "(definitely lost|possibly lost)" >> ../log/memory_check.log
fi

Monthly Tasks

#!/bin/bash
# monthly_maintenance.sh

# Optimize database tables
mysql -u luminari -p luminari -e "
  OPTIMIZE TABLE player_data;
  OPTIMIZE TABLE room_data;
  OPTIMIZE TABLE object_instances;
  ANALYZE TABLE player_data;
  ANALYZE TABLE room_data;
  ANALYZE TABLE object_instances;"

# Clean old log files
find ../log -name "*.log.*" -mtime +30 -delete
find ../log -name "syslog.*" -mtime +30 -delete

# Update system packages
sudo apt-get update && sudo apt-get upgrade -y

# Check for security updates
sudo unattended-upgrades --dry-run

Emergency Procedures

Server Crash Recovery

#!/bin/bash
# crash_recovery.sh

# Check for core dump
if [ -f core ]; then
  echo "Core dump found, analyzing..."
  gdb ../bin/circle core -batch -ex "bt" -ex "quit" > crash_analysis.txt
  mv core core.$(date +%Y%m%d_%H%M%S)
fi

# Check database integrity
mysql -u luminari -p luminari -e "CHECK TABLE player_data;"

# Restart server with logging
nohup ../bin/circle > ../log/recovery.log 2>&1 &

# Monitor startup
tail -f ../log/recovery.log

Data Recovery

#!/bin/bash
# data_recovery.sh

# Restore from latest backup
LATEST_BACKUP=$(ls -t ../backup/full_backup_*.sql | head -1)
if [ -n "$LATEST_BACKUP" ]; then
  echo "Restoring from $LATEST_BACKUP"
  mysql -u luminari -p luminari < "$LATEST_BACKUP"
fi

# Verify data integrity
mysql -u luminari -p luminari -e "
  SELECT COUNT(*) as player_count FROM player_data;
  SELECT COUNT(*) as room_count FROM room_data;"

Monitoring and Alerting

System Monitoring Script

#!/bin/bash
# system_monitor.sh

# Check server status
if ! pgrep circle > /dev/null; then
  echo "ALERT: LuminariMUD server is not running!" | \
    mail -s "Server Down Alert" admin@example.com
fi

# Check database connectivity
if ! mysql -u luminari -p luminari -e "SELECT 1;" > /dev/null 2>&1; then
  echo "ALERT: Database connection failed!" | \
    mail -s "Database Alert" admin@example.com
fi

# Check disk space
DISK_USAGE=$(df / | awk 'NR==2 {print $5}' | sed 's/%//')
if [ $DISK_USAGE -gt 90 ]; then
  echo "ALERT: Disk usage is ${DISK_USAGE}%!" | \
    mail -s "Disk Space Alert" admin@example.com
fi

# Check memory usage
MEM_USAGE=$(free | awk 'NR==2{printf "%.0f", $3*100/$2}')
if [ $MEM_USAGE -gt 90 ]; then
  echo "ALERT: Memory usage is ${MEM_USAGE}%!" | \
    mail -s "Memory Alert" admin@example.com
fi

Log Monitoring

#!/bin/bash
# log_monitor.sh

# Check for critical errors in the last hour
ERRORS=$(grep -c "SYSERR\|CRASH\|FATAL" ../log/syslog)
if [ $ERRORS -gt 10 ]; then
  echo "ALERT: $ERRORS critical errors found in the last check!" | \
    mail -s "Error Alert" admin@example.com
fi

# Check for database errors
DB_ERRORS=$(grep -c "MySQL\|database.*error" ../log/syslog)
if [ $DB_ERRORS -gt 5 ]; then
  echo "ALERT: $DB_ERRORS database errors found!" | \
    mail -s "Database Error Alert" admin@example.com
fi

Security Compliance and Risk Assessment

Current Security Status

Overall Risk Level: LOW (as of January 24, 2025)

Compliance Status

  • ✅ OWASP Top 10 2021: All vulnerabilities addressed
  • ✅ PHP Security Best Practices: Fully implemented
  • ✅ Modern Security Standards: Compliant
  • ✅ Data Protection: Secure handling implemented

Risk Assessment Matrix

Component Before Audit After Audit Risk Reduction
Exploitability HIGH LOW 75%
Impact SEVERE MINIMAL 90%
Likelihood HIGH LOW 80%
Overall Risk CRITICAL LOW 85%

Security Monitoring Checklist

Daily Monitoring

  • Failed authentication attempts
  • Unusual database query patterns
  • File access anomalies
  • Error rate spikes

Weekly Reviews

  • Access log analysis
  • Security event correlation
  • Performance metrics review
  • Backup verification

Monthly Tasks

  • Security patch updates
  • Access control review
  • Incident response testing
  • Security awareness updates

Incident Response Procedures

Security Incident Detection

# Monitor failed authentication attempts
grep "authentication failed" /var/log/apache2/error.log | tail -20

# Check for suspicious file access
grep "403\|404" /var/log/apache2/access.log | grep -E "\.(env|config|backup)"

# Monitor database connection anomalies
grep "database.*error\|connection.*failed" /var/log/php_errors.log

Response Actions

  1. Immediate: Isolate affected systems
  2. Assessment: Determine scope and impact
  3. Containment: Implement temporary controls
  4. Recovery: Restore normal operations
  5. Lessons Learned: Update security controls

Future Security Recommendations

Short-term (Next 3 months)

  • Implement automated security scanning
  • Set up intrusion detection system
  • Enhance logging and monitoring
  • Conduct penetration testing

Long-term (Next 12 months)

  • Security awareness training program
  • Regular security assessments
  • Incident response plan testing
  • Security metrics and reporting

Next Security Review: July 24, 2025

This troubleshooting guide covers the most common issues and maintenance procedures. For specific technical details, refer to the Technical Documentation Master Index.