Troubleshooting

This guide covers common issues and their solutions for LabID production deployments.

Common Issues

Database Connection Errors

Symptoms: - Django application fails to start - Error messages about database connectivity - FATAL: password authentication failed in logs

Diagnostics:

# Check PostgreSQL service status
sudo systemctl status postgresql

# Test database connection
sudo -u labid psql -h localhost -U labid -d labid_prod -c "SELECT 1;"

# Check database configuration
sudo -u labid -i
cd /opt/labid/labid
python manage.py check --database default

Solutions:

1. Verify database service is running:

   sudo systemctl start postgresql
   sudo systemctl enable postgresql
   

2. Check connection parameters in .env file:

   # Verify /opt/labid/labid/.env contains correct settings
   DJANGO_DATABASE_URL=postgres://labid:password@localhost:5432/labid_prod
   

3. Reset database user password:

   sudo -u postgres psql
   ALTER USER labid WITH PASSWORD 'new_password';
   \q
   

4. Check firewall rules:

   sudo ufw status
   # Ensure PostgreSQL port (5432) is accessible if using remote database
   

Celery Workers Not Processing Tasks

Symptoms: - Tasks queue up but don't get processed - No worker processes visible - Tasks timeout or remain in pending state

Diagnostics:

# Check Celery worker status
sudo systemctl status labid-celery-worker labid-celery-worker_archive labid-celery-worker_ingestion

# Check RabbitMQ status
sudo systemctl status rabbitmq-server

# Monitor Celery queues
sudo -u labid -i
cd /opt/labid/labid
python manage.py shell -c "
from celery import Celery
app = Celery('labid')
app.config_from_object('django.conf:settings', namespace='CELERY')
inspect = app.control.inspect()
print('Active workers:', inspect.active())
"

Solutions:

1. Restart Celery services:

   sudo systemctl restart labid-celery-worker labid-celery-worker_archive labid-celery-worker_ingestion labid-celery-beat
   

2. Check RabbitMQ service:

   sudo systemctl restart rabbitmq-server
   
   # Verify RabbitMQ queues
   sudo rabbitmqctl list_queues
   

3. Verify Celery configuration:

   # Check DJANGO_CELERY_BROKER_URL in .env
   DJANGO_CELERY_BROKER_URL=amqp://guest:guest@localhost:5672//
   

4. Clear stuck queues:

   sudo systemctl stop labid-celery-*
   sudo rabbitmqctl purge_queue celery
   sudo systemctl start labid-celery-*
   

Static Files Not Loading

Symptoms: - Admin interface appears unstyled - Missing CSS/JavaScript files - 404 errors for static resources

Diagnostics:

# Check static files collection
sudo -u labid -i
cd /opt/labid/labid
python manage.py collectstatic --dry-run

# Check Nginx static file configuration
sudo nginx -t
curl -I https://your-labid.com/static/admin/css/base.css

Solutions:

1. Collect static files:

   sudo -u labid -i
   cd /opt/labid/labid
   python manage.py collectstatic --noinput
   

2. Check Nginx configuration:

   # Ensure correct static file path in Nginx config
   location ^~ /static/ {
       alias /opt/labid/labid/static_collection/;
   }
   

3. Verify file permissions:

   sudo chown -R labid:labid /opt/labid/labid/static_collection/
   sudo chmod -R 644 /opt/labid/labid/static_collection/
   

4. Restart Nginx:

   sudo systemctl restart nginx
   

UI Shows "Server is down"

Symptoms: - Red "Server is down" banner on the LabID home page immediately after login - Browser console shows: Blocked loading mixed active content "http://..." - Network tab shows API requests going to the wrong hostname, scheme, or port (e.g. http://internal-host:8000)

Cause: The UI was compiled with the wrong VITE_API_URL baked in. Because the UI is a compiled single-page application, this value is embedded in the JavaScript bundle at build time — a mismatch (wrong scheme, internal hostname, or direct gunicorn port) causes the browser to block or mis-route API calls even if the backend itself is running correctly.

Diagnostics:

# 1. Check what API URL the browser is actually using
#    Open DevTools → Network tab, reload the page, look at where /api/ requests go.

# 2. Verify nginx is proxying the API correctly end-to-end
curl -s https://your-labid.com/api/v2/version

# 3. Check what URL is baked into the current build
grep -r "VITE_API_URL\|api/v2" /opt/labid-ui/dist/assets/*.js | head -5

Fix:

1. Edit /opt/labid-ui/.env and set the correct public HTTPS URL:

   VITE_API_URL=https://your-labid.com/api/v2
   VITE_DEPLOYMENT_URL=https://your-labid.com
   

!!! danger "Mixed content" If the site is served over https:// the API URL must also start with https://. Using http:// or an internal hostname causes browsers to block the request as mixed active content.

1. Rebuild the UI:

   sudo -u labid -i
   cd /opt/labid-ui
   npm run build
   

2. Reload Nginx if needed:

   sudo systemctl reload nginx
   

Email Notifications Not Working

Symptoms: - No email notifications sent - SMTP connection errors in logs - Email authentication failures

Diagnostics:

# Test email configuration
sudo -u labid -i
cd /opt/labid/labid
python manage.py sendtestemail admin@your-labid.com

# Check email settings in Django shell
python manage.py shell -c "
from django.conf import settings
print('EMAIL_HOST:', settings.EMAIL_HOST)
print('EMAIL_PORT:', settings.EMAIL_PORT)
print('EMAIL_USE_TLS:', settings.EMAIL_USE_TLS)
"

Solutions:

1. Verify SMTP settings in .env:

   DJANGO_EMAIL_HOST=smtp.example.com
   DJANGO_EMAIL_PORT=587
   DJANGO_EMAIL_USE_TLS=True
   DJANGO_EMAIL_HOST_USER=your-email@example.com
   DJANGO_EMAIL_HOST_PASSWORD=your-password
   

2. Check firewall rules:

   # Ensure SMTP ports (587, 465, 25) are not blocked
   sudo ufw status
   telnet smtp.example.com 587
   

3. Test SMTP connection manually:

   python -c "
   import smtplib
   server = smtplib.SMTP('smtp.example.com', 587)
   server.starttls()
   server.login('username', 'password')
   server.quit()
   print('SMTP connection successful')
   "
   

Log Locations

Application Logs

• Django application: /var/log/labid/django.log
• Gunicorn: journalctl -u labid-gunicorn
• Celery workers: journalctl -u labid-celery-worker
• Celery beat: journalctl -u labid-celery-beat

System Logs

• Nginx: /var/log/nginx/access.log and /var/log/nginx/error.log
• PostgreSQL: /var/log/postgresql/postgresql-*.log
• RabbitMQ: journalctl -u rabbitmq-server
• System: journalctl -xe

Log Analysis Commands

# Monitor real-time Django logs
tail -f /var/log/labid/django.log

# Search for errors in the last hour
journalctl -u labid-gunicorn --since "1 hour ago" | grep -i error

# Check Nginx error patterns
grep -E "(50[0-9]|40[0-9])" /var/log/nginx/access.log | tail -20

# Monitor database slow queries
sudo tail -f /var/log/postgresql/postgresql-*.log | grep "duration:"

Support Resources

• User Documentation
• GitLab Issues
• Community Slack
• Development Documentation