diff --git a/scripts/readme.md b/scripts/readme.md index 9c1d7e4..457147b 100644 --- a/scripts/readme.md +++ b/scripts/readme.md @@ -1,252 +1,507 @@ -# NetBird Backup & Restore Scripts +# Netbird Docker Compose - Backup & Restore Guide -Comprehensive backup and restore solution for NetBird Docker deployments with Zitadel integration. - -## Overview - -These scripts provide automated backup and restore functionality for your NetBird installation, including: -- Docker volumes (database, management data, certificates) -- Configuration files (docker-compose, environment variables) -- Traefik reverse proxy configuration -- Zitadel authentication data - -## Prerequisites - -- Docker and Docker Compose installed -- NetBird deployed via Docker Compose -- Sufficient disk space for backups -- Root or sudo access for Docker operations - -## Files - -### 1. `backup-netbird.sh` -Creates timestamped backups of your complete NetBird installation. - -### 2. `restore-netbird.sh` -Restores NetBird from a backup archive. - -## Installation - -1. Place both scripts in your NetBird installation directory: - ```bash - /home/Dejan/Docker/Netbird-compose/ - ``` - -2. Make scripts executable: - ```bash - chmod +x backup-netbird.sh restore-netbird.sh - ``` - -3. Ensure the backup directory exists: - ```bash - mkdir -p /home/Dejan/Docker/Netbird-compose/backup - ``` - -## Usage - -### Creating a Backup - -Run the backup script from your NetBird directory: - -```bash -cd /home/Dejan/Docker/Netbird-compose -./backup-netbird.sh -``` - -**What happens during backup:** -1. Creates backup directory if it doesn't exist -2. Stops the NetBird Docker stack -3. Exports all Docker volumes to compressed archives -4. Archives configuration files -5. Restarts the NetBird stack -6. Creates a timestamped backup file: `netbird_backup_YYYY-MM-DD_HH-MM-SS.tar.gz` - -**Backup includes:** -- Docker volumes: `netbird_zdb_data`, `netbird_management`, `netbird_zitadel_certs` -- Configuration: `docker-compose.yml`, `*.env` files -- NetBird config: `management.json`, `turnserver.conf` -- Authentication: `machinekey` directory -- Reverse proxy: `traefik-stack` directory -- The backup script itself - -### Restoring from Backup - -#### Option 1: Restore latest backup (automatic) -```bash -cd /home/Dejan/Docker/Netbird-compose -./restore-netbird.sh -``` - -#### Option 2: Restore specific backup -```bash -./restore-netbird.sh netbird_backup_2024-11-24_14-30-00.tar.gz -``` - -Or with full path: -```bash -./restore-netbird.sh /home/Dejan/Docker/Netbird-compose/backup/netbird_backup_2024-11-24_14-30-00.tar.gz -``` - -**What happens during restore:** -1. Validates backup file exists -2. Extracts backup to temporary directory -3. Stops current NetBird stack -4. Restores all Docker volumes -5. Restores configuration files -6. Cleans up temporary files -7. Starts NetBird stack with restored data - -## Backup Schedule - -### Manual Backups -Run before major changes: -- System updates -- Configuration modifications -- Docker Compose upgrades - -### Automated Backups (Recommended) - -Add to crontab for automatic daily backups: - -```bash -crontab -e -``` - -Add this line for daily backup at 2 AM: -```cron -0 2 * * * /home/Dejan/Docker/Netbird-compose/backup-netbird.sh >> /var/log/netbird-backup.log 2>&1 -``` - -For weekly backups (Sunday at 3 AM): -```cron -0 3 * * 0 /home/Dejan/Docker/Netbird-compose/backup-netbird.sh >> /var/log/netbird-backup.log 2>&1 -``` - -## Backup Management - -### View Available Backups -```bash -ls -lh /home/Dejan/Docker/Netbird-compose/backup/ -``` - -### Delete Old Backups -Keep only last 7 backups: -```bash -cd /home/Dejan/Docker/Netbird-compose/backup -ls -1t netbird_backup_*.tar.gz | tail -n +8 | xargs rm -f -``` - -### Backup Retention Script -Create `cleanup-old-backups.sh`: -```bash -#!/bin/bash -BACKUP_DIR="/home/Dejan/Docker/Netbird-compose/backup" -KEEP_LAST=7 -cd "$BACKUP_DIR" -ls -1t netbird_backup_*.tar.gz | tail -n +$((KEEP_LAST + 1)) | xargs -r rm -f -echo "Cleaned up old backups, kept last $KEEP_LAST" -``` - -## Troubleshooting - -### Backup Script Fails -- **Insufficient disk space**: Check available space with `df -h` -- **Docker not running**: Start Docker service: `sudo systemctl start docker` -- **Permission denied**: Run with sudo: `sudo ./backup-netbird.sh` - -### Restore Script Fails -- **Backup file not found**: Verify path and filename -- **Volume restore fails**: Check Docker is running and volumes are accessible -- **Stack won't start**: Check logs: `docker compose logs` - -### Common Issues - -**Stack doesn't start after restore:** -```bash -# Check container logs -docker compose logs -f - -# Verify volumes exist -docker volume ls | grep netbird - -# Try manual restart -docker compose down -docker compose up -d -``` - -**Configuration mismatch:** -```bash -# Verify restored files -ls -la /home/Dejan/Docker/Netbird-compose/ - -# Check environment variables -cat .env -``` - -## Important Notes - -- **Downtime**: Both backup and restore scripts stop the NetBird stack temporarily -- **Backup size**: Varies based on database size and user count (typically 100MB-1GB) -- **Storage**: Ensure adequate backup storage space -- **Testing**: Test restore process in a development environment first -- **Security**: Backup files contain sensitive data - secure appropriately - -## Backup Security - -Protect your backups: - -1. **Restrict permissions:** - ```bash - chmod 700 /home/Dejan/Docker/Netbird-compose/backup - chmod 600 /home/Dejan/Docker/Netbird-compose/backup/*.tar.gz - ``` - -2. **Encrypt backups (optional):** - ```bash - gpg --symmetric --cipher-algo AES256 netbird_backup_*.tar.gz - ``` - -3. **Off-site backup:** - ```bash - # Example: rsync to remote server - rsync -avz backup/ user@backup-server:/backups/netbird/ - ``` - -## Migration Guide - -To migrate NetBird to a new server: - -1. **On old server:** Create backup - ```bash - ./backup-netbird.sh - ``` - -2. **Transfer backup file:** - ```bash - scp backup/netbird_backup_*.tar.gz user@new-server:/tmp/ - ``` - -3. **On new server:** Install Docker, Docker Compose, and NetBird structure - -4. **Restore backup:** - ```bash - ./restore-netbird.sh /tmp/netbird_backup_*.tar.gz - ``` - -5. **Update DNS/IP** if necessary - -## Support - -For issues or questions: -- Check NetBird documentation: https://docs.netbird.io -- Review Docker Compose logs: `docker compose logs` -- Verify volume integrity: `docker volume inspect ` - -## Version History - -- **v1.0** - Initial backup and restore scripts with full volume and config support +Complete documentation for backing up and restoring your Netbird Docker Compose installation. --- -**Created for NetBird deployment at:** `/home/Dejan/Docker/Netbird-compose/` -**Last updated:** November 2024 \ No newline at end of file +## 📋 Table of Contents + +- [Overview](#overview) +- [Prerequisites](#prerequisites) +- [Backup Script](#backup-script) +- [Restore Script](#restore-script) +- [Automated Backups](#automated-backups) +- [Best Practices](#best-practices) +- [Troubleshooting](#troubleshooting) + +--- + +## 🎯 Overview + +This package includes two powerful scripts to manage your Netbird installation: + +- **`backup-netbird.sh`** - Creates timestamped backups of your entire Netbird setup +- **`restore-netbird.sh`** - Restores your Netbird installation from any backup + +Both scripts handle Docker services automatically and include comprehensive error handling. + +--- + +## ⚙️ Prerequisites + +- Root or sudo access +- Docker and Docker Compose installed +- Netbird installation at `/home/Dejan/Docker/Netbird-compose` +- Sufficient disk space for backups + +--- + +## 💾 Backup Script + +### What Gets Backed Up + +The backup script saves: +- All configuration files (`.env`, `dashboard.env`, `zitadel.env`, etc.) +- Docker Compose configuration (`docker-compose.yml`) +- SSL certificates and keys (`acme.json`, `machinekey/`) +- Management configuration (`management.json`) +- Traefik configuration files +- All custom scripts +- Zitadel logs + +**Excluded from backups:** +- The `backup/` directory itself +- `.git/` directory (if present) +- Old `.tar.gz` files + +### How to Use + +#### Basic Usage + +```bash +# Navigate to your Netbird directory +cd /home/Dejan/Docker/Netbird-compose + +# Make the script executable (first time only) +chmod +x backup-netbird.sh + +# Run the backup +./backup-netbird.sh +``` + +#### What Happens During Backup + +1. **Validation** - Checks for root access and directory existence +2. **Service Stop** - Gracefully stops all Docker Compose services +3. **Archive Creation** - Creates a compressed tar.gz archive with timestamp +4. **Service Restart** - Brings all services back online +5. **Verification** - Shows backup size and location + +### Backup Output + +Backups are saved in the `backup/` directory with timestamps: + +``` +backup/ +├── netbird-backup-20241127_143022.tar.gz +├── netbird-backup-20241127_180530.tar.gz +└── netbird-backup-20241128_020000.tar.gz +``` + +**Filename format:** `netbird-backup-YYYYMMDD_HHMMSS.tar.gz` + +### Backup Script Features + +- ✅ Automatic service management +- ✅ Timestamped backups +- ✅ Colored console output +- ✅ Size reporting +- ✅ Error handling at each step +- ✅ Automatic backup directory creation +- ✅ List of recent backups after completion + +### Optional: Automatic Cleanup + +To keep only the last 5 backups, uncomment these lines in the script: + +```bash +KEEP_BACKUPS=5 +print_msg "Cleaning old backups, keeping last $KEEP_BACKUPS..." +cd "$BACKUP_DIR" +ls -t netbird-backup-*.tar.gz | tail -n +$((KEEP_BACKUPS + 1)) | xargs -r rm +``` + +--- + +## 🔄 Restore Script + +### How to Use + +#### Basic Usage + +```bash +# Make the script executable (first time only) +chmod +x restore-netbird.sh + +# Restore from a specific backup +./restore-netbird.sh netbird-backup-20241127_143022.tar.gz +``` + +#### Alternative Ways to Specify Backup + +```bash +# Using just the filename (searches in backup/ directory) +./restore-netbird.sh netbird-backup-20241127_143022.tar.gz + +# Using relative path +./restore-netbird.sh backup/netbird-backup-20241127_143022.tar.gz + +# Using absolute path +./restore-netbird.sh /home/Dejan/Docker/Netbird-compose/backup/netbird-backup-20241127_143022.tar.gz +``` + +#### List Available Backups + +```bash +# Show usage and list available backups +./restore-netbird.sh +``` + +### What Happens During Restore + +1. **Validation** - Verifies backup file exists and is valid +2. **Preview** - Shows backup contents (first 20 files) +3. **Safety Backup** - Creates a pre-restore backup (recommended) +4. **Confirmation** - Asks for your approval before proceeding +5. **Service Stop** - Stops all running Docker services +6. **File Removal** - Removes current files (except backup/ directory) +7. **Extraction** - Extracts all files from backup archive +8. **Permissions** - Sets correct file permissions +9. **Verification** - Checks that critical files were restored +10. **Service Start** - Brings all services back online +11. **Status Report** - Shows running services status + +### Interactive Prompts + +The restore script will ask you two questions: + +``` +Do you want to create a safety backup before restoring? (recommended) [Y/n] +``` +**Recommended:** Press Enter or type `Y` to create a safety backup + +``` +Are you sure you want to restore from this backup? [y/N] +``` +**Type `y`** to proceed with the restore + +### Restore Script Features + +- ✅ Multiple backup file path formats supported +- ✅ Automatic backup file search +- ✅ Pre-restore safety backup +- ✅ Backup content preview +- ✅ Interactive confirmation prompts +- ✅ Automatic permission fixing +- ✅ Critical file verification +- ✅ Service status display +- ✅ Comprehensive error handling + +### Safety Backups + +When you create a safety backup, it's saved as: + +``` +backup/pre-restore-backup-YYYYMMDD_HHMMSS.tar.gz +``` + +This allows you to roll back if something goes wrong during restore. + +--- + +## ⏰ Automated Backups + +### Using Cron + +Set up automatic backups using crontab: + +```bash +# Edit root's crontab +sudo crontab -e + +# Add one of these lines: + +# Daily backup at 2:00 AM +0 2 * * * /home/Dejan/Docker/Netbird-compose/backup-netbird.sh >> /var/log/netbird-backup.log 2>&1 + +# Every 6 hours +0 */6 * * * /home/Dejan/Docker/Netbird-compose/backup-netbird.sh >> /var/log/netbird-backup.log 2>&1 + +# Weekly on Sunday at 3:00 AM +0 3 * * 0 /home/Dejan/Docker/Netbird-compose/backup-netbird.sh >> /var/log/netbird-backup.log 2>&1 +``` + +### Cron Schedule Examples + +| Schedule | Cron Expression | Description | +|----------|----------------|-------------| +| Every day at 2 AM | `0 2 * * *` | Daily backup during low traffic | +| Every 12 hours | `0 */12 * * *` | Twice daily backups | +| Every 6 hours | `0 */6 * * *` | Four times daily | +| Weekly (Sunday 3 AM) | `0 3 * * 0` | Weekly backup | +| Every hour | `0 * * * *` | Hourly backup (generates many files) | + +### Monitoring Automated Backups + +```bash +# View backup log +tail -f /var/log/netbird-backup.log + +# List recent backups +ls -lht /home/Dejan/Docker/Netbird-compose/backup/ | head -10 + +# Check cron is running +sudo systemctl status cron + +# View crontab entries +sudo crontab -l +``` + +--- + +## 🎯 Best Practices + +### Backup Strategy + +1. **Regular Backups** + - Set up daily automated backups via cron + - Create manual backups before major changes + +2. **Retention Policy** + - Keep at least 7 daily backups + - Keep at least 4 weekly backups + - Store critical backups off-site + +3. **Before Updates** + - Always backup before updating Docker images + - Always backup before modifying configuration + - Always backup before system maintenance + +4. **Test Restores** + - Periodically test restore process + - Verify backup integrity + - Document restore procedures + +### Storage Management + +```bash +# Check backup directory size +du -sh /home/Dejan/Docker/Netbird-compose/backup/ + +# Find backups older than 30 days +find /home/Dejan/Docker/Netbird-compose/backup/ -name "*.tar.gz" -mtime +30 + +# Remove backups older than 30 days (use with caution!) +find /home/Dejan/Docker/Netbird-compose/backup/ -name "*.tar.gz" -mtime +30 -delete +``` + +### Security + +- Keep backups in a secure location +- Consider encrypting sensitive backups +- Restrict backup file permissions: `chmod 600 *.tar.gz` +- Store off-site copies for disaster recovery + +--- + +## 🔧 Troubleshooting + +### Common Issues + +#### "Failed to stop services" + +**Problem:** Docker services won't stop + +**Solution:** +```bash +# Force stop all containers +docker-compose down --remove-orphans + +# Or force remove +docker-compose rm -f -s -v +``` + +#### "Backup file not found" + +**Problem:** Restore script can't find the backup + +**Solution:** +```bash +# Check backup directory +ls -la /home/Dejan/Docker/Netbird-compose/backup/ + +# Use absolute path +./restore-netbird.sh /full/path/to/backup.tar.gz +``` + +#### "Permission denied" + +**Problem:** Script can't be executed + +**Solution:** +```bash +# Make scripts executable +chmod +x backup-netbird.sh restore-netbird.sh + +# Run with sudo +sudo ./backup-netbird.sh +``` + +#### "No space left on device" + +**Problem:** Insufficient disk space for backup + +**Solution:** +```bash +# Check disk space +df -h + +# Clean old backups +rm /home/Dejan/Docker/Netbird-compose/backup/netbird-backup-OLD*.tar.gz + +# Clean Docker system +docker system prune -a +``` + +### Verify Backup Integrity + +```bash +# Test if backup archive is valid +tar -tzf backup/netbird-backup-20241127_143022.tar.gz > /dev/null + +# List backup contents +tar -tzf backup/netbird-backup-20241127_143022.tar.gz | less + +# Extract specific file from backup (without full restore) +tar -xzf backup/netbird-backup-20241127_143022.tar.gz docker-compose.yml +``` + +### Manual Restore Steps + +If the restore script fails, you can restore manually: + +```bash +# 1. Stop services +cd /home/Dejan/Docker/Netbird-compose +docker-compose down + +# 2. Backup current state (optional) +tar -czf /tmp/emergency-backup.tar.gz . + +# 3. Extract backup +tar -xzf backup/netbird-backup-YYYYMMDD_HHMMSS.tar.gz + +# 4. Fix permissions +chmod +x *.sh +chmod 600 acme.json +chmod 777 machinekey/ + +# 5. Start services +docker-compose up -d +``` + +--- + +## 📊 Script Workflow Diagrams + +### Backup Process + +``` +Start + ↓ +Check root access + ↓ +Create backup directory (if needed) + ↓ +Stop Docker Compose services + ↓ +Create tar.gz archive with timestamp + ↓ +Restart Docker Compose services + ↓ +Display backup info and size + ↓ +List recent backups + ↓ +End +``` + +### Restore Process + +``` +Start + ↓ +Check root access + ↓ +Validate backup file exists + ↓ +Show backup contents preview + ↓ +Prompt: Create safety backup? (Y/n) + ↓ +Create pre-restore backup (if yes) + ↓ +Prompt: Confirm restore? (y/N) + ↓ +Stop Docker Compose services + ↓ +Remove current files + ↓ +Extract backup archive + ↓ +Set file permissions + ↓ +Verify critical files + ↓ +Start Docker Compose services + ↓ +Display service status + ↓ +End +``` + +--- + +## 📝 File Permissions Reference + +After restore, these permissions are automatically set: + +| File/Directory | Permission | Purpose | +|----------------|-----------|---------| +| `*.sh` | `755` (rwxr-xr-x) | Executable scripts | +| `machinekey/` | `777` (rwxrwxrwx) | Machine key storage | +| `acme.json` | `600` (rw-------) | SSL certificates | +| Other files | `644` (rw-r--r--) | Configuration files | + +--- + +## 🆘 Support + +### Getting Help + +1. Check the troubleshooting section above +2. Verify Docker and Docker Compose are running +3. Check system logs: `journalctl -xe` +4. Check Docker logs: `docker-compose logs` + +### Useful Commands + +```bash +# Check script status +./backup-netbird.sh +./restore-netbird.sh + +# View Docker services +docker-compose ps + +# View Docker logs +docker-compose logs -f + +# Check disk space +df -h + +# List all backups with sizes +ls -lh backup/ +``` + +--- + +## 📄 License + +These scripts are provided as-is for use with Netbird Docker Compose installations. + +--- + +**Last Updated:** November 2024 +**Version:** 1.0 +**Compatible with:** Netbird Docker Compose installations \ No newline at end of file