notebookvb

2026-05-23 06:54:13 +02:00
parent bfa049e0c2
commit 13065aab94
11 changed files with 677 additions and 237 deletions
@@ -0,0 +1,175 @@
+# MongoDB Backup & Restore - Tower / Tower1
+
+Kompletní řešení pro zálohování a obnovu MongoDB na Unraid.  
+**Ověřeno v praxi** na dvou Unraid serverech Tower (192.168.1.76) a Tower1 (192.168.1.50).
+
+---
+
+## 🏗️ Infrastruktura
+
+| Server | IP | Role | MongoDB kontejner |
+|--------|-----|------|-------------------|
+| Tower | 192.168.1.76 | Primární (zdroj záloh) | `MongoDB` |
+| Tower1 | 192.168.1.50 | Záloha (restore cíl) | `MongoDB` |
+
+**SSH:** Tower1 ↔ Tower mají vzájemně nahrané SSH klíče (bez hesla).
+
+---
+
+## 📋 Co backup zachovává
+
+- ✅ Všechny kolekce a dokumenty (BSON formát)
+- ✅ Všechny indexy (unique, compound, sparse, text)
+- ✅ Metadata kolekcí (validators, options)
+- ✅ Komprimace gzipem (`--archive --gzip`)
+
+> **Poznámka k admin DB:** MongoDB je provozováno bez autentifikace, admin DB obsahuje pouze `system.version` (1 dokument, 360 B). Není nutné ji zálohovat pro obnovení dat.
+
+---
+
+## 📁 Struktura záloh
+
+```
+/mnt/user/Backup/Critical/MongoDBBackup/tower/
+├── admin/
+│   └── 2026-05-23_0532/
+│       └── admin.archive.gz  (~360 B)
+├── edc/
+│   └── 2026-05-23_0532/
+│       └── edc.archive.gz    (~436 KB)
+└── ...
+```
+
+**Dostupné přes Windows:** `\\tower\Backup\Critical\MongoDBBackup\`
+
+---
+
+## ⚙️ Konfigurace backup skriptu
+
+Soubor: `mongodbbackup_with_gzip.sh`
+
+| Proměnná | Hodnota | Popis |
+|----------|---------|-------|
+| `CONTAINER_NAME` | `MongoDB` | Jméno Docker kontejneru |
+| `BASE_PATH` | `/mnt/user/Backup/Critical/MongoDBBackup` | Cesta k zálohám |
+| `KEEP_DAYS` | `3` | Smazat zálohy starší než N dní |
+| `KEEP_COUNT` | `3` | Ponechat max N posledních záloh |
+| `WHAT_TO_BACKUP` | `("admin" "edc" ...)` | Seznam DB k zálohování |
+
+**Klíčové:** Backup používá `mongodump --archive --gzip` který streamuje přes stdout - nepotřebuje volume mount v kontejneru.
+
+---
+
+## 🔄 Backup
+
+Skript je registrován v Unraid User Scripts jako **MONGODB_BACKUP** na Tower.
+
+```
+TOOLS → USER SCRIPTS → MONGODB_BACKUP → RUN SCRIPT
+```
+
+**Cesta na Unraidu:**
+```
+/boot/config/plugins/user.scripts/scripts/MONGODB_BACKUP/script
+```
+
+**Očekávaný výstup:**
+```
+Processing database: edc
+SUCCESS: edc backed up successfully
+Dump size: 436K
+Cleaning up old backups for edc...
+```
+
+---
+
+## 🔁 Restore
+
+### Metoda 1: Ze sdíleného share (doporučeno, ověřeno)
+
+Záloha je dostupná na Tower1 přes SMB - `/mnt/user/Backup/Critical/MongoDBBackup/`.  
+Pokud není automaticky namountováno, zkopíruj soubor na Tower1 share `#test`:
+
+```bash
+# Z Windows:
+Copy-Item "\\tower\Backup\Critical\MongoDBBackup\tower\edc\2026-05-23_0532\edc.archive.gz" "\\tower1\#test\edc.archive.gz"
+```
+
+Restore skript na Tower1 (User Scripts → **MONGODB_RESTORE**):
+```bash
+docker exec -i MongoDB mongorestore --archive --gzip --drop < /mnt/user/#test/edc.archive.gz
+```
+
+### Metoda 2: SSH stream Tower → Tower1 (ověřeno, funguje)
+
+```bash
+ssh -o StrictHostKeyChecking=no root@192.168.1.76 \
+  'cat /mnt/user/Backup/Critical/MongoDBBackup/tower/edc/2026-05-23_0532/edc.archive.gz' \
+  | docker exec -i MongoDB mongorestore --archive --gzip --drop
+```
+
+**Cesta skriptu na Tower1:**
+```
+/boot/config/plugins/user.scripts/scripts/MONGODB_RESTORE/script
+```
+
+---
+
+## ⏱️ Reálné časy (ověřeno)
+
+| Databáze | Velikost zálohy | Dokumentů | Čas restore |
+|----------|----------------|-----------|-------------|
+| admin | 360 B | 1 | < 1 min |
+| edc | 436 KB | 4563 | ~7 minut |
+
+> **Pozor:** Restore vypadá že "stojí" - ale probíhá indexace. Například `edc.queries_snapshots` (2091 docs, 5 indexů včetně compound unique) trvá několik minut. **Nezavírat okno!**
+
+---
+
+## ✅ Výsledek úspěšného restore (edc)
+
+```
+finished restoring edc.MDD3003_DateofVisit (381 documents, 0 failures)
+finished restoring edc.queries_snapshots (2091 documents, 0 failures)
+finished restoring edc.queries (2091 documents, 0 failures)
+restoring indexes for collection edc.queries from metadata
+  - queryId_1 (sparse, unique)
+  - queryStatus_1
+  - openedDate_1
+  - site.number_1
+  - subject.label_1
+restoring indexes for collection edc.queries_snapshots from metadata
+  - queryId_1_snapshotDate_1 (compound, unique)
+  - snapshotDate_1
+  - queryStatus_1
+  - site.number_1
+  - subject.label_1
+4563 document(s) restored successfully. 0 document(s) failed to restore.
+Exit: True
+```
+
+---
+
+## 🔧 Troubleshooting
+
+| Problém | Příčina | Řešení |
+|---------|---------|--------|
+| Restore "stojí" minuty | Indexace probíhá | Počkej, nezavírej okno |
+| `--dumpDbUsersAndRoles` error | MongoDB bez auth nemá users/roles | Flag odstraněn ze skriptů |
+| `tar: Cannot stat` | mongodump zapisuje do kontejneru, ne na host | Použít `--archive --gzip` (stdout stream) |
+| SSH stream visí | SSH drží stdin otevřený | Přidat `-n` flag nebo použít lokální soubor |
+| `Permission denied` | Práva na složky | `chmod 755 /mnt/user/Backup/Critical/MongoDBBackup` |
+
+---
+
+## 📝 Historie vývoje a naučené lekce
+
+1. **mongodump `--out` do kontejneru nefunguje** - kontejner zapisuje do svého filesystému, host soubory nevidí. Řešení: `--archive --gzip` streamuje přes stdout.
+
+2. **`--dumpDbUsersAndRoles` selže** na MongoDB bez auth - odstraněno.
+
+3. **tar cesta** - mongodump vytváří `DUMP_DIR/DB_NAME/kolekce.bson`, ne přímo `DUMP_DIR/kolekce.bson`.
+
+4. **Restore je pomalý** - indexace compound a unique indexů trvá. Normální chování, není to chyba.
+
+5. **SSH stream** - funguje ale při pomalé indexaci může paramiko timeout. Bezpečnější je lokální soubor přes share.
@@ -0,0 +1,88 @@
+#!/bin/bash
+set -x
+
+# ==========================================================
+# CONFIGURATION
+# ==========================================================
+CONTAINER_NAME="MongoDB"
+MONGO_HOST="localhost"
+MONGO_PORT="27017"
+
+UNRAID_NAME="tower"
+BASE_PATH="/mnt/user/Backup/Critical/MongoDBBackup"
+
+KEEP_DAYS=3
+KEEP_COUNT=3   # počet posledních záloh k udržení (mimo time-based)
+
+# Databases to back up (+ admin DB pro credentials a roles)
+WHAT_TO_BACKUP=("admin" "edc")
+
+# ==========================================================
+# START
+# ==========================================================
+echo "Starting scheduled backup for: ${WHAT_TO_BACKUP[*]}"
+START_TS=$(date '+%Y-%m-%d %H:%M:%S')
+
+for DB_NAME in "${WHAT_TO_BACKUP[@]}"; do
+    echo "------------------------------------------"
+    echo "Processing database: $DB_NAME"
+
+    DATE=$(date +%Y-%m-%d_%H%M)
+    FINAL_PATH="$BASE_PATH/$UNRAID_NAME/$DB_NAME/$DATE"
+    mkdir -p "$FINAL_PATH"
+
+    DUMP_FILE="$FINAL_PATH/$DB_NAME.archive.gz"
+    ERR_FILE="$FINAL_PATH/$DB_NAME.err"
+
+    # ======================================================
+    # DUMP + GZIP (streamuje přes stdout, nepotřebuje volume mount)
+    # Zachovává: indexy, metadata, validators, all collections
+    # ======================================================
+    docker exec "$CONTAINER_NAME" mongodump \
+        --host="$MONGO_HOST" \
+        --port="$MONGO_PORT" \
+        --db="$DB_NAME" \
+        --archive \
+        --gzip \
+        --numParallelCollections=4 \
+        > "$DUMP_FILE" 2> "$ERR_FILE"
+
+    EXIT_CODE=$?
+
+    # ======================================================
+    # VALIDATION
+    # ======================================================
+    if [ $EXIT_CODE -eq 0 ] && [ -s "$DUMP_FILE" ] && [ ! -s "$ERR_FILE" ]; then
+        echo "SUCCESS: $DB_NAME backed up successfully"
+        echo "Dump size: $(du -h "$DUMP_FILE" | cut -f1)"
+        rm -f "$ERR_FILE"
+    else
+        echo "ERROR: Backup failed for database: $DB_NAME"
+        echo "Exit code: $EXIT_CODE"
+        echo "Dump file:"
+        ls -lh "$DUMP_FILE" 2>/dev/null || echo "  (not created)"
+        echo "Error output:"
+        [ -s "$ERR_FILE" ] && cat "$ERR_FILE" || echo "  (no stderr output)"
+    fi
+
+    # ======================================================
+    # CLEANUP OLD BACKUPS
+    # 1. Smaž zálohy starší než KEEP_DAYS dnů
+    # 2. Z toho co zbyde, ponech jen posledních KEEP_COUNT
+    # ======================================================
+    echo "Cleaning up old backups for $DB_NAME..."
+    find "$BASE_PATH/$UNRAID_NAME/$DB_NAME" \
+        -mindepth 1 -maxdepth 1 -type d -mtime +$KEEP_DAYS \
+        -exec rm -rf {} \;
+
+    # Nech jen posledních KEEP_COUNT záloh
+    ls -td "$BASE_PATH/$UNRAID_NAME/$DB_NAME"/*/ 2>/dev/null \
+        | tail -n +$((KEEP_COUNT + 1)) \
+        | xargs -r rm -rf
+
+done
+
+echo "------------------------------------------"
+echo "All backup tasks completed at $(date)"
+echo "Started at: $START_TS"
+set +x
@@ -0,0 +1,103 @@
+#!/bin/bash
+set -x
+
+# ==========================================================
+# CONFIGURATION
+# ==========================================================
+CONTAINER_NAME="MongoDB"
+MONGO_HOST="localhost"
+MONGO_PORT="27017"
+
+# Cesta k záloze kterou chceš obnovit
+# Např: /mnt/user/Backup/Critical/MongoDBBackup/tower
+BACKUP_BASE_PATH="$1"
+
+# ==========================================================
+# VALIDATION
+# ==========================================================
+if [ -z "$BACKUP_BASE_PATH" ]; then
+    echo "Usage: $0 <backup_base_path>"
+    echo "Example: $0 /mnt/user/Backup/Critical/MongoDBBackup/tower"
+    exit 1
+fi
+
+if [ ! -d "$BACKUP_BASE_PATH" ]; then
+    echo "ERROR: Backup path does not exist: $BACKUP_BASE_PATH"
+    exit 1
+fi
+
+echo "Starting MongoDB restore from: $BACKUP_BASE_PATH"
+START_TS=$(date '+%Y-%m-%d %H:%M:%S')
+
+# ==========================================================
+# RESTORE PROCESS
+# ==========================================================
+
+# Najdi nejnovější backupy pro každou DB
+# Struktura: $BACKUP_BASE_PATH/DB_NAME/YYYY-MM-DD_HHMM/DB_NAME.archive.gz
+
+for DB_DIR in "$BACKUP_BASE_PATH"/*; do
+    if [ ! -d "$DB_DIR" ]; then
+        continue
+    fi
+
+    DB_NAME=$(basename "$DB_DIR")
+    echo "------------------------------------------"
+    echo "Processing database: $DB_NAME"
+
+    # Najdi nejnovější backup pro tuto DB
+    LATEST_BACKUP=$(ls -td "$DB_DIR"/*/ 2>/dev/null | head -1)
+
+    if [ -z "$LATEST_BACKUP" ]; then
+        echo "WARNING: No backup found for $DB_NAME"
+        continue
+    fi
+
+    BACKUP_FILE="$LATEST_BACKUP/${DB_NAME}.archive.gz"
+    ERR_FILE="$LATEST_BACKUP/${DB_NAME}.restore.err"
+
+    if [ ! -f "$BACKUP_FILE" ]; then
+        echo "ERROR: Backup archive not found: $BACKUP_FILE"
+        continue
+    fi
+
+    echo "Found backup: $BACKUP_FILE"
+    echo "Backup date: $(stat -c %y "$BACKUP_FILE" | cut -d' ' -f1-2)"
+    echo "Backup size: $(du -h "$BACKUP_FILE" | cut -f1)"
+
+    # ======================================================
+    # RESTORE TO MONGODB (streamuje přes stdin z hosta)
+    # Restores: indexy, metadata, validators, all collections
+    # ======================================================
+    echo "Restoring $DB_NAME to MongoDB..."
+    docker exec -i "$CONTAINER_NAME" mongorestore \
+        --host="$MONGO_HOST" \
+        --port="$MONGO_PORT" \
+        --archive \
+        --gzip \
+        --drop \
+        --numParallelCollections=4 \
+        < "$BACKUP_FILE" \
+        2> "$ERR_FILE"
+
+    RESTORE_EXIT=$?
+
+    # ======================================================
+    # VALIDATION
+    # ======================================================
+    if [ $RESTORE_EXIT -eq 0 ]; then
+        echo "SUCCESS: $DB_NAME restored successfully"
+        rm -f "$ERR_FILE"
+    else
+        echo "ERROR: Restore failed for database: $DB_NAME"
+        echo "Exit code: $RESTORE_EXIT"
+        echo "Error output:"
+        [ -s "$ERR_FILE" ] && cat "$ERR_FILE" || echo "  (no stderr output)"
+    fi
+
+done
+
+echo "------------------------------------------"
+echo "All restore tasks completed at $(date)"
+echo "Started at: $START_TS"
+set +x
@@ -0,0 +1,133 @@
+#!/bin/bash
+set -x
+
+# ==========================================================
+# VERIFY BACKUP INTEGRITY
+# Kontroluje že se indexy, metadata, a všechny kolekce
+# správně obnovily
+# ==========================================================
+
+CONTAINER_NAME="MongoDB"
+MONGO_HOST="localhost"
+MONGO_PORT="27017"
+
+# Databases to verify
+WHAT_TO_VERIFY=("admin" "fio" "torrents" "OrdinaceDropBoxBackup" "medevio" "kanboard" "medicus" "studie" "puzzle")
+
+echo "Starting backup integrity verification..."
+echo "=========================================="
+
+ALL_OK=true
+
+for DB_NAME in "${WHAT_TO_VERIFY[@]}"; do
+    echo ""
+    echo "Verifying database: $DB_NAME"
+    echo "------------------------------------------"
+
+    # ======================================================
+    # 1. CHECK IF DB EXISTS
+    # ======================================================
+    DB_EXISTS=$(docker exec "$CONTAINER_NAME" mongosh \
+        --host="$MONGO_HOST" \
+        --port="$MONGO_PORT" \
+        --quiet \
+        --eval "db.adminCommand('listDatabases').databases.filter(d => d.name === '$DB_NAME').length > 0")
+
+    if [ "$DB_EXISTS" != "true" ]; then
+        echo "❌ ERROR: Database $DB_NAME does not exist!"
+        ALL_OK=false
+        continue
+    fi
+    echo "✅ Database exists: $DB_NAME"
+
+    # ======================================================
+    # 2. COUNT COLLECTIONS
+    # ======================================================
+    COLLECTION_COUNT=$(docker exec "$CONTAINER_NAME" mongosh \
+        --host="$MONGO_HOST" \
+        --port="$MONGO_PORT" \
+        --quiet \
+        --eval "use $DB_NAME; db.getCollectionNames().length")
+
+    echo "✅ Collections count: $COLLECTION_COUNT"
+
+    # ======================================================
+    # 3. LIST ALL COLLECTIONS
+    # ======================================================
+    echo "   Collections:"
+    docker exec "$CONTAINER_NAME" mongosh \
+        --host="$MONGO_HOST" \
+        --port="$MONGO_PORT" \
+        --quiet \
+        --eval "use $DB_NAME; db.getCollectionNames().forEach(c => print('     - ' + c))"
+
+    # ======================================================
+    # 4. CHECK INDEXES FOR EACH COLLECTION
+    # ======================================================
+    echo "   Indexes per collection:"
+    docker exec "$CONTAINER_NAME" mongosh \
+        --host="$MONGO_HOST" \
+        --port="$MONGO_PORT" \
+        --quiet \
+        --eval "
+use $DB_NAME;
+db.getCollectionNames().forEach(collName => {
+    const indexes = db[collName].getIndexes();
+    print('     ' + collName + ': ' + indexes.length + ' indexes');
+    indexes.forEach(idx => {
+        print('       - ' + JSON.stringify(idx.key));
+    });
+});
+"
+
+    # ======================================================
+    # 5. CHECK COLLECTION STATS (document count, size)
+    # ======================================================
+    echo "   Collection stats:"
+    docker exec "$CONTAINER_NAME" mongosh \
+        --host="$MONGO_HOST" \
+        --port="$MONGO_PORT" \
+        --quiet \
+        --eval "
+use $DB_NAME;
+db.getCollectionNames().forEach(collName => {
+    const stats = db[collName].stats();
+    print('     ' + collName + ':');
+    print('       Documents: ' + stats.count);
+    print('       Size: ' + (stats.size / 1024 / 1024).toFixed(2) + ' MB');
+});
+"
+
+    # ======================================================
+    # 6. VERIFY USERS/ROLES (if admin DB)
+    # ======================================================
+    if [ "$DB_NAME" = "admin" ]; then
+        echo "   Users and Roles:"
+        USER_COUNT=$(docker exec "$CONTAINER_NAME" mongosh \
+            --host="$MONGO_HOST" \
+            --port="$MONGO_PORT" \
+            --quiet \
+            --eval "use admin; db.system.users.find().count()")
+
+        echo "     System users: $USER_COUNT"
+
+        ROLE_COUNT=$(docker exec "$CONTAINER_NAME" mongosh \
+            --host="$MONGO_HOST" \
+            --port="$MONGO_PORT" \
+            --quiet \
+            --eval "use admin; db.system.roles.find().count()")
+
+        echo "     System roles: $ROLE_COUNT"
+    fi
+
+done
+
+echo ""
+echo "=========================================="
+if [ "$ALL_OK" = true ]; then
+    echo "✅ All verifications passed!"
+else
+    echo "❌ Some verifications failed!"
+    exit 1
+fi
+set +x