This commit is contained in:
@@ -86,4 +86,16 @@ The drill restarts MongoDB, API and worker, then force-recreates the stateless a
|
||||
|
||||
Record the host, operator, start/end time, release record checksum, resolved image IDs, preflight output and drill result in the deployment acceptance record. Also verify Docker starts at boot and perform a controlled Debian reboot before Gate A approval. After reboot, run the online preflight and inspect the Workspace health page; do not infer worker health solely from API readiness.
|
||||
|
||||
Keep the `debian-host` and `persistence` records in the restricted evidence store. Start from `deploy/debian-host-acceptance.example.json` and `deploy/persistence-acceptance.example.json`; the examples deliberately fail until every supervised scenario has passed. Bind both records to the expected release identifiers and validate them together:
|
||||
|
||||
```sh
|
||||
python3 deploy/debian_acceptance.py \
|
||||
/secure/acceptance/debian-host.json \
|
||||
/secure/acceptance/persistence.json \
|
||||
--expected-commit FULL_40_CHARACTER_SHA \
|
||||
--expected-release-record-sha256 RELEASE_RECORD_SHA256
|
||||
```
|
||||
|
||||
The validator requires matching archive and image identities, separate operator and reviewer names, the approved host capacity, only ports 80 and 443 recorded as publicly reachable, disabled unaccepted external writes, exact passed scenario sets and no unresolved critical findings. It validates record structure, not the restricted evidence itself. Retain the records, validator output and their checksums outside Git.
|
||||
|
||||
The supplied Docker `json-file` logs are size-capped to protect the small pilot disk, but container recreation removes that container's local log history. Before host acceptance, route GuestOps and Nginx logs to the site's durable restricted logging system, or use a reviewed Docker logging override backed by persistent systemd journal storage. Prove that operators can retrieve pre-recreation logs without exposing request credentials or OAuth callback query strings. Central retention and alerting are completed under milestone 11.
|
||||
|
||||
@@ -113,6 +113,43 @@ systemctl list-timers guestops-backup.timer
|
||||
|
||||
The timer deliberately causes the same brief maintenance interruption as a manual backup. `Persistent=true` runs a missed event after downtime, so choose and communicate the maintenance window. A successful unit only stages an encrypted file locally. Configure an independently monitored off-host transfer, verify the destination checksum, alert on both unit and transfer failure, and test the alert route. Do not add automatic deletion until retention, legal hold and recovery requirements have named owners.
|
||||
|
||||
### Restricted off-host transfer
|
||||
|
||||
The optional transfer service uses rsync over pinned-host SSH. Create a dedicated upload-only account at the restricted store, disable interactive login, agent/port forwarding and access outside the GuestOps backup directory, and keep its private key only in `/etc/guestops` with mode 600. Pin the reviewed server host key; never use `StrictHostKeyChecking=no`.
|
||||
|
||||
Start from `deploy/systemd/backup-transfer.env.example` and store the completed file as `/etc/guestops/backup-transfer.env` with mode 600. The local and remote directories must already exist and remain private. Install and verify the service and its 15-minute retry timer:
|
||||
|
||||
```sh
|
||||
sudo install -m 600 deploy/systemd/backup-transfer.env.example /etc/guestops/backup-transfer.env
|
||||
sudoedit /etc/guestops/backup-transfer.env
|
||||
sudo install -m 644 deploy/systemd/guestops-backup-transfer.service /etc/systemd/system/
|
||||
sudo install -m 644 deploy/systemd/guestops-backup-transfer.timer /etc/systemd/system/
|
||||
sudo systemd-analyze verify /etc/systemd/system/guestops-backup-transfer.service /etc/systemd/system/guestops-backup-transfer.timer
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl start guestops-backup-transfer.service
|
||||
sudo systemctl enable --now guestops-backup-transfer.timer
|
||||
```
|
||||
|
||||
Only files named `guestops-YYYYMMDDTHHMMSSZ.tar.gpg` are eligible. The transfer writes a unique remote partial file, compares the remote and local SHA-256 values, atomically publishes a previously unused final name, verifies it again, and then writes a non-sensitive local marker. An existing remote name is accepted only when its checksum matches. Failed or mismatched transfers are never marked. The service removes local backups older than seven days only when the marker still matches the local checksum; untransferred or changed files are never pruned.
|
||||
|
||||
The restricted store is the durable copy. Configure its independently reviewed retention policy for 35 daily and 12 monthly recovery points. Legal hold must override expiry. The host tool does not delete remote data or enforce remote retention.
|
||||
|
||||
### Zabbix status boundary
|
||||
|
||||
Do not give the Zabbix agent access to Docker, MongoDB credentials or the application owner session. A root-owned systemd probe reads those local sources and atomically publishes aggregate status under `/run/guestops-monitor/status.json`; the agent reads that file only.
|
||||
|
||||
```sh
|
||||
sudo install -m 644 deploy/systemd/guestops-monitor-status.service /etc/systemd/system/
|
||||
sudo install -m 644 deploy/systemd/guestops-monitor-status.timer /etc/systemd/system/
|
||||
sudo install -m 644 deploy/systemd/zabbix-agent-guestops.conf.example /etc/zabbix/zabbix_agentd.d/guestops.conf
|
||||
sudo systemd-analyze verify /etc/systemd/system/guestops-monitor-status.service /etc/systemd/system/guestops-monitor-status.timer
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now guestops-monitor-status.timer
|
||||
sudo systemctl restart zabbix-agent
|
||||
```
|
||||
|
||||
Create dependent Zabbix items from `guestops.status` for HTTPS readiness, certificate days remaining, container state/health, worker heartbeat age, backup age, verified-transfer age, free-disk percentage, persistent journal availability and probe error categories. Alert when the heartbeat is older than three minutes; backup or transfer is older than 30 hours; free disk falls below 25% (warning) or 15% (critical); the certificate has fewer than 30 days (warning) or 14 days (critical); any required container is absent/unhealthy; or the probe/timers fail. Route alerts to the monitoring owner and escalate unacknowledged critical events to the technical owner after 15 minutes. Exercise every trigger and its recovery notification with synthetic conditions.
|
||||
|
||||
Temporary plaintext files are held in private directories and removed on normal completion or exceptions. Process termination or power loss can leave temporary data, stopped services or TTL expiry disabled. After an interrupted run, inspect the dedicated project and remove only its identified abandoned temporary directory after securing any recovery material. Restore the recorded TTL setting (normally true) and restart the services:
|
||||
|
||||
```sh
|
||||
@@ -146,4 +183,20 @@ Restore into new isolated MongoDB and key volumes; preserve the damaged original
|
||||
**A restored database can predate emails, invoices and PMS changes that providers already completed.** Review pending, sending and uncertain records against provider evidence before enabling any worker, including automatic FAQ rules. Do not replay an older approval merely because the restored record says it is pending. Reconcile external effects, validate account sessions and mailbox authorization, and explicitly approve the cutover only after these checks. Rotate credentials if compromise prompted the recovery. Keep the old deployment stopped when enabling the replacement.
|
||||
|
||||
CI exercises a synthetic encrypted backup and isolated restore drill, including actual key decryption and database comparison. A successful CI drill is separate from the required rehearsal on the Debian server with its actual deployment configuration.
|
||||
|
||||
## Milestone 11 acceptance record
|
||||
|
||||
Keep raw backups, restored data, remote paths, host keys, monitoring recipients, screenshots and logs outside Git. Copy `deploy/backup-restore-acceptance.example.json` to the restricted evidence store and replace every placeholder only after completing the supervised exercises. The example deliberately fails.
|
||||
|
||||
The accepted record uses a 24-hour RPO, four-hour RTO, seven-day verified local staging window, 35 daily and 12 monthly off-host recovery points, and the opaque evidence ID `backup-restore`. Bind it to the same release identifiers as the Debian-host and persistence records:
|
||||
|
||||
```sh
|
||||
python3 deploy/backup_restore_acceptance.py \
|
||||
/secure/acceptance/backup-restore.json \
|
||||
--expected-commit FULL_40_CHARACTER_SHA \
|
||||
--expected-release-record-sha256 RELEASE_RECORD_SHA256
|
||||
sha256sum /secure/acceptance/backup-restore.json
|
||||
```
|
||||
|
||||
The validator requires matching local/remote backup checksums, exact immutable image identities, a timed isolated restore, tested retention/legal hold and alert escalation, an image rollback that preserves persistent volumes, return to the approved candidate with external writes disabled, separate independent review, healthy monitoring and no unresolved critical findings. Structural validation does not inspect the restricted evidence or authorize a release by itself.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user