Integrate backup and monitoring operations

This commit is contained in:
mathew 2026-10-01 10:28:25 +01:00
parent 86e548f2fc
commit 158d21ca04
9 changed files with 319 additions and 14 deletions

View File

@ -21,7 +21,7 @@ This summary explains what each milestone delivers and where it currently stands
| 8 | Operational readiness tooling | Release verification, diagnostics, encrypted backup, restore, preflight, and persistence tools. | **Implemented; acceptance required.** The tools must still be run against the exact release on the Debian host. |
| 9 | Gitea and reproducible releases | A checksummed source package, Ansible-controlled installation, recorded image identities, retained artifacts, and approval tagging. | **In progress.** The runner-free deterministic packager is implemented and the Gitea Action is removed; the release-line identity must be confirmed, then the package, Ansible installation evidence, and approval record must be retained. |
| 10 | Debian deployment and persistence | Secure Debian/Compose deployment, HTTPS, persistent database and key volumes, and reboot/recreation proof. | **In progress.** The verified Ansible handoff now covers commit-bound installation, boot services, Nginx validation, listener restrictions and public HTTPS; privileged installation, firewall review, controlled reboot and supervised persistence evidence remain open. |
| 11 | Backups, monitoring, and recovery | Scheduled encrypted backups, verified off-host transfer, Zabbix monitoring, restore, and rollback rehearsal. | **In progress.** Repository tooling is ready; installation and timed operational evidence are blocked until Milestones 9 and 10 pass. |
| 11 | Backups, monitoring, and recovery | Scheduled encrypted backups, verified off-host transfer, Zabbix monitoring, restore, and rollback rehearsal. | **In progress.** The Ansible operations handoff now installs validated systemd units, public-key-only backup support, transfer retry and restricted Zabbix status; secret provisioning, durable-log confirmation, manual backup, timed restore, rollback and independent evidence remain open. |
| 12 | Google mailbox acceptance | End-to-end Gmail consent, import, recovery, reviewed sending, reconciliation, and revocation evidence. | **In progress.** The runbook and validator exist; the live synthetic-data exercise and independent review remain outstanding. |
| 13 | Rezlynx/Guestline adapter | The real PMS provider adapter, mappings, idempotency, reconciliation, and ambiguous-write handling. | **Planned.** Provider contract and sandbox access are still required before implementation and acceptance. |
| 14 | Payment links and status | The real payment-provider integration, webhooks, expiry, replay protection, and reconciliation. | **Planned.** The provider path and sandbox acceptance plan still need to be confirmed and completed. |
@ -61,7 +61,7 @@ This summary explains what each milestone delivers and where it currently stands
| 8 | Operational readiness tooling | A | Implemented / acceptance required | Backup, restore, release, and diagnostic tooling is promoted to local `main`; execute it on the actual Debian host and retain evidence. |
| 9 | Gitea and reproducible releases | A | In progress | `deploy/package_source.py` now packages only an explicit committed ref, verifies matched application versions, produces deterministic gzip output and a SHA-256 source record, and refuses overwrite. Hand that package to a version-selected Ansible playbook following the CMS/CMSFront pattern. Ansible must verify and install it, build commit-tagged images, record their immutable IDs, and deploy without a Gitea runner. Retain the package/install evidence off-host and resolve the release-line/tag identity before approval; the existing `0.1.0` tag remains attached to the foundation release. |
| 10 | Debian deployment and persistence | A | In progress | Compose uses separate named database and shared key volumes, private host configuration, loopback-only API access and bounded logs. The Ansible handoff verifies the source on both controller and host, enables Docker/Nginx at boot, installs and validates the reviewed proxy, rejects exposed API/MongoDB listeners, and requires trusted public HTTPS before selecting the release. Run it on the provisioned Debian host, review the firewall, complete the confirmation-gated persistence drill and controlled reboot, and retain independent evidence. |
| 11 | Backups, monitoring, and recovery | A | In progress | Encrypted backup and isolated restore tooling now includes opt-in systemd scheduling, checksum-verified rsync transfer, a restricted Zabbix status boundary, guarded local retention and a release-bound acceptance validator. Install and test it on Debian, configure the restricted store and alerts, name operational/review owners, and retain independently reviewed evidence from the timed restore and rollback drill. |
| 11 | Backups, monitoring, and recovery | A | In progress | Encrypted backup and isolated restore tooling includes opt-in systemd scheduling, checksum-verified rsync transfer, restricted Zabbix status, guarded local retention and a release-bound acceptance validator. The Ansible operations playbook now verifies the selected release and private-file modes, imports only the recovery public key, validates and installs the units, enables transfer/monitoring, leaves backup scheduling off until manual acceptance, and fetches non-sensitive evidence. Provision secrets and durable logs, configure central alerts/retention, run the manual backup plus timed restore and rollback drills, and retain independent approval. |
| 12 | Google mailbox and reviewed-reply acceptance | B | In progress | The synthetic-data provider runbook, exact scenario set and restricted-record validator are implemented. Complete every scenario against the accepted Debian release and dedicated Google sandbox accounts, independently review the evidence, and retain the validated record. |
| 13 | Rezlynx/Guestline adapter | C | Planned | Obtain the provider contract and sandbox, implement the adapter and mapping, and accept idempotency, stale-data, ambiguous-write, and reconciliation paths. |
| 14 | Payment links and status | C | Planned | Select/confirm the payment-provider path, complete sandbox and webhook acceptance, and prove expiry, replay protection, reconciliation, and support recovery. |

View File

@ -28,4 +28,29 @@ ansible-playbook guestops.yml \
The controller verifies the source package before transfer. The host independently checks the transferred archive checksum, extracts into a commit-specific directory, verifies the package again, selects the commit-tagged images and disabled send/FAQ defaults in the private environment, builds the API and worker images, records immutable image IDs, runs the offline preflight, validates Compose without printing expanded secrets, starts with `--no-build`, and waits for loopback readiness. It then enables Docker and Nginx at boot, validates and installs the reviewed proxy site, rejects public API or MongoDB listeners, and checks the public redirect and certificate-backed HTTPS readiness before changing the `current` symlink. The source archive remains in the restricted controller store; the temporary host copy is removed after success.
This playbook does not provision DNS, TLS, Nginx, firewall rules, backup keys, monitoring, or the private environment. Those remain explicit Milestone 10 and 11 acceptance activities.
The release playbook installs the reviewed Nginx site but does not provision DNS, certificates, firewall rules, durable logging, backup keys, central monitoring, or the private environment. Those remain explicit Milestone 10 and 11 acceptance activities.
## Backup and monitoring operations
After the exact release is deployed, run `guestops-operations.yml`. It installs the repository systemd units, verifies their definitions, imports only the approved public recovery key, enables transfer retry and monitoring, connects the aggregate status file to Zabbix, and retains non-sensitive installation evidence. It never creates a recovery private key or writes transfer credentials.
Before running it, provision these private host files through Ansible Vault or the established secret process:
- `/etc/guestops/backup.env` mode `0600`;
- `/etc/guestops/backup-transfer.env` mode `0600`;
- `/etc/guestops/backup-transfer.key` mode `0600`;
- `/etc/guestops/backup-known-hosts` mode `0644`.
The recovery public key must be available on the controller. Its private key and protected recovery copy must remain off the Debian host. Confirm durable restricted logging separately and pass `guestops_durable_logs_configured=true` only after that review. The daily backup timer defaults to disabled; set `guestops_enable_backup_schedule=true` only after the manual backup and service-recovery exercise passes.
```sh
ansible-playbook guestops-operations.yml \
-l guestops_sandbox \
-e guestops_release_commit=FULL_40_CHARACTER_SHA \
-e guestops_backup_recipient=FULL_40_CHARACTER_GPG_FINGERPRINT \
-e guestops_recovery_public_key=/secure/recovery/guestops-public.asc \
-e guestops_evidence_directory=/secure/evidence/guestops/0.2.1 \
-e guestops_durable_logs_configured=true
```
Run the manual encrypted backup, transfer, isolated restore and rollback exercises separately under the approved maintenance procedure. After they pass, rerun with `guestops_enable_backup_schedule=true`. Configure Zabbix trigger thresholds and escalation recipients in the central Zabbix environment; the playbook exposes only the non-sensitive `guestops.status` item.

View File

@ -0,0 +1,227 @@
---
- name: Install GuestOps backup and monitoring operations
hosts: guestops
become: true
gather_facts: true
vars:
guestops_root: /srv/guestops
guestops_current: "{{ guestops_root }}/current"
guestops_config_root: /etc/guestops
guestops_backup_root: /var/backups/guestops
guestops_gnupg_root: /var/lib/guestops-backup/gnupg
guestops_zabbix_service: zabbix-agent2
guestops_zabbix_include_directory: /etc/zabbix/zabbix_agent2.d
guestops_enable_backup_schedule: false
guestops_durable_logs_configured: false
pre_tasks:
- name: Validate required operational variables
ansible.builtin.assert:
that:
- guestops_release_commit is match('^[0-9a-f]{40}$')
- guestops_backup_recipient is match('^[A-Fa-f0-9]{40}$')
- guestops_recovery_public_key | length > 0
- guestops_evidence_directory | length > 0
- guestops_durable_logs_configured | bool
fail_msg: Release commit, recovery public key/fingerprint, evidence directory and reviewed durable logging are required.
- name: Confirm selected release link
ansible.builtin.stat:
path: "{{ guestops_current }}"
follow: false
register: guestops_selected_release
- name: Require deployed release before operations installation
ansible.builtin.assert:
that:
- guestops_selected_release.stat.exists
- guestops_selected_release.stat.islnk
- guestops_selected_release.stat.lnk_source == guestops_root + '/releases/' + guestops_release_commit
fail_msg: /srv/guestops/current must select the exact approved release commit.
- name: Confirm selected release target exists
ansible.builtin.stat:
path: "{{ guestops_root }}/releases/{{ guestops_release_commit }}"
follow: true
register: guestops_release_target
- name: Require selected release directory
ansible.builtin.assert:
that:
- guestops_release_target.stat.exists
- guestops_release_target.stat.isdir
fail_msg: The selected release target must be an installed directory.
- name: Inspect pre-provisioned private operational files
ansible.builtin.stat:
path: "{{ item.path }}"
follow: false
loop:
- { path: /etc/guestops/backup.env, mode: "0600" }
- { path: /etc/guestops/backup-transfer.env, mode: "0600" }
- { path: /etc/guestops/backup-transfer.key, mode: "0600" }
- { path: /etc/guestops/backup-known-hosts, mode: "0644" }
register: guestops_operational_files
- name: Require private backup and transfer configuration
ansible.builtin.assert:
that:
- item.stat.exists
- item.stat.isreg
- not item.stat.islnk
- item.stat.mode == item.item.mode
fail_msg: "{{ item.item.path }} must be a regular non-symlink file with mode {{ item.item.mode }}."
loop: "{{ guestops_operational_files.results }}"
no_log: true
tasks:
- name: Create private backup directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: root
group: root
mode: "0700"
loop:
- "{{ guestops_backup_root }}"
- "{{ guestops_gnupg_root }}"
- name: Stage recovery public key
ansible.builtin.copy:
src: "{{ guestops_recovery_public_key }}"
dest: /etc/guestops/recovery-public.asc
owner: root
group: root
mode: "0600"
- name: Import recovery public key only
ansible.builtin.command:
argv:
- gpg
- --batch
- --homedir
- "{{ guestops_gnupg_root }}"
- --import
- /etc/guestops/recovery-public.asc
no_log: true
changed_when: true
- name: Verify configured recovery fingerprint
ansible.builtin.command:
argv:
- gpg
- --batch
- --homedir
- "{{ guestops_gnupg_root }}"
- --list-keys
- "{{ guestops_backup_recipient }}"
changed_when: false
no_log: true
- name: Remove staged recovery public key
ansible.builtin.file:
path: /etc/guestops/recovery-public.asc
state: absent
- name: Install GuestOps systemd units
ansible.builtin.copy:
src: "{{ playbook_dir }}/../systemd/{{ item }}"
dest: "/etc/systemd/system/{{ item }}"
owner: root
group: root
mode: "0644"
loop:
- guestops-backup.service
- guestops-backup.timer
- guestops-backup-transfer.service
- guestops-backup-transfer.timer
- guestops-monitor-status.service
- guestops-monitor-status.timer
register: guestops_systemd_units
- name: Verify installed systemd units
ansible.builtin.command:
argv:
- systemd-analyze
- verify
- /etc/systemd/system/guestops-backup.service
- /etc/systemd/system/guestops-backup.timer
- /etc/systemd/system/guestops-backup-transfer.service
- /etc/systemd/system/guestops-backup-transfer.timer
- /etc/systemd/system/guestops-monitor-status.service
- /etc/systemd/system/guestops-monitor-status.timer
changed_when: false
- name: Reload systemd unit definitions
ansible.builtin.systemd_service:
daemon_reload: true
when: guestops_systemd_units.changed
- name: Enable transfer retry and monitoring timers
ansible.builtin.systemd_service:
name: "{{ item }}"
enabled: true
state: started
loop:
- guestops-backup-transfer.timer
- guestops-monitor-status.timer
- name: Select backup schedule state
ansible.builtin.systemd_service:
name: guestops-backup.timer
enabled: "{{ guestops_enable_backup_schedule | bool }}"
state: "{{ 'started' if guestops_enable_backup_schedule | bool else 'stopped' }}"
- name: Require Zabbix include directory
ansible.builtin.stat:
path: "{{ guestops_zabbix_include_directory }}"
register: guestops_zabbix_directory
- name: Confirm Zabbix agent configuration path
ansible.builtin.assert:
that:
- guestops_zabbix_directory.stat.exists
- guestops_zabbix_directory.stat.isdir
fail_msg: Override guestops_zabbix_include_directory for the installed Zabbix agent.
- name: Install restricted Zabbix status item
ansible.builtin.copy:
src: "{{ playbook_dir }}/../systemd/zabbix-agent-guestops.conf.example"
dest: "{{ guestops_zabbix_include_directory }}/guestops.conf"
owner: root
group: root
mode: "0644"
register: guestops_zabbix_configuration
- name: Restart Zabbix agent after configuration change
ansible.builtin.service:
name: "{{ guestops_zabbix_service }}"
enabled: true
state: restarted
when: guestops_zabbix_configuration.changed
- name: Generate initial non-sensitive monitoring status
ansible.builtin.command:
argv: [systemctl, start, guestops-monitor-status.service]
changed_when: true
- name: Read installed timer schedule
ansible.builtin.command:
argv: [systemctl, list-timers, --all, --no-pager, guestops-backup.timer, guestops-backup-transfer.timer, guestops-monitor-status.timer]
register: guestops_timer_status
changed_when: false
- name: Retain monitoring status on the Ansible controller
ansible.builtin.fetch:
src: /run/guestops-monitor/status.json
dest: "{{ guestops_evidence_directory }}/monitor-status.json"
flat: true
- name: Retain timer status on the Ansible controller
ansible.builtin.copy:
content: "{{ guestops_timer_status.stdout }}\n"
dest: "{{ guestops_evidence_directory }}/systemd-timers.txt"
mode: "0600"
delegate_to: localhost
become: false

View File

@ -5,7 +5,7 @@
gather_facts: true
vars:
guestops_root: /opt/guestops
guestops_root: /srv/guestops
guestops_config_root: /etc/guestops
guestops_public_hostname: sandbox-guestops.futuresens.co.uk
guestops_public_origin: "https://{{ guestops_public_hostname }}"

View File

@ -2,15 +2,15 @@
Description=Transfer GuestOps encrypted backups to restricted storage
Wants=network-online.target
After=network-online.target guestops-backup.service
ConditionPathIsDirectory=/srv/guestops
ConditionPathIsDirectory=/srv/guestops/current
ConditionPathIsDirectory=/var/backups/guestops
[Service]
Type=oneshot
WorkingDirectory=/srv/guestops
WorkingDirectory=/srv/guestops/current
EnvironmentFile=/etc/guestops/backup-transfer.env
UMask=0077
ExecStart=/usr/bin/python3 /srv/guestops/deploy/backup_transfer.py --prune-verified --retention-days 7
ExecStart=/usr/bin/python3 /srv/guestops/current/deploy/backup_transfer.py --prune-verified --retention-days 7
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true

View File

@ -2,17 +2,17 @@
Description=GuestOps encrypted maintenance backup
Requires=docker.service
After=docker.service
ConditionPathIsDirectory=/srv/guestops
ConditionPathIsDirectory=/srv/guestops/current
ConditionPathIsDirectory=/var/backups/guestops
ConditionPathIsDirectory=/var/lib/guestops-backup/gnupg
[Service]
Type=oneshot
WorkingDirectory=/srv/guestops
WorkingDirectory=/srv/guestops/current
EnvironmentFile=/etc/guestops/backup.env
Environment=GNUPGHOME=/var/lib/guestops-backup/gnupg
UMask=0077
ExecStart=/usr/bin/python3 /srv/guestops/deploy/ops.py scheduled-backup --confirm-maintenance
ExecStart=/usr/bin/python3 /srv/guestops/current/deploy/ops.py scheduled-backup --confirm-maintenance
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true

View File

@ -2,14 +2,14 @@
Description=Write non-sensitive GuestOps monitoring status
Requires=docker.service
After=docker.service network-online.target
ConditionPathIsDirectory=/srv/guestops
ConditionPathIsDirectory=/srv/guestops/current
ConditionPathIsDirectory=/var/backups/guestops
[Service]
Type=oneshot
WorkingDirectory=/srv/guestops
WorkingDirectory=/srv/guestops/current
UMask=0022
ExecStart=/usr/bin/python3 /srv/guestops/deploy/monitor_status.py
ExecStart=/usr/bin/python3 /srv/guestops/current/deploy/monitor_status.py
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true

View File

@ -87,7 +87,7 @@ Copy the encrypted file off-server to restricted storage after every successful
### Optional systemd schedule
The repository includes an opt-in daily systemd service and timer. They are templates, not automatically installed. The service assumes the reviewed checkout is `/srv/guestops` and stages encrypted files in `/var/backups/guestops`; review and change both unit files together if the host uses different paths.
The repository includes an opt-in daily systemd service and timer. The service follows the Ansible-selected release at `/srv/guestops/current` and stages encrypted files in `/var/backups/guestops`. The application-owned `deploy/ansible/guestops-operations.yml` installs and verifies these units after the exact release is selected; keep the backup timer disabled until the supervised manual backup passes.
Create the staging directory and environment file without storing a private key or passphrase on the server:

View File

@ -0,0 +1,53 @@
from pathlib import Path
import unittest
ROOT = Path(__file__).resolve().parents[1]
PLAYBOOK = ROOT / "deploy" / "ansible" / "guestops-operations.yml"
class AnsibleOperationsTests(unittest.TestCase):
def test_operations_require_release_and_private_inputs(self):
text = PLAYBOOK.read_text(encoding="utf-8")
for required in (
"/srv/guestops/current",
"guestops_release_commit is match('^[0-9a-f]{40}$')",
"guestops_backup_recipient is match('^[A-Fa-f0-9]{40}$')",
"guestops_durable_logs_configured | bool",
"/etc/guestops/backup.env",
"/etc/guestops/backup-transfer.env",
"/etc/guestops/backup-transfer.key",
"/etc/guestops/backup-known-hosts",
"no_log: true",
):
self.assertIn(required, text)
def test_public_key_units_monitoring_and_evidence_are_integrated(self):
text = PLAYBOOK.read_text(encoding="utf-8")
ordered = (
"Import recovery public key only",
"Verify installed systemd units",
"Enable transfer retry and monitoring timers",
"Install restricted Zabbix status item",
"Generate initial non-sensitive monitoring status",
"Retain monitoring status on the Ansible controller",
)
positions = [text.index(item) for item in ordered]
self.assertEqual(positions, sorted(positions))
self.assertIn("guestops_enable_backup_schedule: false", text)
self.assertNotIn("ansible.builtin.shell", text)
self.assertNotIn("PRIVATE KEY", text)
def test_systemd_units_follow_selected_release(self):
for name in (
"guestops-backup.service",
"guestops-backup-transfer.service",
"guestops-monitor-status.service",
):
text = (ROOT / "deploy" / "systemd" / name).read_text(encoding="utf-8")
self.assertIn("/srv/guestops/current", text)
self.assertNotIn("WorkingDirectory=/srv/guestops\n", text)
if __name__ == "__main__":
unittest.main()