GuestOps asible playbook
This commit is contained in:
@@ -1,56 +0,0 @@
|
||||
# GuestOps Ansible handoff
|
||||
|
||||
This playbook is the application-owned deployment contract. Copy or import it into the private Futuresens Ansible repository and bind the `guestops` inventory group there. Do not commit inventory secrets, `.env` contents, provider credentials, hostnames other than the public service name, or restricted evidence paths to GuestOps.
|
||||
|
||||
Required extra variables:
|
||||
|
||||
- `guestops_release_version`: matched application version, for example `0.2.1`;
|
||||
- `guestops_release_commit`: full 40-character packaged commit;
|
||||
- `guestops_archive`: controller path to the generated `.tar.gz`;
|
||||
- `guestops_archive_sha256`: approved lowercase SHA-256;
|
||||
- `guestops_source_record`: controller path to the matching `.source.json`;
|
||||
- `guestops_evidence_directory`: existing restricted controller directory for the fetched release record.
|
||||
|
||||
The target must already have Docker Engine with Compose, Nginx, Python 3, `ss`, a trusted certificate under `/etc/letsencrypt/live/sandbox-guestops.futuresens.co.uk`, and `/etc/guestops/guestops.env` owned by root with mode `0600`. The private environment file supplies MongoDB passwords, image references, provider-file paths and disabled-by-default feature flags. Provision it through Ansible Vault or the existing Futuresens secret process, never through this repository.
|
||||
|
||||
Example invocation from the central Ansible checkout:
|
||||
|
||||
```sh
|
||||
ansible-playbook guestops.yml \
|
||||
-l guestops_sandbox \
|
||||
-e guestops_release_version=0.2.1 \
|
||||
-e guestops_release_commit=FULL_40_CHARACTER_SHA \
|
||||
-e guestops_archive=/secure/releases/GuestOps-0.2.1-COMMIT.tar.gz \
|
||||
-e guestops_archive_sha256=LOWERCASE_SHA256 \
|
||||
-e guestops_source_record=/secure/releases/GuestOps-0.2.1-COMMIT.source.json \
|
||||
-e guestops_evidence_directory=/secure/evidence/guestops/0.2.1
|
||||
```
|
||||
|
||||
The controller verifies the source package before transfer. The host independently checks the transferred archive checksum, extracts into a commit-specific directory, verifies the package again, selects the commit-tagged images and disabled send/FAQ defaults in the private environment, builds the API and worker images, records immutable image IDs, runs the offline preflight, validates Compose without printing expanded secrets, starts with `--no-build`, and waits for loopback readiness. It then enables Docker and Nginx at boot, validates and installs the reviewed proxy site, rejects public API or MongoDB listeners, and checks the public redirect and certificate-backed HTTPS readiness before changing the `current` symlink. The source archive remains in the restricted controller store; the temporary host copy is removed after success.
|
||||
|
||||
The release playbook installs the reviewed Nginx site but does not provision DNS, certificates, firewall rules, durable logging, backup keys, central monitoring, or the private environment. Those remain explicit Milestone 10 and 11 acceptance activities.
|
||||
|
||||
## Backup and monitoring operations
|
||||
|
||||
After the exact release is deployed, run `guestops-operations.yml`. It installs the repository systemd units, verifies their definitions, imports only the approved public recovery key, enables transfer retry and monitoring, connects the aggregate status file to Zabbix, and retains non-sensitive installation evidence. It never creates a recovery private key or writes transfer credentials.
|
||||
|
||||
Before running it, provision these private host files through Ansible Vault or the established secret process:
|
||||
|
||||
- `/etc/guestops/backup.env` mode `0600`;
|
||||
- `/etc/guestops/backup-transfer.env` mode `0600`;
|
||||
- `/etc/guestops/backup-transfer.key` mode `0600`;
|
||||
- `/etc/guestops/backup-known-hosts` mode `0644`.
|
||||
|
||||
The recovery public key must be available on the controller. Its private key and protected recovery copy must remain off the Debian host. Confirm durable restricted logging separately and pass `guestops_durable_logs_configured=true` only after that review. The daily backup timer defaults to disabled; set `guestops_enable_backup_schedule=true` only after the manual backup and service-recovery exercise passes.
|
||||
|
||||
```sh
|
||||
ansible-playbook guestops-operations.yml \
|
||||
-l guestops_sandbox \
|
||||
-e guestops_release_commit=FULL_40_CHARACTER_SHA \
|
||||
-e guestops_backup_recipient=FULL_40_CHARACTER_GPG_FINGERPRINT \
|
||||
-e guestops_recovery_public_key=/secure/recovery/guestops-public.asc \
|
||||
-e guestops_evidence_directory=/secure/evidence/guestops/0.2.1 \
|
||||
-e guestops_durable_logs_configured=true
|
||||
```
|
||||
|
||||
Run the manual encrypted backup, transfer, isolated restore and rollback exercises separately under the approved maintenance procedure. After they pass, rerun with `guestops_enable_backup_schedule=true`. Configure Zabbix trigger thresholds and escalation recipients in the central Zabbix environment; the playbook exposes only the non-sensitive `guestops.status` item.
|
||||
@@ -1,227 +0,0 @@
|
||||
---
|
||||
- name: Install GuestOps backup and monitoring operations
|
||||
hosts: guestops
|
||||
become: true
|
||||
gather_facts: true
|
||||
|
||||
vars:
|
||||
guestops_root: /srv/guestops
|
||||
guestops_current: "{{ guestops_root }}/current"
|
||||
guestops_config_root: /etc/guestops
|
||||
guestops_backup_root: /var/backups/guestops
|
||||
guestops_gnupg_root: /var/lib/guestops-backup/gnupg
|
||||
guestops_zabbix_service: zabbix-agent2
|
||||
guestops_zabbix_include_directory: /etc/zabbix/zabbix_agent2.d
|
||||
guestops_enable_backup_schedule: false
|
||||
guestops_durable_logs_configured: false
|
||||
|
||||
pre_tasks:
|
||||
- name: Validate required operational variables
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- guestops_release_commit is match('^[0-9a-f]{40}$')
|
||||
- guestops_backup_recipient is match('^[A-Fa-f0-9]{40}$')
|
||||
- guestops_recovery_public_key | length > 0
|
||||
- guestops_evidence_directory | length > 0
|
||||
- guestops_durable_logs_configured | bool
|
||||
fail_msg: Release commit, recovery public key/fingerprint, evidence directory and reviewed durable logging are required.
|
||||
|
||||
- name: Confirm selected release link
|
||||
ansible.builtin.stat:
|
||||
path: "{{ guestops_current }}"
|
||||
follow: false
|
||||
register: guestops_selected_release
|
||||
|
||||
- name: Require deployed release before operations installation
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- guestops_selected_release.stat.exists
|
||||
- guestops_selected_release.stat.islnk
|
||||
- guestops_selected_release.stat.lnk_source == guestops_root + '/releases/' + guestops_release_commit
|
||||
fail_msg: /srv/guestops/current must select the exact approved release commit.
|
||||
|
||||
- name: Confirm selected release target exists
|
||||
ansible.builtin.stat:
|
||||
path: "{{ guestops_root }}/releases/{{ guestops_release_commit }}"
|
||||
follow: true
|
||||
register: guestops_release_target
|
||||
|
||||
- name: Require selected release directory
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- guestops_release_target.stat.exists
|
||||
- guestops_release_target.stat.isdir
|
||||
fail_msg: The selected release target must be an installed directory.
|
||||
|
||||
- name: Inspect pre-provisioned private operational files
|
||||
ansible.builtin.stat:
|
||||
path: "{{ item.path }}"
|
||||
follow: false
|
||||
loop:
|
||||
- { path: /etc/guestops/backup.env, mode: "0600" }
|
||||
- { path: /etc/guestops/backup-transfer.env, mode: "0600" }
|
||||
- { path: /etc/guestops/backup-transfer.key, mode: "0600" }
|
||||
- { path: /etc/guestops/backup-known-hosts, mode: "0644" }
|
||||
register: guestops_operational_files
|
||||
|
||||
- name: Require private backup and transfer configuration
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- item.stat.exists
|
||||
- item.stat.isreg
|
||||
- not item.stat.islnk
|
||||
- item.stat.mode == item.item.mode
|
||||
fail_msg: "{{ item.item.path }} must be a regular non-symlink file with mode {{ item.item.mode }}."
|
||||
loop: "{{ guestops_operational_files.results }}"
|
||||
no_log: true
|
||||
|
||||
tasks:
|
||||
- name: Create private backup directories
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0700"
|
||||
loop:
|
||||
- "{{ guestops_backup_root }}"
|
||||
- "{{ guestops_gnupg_root }}"
|
||||
|
||||
- name: Stage recovery public key
|
||||
ansible.builtin.copy:
|
||||
src: "{{ guestops_recovery_public_key }}"
|
||||
dest: /etc/guestops/recovery-public.asc
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0600"
|
||||
|
||||
- name: Import recovery public key only
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- gpg
|
||||
- --batch
|
||||
- --homedir
|
||||
- "{{ guestops_gnupg_root }}"
|
||||
- --import
|
||||
- /etc/guestops/recovery-public.asc
|
||||
no_log: true
|
||||
changed_when: true
|
||||
|
||||
- name: Verify configured recovery fingerprint
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- gpg
|
||||
- --batch
|
||||
- --homedir
|
||||
- "{{ guestops_gnupg_root }}"
|
||||
- --list-keys
|
||||
- "{{ guestops_backup_recipient }}"
|
||||
changed_when: false
|
||||
no_log: true
|
||||
|
||||
- name: Remove staged recovery public key
|
||||
ansible.builtin.file:
|
||||
path: /etc/guestops/recovery-public.asc
|
||||
state: absent
|
||||
|
||||
- name: Install GuestOps systemd units
|
||||
ansible.builtin.copy:
|
||||
src: "{{ playbook_dir }}/../systemd/{{ item }}"
|
||||
dest: "/etc/systemd/system/{{ item }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
loop:
|
||||
- guestops-backup.service
|
||||
- guestops-backup.timer
|
||||
- guestops-backup-transfer.service
|
||||
- guestops-backup-transfer.timer
|
||||
- guestops-monitor-status.service
|
||||
- guestops-monitor-status.timer
|
||||
register: guestops_systemd_units
|
||||
|
||||
- name: Verify installed systemd units
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- systemd-analyze
|
||||
- verify
|
||||
- /etc/systemd/system/guestops-backup.service
|
||||
- /etc/systemd/system/guestops-backup.timer
|
||||
- /etc/systemd/system/guestops-backup-transfer.service
|
||||
- /etc/systemd/system/guestops-backup-transfer.timer
|
||||
- /etc/systemd/system/guestops-monitor-status.service
|
||||
- /etc/systemd/system/guestops-monitor-status.timer
|
||||
changed_when: false
|
||||
|
||||
- name: Reload systemd unit definitions
|
||||
ansible.builtin.systemd_service:
|
||||
daemon_reload: true
|
||||
when: guestops_systemd_units.changed
|
||||
|
||||
- name: Enable transfer retry and monitoring timers
|
||||
ansible.builtin.systemd_service:
|
||||
name: "{{ item }}"
|
||||
enabled: true
|
||||
state: started
|
||||
loop:
|
||||
- guestops-backup-transfer.timer
|
||||
- guestops-monitor-status.timer
|
||||
|
||||
- name: Select backup schedule state
|
||||
ansible.builtin.systemd_service:
|
||||
name: guestops-backup.timer
|
||||
enabled: "{{ guestops_enable_backup_schedule | bool }}"
|
||||
state: "{{ 'started' if guestops_enable_backup_schedule | bool else 'stopped' }}"
|
||||
|
||||
- name: Require Zabbix include directory
|
||||
ansible.builtin.stat:
|
||||
path: "{{ guestops_zabbix_include_directory }}"
|
||||
register: guestops_zabbix_directory
|
||||
|
||||
- name: Confirm Zabbix agent configuration path
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- guestops_zabbix_directory.stat.exists
|
||||
- guestops_zabbix_directory.stat.isdir
|
||||
fail_msg: Override guestops_zabbix_include_directory for the installed Zabbix agent.
|
||||
|
||||
- name: Install restricted Zabbix status item
|
||||
ansible.builtin.copy:
|
||||
src: "{{ playbook_dir }}/../systemd/zabbix-agent-guestops.conf.example"
|
||||
dest: "{{ guestops_zabbix_include_directory }}/guestops.conf"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
register: guestops_zabbix_configuration
|
||||
|
||||
- name: Restart Zabbix agent after configuration change
|
||||
ansible.builtin.service:
|
||||
name: "{{ guestops_zabbix_service }}"
|
||||
enabled: true
|
||||
state: restarted
|
||||
when: guestops_zabbix_configuration.changed
|
||||
|
||||
- name: Generate initial non-sensitive monitoring status
|
||||
ansible.builtin.command:
|
||||
argv: [systemctl, start, guestops-monitor-status.service]
|
||||
changed_when: true
|
||||
|
||||
- name: Read installed timer schedule
|
||||
ansible.builtin.command:
|
||||
argv: [systemctl, list-timers, --all, --no-pager, guestops-backup.timer, guestops-backup-transfer.timer, guestops-monitor-status.timer]
|
||||
register: guestops_timer_status
|
||||
changed_when: false
|
||||
|
||||
- name: Retain monitoring status on the Ansible controller
|
||||
ansible.builtin.fetch:
|
||||
src: /run/guestops-monitor/status.json
|
||||
dest: "{{ guestops_evidence_directory }}/monitor-status.json"
|
||||
flat: true
|
||||
|
||||
- name: Retain timer status on the Ansible controller
|
||||
ansible.builtin.copy:
|
||||
content: "{{ guestops_timer_status.stdout }}\n"
|
||||
dest: "{{ guestops_evidence_directory }}/systemd-timers.txt"
|
||||
mode: "0600"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
@@ -1,402 +0,0 @@
|
||||
---
|
||||
- name: Install a verified GuestOps source release
|
||||
hosts: guestops
|
||||
become: true
|
||||
gather_facts: true
|
||||
|
||||
vars:
|
||||
guestops_root: /srv/guestops
|
||||
guestops_config_root: /etc/guestops
|
||||
guestops_public_hostname: sandbox-guestops.futuresens.co.uk
|
||||
guestops_public_origin: "https://{{ guestops_public_hostname }}"
|
||||
guestops_release_root: "{{ guestops_root }}/releases/{{ guestops_release_commit }}"
|
||||
guestops_staging_archive: "/var/tmp/{{ guestops_archive | basename }}"
|
||||
guestops_staging_record: "/var/tmp/{{ guestops_source_record | basename }}"
|
||||
guestops_api_image: "guestops-api:{{ guestops_release_commit }}"
|
||||
guestops_worker_image: "guestops-worker:{{ guestops_release_commit }}"
|
||||
|
||||
pre_tasks:
|
||||
- name: Validate required release variables
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- guestops_release_version is match('^[0-9]+\\.[0-9]+\\.[0-9]+$')
|
||||
- guestops_release_commit is match('^[0-9a-f]{40}$')
|
||||
- guestops_archive_sha256 is match('^[0-9a-f]{64}$')
|
||||
- guestops_archive | length > 0
|
||||
- guestops_source_record | length > 0
|
||||
- guestops_evidence_directory | length > 0
|
||||
- guestops_public_hostname is match('^[A-Za-z0-9.-]+$')
|
||||
- guestops_public_hostname == 'sandbox-guestops.futuresens.co.uk'
|
||||
fail_msg: Release version, full commit, archive, SHA-256, source record and evidence directory are required.
|
||||
|
||||
- name: Verify source package on the Ansible controller
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- python3
|
||||
- "{{ playbook_dir }}/../verify_source_package.py"
|
||||
- --archive
|
||||
- "{{ guestops_archive }}"
|
||||
- --record
|
||||
- "{{ guestops_source_record }}"
|
||||
- --expected-commit
|
||||
- "{{ guestops_release_commit }}"
|
||||
- --expected-version
|
||||
- "{{ guestops_release_version }}"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
changed_when: false
|
||||
|
||||
- name: Check private GuestOps environment exists
|
||||
ansible.builtin.stat:
|
||||
path: "{{ guestops_config_root }}/guestops.env"
|
||||
register: guestops_environment
|
||||
|
||||
- name: Require a private pre-provisioned environment file
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- guestops_environment.stat.exists
|
||||
- guestops_environment.stat.isreg
|
||||
- guestops_environment.stat.mode == '0600'
|
||||
fail_msg: /etc/guestops/guestops.env must already exist with mode 0600.
|
||||
|
||||
tasks:
|
||||
- name: Require supported Debian host
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_facts.system == 'Linux'
|
||||
- ansible_facts.distribution == 'Debian'
|
||||
- ansible_facts.distribution_major_version | int >= 12
|
||||
- ansible_facts.processor_vcpus | default(0) | int >= 4
|
||||
- ansible_facts.memtotal_mb | default(0) | int >= 7300
|
||||
fail_msg: GuestOps requires Debian 12 or newer with at least 4 CPUs and 7.3 GiB RAM.
|
||||
|
||||
- name: Enable Docker at boot
|
||||
ansible.builtin.service:
|
||||
name: docker
|
||||
enabled: true
|
||||
state: started
|
||||
|
||||
- name: Enable Nginx at boot
|
||||
ansible.builtin.service:
|
||||
name: nginx
|
||||
enabled: true
|
||||
state: started
|
||||
|
||||
- name: Create release and evidence directories
|
||||
ansible.builtin.file:
|
||||
path: "{{ item.path }}"
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "{{ item.mode }}"
|
||||
loop:
|
||||
- { path: "{{ guestops_root }}/releases", mode: "0755" }
|
||||
- { path: "{{ guestops_release_root }}", mode: "0755" }
|
||||
- { path: "{{ guestops_config_root }}/release-records", mode: "0700" }
|
||||
|
||||
- name: Copy source archive to the host
|
||||
ansible.builtin.copy:
|
||||
src: "{{ guestops_archive }}"
|
||||
dest: "{{ guestops_staging_archive }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0600"
|
||||
|
||||
- name: Copy source record to the host
|
||||
ansible.builtin.copy:
|
||||
src: "{{ guestops_source_record }}"
|
||||
dest: "{{ guestops_staging_record }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0600"
|
||||
|
||||
- name: Calculate transferred archive checksum
|
||||
ansible.builtin.stat:
|
||||
path: "{{ guestops_staging_archive }}"
|
||||
checksum_algorithm: sha256
|
||||
register: guestops_transferred_archive
|
||||
|
||||
- name: Reject a changed transferred archive
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- guestops_transferred_archive.stat.checksum == guestops_archive_sha256
|
||||
fail_msg: Transferred GuestOps archive does not match the approved SHA-256.
|
||||
|
||||
- name: Extract committed application files
|
||||
ansible.builtin.unarchive:
|
||||
src: "{{ guestops_staging_archive }}"
|
||||
dest: "{{ guestops_release_root }}"
|
||||
remote_src: true
|
||||
extra_opts:
|
||||
- --strip-components=1
|
||||
|
||||
- name: Verify transferred package and record on the target
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- python3
|
||||
- deploy/verify_source_package.py
|
||||
- --archive
|
||||
- "{{ guestops_staging_archive }}"
|
||||
- --record
|
||||
- "{{ guestops_staging_record }}"
|
||||
- --expected-commit
|
||||
- "{{ guestops_release_commit }}"
|
||||
- --expected-version
|
||||
- "{{ guestops_release_version }}"
|
||||
- --output
|
||||
- "{{ guestops_config_root }}/release-records/{{ guestops_release_commit }}.source-verification.json"
|
||||
chdir: "{{ guestops_release_root }}"
|
||||
changed_when: false
|
||||
|
||||
- name: Retain approved source record on the target
|
||||
ansible.builtin.copy:
|
||||
src: "{{ guestops_staging_record }}"
|
||||
dest: "{{ guestops_config_root }}/release-records/{{ guestops_release_commit }}.source.json"
|
||||
remote_src: true
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0600"
|
||||
|
||||
- name: Select release images and safe pilot defaults in the private environment
|
||||
ansible.builtin.lineinfile:
|
||||
path: "{{ guestops_config_root }}/guestops.env"
|
||||
regexp: "^{{ item.name }}="
|
||||
line: "{{ item.name }}={{ item.value }}"
|
||||
create: false
|
||||
mode: "0600"
|
||||
loop:
|
||||
- { name: GUESTOPS_API_IMAGE, value: "{{ guestops_api_image }}" }
|
||||
- { name: GUESTOPS_WORKER_IMAGE, value: "{{ guestops_worker_image }}" }
|
||||
- { name: GOOGLE_ENABLE_SENDING, value: "false" }
|
||||
- { name: AUTO_REPLY_ENABLE_LIVE, value: "false" }
|
||||
no_log: true
|
||||
|
||||
- name: Copy private deployment environment into the release
|
||||
ansible.builtin.copy:
|
||||
src: "{{ guestops_config_root }}/guestops.env"
|
||||
dest: "{{ guestops_release_root }}/.env"
|
||||
remote_src: true
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0600"
|
||||
|
||||
- name: Build API image from the verified source
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- docker
|
||||
- build
|
||||
- --target
|
||||
- api
|
||||
- --tag
|
||||
- "{{ guestops_api_image }}"
|
||||
- .
|
||||
chdir: "{{ guestops_release_root }}"
|
||||
register: guestops_api_build
|
||||
changed_when: true
|
||||
|
||||
- name: Build worker image from the verified source
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- docker
|
||||
- build
|
||||
- --target
|
||||
- worker
|
||||
- --tag
|
||||
- "{{ guestops_worker_image }}"
|
||||
- .
|
||||
chdir: "{{ guestops_release_root }}"
|
||||
register: guestops_worker_build
|
||||
changed_when: true
|
||||
|
||||
- name: Inspect API image identity
|
||||
ansible.builtin.command:
|
||||
argv: [docker, image, inspect, --format, "{{ '{{.Id}}' }}", "{{ guestops_api_image }}"]
|
||||
register: guestops_api_identity
|
||||
changed_when: false
|
||||
|
||||
- name: Inspect worker image identity
|
||||
ansible.builtin.command:
|
||||
argv: [docker, image, inspect, --format, "{{ '{{.Id}}' }}", "{{ guestops_worker_image }}"]
|
||||
register: guestops_worker_identity
|
||||
changed_when: false
|
||||
|
||||
- name: Create post-build release record
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- python3
|
||||
- deploy/release_record.py
|
||||
- --artifact
|
||||
- "{{ guestops_staging_archive }}"
|
||||
- --commit
|
||||
- "{{ guestops_release_commit }}"
|
||||
- --api-image
|
||||
- "{{ guestops_api_image }}"
|
||||
- --api-id
|
||||
- "{{ guestops_api_identity.stdout }}"
|
||||
- --worker-image
|
||||
- "{{ guestops_worker_image }}"
|
||||
- --worker-id
|
||||
- "{{ guestops_worker_identity.stdout }}"
|
||||
- --output
|
||||
- "{{ guestops_config_root }}/release-records/{{ guestops_release_commit }}.json"
|
||||
chdir: "{{ guestops_release_root }}"
|
||||
|
||||
- name: Run offline deployment preflight
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- python3
|
||||
- deploy/ops.py
|
||||
- preflight
|
||||
- --offline
|
||||
chdir: "{{ guestops_release_root }}"
|
||||
changed_when: false
|
||||
|
||||
- name: Validate Compose without exposing expanded configuration
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- docker
|
||||
- compose
|
||||
- --env-file
|
||||
- "{{ guestops_config_root }}/guestops.env"
|
||||
- config
|
||||
- --quiet
|
||||
chdir: "{{ guestops_release_root }}"
|
||||
changed_when: false
|
||||
|
||||
- name: Start the verified release without rebuilding
|
||||
ansible.builtin.command:
|
||||
argv:
|
||||
- docker
|
||||
- compose
|
||||
- --env-file
|
||||
- "{{ guestops_config_root }}/guestops.env"
|
||||
- up
|
||||
- --detach
|
||||
- --no-build
|
||||
chdir: "{{ guestops_release_root }}"
|
||||
environment:
|
||||
GUESTOPS_API_IMAGE: "{{ guestops_api_image }}"
|
||||
GUESTOPS_WORKER_IMAGE: "{{ guestops_worker_image }}"
|
||||
changed_when: true
|
||||
|
||||
- name: Wait for loopback readiness
|
||||
ansible.builtin.uri:
|
||||
url: http://127.0.0.1:8080/health/ready
|
||||
method: GET
|
||||
status_code: 200
|
||||
return_content: false
|
||||
register: guestops_readiness
|
||||
retries: 30
|
||||
delay: 2
|
||||
until: guestops_readiness.status == 200
|
||||
|
||||
- name: Confirm TLS certificate files exist
|
||||
ansible.builtin.stat:
|
||||
path: "{{ item }}"
|
||||
follow: true
|
||||
loop:
|
||||
- "/etc/letsencrypt/live/{{ guestops_public_hostname }}/fullchain.pem"
|
||||
- "/etc/letsencrypt/live/{{ guestops_public_hostname }}/privkey.pem"
|
||||
register: guestops_certificates
|
||||
|
||||
- name: Require the pre-provisioned TLS certificate
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- guestops_certificates.results | map(attribute='stat.exists') | min
|
||||
- guestops_certificates.results | map(attribute='stat.isreg') | min
|
||||
fail_msg: A valid pre-provisioned Let's Encrypt certificate is required before enabling Nginx.
|
||||
|
||||
- name: Install reviewed GuestOps Nginx site
|
||||
ansible.builtin.copy:
|
||||
src: "{{ playbook_dir }}/../nginx.conf"
|
||||
dest: /etc/nginx/sites-available/guestops.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
register: guestops_nginx_site
|
||||
|
||||
- name: Enable GuestOps Nginx site
|
||||
ansible.builtin.file:
|
||||
src: /etc/nginx/sites-available/guestops.conf
|
||||
dest: /etc/nginx/sites-enabled/guestops.conf
|
||||
state: link
|
||||
register: guestops_nginx_enabled
|
||||
|
||||
- name: Validate Nginx configuration
|
||||
ansible.builtin.command:
|
||||
argv: [nginx, -t]
|
||||
changed_when: false
|
||||
|
||||
- name: Reload Nginx after reviewed configuration change
|
||||
ansible.builtin.service:
|
||||
name: nginx
|
||||
enabled: true
|
||||
state: reloaded
|
||||
when: guestops_nginx_site.changed or guestops_nginx_enabled.changed
|
||||
|
||||
- name: Inspect host TCP listeners
|
||||
ansible.builtin.command:
|
||||
argv: [ss, -ltnH]
|
||||
register: guestops_tcp_listeners
|
||||
changed_when: false
|
||||
|
||||
- name: Reject public API or MongoDB listeners
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- guestops_tcp_listeners.stdout_lines | select('search', ':8080(\\s|$)') | reject('search', '127\\.0\\.0\\.1:8080(\\s|$)') | list | length == 0
|
||||
- guestops_tcp_listeners.stdout_lines | select('search', ':27017(\\s|$)') | list | length == 0
|
||||
fail_msg: API port 8080 must be loopback-only and MongoDB must have no host listener.
|
||||
|
||||
- name: Verify public HTTP redirects to HTTPS
|
||||
ansible.builtin.uri:
|
||||
url: "http://{{ guestops_public_hostname }}/health/ready"
|
||||
method: GET
|
||||
follow_redirects: none
|
||||
status_code: [301, 302, 307, 308]
|
||||
return_content: false
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
|
||||
- name: Verify public HTTPS readiness and certificate trust
|
||||
ansible.builtin.uri:
|
||||
url: "{{ guestops_public_origin }}/health/ready"
|
||||
method: GET
|
||||
status_code: 200
|
||||
return_content: true
|
||||
validate_certs: true
|
||||
register: guestops_public_readiness
|
||||
retries: 10
|
||||
delay: 3
|
||||
until:
|
||||
- guestops_public_readiness.status == 200
|
||||
- guestops_public_readiness.json is defined
|
||||
- guestops_public_readiness.json.status == 'ready'
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
|
||||
- name: Select the current successful release
|
||||
ansible.builtin.file:
|
||||
src: "{{ guestops_release_root }}"
|
||||
dest: "{{ guestops_root }}/current"
|
||||
state: link
|
||||
force: true
|
||||
|
||||
- name: Retain release record on the Ansible controller
|
||||
ansible.builtin.fetch:
|
||||
src: "{{ guestops_config_root }}/release-records/{{ guestops_release_commit }}.json"
|
||||
dest: "{{ guestops_evidence_directory }}/release-record.json"
|
||||
flat: true
|
||||
|
||||
- name: Retain target source verification on the Ansible controller
|
||||
ansible.builtin.fetch:
|
||||
src: "{{ guestops_config_root }}/release-records/{{ guestops_release_commit }}.source-verification.json"
|
||||
dest: "{{ guestops_evidence_directory }}/source-verification.json"
|
||||
flat: true
|
||||
|
||||
- name: Remove transferred staging files after successful deployment
|
||||
ansible.builtin.file:
|
||||
path: "{{ item }}"
|
||||
state: absent
|
||||
loop:
|
||||
- "{{ guestops_staging_archive }}"
|
||||
- "{{ guestops_staging_record }}"
|
||||
@@ -1,81 +0,0 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"system": "guestops-backup-recovery",
|
||||
"evidenceId": "backup-restore",
|
||||
"dataClassification": "synthetic-only",
|
||||
"releaseVersion": "0.2.1",
|
||||
"releaseCommit": "0000000000000000000000000000000000000000",
|
||||
"releaseRecordSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"archiveSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"environment": "https://sandbox-guestops.futuresens.co.uk",
|
||||
"hostIdentifier": "guestops-sandbox-01",
|
||||
"images": {
|
||||
"api": {"reference": "guestops-api:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
|
||||
"worker": {"reference": "guestops-worker:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
|
||||
"mongo": {"reference": "mongo:8.0", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"}
|
||||
},
|
||||
"owners": {
|
||||
"backup": "REPLACE",
|
||||
"monitoring": "REPLACE",
|
||||
"recoveryOperator": "REPLACE",
|
||||
"technicalEscalation": "REPLACE",
|
||||
"retention": "REPLACE",
|
||||
"independentReviewer": "REPLACE"
|
||||
},
|
||||
"startedAt": "2026-10-01T09:00:00Z",
|
||||
"endedAt": "2026-10-01T13:00:00Z",
|
||||
"reviewedAt": "2026-10-01T14:00:00Z",
|
||||
"recoveryObjectives": {
|
||||
"targetRpoHours": 24,
|
||||
"observedRpoHours": 25,
|
||||
"targetRtoMinutes": 240,
|
||||
"observedRtoMinutes": 241
|
||||
},
|
||||
"retention": {
|
||||
"localVerifiedDays": 7,
|
||||
"offHostDaily": 35,
|
||||
"offHostMonthly": 12,
|
||||
"legalHoldOverrideTested": false
|
||||
},
|
||||
"backup": {
|
||||
"createdAt": "2026-10-01T08:00:00Z",
|
||||
"sha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"transferredSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"privateKeyPresentOnHost": false
|
||||
},
|
||||
"rollback": {
|
||||
"previousReleaseCommit": "1111111111111111111111111111111111111111",
|
||||
"previousArchiveSha256": "1111111111111111111111111111111111111111111111111111111111111111",
|
||||
"previousImages": {
|
||||
"api": {"reference": "guestops-api:1111111111111111111111111111111111111111", "id": "sha256:1111111111111111111111111111111111111111111111111111111111111111"},
|
||||
"worker": {"reference": "guestops-worker:1111111111111111111111111111111111111111", "id": "sha256:1111111111111111111111111111111111111111111111111111111111111111"}
|
||||
},
|
||||
"persistentVolumesReplaced": false,
|
||||
"restoredReleaseCommit": "0000000000000000000000000000000000000000",
|
||||
"unresolvedOperations": 1,
|
||||
"finalControls": {
|
||||
"googleSending": "disabled",
|
||||
"faqLiveMode": "disabled",
|
||||
"pmsWrites": "disabled",
|
||||
"paymentCreation": "disabled"
|
||||
}
|
||||
},
|
||||
"monitoringState": "not-configured",
|
||||
"unresolvedCriticalFindings": 1,
|
||||
"scenarios": [
|
||||
{"id": "alert-escalation", "status": "not-run", "evidence": []},
|
||||
{"id": "atomic-off-host-transfer", "status": "not-run", "evidence": []},
|
||||
{"id": "controlled-return-to-service", "status": "not-run", "evidence": []},
|
||||
{"id": "data-protection-recovery", "status": "not-run", "evidence": []},
|
||||
{"id": "database-inventory", "status": "not-run", "evidence": []},
|
||||
{"id": "durable-secret-free-logs", "status": "not-run", "evidence": []},
|
||||
{"id": "encrypted-manual-backup", "status": "not-run", "evidence": []},
|
||||
{"id": "image-rollback", "status": "not-run", "evidence": []},
|
||||
{"id": "isolated-restore", "status": "not-run", "evidence": []},
|
||||
{"id": "monitoring-coverage", "status": "not-run", "evidence": []},
|
||||
{"id": "off-host-checksum", "status": "not-run", "evidence": []},
|
||||
{"id": "production-service-recovery", "status": "not-run", "evidence": []},
|
||||
{"id": "retention-and-legal-hold", "status": "not-run", "evidence": []},
|
||||
{"id": "scheduled-backup", "status": "not-run", "evidence": []}
|
||||
]
|
||||
}
|
||||
@@ -1,220 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate a restricted GuestOps backup, monitoring and recovery record."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
VERSION = "0.2.1"
|
||||
SCENARIOS = {
|
||||
"encrypted-manual-backup",
|
||||
"scheduled-backup",
|
||||
"production-service-recovery",
|
||||
"atomic-off-host-transfer",
|
||||
"off-host-checksum",
|
||||
"retention-and-legal-hold",
|
||||
"monitoring-coverage",
|
||||
"alert-escalation",
|
||||
"durable-secret-free-logs",
|
||||
"isolated-restore",
|
||||
"data-protection-recovery",
|
||||
"database-inventory",
|
||||
"image-rollback",
|
||||
"controlled-return-to-service",
|
||||
}
|
||||
SHA256 = re.compile(r"[0-9a-f]{64}")
|
||||
GIT_SHA = re.compile(r"[0-9a-f]{40}")
|
||||
|
||||
|
||||
def require(condition: bool, message: str) -> None:
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def timestamp(value: object, field: str) -> dt.datetime:
|
||||
require(isinstance(value, str) and value.endswith("Z"),
|
||||
f"{field} must be a UTC timestamp ending in Z.")
|
||||
try:
|
||||
parsed = dt.datetime.fromisoformat(value.removesuffix("Z") + "+00:00")
|
||||
except ValueError as error:
|
||||
raise ValueError(f"{field} is not a valid timestamp.") from error
|
||||
require(parsed.tzinfo == dt.timezone.utc, f"{field} must be UTC.")
|
||||
return parsed
|
||||
|
||||
|
||||
def safe_name(value: object, field: str) -> str:
|
||||
name = str(value or "").strip()
|
||||
require(2 <= len(name) <= 120 and "@" not in name and "/" not in name and "\\" not in name,
|
||||
f"{field} requires a name without an email address or path.")
|
||||
return name
|
||||
|
||||
|
||||
def validate(record: object, expected_commit: str, expected_release_sha256: str) -> None:
|
||||
require(isinstance(record, dict), "Acceptance record must be a JSON object.")
|
||||
require(record.get("schemaVersion") == 1, "Unsupported backup/recovery schema.")
|
||||
require(record.get("system") == "guestops-backup-recovery",
|
||||
"system must be guestops-backup-recovery.")
|
||||
require(record.get("evidenceId") == "backup-restore",
|
||||
"evidenceId must be backup-restore.")
|
||||
require(record.get("dataClassification") == "synthetic-only",
|
||||
"Recovery acceptance must use synthetic data only.")
|
||||
require(record.get("releaseVersion") == VERSION, f"releaseVersion must be {VERSION}.")
|
||||
require(GIT_SHA.fullmatch(str(expected_commit)) is not None,
|
||||
"Expected release commit must be a full lowercase Git SHA.")
|
||||
require(SHA256.fullmatch(str(expected_release_sha256)) is not None,
|
||||
"Expected release-record checksum must be a lowercase SHA-256 digest.")
|
||||
require(record.get("releaseCommit") == expected_commit,
|
||||
"releaseCommit does not match the approved candidate.")
|
||||
require(record.get("releaseRecordSha256") == expected_release_sha256,
|
||||
"releaseRecordSha256 does not match the retained release record.")
|
||||
require(SHA256.fullmatch(str(record.get("archiveSha256", ""))) is not None,
|
||||
"archiveSha256 must be a lowercase SHA-256 digest.")
|
||||
|
||||
origin = urlparse(str(record.get("environment", "")))
|
||||
require(origin.scheme == "https" and origin.hostname and origin.path in ("", "/")
|
||||
and not origin.query and not origin.fragment and origin.username is None
|
||||
and origin.password is None,
|
||||
"environment must be an HTTPS origin without credentials, path, query or fragment.")
|
||||
safe_name(record.get("hostIdentifier"), "hostIdentifier")
|
||||
|
||||
images = record.get("images")
|
||||
require(isinstance(images, dict) and set(images) == {"api", "worker", "mongo"},
|
||||
"images must contain exactly api, worker and mongo.")
|
||||
for name in ("api", "worker"):
|
||||
image = images[name]
|
||||
require(isinstance(image, dict) and set(image) == {"reference", "id"},
|
||||
f"images.{name} must contain exactly reference and id.")
|
||||
require(image["reference"] == f"guestops-{name}:{expected_commit}",
|
||||
f"images.{name}.reference must use the full approved commit.")
|
||||
require(re.fullmatch(r"sha256:[0-9a-f]{64}", str(image["id"])) is not None,
|
||||
f"images.{name}.id must be immutable.")
|
||||
mongo = images["mongo"]
|
||||
require(isinstance(mongo, dict) and set(mongo) == {"reference", "id"}
|
||||
and mongo["reference"] == "mongo:8.0"
|
||||
and re.fullmatch(r"sha256:[0-9a-f]{64}", str(mongo["id"])) is not None,
|
||||
"images.mongo must identify the immutable mongo:8.0 image.")
|
||||
|
||||
owners = record.get("owners")
|
||||
owner_keys = {"backup", "monitoring", "recoveryOperator", "technicalEscalation",
|
||||
"retention", "independentReviewer"}
|
||||
require(isinstance(owners, dict) and set(owners) == owner_keys,
|
||||
"owners must contain the exact operational and review roles.")
|
||||
names = {key: safe_name(value, f"owners.{key}") for key, value in owners.items()}
|
||||
reviewer = names["independentReviewer"].casefold()
|
||||
require(reviewer not in {names[key].casefold() for key in owner_keys - {"independentReviewer"}},
|
||||
"independentReviewer must be different from every operational owner.")
|
||||
|
||||
started = timestamp(record.get("startedAt"), "startedAt")
|
||||
ended = timestamp(record.get("endedAt"), "endedAt")
|
||||
reviewed = timestamp(record.get("reviewedAt"), "reviewedAt")
|
||||
require(started <= ended <= reviewed, "Acceptance timestamps are out of order.")
|
||||
|
||||
recovery = record.get("recoveryObjectives")
|
||||
require(isinstance(recovery, dict) and set(recovery) == {
|
||||
"targetRpoHours", "observedRpoHours", "targetRtoMinutes", "observedRtoMinutes",
|
||||
}, "recoveryObjectives must contain exact target and observed RPO/RTO values.")
|
||||
for field in recovery:
|
||||
require(isinstance(recovery[field], (int, float)) and not isinstance(recovery[field], bool)
|
||||
and recovery[field] >= 0, f"recoveryObjectives.{field} must be non-negative.")
|
||||
require(recovery["targetRpoHours"] == 24 and recovery["observedRpoHours"] <= 24,
|
||||
"Observed RPO must meet the approved 24-hour target.")
|
||||
require(recovery["targetRtoMinutes"] == 240 and recovery["observedRtoMinutes"] <= 240,
|
||||
"Observed RTO must meet the approved four-hour target.")
|
||||
|
||||
retention = record.get("retention")
|
||||
require(retention == {
|
||||
"localVerifiedDays": 7,
|
||||
"offHostDaily": 35,
|
||||
"offHostMonthly": 12,
|
||||
"legalHoldOverrideTested": True,
|
||||
}, "Retention must record seven local days, 35 daily and 12 monthly off-host copies, and legal-hold testing.")
|
||||
|
||||
backup = record.get("backup")
|
||||
require(isinstance(backup, dict) and set(backup) == {
|
||||
"createdAt", "sha256", "transferredSha256", "privateKeyPresentOnHost",
|
||||
}, "backup must contain exact creation, checksum, transfer and private-key fields.")
|
||||
created = timestamp(backup["createdAt"], "backup.createdAt")
|
||||
require(created <= started, "The accepted backup must exist when the timed exercise starts.")
|
||||
require(abs(recovery["observedRpoHours"] - (started - created).total_seconds() / 3600) < 0.01,
|
||||
"Observed RPO must match the backup and exercise timestamps.")
|
||||
require(abs(recovery["observedRtoMinutes"] - (ended - started).total_seconds() / 60) < 0.01,
|
||||
"Observed RTO must match the exercise timestamps.")
|
||||
require(SHA256.fullmatch(str(backup["sha256"])) is not None
|
||||
and backup["transferredSha256"] == backup["sha256"],
|
||||
"Local and transferred backup checksums must match.")
|
||||
require(backup["privateKeyPresentOnHost"] is False,
|
||||
"The recovery private key must not be present on the Debian host.")
|
||||
|
||||
rollback = record.get("rollback")
|
||||
require(isinstance(rollback, dict) and set(rollback) == {
|
||||
"previousReleaseCommit", "previousArchiveSha256", "previousImages",
|
||||
"persistentVolumesReplaced", "restoredReleaseCommit", "unresolvedOperations", "finalControls",
|
||||
}, "rollback must contain the exact rehearsal and final-state fields.")
|
||||
require(GIT_SHA.fullmatch(str(rollback["previousReleaseCommit"])) is not None
|
||||
and rollback["previousReleaseCommit"] != expected_commit,
|
||||
"Rollback must use a different retained previous release.")
|
||||
require(SHA256.fullmatch(str(rollback["previousArchiveSha256"])) is not None,
|
||||
"Rollback requires the previous archive checksum.")
|
||||
previous_images = rollback["previousImages"]
|
||||
require(isinstance(previous_images, dict) and set(previous_images) == {"api", "worker"},
|
||||
"Rollback requires exact previous API and worker images.")
|
||||
for name in ("api", "worker"):
|
||||
image = previous_images[name]
|
||||
require(isinstance(image, dict) and set(image) == {"reference", "id"}
|
||||
and image["reference"] == f"guestops-{name}:{rollback['previousReleaseCommit']}"
|
||||
and re.fullmatch(r"sha256:[0-9a-f]{64}", str(image["id"])) is not None,
|
||||
f"rollback.previousImages.{name} must use the retained previous release identity.")
|
||||
require(rollback["persistentVolumesReplaced"] is False,
|
||||
"Image rollback must not replace persistent volumes.")
|
||||
require(rollback["restoredReleaseCommit"] == expected_commit,
|
||||
"The exercise must finish on the approved candidate.")
|
||||
require(rollback["unresolvedOperations"] == 0,
|
||||
"The exercise must finish without unresolved operations.")
|
||||
require(rollback["finalControls"] == {
|
||||
"googleSending": "disabled", "faqLiveMode": "disabled",
|
||||
"pmsWrites": "disabled", "paymentCreation": "disabled",
|
||||
}, "The exercise must finish with all unaccepted external writes disabled.")
|
||||
|
||||
scenarios = record.get("scenarios")
|
||||
require(isinstance(scenarios, list), "scenarios must be a list.")
|
||||
ids = [item.get("id") for item in scenarios if isinstance(item, dict)]
|
||||
require(len(ids) == len(scenarios) and len(ids) == len(set(ids)) and set(ids) == SCENARIOS,
|
||||
"Acceptance record requires the exact backup/recovery scenario set.")
|
||||
for item in scenarios:
|
||||
scenario_id = item["id"]
|
||||
require(item.get("status") == "pass", f"Scenario {scenario_id} has not passed.")
|
||||
evidence = item.get("evidence")
|
||||
require(isinstance(evidence, list) and 1 <= len(evidence) <= 10 and all(
|
||||
isinstance(value, str) and 3 <= len(value) <= 200 and "@" not in value
|
||||
and "\\" not in value and not value.startswith("/") for value in evidence
|
||||
), f"Scenario {scenario_id} requires safe opaque evidence references.")
|
||||
require(record.get("monitoringState") == "healthy",
|
||||
"Monitoring must be healthy at acceptance completion.")
|
||||
require(record.get("unresolvedCriticalFindings") == 0,
|
||||
"Acceptance cannot pass with unresolved critical findings.")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("record", type=Path)
|
||||
parser.add_argument("--expected-commit", required=True)
|
||||
parser.add_argument("--expected-release-record-sha256", required=True)
|
||||
args = parser.parse_args()
|
||||
validate(json.loads(args.record.read_text(encoding="utf-8")),
|
||||
args.expected_commit, args.expected_release_record_sha256)
|
||||
print("Backup, monitoring and recovery acceptance record is structurally complete and passed. "
|
||||
"This validates the record, not its restricted evidence.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (OSError, ValueError, json.JSONDecodeError) as error:
|
||||
print(f"Backup/recovery acceptance record rejected: {error}", file=__import__("sys").stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -1,238 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Atomically transfer encrypted GuestOps backups to restricted storage."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shlex
|
||||
import shutil
|
||||
import stat
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
import uuid
|
||||
|
||||
|
||||
BACKUP_NAME = re.compile(r"guestops-[0-9]{8}T[0-9]{6}Z\.tar\.gpg")
|
||||
SAFE_HOST = re.compile(r"[A-Za-z0-9.-]{1,253}")
|
||||
SAFE_USER = re.compile(r"[A-Za-z_][A-Za-z0-9_-]{0,31}")
|
||||
SAFE_REMOTE_PATH = re.compile(r"/[A-Za-z0-9._/-]{1,500}")
|
||||
SHA256 = re.compile(r"[0-9a-f]{64}")
|
||||
|
||||
|
||||
def require(condition: bool, message: str) -> None:
|
||||
if not condition:
|
||||
raise RuntimeError(message)
|
||||
|
||||
|
||||
def digest(path: Path) -> str:
|
||||
with path.open("rb") as stream:
|
||||
return hashlib.file_digest(stream, "sha256").hexdigest()
|
||||
|
||||
|
||||
def private_directory(value: str) -> Path:
|
||||
requested = Path(value)
|
||||
require(requested.is_absolute() and not requested.is_symlink(),
|
||||
"BACKUP_DIRECTORY must be an absolute, non-symlink path.")
|
||||
directory = requested.resolve()
|
||||
require(directory.is_dir(), "BACKUP_DIRECTORY must exist.")
|
||||
require(stat.S_IMODE(directory.stat().st_mode) & 0o077 == 0,
|
||||
"BACKUP_DIRECTORY must not be accessible to group or other users.")
|
||||
return directory
|
||||
|
||||
|
||||
def regular_file(value: str, field: str, *, private: bool) -> Path:
|
||||
requested = Path(value)
|
||||
require(requested.is_absolute() and not requested.is_symlink(),
|
||||
f"{field} must be an absolute, non-symlink path.")
|
||||
path = requested.resolve()
|
||||
require(path.is_file(), f"{field} must be an existing regular file.")
|
||||
if private:
|
||||
require(stat.S_IMODE(path.stat().st_mode) & 0o077 == 0,
|
||||
f"{field} must not be accessible to group or other users.")
|
||||
return path
|
||||
|
||||
|
||||
def configuration(environment: dict[str, str]) -> dict[str, object]:
|
||||
directory = private_directory(environment.get("BACKUP_DIRECTORY", ""))
|
||||
host = environment.get("BACKUP_REMOTE_HOST", "")
|
||||
user = environment.get("BACKUP_REMOTE_USER", "")
|
||||
remote = environment.get("BACKUP_REMOTE_DIRECTORY", "")
|
||||
require(SAFE_HOST.fullmatch(host) is not None, "BACKUP_REMOTE_HOST is invalid.")
|
||||
require(SAFE_USER.fullmatch(user) is not None, "BACKUP_REMOTE_USER is invalid.")
|
||||
require(SAFE_REMOTE_PATH.fullmatch(remote) is not None and "//" not in remote
|
||||
and "/../" not in remote + "/" and not remote.endswith("/.."),
|
||||
"BACKUP_REMOTE_DIRECTORY must be a safe absolute path.")
|
||||
identity = regular_file(environment.get("BACKUP_SSH_IDENTITY", ""),
|
||||
"BACKUP_SSH_IDENTITY", private=True)
|
||||
known_hosts = regular_file(environment.get("BACKUP_SSH_KNOWN_HOSTS", ""),
|
||||
"BACKUP_SSH_KNOWN_HOSTS", private=False)
|
||||
require(shutil.which("ssh") is not None and shutil.which("rsync") is not None,
|
||||
"ssh and rsync are required.")
|
||||
return {
|
||||
"directory": directory,
|
||||
"host": host,
|
||||
"user": user,
|
||||
"remote": remote.rstrip("/"),
|
||||
"identity": identity,
|
||||
"known_hosts": known_hosts,
|
||||
}
|
||||
|
||||
|
||||
def ssh_base(config: dict[str, object]) -> list[str]:
|
||||
return [
|
||||
"ssh", "-o", "BatchMode=yes", "-o", "IdentitiesOnly=yes",
|
||||
"-o", "StrictHostKeyChecking=yes", "-o", "ConnectTimeout=15",
|
||||
"-o", f"UserKnownHostsFile={config['known_hosts']}",
|
||||
"-i", str(config["identity"]), f"{config['user']}@{config['host']}",
|
||||
]
|
||||
|
||||
|
||||
def run(args: list[str], *, environment: dict[str, str] | None = None) -> bytes:
|
||||
result = subprocess.run(args, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE, timeout=900, env=environment)
|
||||
require(result.returncode == 0,
|
||||
f"{Path(args[0]).name} step failed; review the restricted operator logs.")
|
||||
return result.stdout
|
||||
|
||||
|
||||
def remote_digest(config: dict[str, object], remote_path: str) -> str | None:
|
||||
command = f"if test -f {remote_path} && test ! -L {remote_path}; then sha256sum -- {remote_path}; fi"
|
||||
output = run([*ssh_base(config), command]).decode("utf-8", "strict").strip()
|
||||
if not output:
|
||||
return None
|
||||
value = output.split()[0]
|
||||
require(SHA256.fullmatch(value) is not None, "Remote checksum response was invalid.")
|
||||
return value
|
||||
|
||||
|
||||
def marker_path(backup: Path) -> Path:
|
||||
return backup.with_name(backup.name + ".transferred.json")
|
||||
|
||||
|
||||
def write_marker(backup: Path, checksum: str) -> None:
|
||||
marker = marker_path(backup)
|
||||
payload = json.dumps({
|
||||
"schemaVersion": 1,
|
||||
"backup": backup.name,
|
||||
"sha256": checksum,
|
||||
"verifiedAt": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
}, sort_keys=True) + "\n"
|
||||
fd, temporary = tempfile.mkstemp(prefix=marker.name + ".", dir=marker.parent)
|
||||
try:
|
||||
os.fchmod(fd, 0o600)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as stream:
|
||||
stream.write(payload)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temporary, marker)
|
||||
finally:
|
||||
try:
|
||||
os.unlink(temporary)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
|
||||
def transfer_one(config: dict[str, object], backup: Path) -> None:
|
||||
require(backup.is_file() and not backup.is_symlink()
|
||||
and BACKUP_NAME.fullmatch(backup.name) is not None,
|
||||
"Refusing to transfer an unexpected backup path.")
|
||||
checksum = digest(backup)
|
||||
remote_final = f"{config['remote']}/{backup.name}"
|
||||
existing = remote_digest(config, remote_final)
|
||||
if existing is not None:
|
||||
require(existing == checksum, "A remote backup with this name has a different checksum.")
|
||||
write_marker(backup, checksum)
|
||||
return
|
||||
|
||||
remote_partial = f"{config['remote']}/.{backup.name}.partial-{uuid.uuid4().hex}"
|
||||
rsh = shlex.join([
|
||||
"ssh", "-o", "BatchMode=yes", "-o", "IdentitiesOnly=yes",
|
||||
"-o", "StrictHostKeyChecking=yes", "-o", "ConnectTimeout=15",
|
||||
"-o", f"UserKnownHostsFile={config['known_hosts']}",
|
||||
"-i", str(config["identity"]),
|
||||
])
|
||||
rsync_environment = os.environ.copy()
|
||||
rsync_environment["RSYNC_RSH"] = rsh
|
||||
try:
|
||||
run(["rsync", "--archive", "--chmod=F600", "--protect-args", "--",
|
||||
str(backup), f"{config['user']}@{config['host']}:{remote_partial}"],
|
||||
environment=rsync_environment)
|
||||
require(remote_digest(config, remote_partial) == checksum,
|
||||
"Transferred backup checksum does not match the local file.")
|
||||
command = (f"test ! -e {remote_final} && mv -T -- {remote_partial} {remote_final} "
|
||||
f"&& chmod 600 -- {remote_final}")
|
||||
run([*ssh_base(config), command])
|
||||
require(remote_digest(config, remote_final) == checksum,
|
||||
"Final remote backup checksum does not match the local file.")
|
||||
write_marker(backup, checksum)
|
||||
except Exception:
|
||||
# The name contains a fresh random suffix and is the only remote object this run may remove.
|
||||
subprocess.run([*ssh_base(config), f"rm -f -- {remote_partial}"], stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=30)
|
||||
raise
|
||||
|
||||
|
||||
def transfer_pending(config: dict[str, object]) -> int:
|
||||
directory = config["directory"]
|
||||
backups = sorted(path for path in directory.iterdir()
|
||||
if path.is_file() and not path.is_symlink()
|
||||
and BACKUP_NAME.fullmatch(path.name) is not None)
|
||||
for backup in backups:
|
||||
marker = marker_path(backup)
|
||||
if marker.is_file() and not marker.is_symlink():
|
||||
try:
|
||||
recorded = json.loads(marker.read_text(encoding="utf-8"))
|
||||
if recorded.get("sha256") == digest(backup):
|
||||
continue
|
||||
except (OSError, ValueError, json.JSONDecodeError):
|
||||
pass
|
||||
transfer_one(config, backup)
|
||||
return len(backups)
|
||||
|
||||
|
||||
def prune_verified(config: dict[str, object], retention_days: int, now: float | None = None) -> int:
|
||||
require(1 <= retention_days <= 365, "Local retention must be between 1 and 365 days.")
|
||||
threshold = (time.time() if now is None else now) - retention_days * 86400
|
||||
removed = 0
|
||||
for backup in config["directory"].iterdir():
|
||||
if not backup.is_file() or backup.is_symlink() or BACKUP_NAME.fullmatch(backup.name) is None:
|
||||
continue
|
||||
marker = marker_path(backup)
|
||||
if backup.stat().st_mtime >= threshold or not marker.is_file() or marker.is_symlink():
|
||||
continue
|
||||
try:
|
||||
recorded = json.loads(marker.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError, json.JSONDecodeError):
|
||||
continue
|
||||
if recorded.get("backup") != backup.name or recorded.get("sha256") != digest(backup):
|
||||
continue
|
||||
backup.unlink()
|
||||
marker.unlink()
|
||||
removed += 1
|
||||
return removed
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--prune-verified", action="store_true")
|
||||
parser.add_argument("--retention-days", type=int, default=7)
|
||||
args = parser.parse_args()
|
||||
config = configuration(dict(os.environ))
|
||||
observed = transfer_pending(config)
|
||||
removed = prune_verified(config, args.retention_days) if args.prune_verified else 0
|
||||
print(f"Backup transfer completed: {observed} encrypted backup(s) inspected; "
|
||||
f"{removed} verified local backup(s) expired.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (OSError, RuntimeError, subprocess.SubprocessError, json.JSONDecodeError) as error:
|
||||
print(f"Backup transfer failed: {error}", file=__import__("sys").stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -1,128 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run a bounded, read-only capacity probe against an approved GuestOps sandbox."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import datetime as dt
|
||||
import http.cookiejar
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import statistics
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
|
||||
PATHS = ("/health/ready", "/api/hotel", "/api/conversations/page")
|
||||
|
||||
|
||||
def require(condition: bool, message: str) -> None:
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def percentile(values: list[float], fraction: float) -> float:
|
||||
ordered = sorted(values)
|
||||
position = max(0, min(len(ordered) - 1, int(len(ordered) * fraction + 0.999999) - 1))
|
||||
return ordered[position]
|
||||
|
||||
|
||||
def summarize(results: list[tuple[bool, float]], concurrency: int) -> dict[str, object]:
|
||||
require(len(results) > 0, "At least one probe result is required.")
|
||||
latencies = [latency for _, latency in results]
|
||||
successes = sum(1 for success, _ in results if success)
|
||||
return {
|
||||
"concurrency": concurrency,
|
||||
"requests": len(results),
|
||||
"successes": successes,
|
||||
"failures": len(results) - successes,
|
||||
"errorRate": round((len(results) - successes) / len(results), 6),
|
||||
"latencyMs": {
|
||||
"median": round(statistics.median(latencies), 2),
|
||||
"p95": round(percentile(latencies, 0.95), 2),
|
||||
"maximum": round(max(latencies), 2),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def login(origin: str, email: str, password: str) -> str:
|
||||
jar = http.cookiejar.CookieJar()
|
||||
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(jar))
|
||||
with opener.open(origin + "/api/session", timeout=15) as response:
|
||||
csrf = json.load(response)["csrfToken"]
|
||||
body = json.dumps({"email": email, "password": password}).encode()
|
||||
request = urllib.request.Request(origin + "/api/auth/login", data=body, method="POST", headers={"Content-Type": "application/json", "X-CSRF-TOKEN": csrf})
|
||||
with opener.open(request, timeout=15) as response:
|
||||
require(response.status == 200, "Sandbox login failed.")
|
||||
cookies = "; ".join(f"{cookie.name}={cookie.value}" for cookie in jar)
|
||||
require("guestops.session=" in cookies, "Sandbox did not issue a GuestOps session cookie.")
|
||||
return cookies
|
||||
|
||||
|
||||
def request_once(origin: str, cookie: str, number: int) -> tuple[bool, float]:
|
||||
path = PATHS[number % len(PATHS)]
|
||||
request = urllib.request.Request(origin + path, headers={"Cookie": cookie, "Accept": "application/json"})
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=20) as response:
|
||||
success = response.status == 200
|
||||
response.read(1024) # Bound local processing; never retain response or guest content.
|
||||
except (OSError, urllib.error.HTTPError):
|
||||
success = False
|
||||
return success, (time.perf_counter() - started) * 1000
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--origin", required=True)
|
||||
parser.add_argument("--requests", type=int, default=100)
|
||||
parser.add_argument("--concurrency", type=int, default=5)
|
||||
parser.add_argument("--release-commit", required=True)
|
||||
parser.add_argument("--release-record-sha256", required=True)
|
||||
parser.add_argument("--output", required=True, type=Path)
|
||||
parser.add_argument("--confirm-sandbox", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
parsed = urllib.parse.urlparse(args.origin)
|
||||
require(args.confirm_sandbox, "Use --confirm-sandbox after confirming the target and maintenance window.")
|
||||
require(parsed.scheme == "https" and parsed.hostname and parsed.path in ("", "/") and not parsed.query and not parsed.fragment and not parsed.username,
|
||||
"Origin must be an HTTPS origin without credentials, path, query or fragment.")
|
||||
require(parsed.hostname != "localhost" and not parsed.hostname.startswith("127."), "Use the deployed HTTPS sandbox, not a development server.")
|
||||
require(1 <= args.concurrency <= 20 and 1 <= args.requests <= 2000, "Probe bounds are 1–20 concurrent workers and 1–2000 requests.")
|
||||
require(re.fullmatch(r"[0-9a-f]{40}", args.release_commit) is not None, "Use a full lowercase release commit SHA.")
|
||||
require(re.fullmatch(r"[0-9a-f]{64}", args.release_record_sha256) is not None, "Use the release-record SHA-256.")
|
||||
require(not args.output.exists() and args.output.parent.is_dir(), "Output must be a new file in an existing restricted directory.")
|
||||
email = os.environ.get("CAPACITY_EMAIL", "")
|
||||
password = os.environ.get("CAPACITY_PASSWORD", "")
|
||||
require(email and password, "Set CAPACITY_EMAIL and CAPACITY_PASSWORD for a dedicated sandbox staff account.")
|
||||
|
||||
cookie = login(args.origin.rstrip("/"), email, password)
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=args.concurrency) as pool:
|
||||
results = list(pool.map(lambda number: request_once(args.origin.rstrip("/"), cookie, number), range(args.requests)))
|
||||
report = {
|
||||
"schemaVersion": 1,
|
||||
"kind": "guestops-read-only-capacity",
|
||||
"recordedAt": dt.datetime.now(dt.timezone.utc).isoformat().replace("+00:00", "Z"),
|
||||
"originHost": parsed.hostname,
|
||||
"releaseCommit": args.release_commit,
|
||||
"releaseRecordSha256": args.release_record_sha256,
|
||||
"paths": list(PATHS),
|
||||
**summarize(results, args.concurrency),
|
||||
}
|
||||
descriptor = os.open(args.output, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
with os.fdopen(descriptor, "w", encoding="utf-8") as output:
|
||||
output.write(json.dumps(report, indent=2, sort_keys=True) + "\n")
|
||||
print(f"Capacity probe completed: {report['successes']}/{report['requests']} successful; p95 {report['latencyMs']['p95']} ms. Review against the approved target before release.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (OSError, ValueError, json.JSONDecodeError) as error:
|
||||
print(f"Capacity probe stopped: {error}", file=__import__("sys").stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -1,45 +0,0 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"system": "guestops-debian-host",
|
||||
"evidenceId": "debian-host",
|
||||
"releaseVersion": "0.2.1",
|
||||
"releaseCommit": "0000000000000000000000000000000000000000",
|
||||
"releaseRecordSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"archiveSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"environment": "https://sandbox-guestops.futuresens.co.uk",
|
||||
"hostIdentifier": "guestops-sandbox-01",
|
||||
"images": {
|
||||
"api": {"reference": "guestops-api:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
|
||||
"worker": {"reference": "guestops-worker:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"}
|
||||
},
|
||||
"hostFacts": {
|
||||
"debianMajor": 12,
|
||||
"cpuCores": 4,
|
||||
"memoryBytes": 8140382208,
|
||||
"freeDiskBytes": 14275686400,
|
||||
"publicTcpPorts": []
|
||||
},
|
||||
"featureControls": {
|
||||
"googleSending": "disabled",
|
||||
"faqLiveMode": "disabled",
|
||||
"pmsWrites": "disabled",
|
||||
"paymentCreation": "disabled"
|
||||
},
|
||||
"operator": "REPLACE",
|
||||
"reviewedBy": "REPLACE",
|
||||
"startedAt": "2026-09-30T09:00:00Z",
|
||||
"endedAt": "2026-09-30T10:00:00Z",
|
||||
"reviewedAt": "2026-09-30T11:00:00Z",
|
||||
"unresolvedCriticalFindings": 1,
|
||||
"scenarios": [
|
||||
{"id": "boot-services", "status": "not-run", "evidence": []},
|
||||
{"id": "controlled-reboot", "status": "not-run", "evidence": []},
|
||||
{"id": "durable-log-retrieval", "status": "not-run", "evidence": []},
|
||||
{"id": "host-baseline", "status": "not-run", "evidence": []},
|
||||
{"id": "https-and-redirect", "status": "not-run", "evidence": []},
|
||||
{"id": "network-exposure", "status": "not-run", "evidence": []},
|
||||
{"id": "proxy-trust", "status": "not-run", "evidence": []},
|
||||
{"id": "secret-free-logs", "status": "not-run", "evidence": []},
|
||||
{"id": "workspace-health", "status": "not-run", "evidence": []}
|
||||
]
|
||||
}
|
||||
@@ -1,206 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate restricted GuestOps Debian-host and persistence acceptance records."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
VERSION = "0.2.1"
|
||||
SYSTEMS = {
|
||||
"guestops-debian-host": {
|
||||
"evidenceId": "debian-host",
|
||||
"scenarios": {
|
||||
"host-baseline",
|
||||
"network-exposure",
|
||||
"https-and-redirect",
|
||||
"proxy-trust",
|
||||
"boot-services",
|
||||
"controlled-reboot",
|
||||
"workspace-health",
|
||||
"durable-log-retrieval",
|
||||
"secret-free-logs",
|
||||
},
|
||||
},
|
||||
"guestops-persistence": {
|
||||
"evidenceId": "persistence",
|
||||
"scenarios": {
|
||||
"separate-volume-layout",
|
||||
"service-restart",
|
||||
"container-recreation",
|
||||
"database-inventory",
|
||||
"data-protection-key",
|
||||
"image-identity",
|
||||
"post-reboot-persistence",
|
||||
},
|
||||
},
|
||||
}
|
||||
SHA256 = re.compile(r"[0-9a-f]{64}")
|
||||
GIT_SHA = re.compile(r"[0-9a-f]{40}")
|
||||
|
||||
|
||||
def require(condition: bool, message: str) -> None:
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def timestamp(value: object, field: str) -> dt.datetime:
|
||||
require(isinstance(value, str) and value.endswith("Z"),
|
||||
f"{field} must be a UTC timestamp ending in Z.")
|
||||
try:
|
||||
parsed = dt.datetime.fromisoformat(value.removesuffix("Z") + "+00:00")
|
||||
except ValueError as error:
|
||||
raise ValueError(f"{field} is not a valid timestamp.") from error
|
||||
require(parsed.tzinfo == dt.timezone.utc, f"{field} must be UTC.")
|
||||
return parsed
|
||||
|
||||
|
||||
def safe_text(value: object, field: str, minimum: int = 2, maximum: int = 160) -> str:
|
||||
text = str(value or "").strip()
|
||||
require(minimum <= len(text) <= maximum and "@" not in text and "\\" not in text,
|
||||
f"{field} must be safe text without an email address or local path.")
|
||||
return text
|
||||
|
||||
|
||||
def validate_common(record: object, expected_commit: str, expected_release_sha256: str) -> str:
|
||||
require(isinstance(record, dict), "Acceptance record must be a JSON object.")
|
||||
require(record.get("schemaVersion") == 1, "Unsupported Debian acceptance schema.")
|
||||
system = record.get("system")
|
||||
require(system in SYSTEMS, "Unknown Debian acceptance record system.")
|
||||
require(record.get("evidenceId") == SYSTEMS[system]["evidenceId"],
|
||||
f"{system} has the wrong evidenceId.")
|
||||
require(record.get("releaseVersion") == VERSION,
|
||||
f"releaseVersion must be {VERSION}.")
|
||||
require(GIT_SHA.fullmatch(str(expected_commit)) is not None,
|
||||
"Expected release commit must be a full lowercase Git SHA.")
|
||||
require(SHA256.fullmatch(str(expected_release_sha256)) is not None,
|
||||
"Expected release-record checksum must be a lowercase SHA-256 digest.")
|
||||
require(record.get("releaseCommit") == expected_commit,
|
||||
"releaseCommit does not match the approved candidate.")
|
||||
require(record.get("releaseRecordSha256") == expected_release_sha256,
|
||||
"releaseRecordSha256 does not match the retained release record.")
|
||||
require(SHA256.fullmatch(str(record.get("archiveSha256", ""))) is not None,
|
||||
"archiveSha256 must be a lowercase SHA-256 digest.")
|
||||
|
||||
origin = urlparse(str(record.get("environment", "")))
|
||||
require(origin.scheme == "https" and origin.hostname and origin.path in ("", "/")
|
||||
and not origin.query and not origin.fragment and origin.username is None
|
||||
and origin.password is None,
|
||||
"environment must be an HTTPS origin without credentials, path, query or fragment.")
|
||||
host = safe_text(record.get("hostIdentifier"), "hostIdentifier")
|
||||
|
||||
images = record.get("images")
|
||||
require(isinstance(images, dict) and set(images) == {"api", "worker"},
|
||||
"images must contain exactly api and worker.")
|
||||
for name in ("api", "worker"):
|
||||
image = images[name]
|
||||
require(isinstance(image, dict) and set(image) == {"reference", "id"},
|
||||
f"images.{name} must contain exactly reference and id.")
|
||||
require(image["reference"] == f"guestops-{name}:{expected_commit}",
|
||||
f"images.{name}.reference must use the full approved commit.")
|
||||
require(re.fullmatch(r"sha256:[0-9a-f]{64}", str(image["id"])) is not None,
|
||||
f"images.{name}.id must be an immutable image ID.")
|
||||
|
||||
operator = safe_text(record.get("operator"), "operator")
|
||||
reviewer = safe_text(record.get("reviewedBy"), "reviewedBy")
|
||||
require(operator.casefold() != reviewer.casefold(),
|
||||
"operator and reviewedBy must be different people.")
|
||||
started = timestamp(record.get("startedAt"), "startedAt")
|
||||
ended = timestamp(record.get("endedAt"), "endedAt")
|
||||
reviewed = timestamp(record.get("reviewedAt"), "reviewedAt")
|
||||
require(started <= ended <= reviewed, "Acceptance timestamps are out of order.")
|
||||
|
||||
scenarios = record.get("scenarios")
|
||||
require(isinstance(scenarios, list), "scenarios must be a list.")
|
||||
ids = [item.get("id") for item in scenarios if isinstance(item, dict)]
|
||||
required = SYSTEMS[system]["scenarios"]
|
||||
require(len(ids) == len(scenarios) and len(ids) == len(set(ids)) and set(ids) == required,
|
||||
f"{system} requires its exact acceptance scenario set.")
|
||||
for item in scenarios:
|
||||
scenario_id = item["id"]
|
||||
require(item.get("status") == "pass", f"Scenario {scenario_id} has not passed.")
|
||||
evidence = item.get("evidence")
|
||||
require(isinstance(evidence, list) and 1 <= len(evidence) <= 10 and all(
|
||||
isinstance(value, str) and 3 <= len(value) <= 200 and "@" not in value
|
||||
and "\\" not in value and not value.startswith("/") for value in evidence
|
||||
), f"Scenario {scenario_id} requires safe opaque evidence references.")
|
||||
|
||||
require(record.get("unresolvedCriticalFindings") == 0,
|
||||
"Acceptance cannot pass with unresolved critical findings.")
|
||||
return host
|
||||
|
||||
|
||||
def validate_host(record: object, expected_commit: str, expected_release_sha256: str) -> str:
|
||||
host = validate_common(record, expected_commit, expected_release_sha256)
|
||||
require(record.get("system") == "guestops-debian-host",
|
||||
"First record must be guestops-debian-host.")
|
||||
facts = record.get("hostFacts")
|
||||
require(isinstance(facts, dict) and set(facts) == {
|
||||
"debianMajor", "cpuCores", "memoryBytes", "freeDiskBytes", "publicTcpPorts",
|
||||
}, "hostFacts must contain the exact reviewed host facts.")
|
||||
require(isinstance(facts["debianMajor"], int) and facts["debianMajor"] >= 12,
|
||||
"Debian 12 or newer is required.")
|
||||
require(isinstance(facts["cpuCores"], int) and facts["cpuCores"] >= 4,
|
||||
"At least four CPU cores are required.")
|
||||
require(isinstance(facts["memoryBytes"], int) and facts["memoryBytes"] >= 7_500_000_000,
|
||||
"At least 7.5 GB of memory is required.")
|
||||
require(isinstance(facts["freeDiskBytes"], int) and facts["freeDiskBytes"] >= 8 * 1024**3,
|
||||
"At least 8 GiB of free disk space is required.")
|
||||
require(facts["publicTcpPorts"] == [80, 443],
|
||||
"Only TCP ports 80 and 443 may be public.")
|
||||
require(record.get("featureControls") == {
|
||||
"googleSending": "disabled",
|
||||
"faqLiveMode": "disabled",
|
||||
"pmsWrites": "disabled",
|
||||
"paymentCreation": "disabled",
|
||||
}, "Unaccepted external writes and FAQ live mode must remain disabled.")
|
||||
return host
|
||||
|
||||
|
||||
def validate_persistence(record: object, expected_commit: str, expected_release_sha256: str) -> str:
|
||||
host = validate_common(record, expected_commit, expected_release_sha256)
|
||||
require(record.get("system") == "guestops-persistence",
|
||||
"Second record must be guestops-persistence.")
|
||||
require(record.get("drillCommand") == "python3 deploy/ops.py persistence-drill --confirm-restart",
|
||||
"drillCommand must identify the confirmation-gated persistence drill.")
|
||||
return host
|
||||
|
||||
|
||||
def validate_pair(host_record: object, persistence_record: object,
|
||||
expected_commit: str, expected_release_sha256: str) -> None:
|
||||
host = validate_host(host_record, expected_commit, expected_release_sha256)
|
||||
persistence_host = validate_persistence(
|
||||
persistence_record, expected_commit, expected_release_sha256)
|
||||
require(host == persistence_host, "Both records must identify the same host.")
|
||||
for field in ("environment", "releaseVersion", "releaseCommit", "releaseRecordSha256",
|
||||
"archiveSha256", "images"):
|
||||
require(host_record.get(field) == persistence_record.get(field),
|
||||
f"Both records must use the same {field}.")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("host_record", type=Path)
|
||||
parser.add_argument("persistence_record", type=Path)
|
||||
parser.add_argument("--expected-commit", required=True)
|
||||
parser.add_argument("--expected-release-record-sha256", required=True)
|
||||
args = parser.parse_args()
|
||||
host_record = json.loads(args.host_record.read_text(encoding="utf-8"))
|
||||
persistence_record = json.loads(args.persistence_record.read_text(encoding="utf-8"))
|
||||
validate_pair(host_record, persistence_record,
|
||||
args.expected_commit, args.expected_release_record_sha256)
|
||||
print("Debian-host and persistence acceptance records are structurally complete and passed. "
|
||||
"This validates the records, not their restricted evidence.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (OSError, ValueError, json.JSONDecodeError) as error:
|
||||
print(f"Debian acceptance records rejected: {error}", file=__import__("sys").stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -1,220 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate the complete release-bound GuestOps Gate B acceptance bundle."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import sys
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
DEPLOY = Path(__file__).resolve().parent
|
||||
if str(DEPLOY) not in sys.path:
|
||||
sys.path.insert(0, str(DEPLOY))
|
||||
|
||||
import automation_acceptance
|
||||
import backup_restore_acceptance
|
||||
import debian_acceptance
|
||||
import desktop_acceptance
|
||||
import google_acceptance
|
||||
import identity_privacy_acceptance
|
||||
import incident_exercise
|
||||
import pilot_approval
|
||||
import pilot_run
|
||||
import verify_release
|
||||
import verify_source_package
|
||||
|
||||
|
||||
VERSION = "0.2.1"
|
||||
SHA256 = re.compile(r"[0-9a-f]{64}")
|
||||
RECORD_KEYS = {
|
||||
"debian-host", "persistence", "backup-restore", "google-mailbox",
|
||||
"automation", "identity-privacy", "inbox-usability", "capacity",
|
||||
"incident-support", "pilot-findings", "pilot-approval",
|
||||
}
|
||||
|
||||
|
||||
def require(condition: bool, message: str) -> None:
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def digest(path: Path) -> str:
|
||||
with path.open("rb") as stream:
|
||||
return hashlib.file_digest(stream, "sha256").hexdigest()
|
||||
|
||||
|
||||
def load_json(path: Path) -> object:
|
||||
require(path.is_file() and not path.is_symlink(), f"Required bundle file is missing or is a symlink: {path.name}")
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def release_binding(record: object, name: str, commit: str, release_sha: str) -> None:
|
||||
require(isinstance(record, dict), f"{name} must be a JSON object.")
|
||||
require(record.get("releaseCommit") == commit, f"{name} uses a different releaseCommit.")
|
||||
require(record.get("releaseRecordSha256") == release_sha,
|
||||
f"{name} uses a different releaseRecordSha256.")
|
||||
|
||||
|
||||
def environment_origin(record: object, name: str) -> str:
|
||||
require(isinstance(record, dict), f"{name} must be a JSON object.")
|
||||
value = str(record.get("environment", "")).rstrip("/")
|
||||
parsed = urlparse(value)
|
||||
require(parsed.scheme == "https" and parsed.hostname and parsed.path in ("", "/"),
|
||||
f"{name} has no valid HTTPS environment.")
|
||||
return value
|
||||
|
||||
|
||||
def validate_capacity(report: object, commit: str, release_sha: str) -> None:
|
||||
require(isinstance(report, dict), "capacity must be a JSON object.")
|
||||
require(report.get("schemaVersion") == 1 and report.get("kind") == "guestops-read-only-capacity",
|
||||
"capacity has an unsupported schema or kind.")
|
||||
require(report.get("releaseCommit") == commit, "capacity uses a different releaseCommit.")
|
||||
require(report.get("releaseRecordSha256") == release_sha,
|
||||
"capacity uses a different releaseRecordSha256.")
|
||||
require(report.get("paths") == ["/health/ready", "/api/hotel", "/api/conversations/page"],
|
||||
"capacity must use the exact read-only path set.")
|
||||
for field in ("concurrency", "requests", "successes", "failures"):
|
||||
require(isinstance(report.get(field), int) and not isinstance(report.get(field), bool),
|
||||
f"capacity.{field} must be an integer.")
|
||||
require(1 <= report["concurrency"] <= 20 and report["requests"] > 0,
|
||||
"capacity has invalid concurrency or request count.")
|
||||
require(report["successes"] + report["failures"] == report["requests"],
|
||||
"capacity success and failure counts do not match requests.")
|
||||
expected_error = round(report["failures"] / report["requests"], 6)
|
||||
require(report.get("errorRate") == expected_error, "capacity.errorRate does not match its counts.")
|
||||
latency = report.get("latencyMs")
|
||||
require(isinstance(latency, dict) and set(latency) == {"median", "p95", "maximum"}
|
||||
and all(isinstance(value, (int, float)) and not isinstance(value, bool) and value >= 0
|
||||
for value in latency.values()), "capacity.latencyMs is invalid.")
|
||||
require(latency["median"] <= latency["p95"] <= latency["maximum"],
|
||||
"capacity latency percentiles are out of order.")
|
||||
|
||||
|
||||
def validate_bundle(
|
||||
source_archive: Path,
|
||||
source_record_path: Path,
|
||||
release_record_path: Path,
|
||||
records: dict[str, tuple[Path, object]],
|
||||
host_metrics_path: Path,
|
||||
) -> dict[str, object]:
|
||||
require(set(records) == RECORD_KEYS, "Gate B bundle requires the exact record set.")
|
||||
release_record = load_json(release_record_path)
|
||||
require(isinstance(release_record, dict), "release-record must be a JSON object.")
|
||||
commit = str(release_record.get("commit", ""))
|
||||
version = str(release_record.get("version", ""))
|
||||
require(version == VERSION, f"Gate B bundle version must be {VERSION}.")
|
||||
release_result = verify_release.validate(source_archive, release_record_path, commit, version)
|
||||
source_result = verify_source_package.validate(source_archive, source_record_path, commit, version)
|
||||
release_sha = str(release_result["releaseRecord"]["sha256"])
|
||||
require(source_result["artifact"]["sha256"] == release_result["archive"]["sha256"],
|
||||
"Source and release records identify different archives.")
|
||||
|
||||
values = {name: value for name, (_, value) in records.items()}
|
||||
for name, record in values.items():
|
||||
release_binding(record, name, commit, release_sha)
|
||||
|
||||
debian_acceptance.validate_pair(values["debian-host"], values["persistence"], commit, release_sha)
|
||||
backup_restore_acceptance.validate(values["backup-restore"], commit, release_sha)
|
||||
google_acceptance.validate(values["google-mailbox"])
|
||||
automation_acceptance.validate(values["automation"])
|
||||
identity_privacy_acceptance.validate(values["identity-privacy"])
|
||||
desktop_acceptance.validate(values["inbox-usability"])
|
||||
validate_capacity(values["capacity"], commit, release_sha)
|
||||
incident_exercise.validate(values["incident-support"])
|
||||
pilot_run.validate(values["pilot-findings"])
|
||||
pilot_approval.validate(values["pilot-approval"])
|
||||
|
||||
environment_names = {
|
||||
"debian-host", "persistence", "backup-restore", "google-mailbox",
|
||||
"automation", "identity-privacy", "inbox-usability", "incident-support",
|
||||
"pilot-findings",
|
||||
}
|
||||
origins = {environment_origin(values[name], name) for name in environment_names}
|
||||
require(len(origins) == 1, "Gate B records use different environments.")
|
||||
origin = next(iter(origins))
|
||||
require(values["capacity"].get("originHost") == urlparse(origin).hostname,
|
||||
"capacity originHost does not match the accepted environment.")
|
||||
|
||||
archive_sha = str(release_result["archive"]["sha256"])
|
||||
for name in ("debian-host", "persistence", "backup-restore"):
|
||||
require(values[name].get("archiveSha256") == archive_sha,
|
||||
f"{name} uses a different archiveSha256.")
|
||||
approved_images = release_record["images"]
|
||||
for name in ("debian-host", "persistence", "backup-restore"):
|
||||
images = values[name].get("images", {})
|
||||
require(images.get("api") == approved_images["api"] and images.get("worker") == approved_images["worker"],
|
||||
f"{name} uses different API or worker image identities.")
|
||||
|
||||
approval_capacity = values["pilot-approval"]["capacity"]
|
||||
capacity = values["capacity"]
|
||||
require(approval_capacity["reportSha256"] == digest(records["capacity"][0]),
|
||||
"Pilot approval capacity report checksum does not match the retained report.")
|
||||
require(approval_capacity["hostMetricsSha256"] == digest(host_metrics_path),
|
||||
"Pilot approval host metrics checksum does not match the retained file.")
|
||||
require(approval_capacity["observedConcurrency"] == capacity["concurrency"],
|
||||
"Pilot approval concurrency does not match the capacity report.")
|
||||
require(approval_capacity["observedP95Ms"] == capacity["latencyMs"]["p95"],
|
||||
"Pilot approval p95 does not match the capacity report.")
|
||||
require(approval_capacity["observedErrorRate"] == capacity["errorRate"],
|
||||
"Pilot approval error rate does not match the capacity report.")
|
||||
require(values["pilot-approval"]["pilot"]["recordSha256"] == digest(records["pilot-findings"][0]),
|
||||
"Pilot approval checksum does not match the retained pilot run record.")
|
||||
|
||||
summary_records = {name: digest(path) for name, (path, _) in sorted(records.items())}
|
||||
return {
|
||||
"archiveSha256": archive_sha,
|
||||
"environmentHost": urlparse(origin).hostname,
|
||||
"releaseCommit": commit,
|
||||
"releaseRecordSha256": release_sha,
|
||||
"records": summary_records,
|
||||
"schemaVersion": 1,
|
||||
"sourceRecordSha256": source_result["sourceRecord"]["sha256"],
|
||||
"validated": True,
|
||||
"version": version,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--source-archive", required=True, type=Path)
|
||||
parser.add_argument("--source-record", required=True, type=Path)
|
||||
parser.add_argument("--release-record", required=True, type=Path)
|
||||
parser.add_argument("--host-metrics", required=True, type=Path)
|
||||
parser.add_argument("--output", required=True, type=Path)
|
||||
for name in sorted(RECORD_KEYS):
|
||||
parser.add_argument("--" + name, required=True, type=Path)
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
require(args.host_metrics.is_file() and not args.host_metrics.is_symlink(),
|
||||
"Host metrics file is missing or is a symlink.")
|
||||
require(not args.output.exists() and args.output.parent.is_dir(),
|
||||
"Output must be a new file in an existing restricted directory.")
|
||||
records = {}
|
||||
for name in RECORD_KEYS:
|
||||
path = getattr(args, name.replace("-", "_"))
|
||||
records[name] = (path, load_json(path))
|
||||
summary = validate_bundle(
|
||||
args.source_archive,
|
||||
args.source_record,
|
||||
args.release_record,
|
||||
records,
|
||||
args.host_metrics,
|
||||
)
|
||||
descriptor = os.open(args.output, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
with os.fdopen(descriptor, "w", encoding="utf-8") as output:
|
||||
output.write(json.dumps(summary, indent=2, sort_keys=True) + "\n")
|
||||
print("Gate B bundle is structurally complete, consistently release-bound and approved. "
|
||||
"This validates records and checksums, not the underlying restricted evidence.")
|
||||
except (OSError, ValueError, json.JSONDecodeError) as error:
|
||||
print(f"Gate B bundle rejected: {error}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -2,7 +2,7 @@
|
||||
"acceptedAt": "2026-01-01T00:00:00Z",
|
||||
"acceptedBy": "REPLACE WITH APPROVER",
|
||||
"endedAt": "2026-01-01T00:00:00Z",
|
||||
"environment": "https://sandbox-guestops.futuresens.co.uk",
|
||||
"environment": "https://guestops-sandbox.futuresens.co.uk",
|
||||
"mailboxLabel": "sandbox mailbox A",
|
||||
"operator": "REPLACE WITH OPERATOR",
|
||||
"releaseCommit": "0000000000000000000000000000000000000000",
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
const guestops = db.getSiblingDB('guestops');
|
||||
guestops.createUser({user: 'guestops', pwd: process.env.MONGO_APP_PASSWORD, roles: [{role: 'readWrite', db: 'guestops'}]});
|
||||
@@ -1,189 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Write non-sensitive GuestOps host status for a read-only monitoring agent."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import socket
|
||||
import ssl
|
||||
import stat
|
||||
import subprocess
|
||||
import tempfile
|
||||
import urllib.request
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
BACKUP_NAME = re.compile(r"guestops-[0-9]{8}T[0-9]{6}Z\.tar\.gpg")
|
||||
MARKER_NAME = re.compile(r"guestops-[0-9]{8}T[0-9]{6}Z\.tar\.gpg\.transferred\.json")
|
||||
AUTH = ('const c=new Mongo("mongodb://127.0.0.1");'
|
||||
'c.getDB("admin").auth(process.env.MONGO_INITDB_ROOT_USERNAME,'
|
||||
'process.env.MONGO_INITDB_ROOT_PASSWORD);')
|
||||
HEARTBEAT = ('const x=c.getDB("guestops").workerheartbeat.findOne({_id:"worker"});'
|
||||
'print(JSON.stringify(x&&x.At?x.At:null));')
|
||||
|
||||
|
||||
def require(condition: bool, message: str) -> None:
|
||||
if not condition:
|
||||
raise RuntimeError(message)
|
||||
|
||||
|
||||
def run(args: list[str], root: Path, timeout: int = 30) -> bytes:
|
||||
result = subprocess.run(args, cwd=root, stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout)
|
||||
require(result.returncode == 0, f"{Path(args[0]).name} probe failed.")
|
||||
return result.stdout
|
||||
|
||||
|
||||
def utc_now() -> dt.datetime:
|
||||
return dt.datetime.now(dt.timezone.utc)
|
||||
|
||||
|
||||
def age_seconds(path: Path, pattern: re.Pattern[str], now: dt.datetime) -> int | None:
|
||||
if not path.is_dir() or path.is_symlink():
|
||||
return None
|
||||
times = [item.stat().st_mtime for item in path.iterdir()
|
||||
if item.is_file() and not item.is_symlink() and pattern.fullmatch(item.name)]
|
||||
return max(0, int(now.timestamp() - max(times))) if times else None
|
||||
|
||||
|
||||
def https_status(origin: str, now: dt.datetime) -> dict[str, object]:
|
||||
parsed = urlparse(origin)
|
||||
require(parsed.scheme == "https" and parsed.hostname and parsed.port in (None, 443)
|
||||
and parsed.path in ("", "/") and not parsed.query and not parsed.fragment
|
||||
and parsed.username is None and parsed.password is None,
|
||||
"GUESTOPS_ORIGIN must be an HTTPS origin without credentials or a path.")
|
||||
context = ssl.create_default_context()
|
||||
with socket.create_connection((parsed.hostname, 443), timeout=10) as connection:
|
||||
with context.wrap_socket(connection, server_hostname=parsed.hostname) as secured:
|
||||
certificate = secured.getpeercert()
|
||||
expires = dt.datetime.strptime(certificate["notAfter"], "%b %d %H:%M:%S %Y %Z").replace(
|
||||
tzinfo=dt.timezone.utc)
|
||||
with urllib.request.urlopen(origin.rstrip("/") + "/health/ready", timeout=15,
|
||||
context=context) as response:
|
||||
ready = response.status == 200 and json.load(response) == {"status": "ready"}
|
||||
return {"ready": ready, "certificateDaysRemaining": max(0, int((expires - now).total_seconds() // 86400))}
|
||||
|
||||
|
||||
def parse_compose(value: bytes) -> dict[str, dict[str, str]]:
|
||||
text = value.decode("utf-8", "strict").strip()
|
||||
if not text:
|
||||
return {}
|
||||
try:
|
||||
parsed = json.loads(text)
|
||||
rows = parsed if isinstance(parsed, list) else [parsed]
|
||||
except json.JSONDecodeError:
|
||||
rows = [json.loads(line) for line in text.splitlines() if line.strip()]
|
||||
result = {}
|
||||
for row in rows:
|
||||
if not isinstance(row, dict):
|
||||
continue
|
||||
service = str(row.get("Service", ""))
|
||||
if service in {"api", "worker", "mongo"}:
|
||||
result[service] = {
|
||||
"state": str(row.get("State", "unknown")).lower(),
|
||||
"health": str(row.get("Health", "none") or "none").lower(),
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def worker_heartbeat_age(root: Path, now: dt.datetime) -> int | None:
|
||||
output = run(["docker", "compose", "exec", "-T", "mongo", "mongosh", "--quiet",
|
||||
"--nodb", "--eval", AUTH + HEARTBEAT], root).decode("utf-8", "strict").strip()
|
||||
value = json.loads(output)
|
||||
if value is None:
|
||||
return None
|
||||
require(isinstance(value, str), "Worker heartbeat response was invalid.")
|
||||
parsed = dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
require(parsed.tzinfo is not None, "Worker heartbeat must contain a timezone.")
|
||||
return max(0, int((now - parsed.astimezone(dt.timezone.utc)).total_seconds()))
|
||||
|
||||
|
||||
def build_status(root: Path, backup_directory: Path, origin: str,
|
||||
now: dt.datetime | None = None) -> tuple[dict[str, object], list[str]]:
|
||||
moment = now or utc_now()
|
||||
errors: list[str] = []
|
||||
try:
|
||||
https = https_status(origin, moment)
|
||||
except Exception:
|
||||
https = {"ready": False, "certificateDaysRemaining": None}
|
||||
errors.append("https-probe-failed")
|
||||
try:
|
||||
containers = parse_compose(run(["docker", "compose", "ps", "--format", "json"], root))
|
||||
if set(containers) != {"api", "worker", "mongo"}:
|
||||
errors.append("container-set-incomplete")
|
||||
except Exception:
|
||||
containers = {}
|
||||
errors.append("container-probe-failed")
|
||||
try:
|
||||
heartbeat_age = worker_heartbeat_age(root, moment)
|
||||
if heartbeat_age is None:
|
||||
errors.append("worker-heartbeat-missing")
|
||||
except Exception:
|
||||
heartbeat_age = None
|
||||
errors.append("worker-heartbeat-probe-failed")
|
||||
disk = shutil.disk_usage(root)
|
||||
status = {
|
||||
"schemaVersion": 1,
|
||||
"generatedAt": moment.isoformat().replace("+00:00", "Z"),
|
||||
"https": https,
|
||||
"containers": containers,
|
||||
"workerHeartbeatAgeSeconds": heartbeat_age,
|
||||
"backupAgeSeconds": age_seconds(backup_directory, BACKUP_NAME, moment),
|
||||
"verifiedTransferAgeSeconds": age_seconds(backup_directory, MARKER_NAME, moment),
|
||||
"diskFreePercent": round(disk.free * 100 / disk.total, 2),
|
||||
"persistentJournal": Path("/var/log/journal").is_dir(),
|
||||
"errors": sorted(set(errors)),
|
||||
}
|
||||
return status, errors
|
||||
|
||||
|
||||
def write_status(path: Path, status: dict[str, object]) -> None:
|
||||
require(path.is_absolute() and not path.is_symlink(),
|
||||
"Status output must be an absolute, non-symlink path.")
|
||||
parent = path.parent.resolve()
|
||||
require(parent.is_dir() and not path.parent.is_symlink(), "Status output directory must exist.")
|
||||
fd, temporary = tempfile.mkstemp(prefix=path.name + ".", dir=parent)
|
||||
try:
|
||||
os.fchmod(fd, 0o644)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as stream:
|
||||
json.dump(status, stream, sort_keys=True, separators=(",", ":"))
|
||||
stream.write("\n")
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temporary, path)
|
||||
finally:
|
||||
try:
|
||||
os.unlink(temporary)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--root", type=Path, default=Path(__file__).resolve().parents[1])
|
||||
parser.add_argument("--backup-directory", type=Path, default=Path("/var/backups/guestops"))
|
||||
parser.add_argument("--origin", default="https://sandbox-guestops.futuresens.co.uk")
|
||||
parser.add_argument("--output", type=Path, default=Path("/run/guestops-monitor/status.json"))
|
||||
args = parser.parse_args()
|
||||
root = args.root.resolve()
|
||||
require(root.is_dir(), "GuestOps root must exist.")
|
||||
backup_directory = args.backup_directory.resolve()
|
||||
status, errors = build_status(root, backup_directory, args.origin)
|
||||
write_status(args.output, status)
|
||||
print("GuestOps monitoring status updated." if not errors
|
||||
else "GuestOps monitoring status updated with failed probes.")
|
||||
raise SystemExit(1 if errors else 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (OSError, RuntimeError, ValueError, subprocess.SubprocessError, json.JSONDecodeError) as error:
|
||||
print(f"GuestOps monitoring probe failed: {error}", file=__import__("sys").stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -1,31 +0,0 @@
|
||||
# Install within the existing Nginx configuration. Obtain the certificate first.
|
||||
server {
|
||||
listen 80;
|
||||
server_name sandbox-guestops.futuresens.co.uk;
|
||||
location /.well-known/acme-challenge/ { root /var/www/letsencrypt; }
|
||||
location / { return 301 https://$host$request_uri; }
|
||||
}
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name sandbox-guestops.futuresens.co.uk;
|
||||
ssl_certificate /etc/letsencrypt/live/sandbox-guestops.futuresens.co.uk/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/sandbox-guestops.futuresens.co.uk/privkey.pem;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
add_header Strict-Transport-Security "max-age=31536000" always;
|
||||
client_max_body_size 128k;
|
||||
# OAuth authorization codes and state must not appear in access logs.
|
||||
location = /api/integrations/google/callback {
|
||||
access_log off;
|
||||
proxy_pass http://127.0.0.1:8080;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
}
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:8080;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_set_header X-Forwarded-For $remote_addr;
|
||||
proxy_read_timeout 60s;
|
||||
}
|
||||
}
|
||||
-344
@@ -1,344 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""GuestOps dedicated-Compose operations. Never prints credentials or provider data."""
|
||||
import argparse
|
||||
import contextlib
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import time
|
||||
import urllib.request
|
||||
import uuid
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
FILES = {"mongo.archive.gz", "keys.tar.gz", "configuration.json", "manifest.json"}
|
||||
LIMIT = 8 * 1024**3
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise RuntimeError(message)
|
||||
|
||||
|
||||
def run(args, *, output=None, input_file=None, timeout=900):
|
||||
# Provider output may contain secrets; errors identify only the program.
|
||||
result = subprocess.run(args, cwd=ROOT, stdin=input_file or subprocess.DEVNULL,
|
||||
stdout=output or subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout)
|
||||
require(result.returncode == 0, f"{args[0]} step failed. Check the private operator environment; no command output was logged.")
|
||||
return result.stdout or b""
|
||||
|
||||
|
||||
def compose(*args, **kwargs):
|
||||
return run(["docker", "compose", *args], **kwargs)
|
||||
|
||||
|
||||
def digest(path):
|
||||
with path.open("rb") as stream:
|
||||
return hashlib.file_digest(stream, "sha256").hexdigest()
|
||||
|
||||
|
||||
def image_id(image):
|
||||
require(not image.startswith("-"), "Invalid image reference.")
|
||||
return run(["docker", "image", "inspect", "--format", "{{.Id}}", image]).decode().strip()
|
||||
|
||||
|
||||
def provider_configured(config, target):
|
||||
section = "Pms" if target == "/run/guestops/pms.json" else "Payments"
|
||||
# Only the exact shipped empty template is public; unknown settings fail closed.
|
||||
return config not in ({}, {section: {"Hotels": {}}})
|
||||
|
||||
|
||||
def persistence_layout(config):
|
||||
"""Return the resolved named volumes required to survive recreation."""
|
||||
services = config["services"]
|
||||
declared = config.get("volumes", {})
|
||||
|
||||
def volume_for(service, target):
|
||||
matches = [v for v in services[service].get("volumes", []) if v["target"] == target]
|
||||
require(len(matches) == 1 and matches[0]["type"] == "volume" and matches[0].get("source"),
|
||||
f"{service} requires one named persistent volume at {target}.")
|
||||
source = matches[0]["source"]
|
||||
require(source in declared, f"{service} volume {source} must be declared at the top level.")
|
||||
return declared[source].get("name") or source
|
||||
|
||||
api_keys = volume_for("api", "/var/lib/guestops/keys")
|
||||
worker_keys = volume_for("worker", "/var/lib/guestops/keys")
|
||||
require(api_keys == worker_keys, "API and worker key volumes differ.")
|
||||
mongo_data = volume_for("mongo", "/data/db")
|
||||
require(api_keys != mongo_data, "Database and key data require separate named volumes.")
|
||||
return {"keys": api_keys, "database": mongo_data}
|
||||
|
||||
|
||||
AUTH = 'const c=new Mongo("mongodb://127.0.0.1"); c.getDB("admin").auth(process.env.MONGO_INITDB_ROOT_USERNAME,process.env.MONGO_INITDB_ROOT_PASSWORD);'
|
||||
INVENTORY = 'const d=c.getDB("guestops"); print(JSON.stringify({bytes:d.stats().storageSize+d.stats().indexSize,collections:Object.fromEntries(d.getCollectionNames().filter(n=>!n.startsWith("system.")).sort().map(n=>[n,{count:d.getCollection(n).countDocuments({}),indexes:d.getCollection(n).getIndexes().map(i=>{delete i.ns;return i;}).sort((a,b)=>a.name.localeCompare(b.name))}]))}));'
|
||||
|
||||
|
||||
def mongo(script):
|
||||
return compose("exec", "-T", "mongo", "mongosh", "--quiet", "--nodb", "--eval", AUTH + script)
|
||||
|
||||
|
||||
def configuration():
|
||||
config = json.loads(compose("config", "--format", "json"))
|
||||
services = config["services"]
|
||||
require(set(services) == {"api", "worker", "mongo"}, "This tool supports the dedicated three-service GuestOps Compose deployment only.")
|
||||
require(not services["mongo"].get("ports"), "MongoDB must not have published ports.")
|
||||
ports = services["api"].get("ports", [])
|
||||
require(len(ports) == 1 and ports[0].get("host_ip") == "127.0.0.1" and int(ports[0]["target"]) == 8080, "API must publish only port 8080 on loopback.")
|
||||
require(not services["worker"].get("ports"), "Worker must not publish ports.")
|
||||
for name in ("api", "worker"):
|
||||
env = services[name]["environment"]
|
||||
require(env.get("Mongo__Database") == "guestops" and "@mongo:27017/guestops?" in env.get("Mongo__ConnectionString", ""), "Backup supports only the dedicated Compose guestops database.")
|
||||
require(str(env.get("Preview", "false")).lower() != "true", "Production preview is forbidden.")
|
||||
require(env.get("Keys__Path") == "/var/lib/guestops/keys", "Unexpected key directory.")
|
||||
require(services["api"]["environment"].get("ASPNETCORE_ENVIRONMENT") == "Production", "API must use Production environment.")
|
||||
require(re.fullmatch(r"https://[A-Za-z0-9.-]+", services["api"]["environment"].get("PublicUrl", "")), "PublicUrl must be an HTTPS hostname without a path.")
|
||||
persistence_layout(config)
|
||||
return config
|
||||
|
||||
|
||||
def wait_for(action, message, attempts=60):
|
||||
for attempt in range(attempts):
|
||||
try:
|
||||
action()
|
||||
return
|
||||
except Exception:
|
||||
if attempt == attempts - 1:
|
||||
raise RuntimeError(message)
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
def volume_identity(name):
|
||||
require(not name.startswith("-"), "Invalid volume name.")
|
||||
details = json.loads(run(["docker", "volume", "inspect", name]))
|
||||
require(len(details) == 1 and details[0].get("Name") == name, "Named volume inspection failed.")
|
||||
return {"name": name, "driver": details[0].get("Driver"), "scope": details[0].get("Scope")}
|
||||
|
||||
|
||||
def readiness():
|
||||
with urllib.request.urlopen("http://127.0.0.1:8080/health/ready", timeout=10) as response:
|
||||
require(json.load(response) == {"status": "ready"}, "Loopback readiness failed.")
|
||||
|
||||
|
||||
def persistence_drill(args):
|
||||
require(args.confirm_restart, "Persistence drill requires --confirm-restart: all services will be restarted and application containers recreated.")
|
||||
config = configuration()
|
||||
layout = persistence_layout(config)
|
||||
before_volumes = {purpose: volume_identity(name) for purpose, name in layout.items()}
|
||||
before_inventory = json.loads(mongo(INVENTORY))
|
||||
probe = compose("run", "--rm", "--no-deps", "-T", "-e", "Logging__LogLevel__Default=None", "api", "--backup-probe").decode().strip()
|
||||
require(re.fullmatch(r"[A-Za-z0-9_-]{20,4096}", probe), "Key probe did not return a valid protected value.")
|
||||
|
||||
# Restart the database first, then its clients, so recovery is exercised in a known order.
|
||||
compose("restart", "-t", "150", "mongo")
|
||||
wait_for(lambda: mongo('const r=c.getDB("admin").runCommand({ping:1});if(!r.ok)quit(1);'), "MongoDB did not recover after restart.")
|
||||
compose("restart", "-t", "150", "api", "worker")
|
||||
wait_for(readiness, "API readiness did not recover after restart.")
|
||||
|
||||
# Recreate stateless containers as an image upgrade would, without rebuilding or changing data volumes.
|
||||
compose("up", "-d", "--no-build", "--force-recreate", "api", "worker")
|
||||
wait_for(readiness, "API readiness did not recover after container recreation.")
|
||||
compose("run", "--rm", "--no-deps", "-T", "-e", "Logging__LogLevel__Default=None", "-e", "BACKUP_PROBE=" + probe, "api", "--verify-backup-probe")
|
||||
|
||||
after_volumes = {purpose: volume_identity(name) for purpose, name in layout.items()}
|
||||
after_inventory = json.loads(mongo(INVENTORY))
|
||||
require(after_volumes == before_volumes, "A persistent volume identity changed during the drill.")
|
||||
require(after_inventory["collections"] == before_inventory["collections"], "Database collection counts or indexes changed during the drill.")
|
||||
print("Persistence drill passed: named volumes, database counts/indexes, data-protection keys and readiness survived restart and container recreation.")
|
||||
|
||||
|
||||
def preflight(args):
|
||||
config = configuration()
|
||||
env_file = ROOT / ".env"
|
||||
require(env_file.is_file() and not env_file.is_symlink(), "Create a private .env file before deployment.")
|
||||
require(stat.S_IMODE(env_file.stat().st_mode) & 0o077 == 0, ".env must not be accessible to group or other users.")
|
||||
require(shutil.disk_usage(ROOT).free >= 8 * 1024**3, "Keep at least 8 GiB free before deployment; allow extra room for backups.")
|
||||
for name, service in config["services"].items():
|
||||
image_id(service["image"])
|
||||
for volume in service.get("volumes", []):
|
||||
if volume["type"] == "bind":
|
||||
require(Path(volume["source"]).exists(), f"A required {name} bind mount is missing.")
|
||||
if volume["target"] in ("/run/guestops/pms.json", "/run/guestops/payments.json"):
|
||||
provider = Path(volume["source"])
|
||||
if provider_configured(json.loads(provider.read_text()), volume["target"]):
|
||||
require(stat.S_IMODE(provider.stat().st_mode) & 0o077 == 0, "Configured provider files must not be accessible to group or other users.")
|
||||
if not args.offline:
|
||||
url = config["services"]["api"]["environment"]["PublicUrl"]
|
||||
require(re.fullmatch(r"https://[A-Za-z0-9.-]+", url), "PublicUrl must be an HTTPS hostname without a path.")
|
||||
with urllib.request.urlopen(url + "/health/ready", timeout=15) as response:
|
||||
require(response.url == url + "/health/ready" and json.load(response) == {"status": "ready"}, "Public HTTPS readiness failed.")
|
||||
print("Preflight passed: private database, loopback API, production settings, images, mounts and disk headroom" + ("; HTTPS not checked." if args.offline else "; public HTTPS and database readiness checked."))
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def maintenance_lock():
|
||||
import fcntl
|
||||
with (ROOT / ".guestops-maintenance.lock").open("a") as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
yield
|
||||
|
||||
|
||||
def backup(args):
|
||||
require(args.confirm_maintenance, "Backup requires --confirm-maintenance: API and workers will briefly stop.")
|
||||
require(re.fullmatch(r"[A-Fa-f0-9]{40}", args.recipient), "Use a verified full 40-character GPG recipient fingerprint.")
|
||||
run(["gpg", "--batch", "--list-keys", args.recipient])
|
||||
destination = Path(args.output).resolve()
|
||||
require(not destination.exists(), "Backup output already exists; choose a new filename.")
|
||||
require(destination.parent.is_dir(), "Create the private backup directory first.")
|
||||
require(stat.S_IMODE(destination.parent.stat().st_mode) & 0o077 == 0, "Backup directory must be private (mode 700).")
|
||||
config = configuration()
|
||||
runtime = {}
|
||||
for service in ("api", "worker", "mongo"):
|
||||
cid = compose("ps", "--status", "running", "-q", service).decode().strip()
|
||||
require(re.fullmatch(r"[a-f0-9]{12,64}", cid), f"Exactly one running {service} container is required.")
|
||||
runtime[service] = json.loads(run(["docker", "inspect", cid]))[0]
|
||||
require(shutil.disk_usage(destination.parent).free >= 3 * json.loads(mongo(INVENTORY))["bytes"] + 1024**3, "Insufficient free space for a consistent encrypted backup.")
|
||||
partial = destination.with_name(destination.name + ".partial-" + uuid.uuid4().hex)
|
||||
with maintenance_lock(), tempfile.TemporaryDirectory(prefix="guestops-backup-", dir=destination.parent) as folder:
|
||||
folder = Path(folder)
|
||||
stopped = False
|
||||
ttl = None
|
||||
try:
|
||||
# Set before stopping so partial stop failures also attempt recovery.
|
||||
stopped = True
|
||||
compose("stop", "-t", "150", "api", "worker")
|
||||
ttl = json.loads(mongo('print(JSON.stringify(c.getDB("admin").runCommand({getParameter:1,ttlMonitorEnabled:1}).ttlMonitorEnabled));'))
|
||||
require(isinstance(ttl, bool), "Cannot determine MongoDB TTL monitor state.")
|
||||
mongo('const r=c.getDB("admin").runCommand({setParameter:1,ttlMonitorEnabled:false});if(!r.ok)quit(1);')
|
||||
inventory = json.loads(mongo(INVENTORY))
|
||||
dump = 'set -eu; case "$MONGO_INITDB_ROOT_PASSWORD" in ""|*[!0-9a-fA-F]*) exit 1;; esac; umask 077; cfg=$(mktemp); trap \'rm -f "$cfg"\' EXIT; printf \'password: "%s"\\n\' "$MONGO_INITDB_ROOT_PASSWORD" > "$cfg"; mongodump --config "$cfg" --username "$MONGO_INITDB_ROOT_USERNAME" --authenticationDatabase admin --db guestops --archive --gzip'
|
||||
with (folder / "mongo.archive.gz").open("wb") as output:
|
||||
compose("exec", "-T", "mongo", "sh", "-c", dump, output=output)
|
||||
probe = compose("run", "--rm", "--no-deps", "-T", "-e", "Logging__LogLevel__Default=None", "api", "--backup-probe").decode().strip()
|
||||
require(re.fullmatch(r"[A-Za-z0-9_-]{20,4096}", probe), "Key probe did not return a valid protected value.")
|
||||
with (folder / "keys.tar.gz").open("wb") as output:
|
||||
compose("run", "--rm", "--no-deps", "-T", "--entrypoint", "tar", "api", "-C", "/var/lib/guestops/keys", "-czf", "-", ".", output=output)
|
||||
mounted = {}
|
||||
for volume in config["services"]["api"].get("volumes", []):
|
||||
if volume["type"] == "bind" and volume["target"] in ("/run/guestops/pms.json", "/run/guestops/payments.json"):
|
||||
mounted[volume["target"]] = Path(volume["source"]).read_text()
|
||||
(folder / "configuration.json").write_text(json.dumps({"compose": config, "runtime": runtime, "providerFiles": mounted}))
|
||||
manifest = {"format": 1, "createdAt": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "inventory": inventory, "probe": probe, "apiImageId": runtime["api"]["Image"], "mongoImageId": runtime["mongo"]["Image"], "sha256": {name: digest(folder / name) for name in FILES - {"manifest.json"}}}
|
||||
(folder / "manifest.json").write_text(json.dumps(manifest))
|
||||
finally:
|
||||
try:
|
||||
if ttl is not None:
|
||||
mongo('const r=c.getDB("admin").runCommand({setParameter:1,ttlMonitorEnabled:' + str(ttl).lower() + '});if(!r.ok)quit(1);')
|
||||
finally:
|
||||
if stopped:
|
||||
compose("start", "api", "worker")
|
||||
try:
|
||||
with tarfile.open(folder / "bundle.tar", "w") as archive:
|
||||
for name in sorted(FILES):
|
||||
archive.add(folder / name, arcname=name, recursive=False)
|
||||
run(["gpg", "--batch", "--trust-model", "always", "--recipient", args.recipient, "--output", str(partial), "--encrypt", str(folder / "bundle.tar")])
|
||||
os.link(partial, destination) # Atomic publication, never overwrite an existing backup.
|
||||
finally:
|
||||
partial.unlink(missing_ok=True)
|
||||
print("Encrypted backup complete. Copy it off-server and perform a restore drill.")
|
||||
|
||||
|
||||
def scheduled_backup(args):
|
||||
require(args.confirm_maintenance, "Scheduled backup requires --confirm-maintenance because API and workers will briefly stop.")
|
||||
recipient = os.environ.get("BACKUP_RECIPIENT", "")
|
||||
directory_value = os.environ.get("BACKUP_DIRECTORY", "")
|
||||
require(re.fullmatch(r"[A-Fa-f0-9]{40}", recipient), "BACKUP_RECIPIENT must be a verified full GPG fingerprint.")
|
||||
require(directory_value != "", "BACKUP_DIRECTORY must name the private off-checkout staging directory.")
|
||||
requested_directory = Path(directory_value)
|
||||
require(requested_directory.is_absolute() and not requested_directory.is_symlink(), "BACKUP_DIRECTORY must be an absolute, non-symlink path.")
|
||||
directory = requested_directory.resolve()
|
||||
require(directory.is_dir(), "BACKUP_DIRECTORY must be an existing real directory.")
|
||||
require(ROOT not in directory.parents and directory != ROOT, "Scheduled backups must be staged outside the application checkout.")
|
||||
require(stat.S_IMODE(directory.stat().st_mode) & 0o077 == 0, "BACKUP_DIRECTORY must be private (mode 700).")
|
||||
timestamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime())
|
||||
destination = directory / f"guestops-{timestamp}.tar.gpg"
|
||||
backup(argparse.Namespace(confirm_maintenance=True, recipient=recipient, output=str(destination)))
|
||||
print(f"Scheduled backup staged as {destination.name}. Off-host transfer and alert verification remain required.")
|
||||
|
||||
|
||||
def unpack(bundle, folder):
|
||||
with tarfile.open(bundle, "r:") as archive:
|
||||
members = archive.getmembers()
|
||||
require(len(members) == len(FILES) and {m.name for m in members} == FILES, "Unexpected backup members.")
|
||||
require(all(m.isfile() and 0 <= m.size <= LIMIT for m in members) and sum(m.size for m in members) <= LIMIT, "Invalid or oversized backup members.")
|
||||
for member in members:
|
||||
with archive.extractfile(member) as source, (folder / member.name).open("xb") as output:
|
||||
shutil.copyfileobj(source, output)
|
||||
manifest = json.loads((folder / "manifest.json").read_text())
|
||||
require(manifest.get("format") == 1 and set(manifest.get("sha256", {})) == FILES - {"manifest.json"}, "Unsupported backup format.")
|
||||
for name, expected in manifest["sha256"].items():
|
||||
require(digest(folder / name) == expected, "Backup checksum verification failed.")
|
||||
return manifest
|
||||
|
||||
|
||||
def restore_drill(args):
|
||||
backup_file = Path(args.backup).resolve()
|
||||
require(backup_file.is_file(), "Backup file is missing.")
|
||||
with tempfile.TemporaryDirectory(prefix="guestops-restore-", dir=ROOT) as temp:
|
||||
folder = Path(temp)
|
||||
run(["gpg", "--batch", "--max-output", str(LIMIT), "--output", str(folder / "bundle.tar"), "--decrypt", str(backup_file)])
|
||||
require((folder / "bundle.tar").stat().st_size <= LIMIT, "Decrypted bundle exceeds the 8 GiB pilot limit.")
|
||||
manifest = unpack(folder / "bundle.tar", folder)
|
||||
require(image_id(args.api_image) == manifest["apiImageId"] and image_id(args.mongo_image) == manifest["mongoImageId"], "Load the exact trusted API and MongoDB images used for this backup.")
|
||||
require(shutil.disk_usage(ROOT).free > 3 * manifest["inventory"]["bytes"] + 1024**3, "Insufficient restore drill space.")
|
||||
keys = folder / "keys"
|
||||
keys.mkdir(mode=0o700)
|
||||
with tarfile.open(folder / "keys.tar.gz", "r:gz") as archive:
|
||||
total = 0
|
||||
for member in archive:
|
||||
if member.name in (".", "./") and member.isdir():
|
||||
continue
|
||||
name = member.name.removeprefix("./")
|
||||
total += member.size
|
||||
require(member.isfile() and re.fullmatch(r"[A-Za-z0-9_-]+\.xml", name) and member.size < 1024**2 and total < 16 * 1024**2, "Invalid key archive.")
|
||||
with archive.extractfile(member) as source, (keys / name).open("xb") as output:
|
||||
shutil.copyfileobj(source, output)
|
||||
run(["docker", "run", "--rm", "--pull", "never", "--network", "none", "--user", "0:0", "--read-only", "--mount", f"type=bind,src={keys},dst=/var/lib/guestops/keys,readonly", "-e", "Logging__LogLevel__Default=None", "-e", "BACKUP_PROBE=" + manifest["probe"], args.api_image, "--verify-backup-probe"])
|
||||
cid = run(["docker", "run", "-d", "--pull", "never", "--network", "none", "--memory", "1g", "--label", "guestops.restore-drill=true", args.mongo_image, "--bind_ip", "127.0.0.1", "--setParameter", "ttlMonitorEnabled=false"]).decode().strip()
|
||||
require(re.fullmatch(r"[a-f0-9]{64}", cid), "Unexpected restore container identifier.")
|
||||
try:
|
||||
for attempt in range(30):
|
||||
try:
|
||||
run(["docker", "exec", cid, "mongosh", "--quiet", "--eval", "db.adminCommand({ping:1})"], timeout=10)
|
||||
break
|
||||
except RuntimeError:
|
||||
if attempt == 29:
|
||||
raise
|
||||
time.sleep(1)
|
||||
with (folder / "mongo.archive.gz").open("rb") as source:
|
||||
run(["docker", "exec", "-i", cid, "mongorestore", "--archive", "--gzip", "--nsInclude", "guestops.*"], input_file=source)
|
||||
restored = json.loads(run(["docker", "exec", cid, "mongosh", "--quiet", "--nodb", "--eval", 'const c=new Mongo("mongodb://127.0.0.1");' + INVENTORY]))
|
||||
require(restored["collections"] == manifest["inventory"]["collections"], "Restored collection counts or indexes differ.")
|
||||
finally:
|
||||
run(["docker", "rm", "-f", "-v", cid]) # Only the exact container created above and its anonymous volumes.
|
||||
print("Restore drill passed: collection counts, indexes and actual key decryption verified in isolated containers. Production was not restored or modified.")
|
||||
|
||||
|
||||
def main():
|
||||
require(os.name == "posix", "Run deployment operations on Linux.")
|
||||
os.umask(0o077)
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
subs = parser.add_subparsers(dest="command", required=True)
|
||||
check = subs.add_parser("preflight"); check.add_argument("--offline", action="store_true")
|
||||
persistence = subs.add_parser("persistence-drill"); persistence.add_argument("--confirm-restart", action="store_true")
|
||||
save = subs.add_parser("backup"); save.add_argument("--recipient", required=True); save.add_argument("--output", required=True); save.add_argument("--confirm-maintenance", action="store_true")
|
||||
scheduled = subs.add_parser("scheduled-backup"); scheduled.add_argument("--confirm-maintenance", action="store_true")
|
||||
drill = subs.add_parser("restore-drill"); drill.add_argument("backup"); drill.add_argument("--api-image", required=True); drill.add_argument("--mongo-image", default="mongo:8.0")
|
||||
args = parser.parse_args()
|
||||
{"preflight": preflight, "persistence-drill": persistence_drill, "backup": backup, "scheduled-backup": scheduled_backup, "restore-drill": restore_drill}[args.command](args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Exception as error:
|
||||
# Never include subprocess output, config values or parsed guest data.
|
||||
print(str(error) if isinstance(error, RuntimeError) else "Operation failed (" + type(error).__name__ + "). No sensitive details were logged.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
@@ -5,7 +5,6 @@ from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
@@ -44,11 +43,7 @@ def versions(repository: Path, commit: str) -> tuple[str, str]:
|
||||
return dotnet_version, web_version
|
||||
|
||||
|
||||
def sha256_bytes(value: bytes) -> str:
|
||||
return hashlib.sha256(value).hexdigest()
|
||||
|
||||
|
||||
def build_package(repository: Path, ref: str, output_directory: Path) -> tuple[Path, Path, dict[str, object]]:
|
||||
def build_package(repository: Path, ref: str, output_directory: Path) -> tuple[Path, dict[str, str]]:
|
||||
repository = repository.resolve()
|
||||
output_directory = output_directory.resolve()
|
||||
if not (repository / ".git").exists():
|
||||
@@ -66,7 +61,7 @@ def build_package(repository: Path, ref: str, output_directory: Path) -> tuple[P
|
||||
if SEMVER.fullmatch(dotnet_version) is None:
|
||||
raise ValueError("The committed version must use MAJOR.MINOR.PATCH.")
|
||||
|
||||
prefix = f"GuestOps-{dotnet_version}/"
|
||||
prefix = f"guestops_{dotnet_version}/"
|
||||
tar_bytes = bytes(
|
||||
git(
|
||||
repository,
|
||||
@@ -82,28 +77,13 @@ def build_package(repository: Path, ref: str, output_directory: Path) -> tuple[P
|
||||
stream.write(tar_bytes)
|
||||
archive_bytes = compressed.getvalue()
|
||||
|
||||
stem = f"GuestOps-{dotnet_version}-{commit[:12]}"
|
||||
archive = output_directory / f"{stem}.tar.gz"
|
||||
record_path = output_directory / f"{stem}.source.json"
|
||||
if archive.exists() or record_path.exists():
|
||||
raise ValueError("The output package or source record already exists; releases are not overwritten.")
|
||||
archive = output_directory / f"guestops_{dotnet_version}.tar.gz"
|
||||
if archive.exists():
|
||||
raise ValueError("The output package already exists; published versions are not overwritten.")
|
||||
|
||||
output_directory.mkdir(parents=True, exist_ok=True)
|
||||
archive.write_bytes(archive_bytes)
|
||||
record: dict[str, object] = {
|
||||
"artifact": {
|
||||
"name": archive.name,
|
||||
"sha256": sha256_bytes(archive_bytes),
|
||||
"size": len(archive_bytes),
|
||||
},
|
||||
"commit": commit,
|
||||
"schemaVersion": 1,
|
||||
"version": dotnet_version,
|
||||
}
|
||||
record_path.write_text(
|
||||
json.dumps(record, indent=2, sort_keys=True) + "\n", encoding="utf-8"
|
||||
)
|
||||
return archive, record_path, record
|
||||
return archive, {"commit": commit, "version": dotnet_version}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
@@ -113,11 +93,11 @@ def main() -> None:
|
||||
parser.add_argument("--output-directory", required=True, type=Path)
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
archive, record, details = build_package(args.repository, args.ref, args.output_directory)
|
||||
archive, details = build_package(args.repository, args.ref, args.output_directory)
|
||||
except (OSError, ValueError, ET.ParseError, json.JSONDecodeError, subprocess.SubprocessError) as error:
|
||||
print(f"Source packaging failed: {error}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(json.dumps({"archive": str(archive), "record": str(record), **details}, indent=2))
|
||||
print(json.dumps({"archive": str(archive), **details}, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"system": "guestops-persistence",
|
||||
"evidenceId": "persistence",
|
||||
"releaseVersion": "0.2.1",
|
||||
"releaseCommit": "0000000000000000000000000000000000000000",
|
||||
"releaseRecordSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"archiveSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"environment": "https://sandbox-guestops.futuresens.co.uk",
|
||||
"hostIdentifier": "guestops-sandbox-01",
|
||||
"images": {
|
||||
"api": {"reference": "guestops-api:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
|
||||
"worker": {"reference": "guestops-worker:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"}
|
||||
},
|
||||
"drillCommand": "python3 deploy/ops.py persistence-drill --confirm-restart",
|
||||
"operator": "REPLACE",
|
||||
"reviewedBy": "REPLACE",
|
||||
"startedAt": "2026-09-30T09:00:00Z",
|
||||
"endedAt": "2026-09-30T10:00:00Z",
|
||||
"reviewedAt": "2026-09-30T11:00:00Z",
|
||||
"unresolvedCriticalFindings": 1,
|
||||
"scenarios": [
|
||||
{"id": "container-recreation", "status": "not-run", "evidence": []},
|
||||
{"id": "data-protection-key", "status": "not-run", "evidence": []},
|
||||
{"id": "database-inventory", "status": "not-run", "evidence": []},
|
||||
{"id": "image-identity", "status": "not-run", "evidence": []},
|
||||
{"id": "post-reboot-persistence", "status": "not-run", "evidence": []},
|
||||
{"id": "separate-volume-layout", "status": "not-run", "evidence": []},
|
||||
{"id": "service-restart", "status": "not-run", "evidence": []}
|
||||
]
|
||||
}
|
||||
@@ -1,76 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Bind a reviewed GuestOps source package to its installed image identities."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def project_versions() -> tuple[str, str]:
|
||||
props = (ROOT / "Directory.Build.props").read_text(encoding="utf-8")
|
||||
match = re.search(r"<Version>([^<]+)</Version>", props)
|
||||
if match is None:
|
||||
raise ValueError("Directory.Build.props does not contain a Version element")
|
||||
package = json.loads((ROOT / "web" / "package.json").read_text(encoding="utf-8"))
|
||||
return match.group(1), str(package["version"])
|
||||
|
||||
|
||||
def sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--artifact", required=True, type=Path)
|
||||
parser.add_argument("--commit", required=True)
|
||||
parser.add_argument("--api-image", required=True)
|
||||
parser.add_argument("--api-id", required=True)
|
||||
parser.add_argument("--worker-image", required=True)
|
||||
parser.add_argument("--worker-id", required=True)
|
||||
parser.add_argument("--output", required=True, type=Path)
|
||||
args = parser.parse_args()
|
||||
|
||||
commit = args.commit.lower()
|
||||
if re.fullmatch(r"[0-9a-f]{40}", commit) is None:
|
||||
parser.error("--commit must be a full 40-character Git SHA")
|
||||
if not args.artifact.is_file():
|
||||
parser.error("--artifact must name an existing file")
|
||||
|
||||
dotnet_version, web_version = project_versions()
|
||||
if dotnet_version != web_version:
|
||||
parser.error(
|
||||
f"release versions differ: .NET={dotnet_version}, web={web_version}"
|
||||
)
|
||||
|
||||
record = {
|
||||
"artifact": {
|
||||
"name": args.artifact.name,
|
||||
"sha256": sha256(args.artifact),
|
||||
"size": args.artifact.stat().st_size,
|
||||
},
|
||||
"commit": commit,
|
||||
"images": {
|
||||
"api": {"id": args.api_id, "reference": args.api_image},
|
||||
"worker": {"id": args.worker_id, "reference": args.worker_image},
|
||||
},
|
||||
"schemaVersion": 1,
|
||||
"version": dotnet_version,
|
||||
}
|
||||
args.output.write_text(
|
||||
json.dumps(record, indent=2, sort_keys=True) + "\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,6 +0,0 @@
|
||||
BACKUP_DIRECTORY=/var/backups/guestops
|
||||
BACKUP_REMOTE_HOST=restricted-store.example.invalid
|
||||
BACKUP_REMOTE_USER=guestops_upload
|
||||
BACKUP_REMOTE_DIRECTORY=/restricted/guestops/backups
|
||||
BACKUP_SSH_IDENTITY=/etc/guestops/backup-transfer.key
|
||||
BACKUP_SSH_KNOWN_HOSTS=/etc/guestops/backup-known-hosts
|
||||
@@ -1,2 +0,0 @@
|
||||
BACKUP_RECIPIENT=0123456789ABCDEF0123456789ABCDEF01234567
|
||||
BACKUP_DIRECTORY=/var/backups/guestops
|
||||
@@ -1,19 +0,0 @@
|
||||
[Unit]
|
||||
Description=Transfer GuestOps encrypted backups to restricted storage
|
||||
Wants=network-online.target
|
||||
After=network-online.target guestops-backup.service
|
||||
ConditionPathIsDirectory=/srv/guestops/current
|
||||
ConditionPathIsDirectory=/var/backups/guestops
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=/srv/guestops/current
|
||||
EnvironmentFile=/etc/guestops/backup-transfer.env
|
||||
UMask=0077
|
||||
ExecStart=/usr/bin/python3 /srv/guestops/current/deploy/backup_transfer.py --prune-verified --retention-days 7
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectHome=true
|
||||
ProtectSystem=full
|
||||
ReadWritePaths=/var/backups/guestops
|
||||
TimeoutStartSec=30min
|
||||
@@ -1,11 +0,0 @@
|
||||
[Unit]
|
||||
Description=Retry GuestOps off-host backup transfer
|
||||
|
||||
[Timer]
|
||||
OnBootSec=10min
|
||||
OnUnitActiveSec=15min
|
||||
Persistent=true
|
||||
Unit=guestops-backup-transfer.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -1,21 +0,0 @@
|
||||
[Unit]
|
||||
Description=GuestOps encrypted maintenance backup
|
||||
Requires=docker.service
|
||||
After=docker.service
|
||||
ConditionPathIsDirectory=/srv/guestops/current
|
||||
ConditionPathIsDirectory=/var/backups/guestops
|
||||
ConditionPathIsDirectory=/var/lib/guestops-backup/gnupg
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=/srv/guestops/current
|
||||
EnvironmentFile=/etc/guestops/backup.env
|
||||
Environment=GNUPGHOME=/var/lib/guestops-backup/gnupg
|
||||
UMask=0077
|
||||
ExecStart=/usr/bin/python3 /srv/guestops/current/deploy/ops.py scheduled-backup --confirm-maintenance
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectHome=true
|
||||
ProtectSystem=full
|
||||
ReadWritePaths=/var/backups/guestops /var/lib/guestops-backup/gnupg
|
||||
TimeoutStartSec=45min
|
||||
@@ -1,11 +0,0 @@
|
||||
[Unit]
|
||||
Description=Run the GuestOps encrypted backup each day
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 02:17:00 UTC
|
||||
RandomizedDelaySec=30min
|
||||
Persistent=true
|
||||
Unit=guestops-backup.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -1,21 +0,0 @@
|
||||
[Unit]
|
||||
Description=Write non-sensitive GuestOps monitoring status
|
||||
Requires=docker.service
|
||||
After=docker.service network-online.target
|
||||
ConditionPathIsDirectory=/srv/guestops/current
|
||||
ConditionPathIsDirectory=/var/backups/guestops
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=/srv/guestops/current
|
||||
UMask=0022
|
||||
ExecStart=/usr/bin/python3 /srv/guestops/current/deploy/monitor_status.py
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectHome=true
|
||||
ProtectSystem=strict
|
||||
RuntimeDirectory=guestops-monitor
|
||||
RuntimeDirectoryMode=0755
|
||||
RuntimeDirectoryPreserve=yes
|
||||
ReadWritePaths=/run/guestops-monitor
|
||||
TimeoutStartSec=2min
|
||||
@@ -1,11 +0,0 @@
|
||||
[Unit]
|
||||
Description=Refresh GuestOps monitoring status each minute
|
||||
|
||||
[Timer]
|
||||
OnBootSec=2min
|
||||
OnUnitActiveSec=1min
|
||||
Persistent=true
|
||||
Unit=guestops-monitor-status.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -1,2 +0,0 @@
|
||||
# The privileged systemd probe owns Docker/database access. Zabbix reads only this aggregate JSON.
|
||||
UserParameter=guestops.status,cat /run/guestops-monitor/status.json
|
||||
@@ -1,118 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Verify a GuestOps source package and its immutable release record."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
|
||||
|
||||
SHA = re.compile(r"[0-9a-f]{40}")
|
||||
DIGEST = re.compile(r"sha256:[0-9a-f]{64}")
|
||||
|
||||
|
||||
def require(condition: bool, message: str) -> None:
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def loaded_image_id(reference: str) -> str:
|
||||
result = subprocess.run(
|
||||
["docker", "image", "inspect", "--format", "{{.Id}}", reference],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def validate(
|
||||
archive: Path,
|
||||
record_path: Path,
|
||||
expected_commit: str,
|
||||
expected_version: str,
|
||||
verify_loaded_images: bool = False,
|
||||
) -> dict[str, object]:
|
||||
require(archive.is_file(), "Release archive does not exist.")
|
||||
require(record_path.is_file(), "Release record does not exist.")
|
||||
require(SHA.fullmatch(expected_commit) is not None, "Expected commit must be a full lowercase Git SHA.")
|
||||
require(re.fullmatch(r"[0-9]+\.[0-9]+\.[0-9]+", expected_version) is not None,
|
||||
"Expected version must use MAJOR.MINOR.PATCH.")
|
||||
|
||||
record = json.loads(record_path.read_text(encoding="utf-8"))
|
||||
require(isinstance(record, dict), "Release record must be a JSON object.")
|
||||
require(record.get("schemaVersion") == 1, "Unsupported release-record schema.")
|
||||
require(record.get("commit") == expected_commit, "Release-record commit does not match the approved candidate.")
|
||||
require(record.get("version") == expected_version, "Release-record version does not match the approved version.")
|
||||
|
||||
artifact = record.get("artifact")
|
||||
require(isinstance(artifact, dict), "Release record has no artifact object.")
|
||||
require(artifact.get("name") == archive.name, "Release archive filename does not match the record.")
|
||||
require(artifact.get("size") == archive.stat().st_size, "Release archive size does not match the record.")
|
||||
archive_sha = sha256(archive)
|
||||
require(artifact.get("sha256") == archive_sha, "Release archive SHA-256 does not match the record.")
|
||||
|
||||
images = record.get("images")
|
||||
require(isinstance(images, dict) and set(images) == {"api", "worker"},
|
||||
"Release record must contain exactly API and worker images.")
|
||||
expected_references = {
|
||||
"api": f"guestops-api:{expected_commit}",
|
||||
"worker": f"guestops-worker:{expected_commit}",
|
||||
}
|
||||
verified_images: dict[str, dict[str, str]] = {}
|
||||
for name, reference in expected_references.items():
|
||||
image = images.get(name)
|
||||
require(isinstance(image, dict), f"Release record has no {name} image object.")
|
||||
require(image.get("reference") == reference, f"{name} image reference is not bound to the full candidate SHA.")
|
||||
image_id = str(image.get("id", ""))
|
||||
require(DIGEST.fullmatch(image_id) is not None, f"{name} image ID is not an immutable SHA-256 digest.")
|
||||
if verify_loaded_images:
|
||||
require(loaded_image_id(reference) == image_id, f"Loaded {name} image ID does not match the release record.")
|
||||
verified_images[name] = {"reference": reference, "id": image_id}
|
||||
|
||||
return {
|
||||
"schemaVersion": 1,
|
||||
"verified": True,
|
||||
"commit": expected_commit,
|
||||
"version": expected_version,
|
||||
"archive": {"name": archive.name, "size": archive.stat().st_size, "sha256": archive_sha},
|
||||
"releaseRecord": {"name": record_path.name, "sha256": sha256(record_path)},
|
||||
"images": verified_images,
|
||||
"loadedImageIdsVerified": verify_loaded_images,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--archive", required=True, type=Path)
|
||||
parser.add_argument("--record", required=True, type=Path)
|
||||
parser.add_argument("--commit", required=True)
|
||||
parser.add_argument("--version", required=True)
|
||||
parser.add_argument("--verify-loaded-images", action="store_true")
|
||||
parser.add_argument("--output", type=Path, help="Optional path for the non-sensitive verification summary.")
|
||||
args = parser.parse_args()
|
||||
result = validate(args.archive, args.record, args.commit, args.version, args.verify_loaded_images)
|
||||
rendered = json.dumps(result, indent=2, sort_keys=True) + "\n"
|
||||
if args.output:
|
||||
args.output.write_text(rendered, encoding="utf-8")
|
||||
print(rendered, end="")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (OSError, ValueError, json.JSONDecodeError, subprocess.SubprocessError) as error:
|
||||
print(f"Release verification failed: {error}", file=__import__("sys").stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -1,104 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Verify a GuestOps source archive against its deterministic source record."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import sys
|
||||
|
||||
|
||||
FULL_SHA = re.compile(r"[0-9a-f]{40}")
|
||||
SHA256 = re.compile(r"[0-9a-f]{64}")
|
||||
SEMVER = re.compile(r"[0-9]+\.[0-9]+\.[0-9]+")
|
||||
|
||||
|
||||
def require(condition: bool, message: str) -> None:
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def validate(
|
||||
archive: Path,
|
||||
record_path: Path,
|
||||
expected_commit: str | None = None,
|
||||
expected_version: str | None = None,
|
||||
) -> dict[str, object]:
|
||||
require(archive.is_file(), "Source archive does not exist.")
|
||||
require(record_path.is_file(), "Source record does not exist.")
|
||||
record = json.loads(record_path.read_text(encoding="utf-8"))
|
||||
require(isinstance(record, dict), "Source record must be a JSON object.")
|
||||
require(record.get("schemaVersion") == 1, "Unsupported source-record schema.")
|
||||
|
||||
commit = record.get("commit")
|
||||
version = record.get("version")
|
||||
require(isinstance(commit, str) and FULL_SHA.fullmatch(commit) is not None,
|
||||
"Source record commit must be a full lowercase Git SHA.")
|
||||
require(isinstance(version, str) and SEMVER.fullmatch(version) is not None,
|
||||
"Source record version must use MAJOR.MINOR.PATCH.")
|
||||
if expected_commit is not None:
|
||||
require(FULL_SHA.fullmatch(expected_commit) is not None,
|
||||
"Expected commit must be a full lowercase Git SHA.")
|
||||
require(commit == expected_commit, "Source-record commit does not match the selected release.")
|
||||
if expected_version is not None:
|
||||
require(SEMVER.fullmatch(expected_version) is not None,
|
||||
"Expected version must use MAJOR.MINOR.PATCH.")
|
||||
require(version == expected_version, "Source-record version does not match the selected release.")
|
||||
|
||||
artifact = record.get("artifact")
|
||||
require(isinstance(artifact, dict), "Source record has no artifact object.")
|
||||
require(artifact.get("name") == archive.name, "Source archive filename does not match the record.")
|
||||
require(artifact.get("size") == archive.stat().st_size, "Source archive size does not match the record.")
|
||||
recorded_sha = artifact.get("sha256")
|
||||
require(isinstance(recorded_sha, str) and SHA256.fullmatch(recorded_sha) is not None,
|
||||
"Source archive record must contain a lowercase SHA-256 digest.")
|
||||
actual_sha = sha256(archive)
|
||||
require(recorded_sha == actual_sha, "Source archive SHA-256 does not match the record.")
|
||||
|
||||
return {
|
||||
"artifact": {"name": archive.name, "sha256": actual_sha, "size": archive.stat().st_size},
|
||||
"commit": commit,
|
||||
"schemaVersion": 1,
|
||||
"sourceRecord": {"name": record_path.name, "sha256": sha256(record_path)},
|
||||
"verified": True,
|
||||
"version": version,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--archive", required=True, type=Path)
|
||||
parser.add_argument("--record", required=True, type=Path)
|
||||
parser.add_argument("--expected-commit")
|
||||
parser.add_argument("--expected-version")
|
||||
parser.add_argument("--output", type=Path)
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
result = validate(
|
||||
args.archive,
|
||||
args.record,
|
||||
args.expected_commit,
|
||||
args.expected_version,
|
||||
)
|
||||
except (OSError, ValueError, json.JSONDecodeError) as error:
|
||||
print(f"Source package verification failed: {error}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
rendered = json.dumps(result, indent=2, sort_keys=True) + "\n"
|
||||
if args.output:
|
||||
args.output.write_text(rendered, encoding="utf-8")
|
||||
print(rendered, end="")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user