GuestOps asible playbook

This commit is contained in:
2026-10-21 11:55:33 +01:00
parent c5c6ec1453
commit db8422afda
61 changed files with 108 additions and 4417 deletions
-56
View File
@@ -1,56 +0,0 @@
# GuestOps Ansible handoff
This playbook is the application-owned deployment contract. Copy or import it into the private Futuresens Ansible repository and bind the `guestops` inventory group there. Do not commit inventory secrets, `.env` contents, provider credentials, hostnames other than the public service name, or restricted evidence paths to GuestOps.
Required extra variables:
- `guestops_release_version`: matched application version, for example `0.2.1`;
- `guestops_release_commit`: full 40-character packaged commit;
- `guestops_archive`: controller path to the generated `.tar.gz`;
- `guestops_archive_sha256`: approved lowercase SHA-256;
- `guestops_source_record`: controller path to the matching `.source.json`;
- `guestops_evidence_directory`: existing restricted controller directory for the fetched release record.
The target must already have Docker Engine with Compose, Nginx, Python 3, `ss`, a trusted certificate under `/etc/letsencrypt/live/sandbox-guestops.futuresens.co.uk`, and `/etc/guestops/guestops.env` owned by root with mode `0600`. The private environment file supplies MongoDB passwords, image references, provider-file paths and disabled-by-default feature flags. Provision it through Ansible Vault or the existing Futuresens secret process, never through this repository.
Example invocation from the central Ansible checkout:
```sh
ansible-playbook guestops.yml \
-l guestops_sandbox \
-e guestops_release_version=0.2.1 \
-e guestops_release_commit=FULL_40_CHARACTER_SHA \
-e guestops_archive=/secure/releases/GuestOps-0.2.1-COMMIT.tar.gz \
-e guestops_archive_sha256=LOWERCASE_SHA256 \
-e guestops_source_record=/secure/releases/GuestOps-0.2.1-COMMIT.source.json \
-e guestops_evidence_directory=/secure/evidence/guestops/0.2.1
```
The controller verifies the source package before transfer. The host independently checks the transferred archive checksum, extracts into a commit-specific directory, verifies the package again, selects the commit-tagged images and disabled send/FAQ defaults in the private environment, builds the API and worker images, records immutable image IDs, runs the offline preflight, validates Compose without printing expanded secrets, starts with `--no-build`, and waits for loopback readiness. It then enables Docker and Nginx at boot, validates and installs the reviewed proxy site, rejects public API or MongoDB listeners, and checks the public redirect and certificate-backed HTTPS readiness before changing the `current` symlink. The source archive remains in the restricted controller store; the temporary host copy is removed after success.
The release playbook installs the reviewed Nginx site but does not provision DNS, certificates, firewall rules, durable logging, backup keys, central monitoring, or the private environment. Those remain explicit Milestone 10 and 11 acceptance activities.
## Backup and monitoring operations
After the exact release is deployed, run `guestops-operations.yml`. It installs the repository systemd units, verifies their definitions, imports only the approved public recovery key, enables transfer retry and monitoring, connects the aggregate status file to Zabbix, and retains non-sensitive installation evidence. It never creates a recovery private key or writes transfer credentials.
Before running it, provision these private host files through Ansible Vault or the established secret process:
- `/etc/guestops/backup.env` mode `0600`;
- `/etc/guestops/backup-transfer.env` mode `0600`;
- `/etc/guestops/backup-transfer.key` mode `0600`;
- `/etc/guestops/backup-known-hosts` mode `0644`.
The recovery public key must be available on the controller. Its private key and protected recovery copy must remain off the Debian host. Confirm durable restricted logging separately and pass `guestops_durable_logs_configured=true` only after that review. The daily backup timer defaults to disabled; set `guestops_enable_backup_schedule=true` only after the manual backup and service-recovery exercise passes.
```sh
ansible-playbook guestops-operations.yml \
-l guestops_sandbox \
-e guestops_release_commit=FULL_40_CHARACTER_SHA \
-e guestops_backup_recipient=FULL_40_CHARACTER_GPG_FINGERPRINT \
-e guestops_recovery_public_key=/secure/recovery/guestops-public.asc \
-e guestops_evidence_directory=/secure/evidence/guestops/0.2.1 \
-e guestops_durable_logs_configured=true
```
Run the manual encrypted backup, transfer, isolated restore and rollback exercises separately under the approved maintenance procedure. After they pass, rerun with `guestops_enable_backup_schedule=true`. Configure Zabbix trigger thresholds and escalation recipients in the central Zabbix environment; the playbook exposes only the non-sensitive `guestops.status` item.
-227
View File
@@ -1,227 +0,0 @@
---
- name: Install GuestOps backup and monitoring operations
hosts: guestops
become: true
gather_facts: true
vars:
guestops_root: /srv/guestops
guestops_current: "{{ guestops_root }}/current"
guestops_config_root: /etc/guestops
guestops_backup_root: /var/backups/guestops
guestops_gnupg_root: /var/lib/guestops-backup/gnupg
guestops_zabbix_service: zabbix-agent2
guestops_zabbix_include_directory: /etc/zabbix/zabbix_agent2.d
guestops_enable_backup_schedule: false
guestops_durable_logs_configured: false
pre_tasks:
- name: Validate required operational variables
ansible.builtin.assert:
that:
- guestops_release_commit is match('^[0-9a-f]{40}$')
- guestops_backup_recipient is match('^[A-Fa-f0-9]{40}$')
- guestops_recovery_public_key | length > 0
- guestops_evidence_directory | length > 0
- guestops_durable_logs_configured | bool
fail_msg: Release commit, recovery public key/fingerprint, evidence directory and reviewed durable logging are required.
- name: Confirm selected release link
ansible.builtin.stat:
path: "{{ guestops_current }}"
follow: false
register: guestops_selected_release
- name: Require deployed release before operations installation
ansible.builtin.assert:
that:
- guestops_selected_release.stat.exists
- guestops_selected_release.stat.islnk
- guestops_selected_release.stat.lnk_source == guestops_root + '/releases/' + guestops_release_commit
fail_msg: /srv/guestops/current must select the exact approved release commit.
- name: Confirm selected release target exists
ansible.builtin.stat:
path: "{{ guestops_root }}/releases/{{ guestops_release_commit }}"
follow: true
register: guestops_release_target
- name: Require selected release directory
ansible.builtin.assert:
that:
- guestops_release_target.stat.exists
- guestops_release_target.stat.isdir
fail_msg: The selected release target must be an installed directory.
- name: Inspect pre-provisioned private operational files
ansible.builtin.stat:
path: "{{ item.path }}"
follow: false
loop:
- { path: /etc/guestops/backup.env, mode: "0600" }
- { path: /etc/guestops/backup-transfer.env, mode: "0600" }
- { path: /etc/guestops/backup-transfer.key, mode: "0600" }
- { path: /etc/guestops/backup-known-hosts, mode: "0644" }
register: guestops_operational_files
- name: Require private backup and transfer configuration
ansible.builtin.assert:
that:
- item.stat.exists
- item.stat.isreg
- not item.stat.islnk
- item.stat.mode == item.item.mode
fail_msg: "{{ item.item.path }} must be a regular non-symlink file with mode {{ item.item.mode }}."
loop: "{{ guestops_operational_files.results }}"
no_log: true
tasks:
- name: Create private backup directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: root
group: root
mode: "0700"
loop:
- "{{ guestops_backup_root }}"
- "{{ guestops_gnupg_root }}"
- name: Stage recovery public key
ansible.builtin.copy:
src: "{{ guestops_recovery_public_key }}"
dest: /etc/guestops/recovery-public.asc
owner: root
group: root
mode: "0600"
- name: Import recovery public key only
ansible.builtin.command:
argv:
- gpg
- --batch
- --homedir
- "{{ guestops_gnupg_root }}"
- --import
- /etc/guestops/recovery-public.asc
no_log: true
changed_when: true
- name: Verify configured recovery fingerprint
ansible.builtin.command:
argv:
- gpg
- --batch
- --homedir
- "{{ guestops_gnupg_root }}"
- --list-keys
- "{{ guestops_backup_recipient }}"
changed_when: false
no_log: true
- name: Remove staged recovery public key
ansible.builtin.file:
path: /etc/guestops/recovery-public.asc
state: absent
- name: Install GuestOps systemd units
ansible.builtin.copy:
src: "{{ playbook_dir }}/../systemd/{{ item }}"
dest: "/etc/systemd/system/{{ item }}"
owner: root
group: root
mode: "0644"
loop:
- guestops-backup.service
- guestops-backup.timer
- guestops-backup-transfer.service
- guestops-backup-transfer.timer
- guestops-monitor-status.service
- guestops-monitor-status.timer
register: guestops_systemd_units
- name: Verify installed systemd units
ansible.builtin.command:
argv:
- systemd-analyze
- verify
- /etc/systemd/system/guestops-backup.service
- /etc/systemd/system/guestops-backup.timer
- /etc/systemd/system/guestops-backup-transfer.service
- /etc/systemd/system/guestops-backup-transfer.timer
- /etc/systemd/system/guestops-monitor-status.service
- /etc/systemd/system/guestops-monitor-status.timer
changed_when: false
- name: Reload systemd unit definitions
ansible.builtin.systemd_service:
daemon_reload: true
when: guestops_systemd_units.changed
- name: Enable transfer retry and monitoring timers
ansible.builtin.systemd_service:
name: "{{ item }}"
enabled: true
state: started
loop:
- guestops-backup-transfer.timer
- guestops-monitor-status.timer
- name: Select backup schedule state
ansible.builtin.systemd_service:
name: guestops-backup.timer
enabled: "{{ guestops_enable_backup_schedule | bool }}"
state: "{{ 'started' if guestops_enable_backup_schedule | bool else 'stopped' }}"
- name: Require Zabbix include directory
ansible.builtin.stat:
path: "{{ guestops_zabbix_include_directory }}"
register: guestops_zabbix_directory
- name: Confirm Zabbix agent configuration path
ansible.builtin.assert:
that:
- guestops_zabbix_directory.stat.exists
- guestops_zabbix_directory.stat.isdir
fail_msg: Override guestops_zabbix_include_directory for the installed Zabbix agent.
- name: Install restricted Zabbix status item
ansible.builtin.copy:
src: "{{ playbook_dir }}/../systemd/zabbix-agent-guestops.conf.example"
dest: "{{ guestops_zabbix_include_directory }}/guestops.conf"
owner: root
group: root
mode: "0644"
register: guestops_zabbix_configuration
- name: Restart Zabbix agent after configuration change
ansible.builtin.service:
name: "{{ guestops_zabbix_service }}"
enabled: true
state: restarted
when: guestops_zabbix_configuration.changed
- name: Generate initial non-sensitive monitoring status
ansible.builtin.command:
argv: [systemctl, start, guestops-monitor-status.service]
changed_when: true
- name: Read installed timer schedule
ansible.builtin.command:
argv: [systemctl, list-timers, --all, --no-pager, guestops-backup.timer, guestops-backup-transfer.timer, guestops-monitor-status.timer]
register: guestops_timer_status
changed_when: false
- name: Retain monitoring status on the Ansible controller
ansible.builtin.fetch:
src: /run/guestops-monitor/status.json
dest: "{{ guestops_evidence_directory }}/monitor-status.json"
flat: true
- name: Retain timer status on the Ansible controller
ansible.builtin.copy:
content: "{{ guestops_timer_status.stdout }}\n"
dest: "{{ guestops_evidence_directory }}/systemd-timers.txt"
mode: "0600"
delegate_to: localhost
become: false
-402
View File
@@ -1,402 +0,0 @@
---
- name: Install a verified GuestOps source release
hosts: guestops
become: true
gather_facts: true
vars:
guestops_root: /srv/guestops
guestops_config_root: /etc/guestops
guestops_public_hostname: sandbox-guestops.futuresens.co.uk
guestops_public_origin: "https://{{ guestops_public_hostname }}"
guestops_release_root: "{{ guestops_root }}/releases/{{ guestops_release_commit }}"
guestops_staging_archive: "/var/tmp/{{ guestops_archive | basename }}"
guestops_staging_record: "/var/tmp/{{ guestops_source_record | basename }}"
guestops_api_image: "guestops-api:{{ guestops_release_commit }}"
guestops_worker_image: "guestops-worker:{{ guestops_release_commit }}"
pre_tasks:
- name: Validate required release variables
ansible.builtin.assert:
that:
- guestops_release_version is match('^[0-9]+\\.[0-9]+\\.[0-9]+$')
- guestops_release_commit is match('^[0-9a-f]{40}$')
- guestops_archive_sha256 is match('^[0-9a-f]{64}$')
- guestops_archive | length > 0
- guestops_source_record | length > 0
- guestops_evidence_directory | length > 0
- guestops_public_hostname is match('^[A-Za-z0-9.-]+$')
- guestops_public_hostname == 'sandbox-guestops.futuresens.co.uk'
fail_msg: Release version, full commit, archive, SHA-256, source record and evidence directory are required.
- name: Verify source package on the Ansible controller
ansible.builtin.command:
argv:
- python3
- "{{ playbook_dir }}/../verify_source_package.py"
- --archive
- "{{ guestops_archive }}"
- --record
- "{{ guestops_source_record }}"
- --expected-commit
- "{{ guestops_release_commit }}"
- --expected-version
- "{{ guestops_release_version }}"
delegate_to: localhost
become: false
changed_when: false
- name: Check private GuestOps environment exists
ansible.builtin.stat:
path: "{{ guestops_config_root }}/guestops.env"
register: guestops_environment
- name: Require a private pre-provisioned environment file
ansible.builtin.assert:
that:
- guestops_environment.stat.exists
- guestops_environment.stat.isreg
- guestops_environment.stat.mode == '0600'
fail_msg: /etc/guestops/guestops.env must already exist with mode 0600.
tasks:
- name: Require supported Debian host
ansible.builtin.assert:
that:
- ansible_facts.system == 'Linux'
- ansible_facts.distribution == 'Debian'
- ansible_facts.distribution_major_version | int >= 12
- ansible_facts.processor_vcpus | default(0) | int >= 4
- ansible_facts.memtotal_mb | default(0) | int >= 7300
fail_msg: GuestOps requires Debian 12 or newer with at least 4 CPUs and 7.3 GiB RAM.
- name: Enable Docker at boot
ansible.builtin.service:
name: docker
enabled: true
state: started
- name: Enable Nginx at boot
ansible.builtin.service:
name: nginx
enabled: true
state: started
- name: Create release and evidence directories
ansible.builtin.file:
path: "{{ item.path }}"
state: directory
owner: root
group: root
mode: "{{ item.mode }}"
loop:
- { path: "{{ guestops_root }}/releases", mode: "0755" }
- { path: "{{ guestops_release_root }}", mode: "0755" }
- { path: "{{ guestops_config_root }}/release-records", mode: "0700" }
- name: Copy source archive to the host
ansible.builtin.copy:
src: "{{ guestops_archive }}"
dest: "{{ guestops_staging_archive }}"
owner: root
group: root
mode: "0600"
- name: Copy source record to the host
ansible.builtin.copy:
src: "{{ guestops_source_record }}"
dest: "{{ guestops_staging_record }}"
owner: root
group: root
mode: "0600"
- name: Calculate transferred archive checksum
ansible.builtin.stat:
path: "{{ guestops_staging_archive }}"
checksum_algorithm: sha256
register: guestops_transferred_archive
- name: Reject a changed transferred archive
ansible.builtin.assert:
that:
- guestops_transferred_archive.stat.checksum == guestops_archive_sha256
fail_msg: Transferred GuestOps archive does not match the approved SHA-256.
- name: Extract committed application files
ansible.builtin.unarchive:
src: "{{ guestops_staging_archive }}"
dest: "{{ guestops_release_root }}"
remote_src: true
extra_opts:
- --strip-components=1
- name: Verify transferred package and record on the target
ansible.builtin.command:
argv:
- python3
- deploy/verify_source_package.py
- --archive
- "{{ guestops_staging_archive }}"
- --record
- "{{ guestops_staging_record }}"
- --expected-commit
- "{{ guestops_release_commit }}"
- --expected-version
- "{{ guestops_release_version }}"
- --output
- "{{ guestops_config_root }}/release-records/{{ guestops_release_commit }}.source-verification.json"
chdir: "{{ guestops_release_root }}"
changed_when: false
- name: Retain approved source record on the target
ansible.builtin.copy:
src: "{{ guestops_staging_record }}"
dest: "{{ guestops_config_root }}/release-records/{{ guestops_release_commit }}.source.json"
remote_src: true
owner: root
group: root
mode: "0600"
- name: Select release images and safe pilot defaults in the private environment
ansible.builtin.lineinfile:
path: "{{ guestops_config_root }}/guestops.env"
regexp: "^{{ item.name }}="
line: "{{ item.name }}={{ item.value }}"
create: false
mode: "0600"
loop:
- { name: GUESTOPS_API_IMAGE, value: "{{ guestops_api_image }}" }
- { name: GUESTOPS_WORKER_IMAGE, value: "{{ guestops_worker_image }}" }
- { name: GOOGLE_ENABLE_SENDING, value: "false" }
- { name: AUTO_REPLY_ENABLE_LIVE, value: "false" }
no_log: true
- name: Copy private deployment environment into the release
ansible.builtin.copy:
src: "{{ guestops_config_root }}/guestops.env"
dest: "{{ guestops_release_root }}/.env"
remote_src: true
owner: root
group: root
mode: "0600"
- name: Build API image from the verified source
ansible.builtin.command:
argv:
- docker
- build
- --target
- api
- --tag
- "{{ guestops_api_image }}"
- .
chdir: "{{ guestops_release_root }}"
register: guestops_api_build
changed_when: true
- name: Build worker image from the verified source
ansible.builtin.command:
argv:
- docker
- build
- --target
- worker
- --tag
- "{{ guestops_worker_image }}"
- .
chdir: "{{ guestops_release_root }}"
register: guestops_worker_build
changed_when: true
- name: Inspect API image identity
ansible.builtin.command:
argv: [docker, image, inspect, --format, "{{ '{{.Id}}' }}", "{{ guestops_api_image }}"]
register: guestops_api_identity
changed_when: false
- name: Inspect worker image identity
ansible.builtin.command:
argv: [docker, image, inspect, --format, "{{ '{{.Id}}' }}", "{{ guestops_worker_image }}"]
register: guestops_worker_identity
changed_when: false
- name: Create post-build release record
ansible.builtin.command:
argv:
- python3
- deploy/release_record.py
- --artifact
- "{{ guestops_staging_archive }}"
- --commit
- "{{ guestops_release_commit }}"
- --api-image
- "{{ guestops_api_image }}"
- --api-id
- "{{ guestops_api_identity.stdout }}"
- --worker-image
- "{{ guestops_worker_image }}"
- --worker-id
- "{{ guestops_worker_identity.stdout }}"
- --output
- "{{ guestops_config_root }}/release-records/{{ guestops_release_commit }}.json"
chdir: "{{ guestops_release_root }}"
- name: Run offline deployment preflight
ansible.builtin.command:
argv:
- python3
- deploy/ops.py
- preflight
- --offline
chdir: "{{ guestops_release_root }}"
changed_when: false
- name: Validate Compose without exposing expanded configuration
ansible.builtin.command:
argv:
- docker
- compose
- --env-file
- "{{ guestops_config_root }}/guestops.env"
- config
- --quiet
chdir: "{{ guestops_release_root }}"
changed_when: false
- name: Start the verified release without rebuilding
ansible.builtin.command:
argv:
- docker
- compose
- --env-file
- "{{ guestops_config_root }}/guestops.env"
- up
- --detach
- --no-build
chdir: "{{ guestops_release_root }}"
environment:
GUESTOPS_API_IMAGE: "{{ guestops_api_image }}"
GUESTOPS_WORKER_IMAGE: "{{ guestops_worker_image }}"
changed_when: true
- name: Wait for loopback readiness
ansible.builtin.uri:
url: http://127.0.0.1:8080/health/ready
method: GET
status_code: 200
return_content: false
register: guestops_readiness
retries: 30
delay: 2
until: guestops_readiness.status == 200
- name: Confirm TLS certificate files exist
ansible.builtin.stat:
path: "{{ item }}"
follow: true
loop:
- "/etc/letsencrypt/live/{{ guestops_public_hostname }}/fullchain.pem"
- "/etc/letsencrypt/live/{{ guestops_public_hostname }}/privkey.pem"
register: guestops_certificates
- name: Require the pre-provisioned TLS certificate
ansible.builtin.assert:
that:
- guestops_certificates.results | map(attribute='stat.exists') | min
- guestops_certificates.results | map(attribute='stat.isreg') | min
fail_msg: A valid pre-provisioned Let's Encrypt certificate is required before enabling Nginx.
- name: Install reviewed GuestOps Nginx site
ansible.builtin.copy:
src: "{{ playbook_dir }}/../nginx.conf"
dest: /etc/nginx/sites-available/guestops.conf
owner: root
group: root
mode: "0644"
register: guestops_nginx_site
- name: Enable GuestOps Nginx site
ansible.builtin.file:
src: /etc/nginx/sites-available/guestops.conf
dest: /etc/nginx/sites-enabled/guestops.conf
state: link
register: guestops_nginx_enabled
- name: Validate Nginx configuration
ansible.builtin.command:
argv: [nginx, -t]
changed_when: false
- name: Reload Nginx after reviewed configuration change
ansible.builtin.service:
name: nginx
enabled: true
state: reloaded
when: guestops_nginx_site.changed or guestops_nginx_enabled.changed
- name: Inspect host TCP listeners
ansible.builtin.command:
argv: [ss, -ltnH]
register: guestops_tcp_listeners
changed_when: false
- name: Reject public API or MongoDB listeners
ansible.builtin.assert:
that:
- guestops_tcp_listeners.stdout_lines | select('search', ':8080(\\s|$)') | reject('search', '127\\.0\\.0\\.1:8080(\\s|$)') | list | length == 0
- guestops_tcp_listeners.stdout_lines | select('search', ':27017(\\s|$)') | list | length == 0
fail_msg: API port 8080 must be loopback-only and MongoDB must have no host listener.
- name: Verify public HTTP redirects to HTTPS
ansible.builtin.uri:
url: "http://{{ guestops_public_hostname }}/health/ready"
method: GET
follow_redirects: none
status_code: [301, 302, 307, 308]
return_content: false
delegate_to: localhost
become: false
- name: Verify public HTTPS readiness and certificate trust
ansible.builtin.uri:
url: "{{ guestops_public_origin }}/health/ready"
method: GET
status_code: 200
return_content: true
validate_certs: true
register: guestops_public_readiness
retries: 10
delay: 3
until:
- guestops_public_readiness.status == 200
- guestops_public_readiness.json is defined
- guestops_public_readiness.json.status == 'ready'
delegate_to: localhost
become: false
- name: Select the current successful release
ansible.builtin.file:
src: "{{ guestops_release_root }}"
dest: "{{ guestops_root }}/current"
state: link
force: true
- name: Retain release record on the Ansible controller
ansible.builtin.fetch:
src: "{{ guestops_config_root }}/release-records/{{ guestops_release_commit }}.json"
dest: "{{ guestops_evidence_directory }}/release-record.json"
flat: true
- name: Retain target source verification on the Ansible controller
ansible.builtin.fetch:
src: "{{ guestops_config_root }}/release-records/{{ guestops_release_commit }}.source-verification.json"
dest: "{{ guestops_evidence_directory }}/source-verification.json"
flat: true
- name: Remove transferred staging files after successful deployment
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- "{{ guestops_staging_archive }}"
- "{{ guestops_staging_record }}"
@@ -1,81 +0,0 @@
{
"schemaVersion": 1,
"system": "guestops-backup-recovery",
"evidenceId": "backup-restore",
"dataClassification": "synthetic-only",
"releaseVersion": "0.2.1",
"releaseCommit": "0000000000000000000000000000000000000000",
"releaseRecordSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"archiveSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"environment": "https://sandbox-guestops.futuresens.co.uk",
"hostIdentifier": "guestops-sandbox-01",
"images": {
"api": {"reference": "guestops-api:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
"worker": {"reference": "guestops-worker:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
"mongo": {"reference": "mongo:8.0", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"}
},
"owners": {
"backup": "REPLACE",
"monitoring": "REPLACE",
"recoveryOperator": "REPLACE",
"technicalEscalation": "REPLACE",
"retention": "REPLACE",
"independentReviewer": "REPLACE"
},
"startedAt": "2026-10-01T09:00:00Z",
"endedAt": "2026-10-01T13:00:00Z",
"reviewedAt": "2026-10-01T14:00:00Z",
"recoveryObjectives": {
"targetRpoHours": 24,
"observedRpoHours": 25,
"targetRtoMinutes": 240,
"observedRtoMinutes": 241
},
"retention": {
"localVerifiedDays": 7,
"offHostDaily": 35,
"offHostMonthly": 12,
"legalHoldOverrideTested": false
},
"backup": {
"createdAt": "2026-10-01T08:00:00Z",
"sha256": "0000000000000000000000000000000000000000000000000000000000000000",
"transferredSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"privateKeyPresentOnHost": false
},
"rollback": {
"previousReleaseCommit": "1111111111111111111111111111111111111111",
"previousArchiveSha256": "1111111111111111111111111111111111111111111111111111111111111111",
"previousImages": {
"api": {"reference": "guestops-api:1111111111111111111111111111111111111111", "id": "sha256:1111111111111111111111111111111111111111111111111111111111111111"},
"worker": {"reference": "guestops-worker:1111111111111111111111111111111111111111", "id": "sha256:1111111111111111111111111111111111111111111111111111111111111111"}
},
"persistentVolumesReplaced": false,
"restoredReleaseCommit": "0000000000000000000000000000000000000000",
"unresolvedOperations": 1,
"finalControls": {
"googleSending": "disabled",
"faqLiveMode": "disabled",
"pmsWrites": "disabled",
"paymentCreation": "disabled"
}
},
"monitoringState": "not-configured",
"unresolvedCriticalFindings": 1,
"scenarios": [
{"id": "alert-escalation", "status": "not-run", "evidence": []},
{"id": "atomic-off-host-transfer", "status": "not-run", "evidence": []},
{"id": "controlled-return-to-service", "status": "not-run", "evidence": []},
{"id": "data-protection-recovery", "status": "not-run", "evidence": []},
{"id": "database-inventory", "status": "not-run", "evidence": []},
{"id": "durable-secret-free-logs", "status": "not-run", "evidence": []},
{"id": "encrypted-manual-backup", "status": "not-run", "evidence": []},
{"id": "image-rollback", "status": "not-run", "evidence": []},
{"id": "isolated-restore", "status": "not-run", "evidence": []},
{"id": "monitoring-coverage", "status": "not-run", "evidence": []},
{"id": "off-host-checksum", "status": "not-run", "evidence": []},
{"id": "production-service-recovery", "status": "not-run", "evidence": []},
{"id": "retention-and-legal-hold", "status": "not-run", "evidence": []},
{"id": "scheduled-backup", "status": "not-run", "evidence": []}
]
}
-220
View File
@@ -1,220 +0,0 @@
#!/usr/bin/env python3
"""Validate a restricted GuestOps backup, monitoring and recovery record."""
from __future__ import annotations
import argparse
import datetime as dt
import json
from pathlib import Path
import re
from urllib.parse import urlparse
VERSION = "0.2.1"
SCENARIOS = {
"encrypted-manual-backup",
"scheduled-backup",
"production-service-recovery",
"atomic-off-host-transfer",
"off-host-checksum",
"retention-and-legal-hold",
"monitoring-coverage",
"alert-escalation",
"durable-secret-free-logs",
"isolated-restore",
"data-protection-recovery",
"database-inventory",
"image-rollback",
"controlled-return-to-service",
}
SHA256 = re.compile(r"[0-9a-f]{64}")
GIT_SHA = re.compile(r"[0-9a-f]{40}")
def require(condition: bool, message: str) -> None:
if not condition:
raise ValueError(message)
def timestamp(value: object, field: str) -> dt.datetime:
require(isinstance(value, str) and value.endswith("Z"),
f"{field} must be a UTC timestamp ending in Z.")
try:
parsed = dt.datetime.fromisoformat(value.removesuffix("Z") + "+00:00")
except ValueError as error:
raise ValueError(f"{field} is not a valid timestamp.") from error
require(parsed.tzinfo == dt.timezone.utc, f"{field} must be UTC.")
return parsed
def safe_name(value: object, field: str) -> str:
name = str(value or "").strip()
require(2 <= len(name) <= 120 and "@" not in name and "/" not in name and "\\" not in name,
f"{field} requires a name without an email address or path.")
return name
def validate(record: object, expected_commit: str, expected_release_sha256: str) -> None:
require(isinstance(record, dict), "Acceptance record must be a JSON object.")
require(record.get("schemaVersion") == 1, "Unsupported backup/recovery schema.")
require(record.get("system") == "guestops-backup-recovery",
"system must be guestops-backup-recovery.")
require(record.get("evidenceId") == "backup-restore",
"evidenceId must be backup-restore.")
require(record.get("dataClassification") == "synthetic-only",
"Recovery acceptance must use synthetic data only.")
require(record.get("releaseVersion") == VERSION, f"releaseVersion must be {VERSION}.")
require(GIT_SHA.fullmatch(str(expected_commit)) is not None,
"Expected release commit must be a full lowercase Git SHA.")
require(SHA256.fullmatch(str(expected_release_sha256)) is not None,
"Expected release-record checksum must be a lowercase SHA-256 digest.")
require(record.get("releaseCommit") == expected_commit,
"releaseCommit does not match the approved candidate.")
require(record.get("releaseRecordSha256") == expected_release_sha256,
"releaseRecordSha256 does not match the retained release record.")
require(SHA256.fullmatch(str(record.get("archiveSha256", ""))) is not None,
"archiveSha256 must be a lowercase SHA-256 digest.")
origin = urlparse(str(record.get("environment", "")))
require(origin.scheme == "https" and origin.hostname and origin.path in ("", "/")
and not origin.query and not origin.fragment and origin.username is None
and origin.password is None,
"environment must be an HTTPS origin without credentials, path, query or fragment.")
safe_name(record.get("hostIdentifier"), "hostIdentifier")
images = record.get("images")
require(isinstance(images, dict) and set(images) == {"api", "worker", "mongo"},
"images must contain exactly api, worker and mongo.")
for name in ("api", "worker"):
image = images[name]
require(isinstance(image, dict) and set(image) == {"reference", "id"},
f"images.{name} must contain exactly reference and id.")
require(image["reference"] == f"guestops-{name}:{expected_commit}",
f"images.{name}.reference must use the full approved commit.")
require(re.fullmatch(r"sha256:[0-9a-f]{64}", str(image["id"])) is not None,
f"images.{name}.id must be immutable.")
mongo = images["mongo"]
require(isinstance(mongo, dict) and set(mongo) == {"reference", "id"}
and mongo["reference"] == "mongo:8.0"
and re.fullmatch(r"sha256:[0-9a-f]{64}", str(mongo["id"])) is not None,
"images.mongo must identify the immutable mongo:8.0 image.")
owners = record.get("owners")
owner_keys = {"backup", "monitoring", "recoveryOperator", "technicalEscalation",
"retention", "independentReviewer"}
require(isinstance(owners, dict) and set(owners) == owner_keys,
"owners must contain the exact operational and review roles.")
names = {key: safe_name(value, f"owners.{key}") for key, value in owners.items()}
reviewer = names["independentReviewer"].casefold()
require(reviewer not in {names[key].casefold() for key in owner_keys - {"independentReviewer"}},
"independentReviewer must be different from every operational owner.")
started = timestamp(record.get("startedAt"), "startedAt")
ended = timestamp(record.get("endedAt"), "endedAt")
reviewed = timestamp(record.get("reviewedAt"), "reviewedAt")
require(started <= ended <= reviewed, "Acceptance timestamps are out of order.")
recovery = record.get("recoveryObjectives")
require(isinstance(recovery, dict) and set(recovery) == {
"targetRpoHours", "observedRpoHours", "targetRtoMinutes", "observedRtoMinutes",
}, "recoveryObjectives must contain exact target and observed RPO/RTO values.")
for field in recovery:
require(isinstance(recovery[field], (int, float)) and not isinstance(recovery[field], bool)
and recovery[field] >= 0, f"recoveryObjectives.{field} must be non-negative.")
require(recovery["targetRpoHours"] == 24 and recovery["observedRpoHours"] <= 24,
"Observed RPO must meet the approved 24-hour target.")
require(recovery["targetRtoMinutes"] == 240 and recovery["observedRtoMinutes"] <= 240,
"Observed RTO must meet the approved four-hour target.")
retention = record.get("retention")
require(retention == {
"localVerifiedDays": 7,
"offHostDaily": 35,
"offHostMonthly": 12,
"legalHoldOverrideTested": True,
}, "Retention must record seven local days, 35 daily and 12 monthly off-host copies, and legal-hold testing.")
backup = record.get("backup")
require(isinstance(backup, dict) and set(backup) == {
"createdAt", "sha256", "transferredSha256", "privateKeyPresentOnHost",
}, "backup must contain exact creation, checksum, transfer and private-key fields.")
created = timestamp(backup["createdAt"], "backup.createdAt")
require(created <= started, "The accepted backup must exist when the timed exercise starts.")
require(abs(recovery["observedRpoHours"] - (started - created).total_seconds() / 3600) < 0.01,
"Observed RPO must match the backup and exercise timestamps.")
require(abs(recovery["observedRtoMinutes"] - (ended - started).total_seconds() / 60) < 0.01,
"Observed RTO must match the exercise timestamps.")
require(SHA256.fullmatch(str(backup["sha256"])) is not None
and backup["transferredSha256"] == backup["sha256"],
"Local and transferred backup checksums must match.")
require(backup["privateKeyPresentOnHost"] is False,
"The recovery private key must not be present on the Debian host.")
rollback = record.get("rollback")
require(isinstance(rollback, dict) and set(rollback) == {
"previousReleaseCommit", "previousArchiveSha256", "previousImages",
"persistentVolumesReplaced", "restoredReleaseCommit", "unresolvedOperations", "finalControls",
}, "rollback must contain the exact rehearsal and final-state fields.")
require(GIT_SHA.fullmatch(str(rollback["previousReleaseCommit"])) is not None
and rollback["previousReleaseCommit"] != expected_commit,
"Rollback must use a different retained previous release.")
require(SHA256.fullmatch(str(rollback["previousArchiveSha256"])) is not None,
"Rollback requires the previous archive checksum.")
previous_images = rollback["previousImages"]
require(isinstance(previous_images, dict) and set(previous_images) == {"api", "worker"},
"Rollback requires exact previous API and worker images.")
for name in ("api", "worker"):
image = previous_images[name]
require(isinstance(image, dict) and set(image) == {"reference", "id"}
and image["reference"] == f"guestops-{name}:{rollback['previousReleaseCommit']}"
and re.fullmatch(r"sha256:[0-9a-f]{64}", str(image["id"])) is not None,
f"rollback.previousImages.{name} must use the retained previous release identity.")
require(rollback["persistentVolumesReplaced"] is False,
"Image rollback must not replace persistent volumes.")
require(rollback["restoredReleaseCommit"] == expected_commit,
"The exercise must finish on the approved candidate.")
require(rollback["unresolvedOperations"] == 0,
"The exercise must finish without unresolved operations.")
require(rollback["finalControls"] == {
"googleSending": "disabled", "faqLiveMode": "disabled",
"pmsWrites": "disabled", "paymentCreation": "disabled",
}, "The exercise must finish with all unaccepted external writes disabled.")
scenarios = record.get("scenarios")
require(isinstance(scenarios, list), "scenarios must be a list.")
ids = [item.get("id") for item in scenarios if isinstance(item, dict)]
require(len(ids) == len(scenarios) and len(ids) == len(set(ids)) and set(ids) == SCENARIOS,
"Acceptance record requires the exact backup/recovery scenario set.")
for item in scenarios:
scenario_id = item["id"]
require(item.get("status") == "pass", f"Scenario {scenario_id} has not passed.")
evidence = item.get("evidence")
require(isinstance(evidence, list) and 1 <= len(evidence) <= 10 and all(
isinstance(value, str) and 3 <= len(value) <= 200 and "@" not in value
and "\\" not in value and not value.startswith("/") for value in evidence
), f"Scenario {scenario_id} requires safe opaque evidence references.")
require(record.get("monitoringState") == "healthy",
"Monitoring must be healthy at acceptance completion.")
require(record.get("unresolvedCriticalFindings") == 0,
"Acceptance cannot pass with unresolved critical findings.")
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("record", type=Path)
parser.add_argument("--expected-commit", required=True)
parser.add_argument("--expected-release-record-sha256", required=True)
args = parser.parse_args()
validate(json.loads(args.record.read_text(encoding="utf-8")),
args.expected_commit, args.expected_release_record_sha256)
print("Backup, monitoring and recovery acceptance record is structurally complete and passed. "
"This validates the record, not its restricted evidence.")
if __name__ == "__main__":
try:
main()
except (OSError, ValueError, json.JSONDecodeError) as error:
print(f"Backup/recovery acceptance record rejected: {error}", file=__import__("sys").stderr)
raise SystemExit(1)
-238
View File
@@ -1,238 +0,0 @@
#!/usr/bin/env python3
"""Atomically transfer encrypted GuestOps backups to restricted storage."""
from __future__ import annotations
import argparse
import hashlib
import json
import os
from pathlib import Path
import re
import shlex
import shutil
import stat
import subprocess
import tempfile
import time
import uuid
BACKUP_NAME = re.compile(r"guestops-[0-9]{8}T[0-9]{6}Z\.tar\.gpg")
SAFE_HOST = re.compile(r"[A-Za-z0-9.-]{1,253}")
SAFE_USER = re.compile(r"[A-Za-z_][A-Za-z0-9_-]{0,31}")
SAFE_REMOTE_PATH = re.compile(r"/[A-Za-z0-9._/-]{1,500}")
SHA256 = re.compile(r"[0-9a-f]{64}")
def require(condition: bool, message: str) -> None:
if not condition:
raise RuntimeError(message)
def digest(path: Path) -> str:
with path.open("rb") as stream:
return hashlib.file_digest(stream, "sha256").hexdigest()
def private_directory(value: str) -> Path:
requested = Path(value)
require(requested.is_absolute() and not requested.is_symlink(),
"BACKUP_DIRECTORY must be an absolute, non-symlink path.")
directory = requested.resolve()
require(directory.is_dir(), "BACKUP_DIRECTORY must exist.")
require(stat.S_IMODE(directory.stat().st_mode) & 0o077 == 0,
"BACKUP_DIRECTORY must not be accessible to group or other users.")
return directory
def regular_file(value: str, field: str, *, private: bool) -> Path:
requested = Path(value)
require(requested.is_absolute() and not requested.is_symlink(),
f"{field} must be an absolute, non-symlink path.")
path = requested.resolve()
require(path.is_file(), f"{field} must be an existing regular file.")
if private:
require(stat.S_IMODE(path.stat().st_mode) & 0o077 == 0,
f"{field} must not be accessible to group or other users.")
return path
def configuration(environment: dict[str, str]) -> dict[str, object]:
directory = private_directory(environment.get("BACKUP_DIRECTORY", ""))
host = environment.get("BACKUP_REMOTE_HOST", "")
user = environment.get("BACKUP_REMOTE_USER", "")
remote = environment.get("BACKUP_REMOTE_DIRECTORY", "")
require(SAFE_HOST.fullmatch(host) is not None, "BACKUP_REMOTE_HOST is invalid.")
require(SAFE_USER.fullmatch(user) is not None, "BACKUP_REMOTE_USER is invalid.")
require(SAFE_REMOTE_PATH.fullmatch(remote) is not None and "//" not in remote
and "/../" not in remote + "/" and not remote.endswith("/.."),
"BACKUP_REMOTE_DIRECTORY must be a safe absolute path.")
identity = regular_file(environment.get("BACKUP_SSH_IDENTITY", ""),
"BACKUP_SSH_IDENTITY", private=True)
known_hosts = regular_file(environment.get("BACKUP_SSH_KNOWN_HOSTS", ""),
"BACKUP_SSH_KNOWN_HOSTS", private=False)
require(shutil.which("ssh") is not None and shutil.which("rsync") is not None,
"ssh and rsync are required.")
return {
"directory": directory,
"host": host,
"user": user,
"remote": remote.rstrip("/"),
"identity": identity,
"known_hosts": known_hosts,
}
def ssh_base(config: dict[str, object]) -> list[str]:
return [
"ssh", "-o", "BatchMode=yes", "-o", "IdentitiesOnly=yes",
"-o", "StrictHostKeyChecking=yes", "-o", "ConnectTimeout=15",
"-o", f"UserKnownHostsFile={config['known_hosts']}",
"-i", str(config["identity"]), f"{config['user']}@{config['host']}",
]
def run(args: list[str], *, environment: dict[str, str] | None = None) -> bytes:
result = subprocess.run(args, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, timeout=900, env=environment)
require(result.returncode == 0,
f"{Path(args[0]).name} step failed; review the restricted operator logs.")
return result.stdout
def remote_digest(config: dict[str, object], remote_path: str) -> str | None:
command = f"if test -f {remote_path} && test ! -L {remote_path}; then sha256sum -- {remote_path}; fi"
output = run([*ssh_base(config), command]).decode("utf-8", "strict").strip()
if not output:
return None
value = output.split()[0]
require(SHA256.fullmatch(value) is not None, "Remote checksum response was invalid.")
return value
def marker_path(backup: Path) -> Path:
return backup.with_name(backup.name + ".transferred.json")
def write_marker(backup: Path, checksum: str) -> None:
marker = marker_path(backup)
payload = json.dumps({
"schemaVersion": 1,
"backup": backup.name,
"sha256": checksum,
"verifiedAt": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
}, sort_keys=True) + "\n"
fd, temporary = tempfile.mkstemp(prefix=marker.name + ".", dir=marker.parent)
try:
os.fchmod(fd, 0o600)
with os.fdopen(fd, "w", encoding="utf-8") as stream:
stream.write(payload)
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary, marker)
finally:
try:
os.unlink(temporary)
except FileNotFoundError:
pass
def transfer_one(config: dict[str, object], backup: Path) -> None:
require(backup.is_file() and not backup.is_symlink()
and BACKUP_NAME.fullmatch(backup.name) is not None,
"Refusing to transfer an unexpected backup path.")
checksum = digest(backup)
remote_final = f"{config['remote']}/{backup.name}"
existing = remote_digest(config, remote_final)
if existing is not None:
require(existing == checksum, "A remote backup with this name has a different checksum.")
write_marker(backup, checksum)
return
remote_partial = f"{config['remote']}/.{backup.name}.partial-{uuid.uuid4().hex}"
rsh = shlex.join([
"ssh", "-o", "BatchMode=yes", "-o", "IdentitiesOnly=yes",
"-o", "StrictHostKeyChecking=yes", "-o", "ConnectTimeout=15",
"-o", f"UserKnownHostsFile={config['known_hosts']}",
"-i", str(config["identity"]),
])
rsync_environment = os.environ.copy()
rsync_environment["RSYNC_RSH"] = rsh
try:
run(["rsync", "--archive", "--chmod=F600", "--protect-args", "--",
str(backup), f"{config['user']}@{config['host']}:{remote_partial}"],
environment=rsync_environment)
require(remote_digest(config, remote_partial) == checksum,
"Transferred backup checksum does not match the local file.")
command = (f"test ! -e {remote_final} && mv -T -- {remote_partial} {remote_final} "
f"&& chmod 600 -- {remote_final}")
run([*ssh_base(config), command])
require(remote_digest(config, remote_final) == checksum,
"Final remote backup checksum does not match the local file.")
write_marker(backup, checksum)
except Exception:
# The name contains a fresh random suffix and is the only remote object this run may remove.
subprocess.run([*ssh_base(config), f"rm -f -- {remote_partial}"], stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=30)
raise
def transfer_pending(config: dict[str, object]) -> int:
directory = config["directory"]
backups = sorted(path for path in directory.iterdir()
if path.is_file() and not path.is_symlink()
and BACKUP_NAME.fullmatch(path.name) is not None)
for backup in backups:
marker = marker_path(backup)
if marker.is_file() and not marker.is_symlink():
try:
recorded = json.loads(marker.read_text(encoding="utf-8"))
if recorded.get("sha256") == digest(backup):
continue
except (OSError, ValueError, json.JSONDecodeError):
pass
transfer_one(config, backup)
return len(backups)
def prune_verified(config: dict[str, object], retention_days: int, now: float | None = None) -> int:
require(1 <= retention_days <= 365, "Local retention must be between 1 and 365 days.")
threshold = (time.time() if now is None else now) - retention_days * 86400
removed = 0
for backup in config["directory"].iterdir():
if not backup.is_file() or backup.is_symlink() or BACKUP_NAME.fullmatch(backup.name) is None:
continue
marker = marker_path(backup)
if backup.stat().st_mtime >= threshold or not marker.is_file() or marker.is_symlink():
continue
try:
recorded = json.loads(marker.read_text(encoding="utf-8"))
except (OSError, ValueError, json.JSONDecodeError):
continue
if recorded.get("backup") != backup.name or recorded.get("sha256") != digest(backup):
continue
backup.unlink()
marker.unlink()
removed += 1
return removed
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--prune-verified", action="store_true")
parser.add_argument("--retention-days", type=int, default=7)
args = parser.parse_args()
config = configuration(dict(os.environ))
observed = transfer_pending(config)
removed = prune_verified(config, args.retention_days) if args.prune_verified else 0
print(f"Backup transfer completed: {observed} encrypted backup(s) inspected; "
f"{removed} verified local backup(s) expired.")
if __name__ == "__main__":
try:
main()
except (OSError, RuntimeError, subprocess.SubprocessError, json.JSONDecodeError) as error:
print(f"Backup transfer failed: {error}", file=__import__("sys").stderr)
raise SystemExit(1)
-128
View File
@@ -1,128 +0,0 @@
#!/usr/bin/env python3
"""Run a bounded, read-only capacity probe against an approved GuestOps sandbox."""
from __future__ import annotations
import argparse
import concurrent.futures
import datetime as dt
import http.cookiejar
import json
import os
from pathlib import Path
import re
import statistics
import time
import urllib.error
import urllib.parse
import urllib.request
PATHS = ("/health/ready", "/api/hotel", "/api/conversations/page")
def require(condition: bool, message: str) -> None:
if not condition:
raise ValueError(message)
def percentile(values: list[float], fraction: float) -> float:
ordered = sorted(values)
position = max(0, min(len(ordered) - 1, int(len(ordered) * fraction + 0.999999) - 1))
return ordered[position]
def summarize(results: list[tuple[bool, float]], concurrency: int) -> dict[str, object]:
require(len(results) > 0, "At least one probe result is required.")
latencies = [latency for _, latency in results]
successes = sum(1 for success, _ in results if success)
return {
"concurrency": concurrency,
"requests": len(results),
"successes": successes,
"failures": len(results) - successes,
"errorRate": round((len(results) - successes) / len(results), 6),
"latencyMs": {
"median": round(statistics.median(latencies), 2),
"p95": round(percentile(latencies, 0.95), 2),
"maximum": round(max(latencies), 2),
},
}
def login(origin: str, email: str, password: str) -> str:
jar = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(jar))
with opener.open(origin + "/api/session", timeout=15) as response:
csrf = json.load(response)["csrfToken"]
body = json.dumps({"email": email, "password": password}).encode()
request = urllib.request.Request(origin + "/api/auth/login", data=body, method="POST", headers={"Content-Type": "application/json", "X-CSRF-TOKEN": csrf})
with opener.open(request, timeout=15) as response:
require(response.status == 200, "Sandbox login failed.")
cookies = "; ".join(f"{cookie.name}={cookie.value}" for cookie in jar)
require("guestops.session=" in cookies, "Sandbox did not issue a GuestOps session cookie.")
return cookies
def request_once(origin: str, cookie: str, number: int) -> tuple[bool, float]:
path = PATHS[number % len(PATHS)]
request = urllib.request.Request(origin + path, headers={"Cookie": cookie, "Accept": "application/json"})
started = time.perf_counter()
try:
with urllib.request.urlopen(request, timeout=20) as response:
success = response.status == 200
response.read(1024) # Bound local processing; never retain response or guest content.
except (OSError, urllib.error.HTTPError):
success = False
return success, (time.perf_counter() - started) * 1000
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--origin", required=True)
parser.add_argument("--requests", type=int, default=100)
parser.add_argument("--concurrency", type=int, default=5)
parser.add_argument("--release-commit", required=True)
parser.add_argument("--release-record-sha256", required=True)
parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--confirm-sandbox", action="store_true")
args = parser.parse_args()
parsed = urllib.parse.urlparse(args.origin)
require(args.confirm_sandbox, "Use --confirm-sandbox after confirming the target and maintenance window.")
require(parsed.scheme == "https" and parsed.hostname and parsed.path in ("", "/") and not parsed.query and not parsed.fragment and not parsed.username,
"Origin must be an HTTPS origin without credentials, path, query or fragment.")
require(parsed.hostname != "localhost" and not parsed.hostname.startswith("127."), "Use the deployed HTTPS sandbox, not a development server.")
require(1 <= args.concurrency <= 20 and 1 <= args.requests <= 2000, "Probe bounds are 1–20 concurrent workers and 1–2000 requests.")
require(re.fullmatch(r"[0-9a-f]{40}", args.release_commit) is not None, "Use a full lowercase release commit SHA.")
require(re.fullmatch(r"[0-9a-f]{64}", args.release_record_sha256) is not None, "Use the release-record SHA-256.")
require(not args.output.exists() and args.output.parent.is_dir(), "Output must be a new file in an existing restricted directory.")
email = os.environ.get("CAPACITY_EMAIL", "")
password = os.environ.get("CAPACITY_PASSWORD", "")
require(email and password, "Set CAPACITY_EMAIL and CAPACITY_PASSWORD for a dedicated sandbox staff account.")
cookie = login(args.origin.rstrip("/"), email, password)
with concurrent.futures.ThreadPoolExecutor(max_workers=args.concurrency) as pool:
results = list(pool.map(lambda number: request_once(args.origin.rstrip("/"), cookie, number), range(args.requests)))
report = {
"schemaVersion": 1,
"kind": "guestops-read-only-capacity",
"recordedAt": dt.datetime.now(dt.timezone.utc).isoformat().replace("+00:00", "Z"),
"originHost": parsed.hostname,
"releaseCommit": args.release_commit,
"releaseRecordSha256": args.release_record_sha256,
"paths": list(PATHS),
**summarize(results, args.concurrency),
}
descriptor = os.open(args.output, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(descriptor, "w", encoding="utf-8") as output:
output.write(json.dumps(report, indent=2, sort_keys=True) + "\n")
print(f"Capacity probe completed: {report['successes']}/{report['requests']} successful; p95 {report['latencyMs']['p95']} ms. Review against the approved target before release.")
if __name__ == "__main__":
try:
main()
except (OSError, ValueError, json.JSONDecodeError) as error:
print(f"Capacity probe stopped: {error}", file=__import__("sys").stderr)
raise SystemExit(1)
@@ -1,45 +0,0 @@
{
"schemaVersion": 1,
"system": "guestops-debian-host",
"evidenceId": "debian-host",
"releaseVersion": "0.2.1",
"releaseCommit": "0000000000000000000000000000000000000000",
"releaseRecordSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"archiveSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"environment": "https://sandbox-guestops.futuresens.co.uk",
"hostIdentifier": "guestops-sandbox-01",
"images": {
"api": {"reference": "guestops-api:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
"worker": {"reference": "guestops-worker:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"}
},
"hostFacts": {
"debianMajor": 12,
"cpuCores": 4,
"memoryBytes": 8140382208,
"freeDiskBytes": 14275686400,
"publicTcpPorts": []
},
"featureControls": {
"googleSending": "disabled",
"faqLiveMode": "disabled",
"pmsWrites": "disabled",
"paymentCreation": "disabled"
},
"operator": "REPLACE",
"reviewedBy": "REPLACE",
"startedAt": "2026-09-30T09:00:00Z",
"endedAt": "2026-09-30T10:00:00Z",
"reviewedAt": "2026-09-30T11:00:00Z",
"unresolvedCriticalFindings": 1,
"scenarios": [
{"id": "boot-services", "status": "not-run", "evidence": []},
{"id": "controlled-reboot", "status": "not-run", "evidence": []},
{"id": "durable-log-retrieval", "status": "not-run", "evidence": []},
{"id": "host-baseline", "status": "not-run", "evidence": []},
{"id": "https-and-redirect", "status": "not-run", "evidence": []},
{"id": "network-exposure", "status": "not-run", "evidence": []},
{"id": "proxy-trust", "status": "not-run", "evidence": []},
{"id": "secret-free-logs", "status": "not-run", "evidence": []},
{"id": "workspace-health", "status": "not-run", "evidence": []}
]
}
-206
View File
@@ -1,206 +0,0 @@
#!/usr/bin/env python3
"""Validate restricted GuestOps Debian-host and persistence acceptance records."""
from __future__ import annotations
import argparse
import datetime as dt
import json
from pathlib import Path
import re
from urllib.parse import urlparse
VERSION = "0.2.1"
SYSTEMS = {
"guestops-debian-host": {
"evidenceId": "debian-host",
"scenarios": {
"host-baseline",
"network-exposure",
"https-and-redirect",
"proxy-trust",
"boot-services",
"controlled-reboot",
"workspace-health",
"durable-log-retrieval",
"secret-free-logs",
},
},
"guestops-persistence": {
"evidenceId": "persistence",
"scenarios": {
"separate-volume-layout",
"service-restart",
"container-recreation",
"database-inventory",
"data-protection-key",
"image-identity",
"post-reboot-persistence",
},
},
}
SHA256 = re.compile(r"[0-9a-f]{64}")
GIT_SHA = re.compile(r"[0-9a-f]{40}")
def require(condition: bool, message: str) -> None:
if not condition:
raise ValueError(message)
def timestamp(value: object, field: str) -> dt.datetime:
require(isinstance(value, str) and value.endswith("Z"),
f"{field} must be a UTC timestamp ending in Z.")
try:
parsed = dt.datetime.fromisoformat(value.removesuffix("Z") + "+00:00")
except ValueError as error:
raise ValueError(f"{field} is not a valid timestamp.") from error
require(parsed.tzinfo == dt.timezone.utc, f"{field} must be UTC.")
return parsed
def safe_text(value: object, field: str, minimum: int = 2, maximum: int = 160) -> str:
text = str(value or "").strip()
require(minimum <= len(text) <= maximum and "@" not in text and "\\" not in text,
f"{field} must be safe text without an email address or local path.")
return text
def validate_common(record: object, expected_commit: str, expected_release_sha256: str) -> str:
require(isinstance(record, dict), "Acceptance record must be a JSON object.")
require(record.get("schemaVersion") == 1, "Unsupported Debian acceptance schema.")
system = record.get("system")
require(system in SYSTEMS, "Unknown Debian acceptance record system.")
require(record.get("evidenceId") == SYSTEMS[system]["evidenceId"],
f"{system} has the wrong evidenceId.")
require(record.get("releaseVersion") == VERSION,
f"releaseVersion must be {VERSION}.")
require(GIT_SHA.fullmatch(str(expected_commit)) is not None,
"Expected release commit must be a full lowercase Git SHA.")
require(SHA256.fullmatch(str(expected_release_sha256)) is not None,
"Expected release-record checksum must be a lowercase SHA-256 digest.")
require(record.get("releaseCommit") == expected_commit,
"releaseCommit does not match the approved candidate.")
require(record.get("releaseRecordSha256") == expected_release_sha256,
"releaseRecordSha256 does not match the retained release record.")
require(SHA256.fullmatch(str(record.get("archiveSha256", ""))) is not None,
"archiveSha256 must be a lowercase SHA-256 digest.")
origin = urlparse(str(record.get("environment", "")))
require(origin.scheme == "https" and origin.hostname and origin.path in ("", "/")
and not origin.query and not origin.fragment and origin.username is None
and origin.password is None,
"environment must be an HTTPS origin without credentials, path, query or fragment.")
host = safe_text(record.get("hostIdentifier"), "hostIdentifier")
images = record.get("images")
require(isinstance(images, dict) and set(images) == {"api", "worker"},
"images must contain exactly api and worker.")
for name in ("api", "worker"):
image = images[name]
require(isinstance(image, dict) and set(image) == {"reference", "id"},
f"images.{name} must contain exactly reference and id.")
require(image["reference"] == f"guestops-{name}:{expected_commit}",
f"images.{name}.reference must use the full approved commit.")
require(re.fullmatch(r"sha256:[0-9a-f]{64}", str(image["id"])) is not None,
f"images.{name}.id must be an immutable image ID.")
operator = safe_text(record.get("operator"), "operator")
reviewer = safe_text(record.get("reviewedBy"), "reviewedBy")
require(operator.casefold() != reviewer.casefold(),
"operator and reviewedBy must be different people.")
started = timestamp(record.get("startedAt"), "startedAt")
ended = timestamp(record.get("endedAt"), "endedAt")
reviewed = timestamp(record.get("reviewedAt"), "reviewedAt")
require(started <= ended <= reviewed, "Acceptance timestamps are out of order.")
scenarios = record.get("scenarios")
require(isinstance(scenarios, list), "scenarios must be a list.")
ids = [item.get("id") for item in scenarios if isinstance(item, dict)]
required = SYSTEMS[system]["scenarios"]
require(len(ids) == len(scenarios) and len(ids) == len(set(ids)) and set(ids) == required,
f"{system} requires its exact acceptance scenario set.")
for item in scenarios:
scenario_id = item["id"]
require(item.get("status") == "pass", f"Scenario {scenario_id} has not passed.")
evidence = item.get("evidence")
require(isinstance(evidence, list) and 1 <= len(evidence) <= 10 and all(
isinstance(value, str) and 3 <= len(value) <= 200 and "@" not in value
and "\\" not in value and not value.startswith("/") for value in evidence
), f"Scenario {scenario_id} requires safe opaque evidence references.")
require(record.get("unresolvedCriticalFindings") == 0,
"Acceptance cannot pass with unresolved critical findings.")
return host
def validate_host(record: object, expected_commit: str, expected_release_sha256: str) -> str:
host = validate_common(record, expected_commit, expected_release_sha256)
require(record.get("system") == "guestops-debian-host",
"First record must be guestops-debian-host.")
facts = record.get("hostFacts")
require(isinstance(facts, dict) and set(facts) == {
"debianMajor", "cpuCores", "memoryBytes", "freeDiskBytes", "publicTcpPorts",
}, "hostFacts must contain the exact reviewed host facts.")
require(isinstance(facts["debianMajor"], int) and facts["debianMajor"] >= 12,
"Debian 12 or newer is required.")
require(isinstance(facts["cpuCores"], int) and facts["cpuCores"] >= 4,
"At least four CPU cores are required.")
require(isinstance(facts["memoryBytes"], int) and facts["memoryBytes"] >= 7_500_000_000,
"At least 7.5 GB of memory is required.")
require(isinstance(facts["freeDiskBytes"], int) and facts["freeDiskBytes"] >= 8 * 1024**3,
"At least 8 GiB of free disk space is required.")
require(facts["publicTcpPorts"] == [80, 443],
"Only TCP ports 80 and 443 may be public.")
require(record.get("featureControls") == {
"googleSending": "disabled",
"faqLiveMode": "disabled",
"pmsWrites": "disabled",
"paymentCreation": "disabled",
}, "Unaccepted external writes and FAQ live mode must remain disabled.")
return host
def validate_persistence(record: object, expected_commit: str, expected_release_sha256: str) -> str:
host = validate_common(record, expected_commit, expected_release_sha256)
require(record.get("system") == "guestops-persistence",
"Second record must be guestops-persistence.")
require(record.get("drillCommand") == "python3 deploy/ops.py persistence-drill --confirm-restart",
"drillCommand must identify the confirmation-gated persistence drill.")
return host
def validate_pair(host_record: object, persistence_record: object,
expected_commit: str, expected_release_sha256: str) -> None:
host = validate_host(host_record, expected_commit, expected_release_sha256)
persistence_host = validate_persistence(
persistence_record, expected_commit, expected_release_sha256)
require(host == persistence_host, "Both records must identify the same host.")
for field in ("environment", "releaseVersion", "releaseCommit", "releaseRecordSha256",
"archiveSha256", "images"):
require(host_record.get(field) == persistence_record.get(field),
f"Both records must use the same {field}.")
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("host_record", type=Path)
parser.add_argument("persistence_record", type=Path)
parser.add_argument("--expected-commit", required=True)
parser.add_argument("--expected-release-record-sha256", required=True)
args = parser.parse_args()
host_record = json.loads(args.host_record.read_text(encoding="utf-8"))
persistence_record = json.loads(args.persistence_record.read_text(encoding="utf-8"))
validate_pair(host_record, persistence_record,
args.expected_commit, args.expected_release_record_sha256)
print("Debian-host and persistence acceptance records are structurally complete and passed. "
"This validates the records, not their restricted evidence.")
if __name__ == "__main__":
try:
main()
except (OSError, ValueError, json.JSONDecodeError) as error:
print(f"Debian acceptance records rejected: {error}", file=__import__("sys").stderr)
raise SystemExit(1)
-220
View File
@@ -1,220 +0,0 @@
#!/usr/bin/env python3
"""Validate the complete release-bound GuestOps Gate B acceptance bundle."""
from __future__ import annotations
import argparse
import hashlib
import json
import os
from pathlib import Path
import re
import sys
from urllib.parse import urlparse
DEPLOY = Path(__file__).resolve().parent
if str(DEPLOY) not in sys.path:
sys.path.insert(0, str(DEPLOY))
import automation_acceptance
import backup_restore_acceptance
import debian_acceptance
import desktop_acceptance
import google_acceptance
import identity_privacy_acceptance
import incident_exercise
import pilot_approval
import pilot_run
import verify_release
import verify_source_package
VERSION = "0.2.1"
SHA256 = re.compile(r"[0-9a-f]{64}")
RECORD_KEYS = {
"debian-host", "persistence", "backup-restore", "google-mailbox",
"automation", "identity-privacy", "inbox-usability", "capacity",
"incident-support", "pilot-findings", "pilot-approval",
}
def require(condition: bool, message: str) -> None:
if not condition:
raise ValueError(message)
def digest(path: Path) -> str:
with path.open("rb") as stream:
return hashlib.file_digest(stream, "sha256").hexdigest()
def load_json(path: Path) -> object:
require(path.is_file() and not path.is_symlink(), f"Required bundle file is missing or is a symlink: {path.name}")
return json.loads(path.read_text(encoding="utf-8"))
def release_binding(record: object, name: str, commit: str, release_sha: str) -> None:
require(isinstance(record, dict), f"{name} must be a JSON object.")
require(record.get("releaseCommit") == commit, f"{name} uses a different releaseCommit.")
require(record.get("releaseRecordSha256") == release_sha,
f"{name} uses a different releaseRecordSha256.")
def environment_origin(record: object, name: str) -> str:
require(isinstance(record, dict), f"{name} must be a JSON object.")
value = str(record.get("environment", "")).rstrip("/")
parsed = urlparse(value)
require(parsed.scheme == "https" and parsed.hostname and parsed.path in ("", "/"),
f"{name} has no valid HTTPS environment.")
return value
def validate_capacity(report: object, commit: str, release_sha: str) -> None:
require(isinstance(report, dict), "capacity must be a JSON object.")
require(report.get("schemaVersion") == 1 and report.get("kind") == "guestops-read-only-capacity",
"capacity has an unsupported schema or kind.")
require(report.get("releaseCommit") == commit, "capacity uses a different releaseCommit.")
require(report.get("releaseRecordSha256") == release_sha,
"capacity uses a different releaseRecordSha256.")
require(report.get("paths") == ["/health/ready", "/api/hotel", "/api/conversations/page"],
"capacity must use the exact read-only path set.")
for field in ("concurrency", "requests", "successes", "failures"):
require(isinstance(report.get(field), int) and not isinstance(report.get(field), bool),
f"capacity.{field} must be an integer.")
require(1 <= report["concurrency"] <= 20 and report["requests"] > 0,
"capacity has invalid concurrency or request count.")
require(report["successes"] + report["failures"] == report["requests"],
"capacity success and failure counts do not match requests.")
expected_error = round(report["failures"] / report["requests"], 6)
require(report.get("errorRate") == expected_error, "capacity.errorRate does not match its counts.")
latency = report.get("latencyMs")
require(isinstance(latency, dict) and set(latency) == {"median", "p95", "maximum"}
and all(isinstance(value, (int, float)) and not isinstance(value, bool) and value >= 0
for value in latency.values()), "capacity.latencyMs is invalid.")
require(latency["median"] <= latency["p95"] <= latency["maximum"],
"capacity latency percentiles are out of order.")
def validate_bundle(
source_archive: Path,
source_record_path: Path,
release_record_path: Path,
records: dict[str, tuple[Path, object]],
host_metrics_path: Path,
) -> dict[str, object]:
require(set(records) == RECORD_KEYS, "Gate B bundle requires the exact record set.")
release_record = load_json(release_record_path)
require(isinstance(release_record, dict), "release-record must be a JSON object.")
commit = str(release_record.get("commit", ""))
version = str(release_record.get("version", ""))
require(version == VERSION, f"Gate B bundle version must be {VERSION}.")
release_result = verify_release.validate(source_archive, release_record_path, commit, version)
source_result = verify_source_package.validate(source_archive, source_record_path, commit, version)
release_sha = str(release_result["releaseRecord"]["sha256"])
require(source_result["artifact"]["sha256"] == release_result["archive"]["sha256"],
"Source and release records identify different archives.")
values = {name: value for name, (_, value) in records.items()}
for name, record in values.items():
release_binding(record, name, commit, release_sha)
debian_acceptance.validate_pair(values["debian-host"], values["persistence"], commit, release_sha)
backup_restore_acceptance.validate(values["backup-restore"], commit, release_sha)
google_acceptance.validate(values["google-mailbox"])
automation_acceptance.validate(values["automation"])
identity_privacy_acceptance.validate(values["identity-privacy"])
desktop_acceptance.validate(values["inbox-usability"])
validate_capacity(values["capacity"], commit, release_sha)
incident_exercise.validate(values["incident-support"])
pilot_run.validate(values["pilot-findings"])
pilot_approval.validate(values["pilot-approval"])
environment_names = {
"debian-host", "persistence", "backup-restore", "google-mailbox",
"automation", "identity-privacy", "inbox-usability", "incident-support",
"pilot-findings",
}
origins = {environment_origin(values[name], name) for name in environment_names}
require(len(origins) == 1, "Gate B records use different environments.")
origin = next(iter(origins))
require(values["capacity"].get("originHost") == urlparse(origin).hostname,
"capacity originHost does not match the accepted environment.")
archive_sha = str(release_result["archive"]["sha256"])
for name in ("debian-host", "persistence", "backup-restore"):
require(values[name].get("archiveSha256") == archive_sha,
f"{name} uses a different archiveSha256.")
approved_images = release_record["images"]
for name in ("debian-host", "persistence", "backup-restore"):
images = values[name].get("images", {})
require(images.get("api") == approved_images["api"] and images.get("worker") == approved_images["worker"],
f"{name} uses different API or worker image identities.")
approval_capacity = values["pilot-approval"]["capacity"]
capacity = values["capacity"]
require(approval_capacity["reportSha256"] == digest(records["capacity"][0]),
"Pilot approval capacity report checksum does not match the retained report.")
require(approval_capacity["hostMetricsSha256"] == digest(host_metrics_path),
"Pilot approval host metrics checksum does not match the retained file.")
require(approval_capacity["observedConcurrency"] == capacity["concurrency"],
"Pilot approval concurrency does not match the capacity report.")
require(approval_capacity["observedP95Ms"] == capacity["latencyMs"]["p95"],
"Pilot approval p95 does not match the capacity report.")
require(approval_capacity["observedErrorRate"] == capacity["errorRate"],
"Pilot approval error rate does not match the capacity report.")
require(values["pilot-approval"]["pilot"]["recordSha256"] == digest(records["pilot-findings"][0]),
"Pilot approval checksum does not match the retained pilot run record.")
summary_records = {name: digest(path) for name, (path, _) in sorted(records.items())}
return {
"archiveSha256": archive_sha,
"environmentHost": urlparse(origin).hostname,
"releaseCommit": commit,
"releaseRecordSha256": release_sha,
"records": summary_records,
"schemaVersion": 1,
"sourceRecordSha256": source_result["sourceRecord"]["sha256"],
"validated": True,
"version": version,
}
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--source-archive", required=True, type=Path)
parser.add_argument("--source-record", required=True, type=Path)
parser.add_argument("--release-record", required=True, type=Path)
parser.add_argument("--host-metrics", required=True, type=Path)
parser.add_argument("--output", required=True, type=Path)
for name in sorted(RECORD_KEYS):
parser.add_argument("--" + name, required=True, type=Path)
args = parser.parse_args()
try:
require(args.host_metrics.is_file() and not args.host_metrics.is_symlink(),
"Host metrics file is missing or is a symlink.")
require(not args.output.exists() and args.output.parent.is_dir(),
"Output must be a new file in an existing restricted directory.")
records = {}
for name in RECORD_KEYS:
path = getattr(args, name.replace("-", "_"))
records[name] = (path, load_json(path))
summary = validate_bundle(
args.source_archive,
args.source_record,
args.release_record,
records,
args.host_metrics,
)
descriptor = os.open(args.output, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(descriptor, "w", encoding="utf-8") as output:
output.write(json.dumps(summary, indent=2, sort_keys=True) + "\n")
print("Gate B bundle is structurally complete, consistently release-bound and approved. "
"This validates records and checksums, not the underlying restricted evidence.")
except (OSError, ValueError, json.JSONDecodeError) as error:
print(f"Gate B bundle rejected: {error}", file=sys.stderr)
raise SystemExit(1)
if __name__ == "__main__":
main()
+1 -1
View File
@@ -2,7 +2,7 @@
"acceptedAt": "2026-01-01T00:00:00Z",
"acceptedBy": "REPLACE WITH APPROVER",
"endedAt": "2026-01-01T00:00:00Z",
"environment": "https://sandbox-guestops.futuresens.co.uk",
"environment": "https://guestops-sandbox.futuresens.co.uk",
"mailboxLabel": "sandbox mailbox A",
"operator": "REPLACE WITH OPERATOR",
"releaseCommit": "0000000000000000000000000000000000000000",
-2
View File
@@ -1,2 +0,0 @@
const guestops = db.getSiblingDB('guestops');
guestops.createUser({user: 'guestops', pwd: process.env.MONGO_APP_PASSWORD, roles: [{role: 'readWrite', db: 'guestops'}]});
-189
View File
@@ -1,189 +0,0 @@
#!/usr/bin/env python3
"""Write non-sensitive GuestOps host status for a read-only monitoring agent."""
from __future__ import annotations
import argparse
import datetime as dt
import json
import os
from pathlib import Path
import re
import shutil
import socket
import ssl
import stat
import subprocess
import tempfile
import urllib.request
from urllib.parse import urlparse
BACKUP_NAME = re.compile(r"guestops-[0-9]{8}T[0-9]{6}Z\.tar\.gpg")
MARKER_NAME = re.compile(r"guestops-[0-9]{8}T[0-9]{6}Z\.tar\.gpg\.transferred\.json")
AUTH = ('const c=new Mongo("mongodb://127.0.0.1");'
'c.getDB("admin").auth(process.env.MONGO_INITDB_ROOT_USERNAME,'
'process.env.MONGO_INITDB_ROOT_PASSWORD);')
HEARTBEAT = ('const x=c.getDB("guestops").workerheartbeat.findOne({_id:"worker"});'
'print(JSON.stringify(x&&x.At?x.At:null));')
def require(condition: bool, message: str) -> None:
if not condition:
raise RuntimeError(message)
def run(args: list[str], root: Path, timeout: int = 30) -> bytes:
result = subprocess.run(args, cwd=root, stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout)
require(result.returncode == 0, f"{Path(args[0]).name} probe failed.")
return result.stdout
def utc_now() -> dt.datetime:
return dt.datetime.now(dt.timezone.utc)
def age_seconds(path: Path, pattern: re.Pattern[str], now: dt.datetime) -> int | None:
if not path.is_dir() or path.is_symlink():
return None
times = [item.stat().st_mtime for item in path.iterdir()
if item.is_file() and not item.is_symlink() and pattern.fullmatch(item.name)]
return max(0, int(now.timestamp() - max(times))) if times else None
def https_status(origin: str, now: dt.datetime) -> dict[str, object]:
parsed = urlparse(origin)
require(parsed.scheme == "https" and parsed.hostname and parsed.port in (None, 443)
and parsed.path in ("", "/") and not parsed.query and not parsed.fragment
and parsed.username is None and parsed.password is None,
"GUESTOPS_ORIGIN must be an HTTPS origin without credentials or a path.")
context = ssl.create_default_context()
with socket.create_connection((parsed.hostname, 443), timeout=10) as connection:
with context.wrap_socket(connection, server_hostname=parsed.hostname) as secured:
certificate = secured.getpeercert()
expires = dt.datetime.strptime(certificate["notAfter"], "%b %d %H:%M:%S %Y %Z").replace(
tzinfo=dt.timezone.utc)
with urllib.request.urlopen(origin.rstrip("/") + "/health/ready", timeout=15,
context=context) as response:
ready = response.status == 200 and json.load(response) == {"status": "ready"}
return {"ready": ready, "certificateDaysRemaining": max(0, int((expires - now).total_seconds() // 86400))}
def parse_compose(value: bytes) -> dict[str, dict[str, str]]:
text = value.decode("utf-8", "strict").strip()
if not text:
return {}
try:
parsed = json.loads(text)
rows = parsed if isinstance(parsed, list) else [parsed]
except json.JSONDecodeError:
rows = [json.loads(line) for line in text.splitlines() if line.strip()]
result = {}
for row in rows:
if not isinstance(row, dict):
continue
service = str(row.get("Service", ""))
if service in {"api", "worker", "mongo"}:
result[service] = {
"state": str(row.get("State", "unknown")).lower(),
"health": str(row.get("Health", "none") or "none").lower(),
}
return result
def worker_heartbeat_age(root: Path, now: dt.datetime) -> int | None:
output = run(["docker", "compose", "exec", "-T", "mongo", "mongosh", "--quiet",
"--nodb", "--eval", AUTH + HEARTBEAT], root).decode("utf-8", "strict").strip()
value = json.loads(output)
if value is None:
return None
require(isinstance(value, str), "Worker heartbeat response was invalid.")
parsed = dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
require(parsed.tzinfo is not None, "Worker heartbeat must contain a timezone.")
return max(0, int((now - parsed.astimezone(dt.timezone.utc)).total_seconds()))
def build_status(root: Path, backup_directory: Path, origin: str,
now: dt.datetime | None = None) -> tuple[dict[str, object], list[str]]:
moment = now or utc_now()
errors: list[str] = []
try:
https = https_status(origin, moment)
except Exception:
https = {"ready": False, "certificateDaysRemaining": None}
errors.append("https-probe-failed")
try:
containers = parse_compose(run(["docker", "compose", "ps", "--format", "json"], root))
if set(containers) != {"api", "worker", "mongo"}:
errors.append("container-set-incomplete")
except Exception:
containers = {}
errors.append("container-probe-failed")
try:
heartbeat_age = worker_heartbeat_age(root, moment)
if heartbeat_age is None:
errors.append("worker-heartbeat-missing")
except Exception:
heartbeat_age = None
errors.append("worker-heartbeat-probe-failed")
disk = shutil.disk_usage(root)
status = {
"schemaVersion": 1,
"generatedAt": moment.isoformat().replace("+00:00", "Z"),
"https": https,
"containers": containers,
"workerHeartbeatAgeSeconds": heartbeat_age,
"backupAgeSeconds": age_seconds(backup_directory, BACKUP_NAME, moment),
"verifiedTransferAgeSeconds": age_seconds(backup_directory, MARKER_NAME, moment),
"diskFreePercent": round(disk.free * 100 / disk.total, 2),
"persistentJournal": Path("/var/log/journal").is_dir(),
"errors": sorted(set(errors)),
}
return status, errors
def write_status(path: Path, status: dict[str, object]) -> None:
require(path.is_absolute() and not path.is_symlink(),
"Status output must be an absolute, non-symlink path.")
parent = path.parent.resolve()
require(parent.is_dir() and not path.parent.is_symlink(), "Status output directory must exist.")
fd, temporary = tempfile.mkstemp(prefix=path.name + ".", dir=parent)
try:
os.fchmod(fd, 0o644)
with os.fdopen(fd, "w", encoding="utf-8") as stream:
json.dump(status, stream, sort_keys=True, separators=(",", ":"))
stream.write("\n")
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary, path)
finally:
try:
os.unlink(temporary)
except FileNotFoundError:
pass
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--root", type=Path, default=Path(__file__).resolve().parents[1])
parser.add_argument("--backup-directory", type=Path, default=Path("/var/backups/guestops"))
parser.add_argument("--origin", default="https://sandbox-guestops.futuresens.co.uk")
parser.add_argument("--output", type=Path, default=Path("/run/guestops-monitor/status.json"))
args = parser.parse_args()
root = args.root.resolve()
require(root.is_dir(), "GuestOps root must exist.")
backup_directory = args.backup_directory.resolve()
status, errors = build_status(root, backup_directory, args.origin)
write_status(args.output, status)
print("GuestOps monitoring status updated." if not errors
else "GuestOps monitoring status updated with failed probes.")
raise SystemExit(1 if errors else 0)
if __name__ == "__main__":
try:
main()
except (OSError, RuntimeError, ValueError, subprocess.SubprocessError, json.JSONDecodeError) as error:
print(f"GuestOps monitoring probe failed: {error}", file=__import__("sys").stderr)
raise SystemExit(1)
-31
View File
@@ -1,31 +0,0 @@
# Install within the existing Nginx configuration. Obtain the certificate first.
server {
listen 80;
server_name sandbox-guestops.futuresens.co.uk;
location /.well-known/acme-challenge/ { root /var/www/letsencrypt; }
location / { return 301 https://$host$request_uri; }
}
server {
listen 443 ssl;
server_name sandbox-guestops.futuresens.co.uk;
ssl_certificate /etc/letsencrypt/live/sandbox-guestops.futuresens.co.uk/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/sandbox-guestops.futuresens.co.uk/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
add_header Strict-Transport-Security "max-age=31536000" always;
client_max_body_size 128k;
# OAuth authorization codes and state must not appear in access logs.
location = /api/integrations/google/callback {
access_log off;
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto https;
}
location / {
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_read_timeout 60s;
}
}
-344
View File
@@ -1,344 +0,0 @@
#!/usr/bin/env python3
"""GuestOps dedicated-Compose operations. Never prints credentials or provider data."""
import argparse
import contextlib
import hashlib
import json
import os
from pathlib import Path
import re
import shutil
import stat
import subprocess
import sys
import tarfile
import tempfile
import time
import urllib.request
import uuid
ROOT = Path(__file__).resolve().parents[1]
FILES = {"mongo.archive.gz", "keys.tar.gz", "configuration.json", "manifest.json"}
LIMIT = 8 * 1024**3
def require(condition, message):
if not condition:
raise RuntimeError(message)
def run(args, *, output=None, input_file=None, timeout=900):
# Provider output may contain secrets; errors identify only the program.
result = subprocess.run(args, cwd=ROOT, stdin=input_file or subprocess.DEVNULL,
stdout=output or subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout)
require(result.returncode == 0, f"{args[0]} step failed. Check the private operator environment; no command output was logged.")
return result.stdout or b""
def compose(*args, **kwargs):
return run(["docker", "compose", *args], **kwargs)
def digest(path):
with path.open("rb") as stream:
return hashlib.file_digest(stream, "sha256").hexdigest()
def image_id(image):
require(not image.startswith("-"), "Invalid image reference.")
return run(["docker", "image", "inspect", "--format", "{{.Id}}", image]).decode().strip()
def provider_configured(config, target):
section = "Pms" if target == "/run/guestops/pms.json" else "Payments"
# Only the exact shipped empty template is public; unknown settings fail closed.
return config not in ({}, {section: {"Hotels": {}}})
def persistence_layout(config):
"""Return the resolved named volumes required to survive recreation."""
services = config["services"]
declared = config.get("volumes", {})
def volume_for(service, target):
matches = [v for v in services[service].get("volumes", []) if v["target"] == target]
require(len(matches) == 1 and matches[0]["type"] == "volume" and matches[0].get("source"),
f"{service} requires one named persistent volume at {target}.")
source = matches[0]["source"]
require(source in declared, f"{service} volume {source} must be declared at the top level.")
return declared[source].get("name") or source
api_keys = volume_for("api", "/var/lib/guestops/keys")
worker_keys = volume_for("worker", "/var/lib/guestops/keys")
require(api_keys == worker_keys, "API and worker key volumes differ.")
mongo_data = volume_for("mongo", "/data/db")
require(api_keys != mongo_data, "Database and key data require separate named volumes.")
return {"keys": api_keys, "database": mongo_data}
AUTH = 'const c=new Mongo("mongodb://127.0.0.1"); c.getDB("admin").auth(process.env.MONGO_INITDB_ROOT_USERNAME,process.env.MONGO_INITDB_ROOT_PASSWORD);'
INVENTORY = 'const d=c.getDB("guestops"); print(JSON.stringify({bytes:d.stats().storageSize+d.stats().indexSize,collections:Object.fromEntries(d.getCollectionNames().filter(n=>!n.startsWith("system.")).sort().map(n=>[n,{count:d.getCollection(n).countDocuments({}),indexes:d.getCollection(n).getIndexes().map(i=>{delete i.ns;return i;}).sort((a,b)=>a.name.localeCompare(b.name))}]))}));'
def mongo(script):
return compose("exec", "-T", "mongo", "mongosh", "--quiet", "--nodb", "--eval", AUTH + script)
def configuration():
config = json.loads(compose("config", "--format", "json"))
services = config["services"]
require(set(services) == {"api", "worker", "mongo"}, "This tool supports the dedicated three-service GuestOps Compose deployment only.")
require(not services["mongo"].get("ports"), "MongoDB must not have published ports.")
ports = services["api"].get("ports", [])
require(len(ports) == 1 and ports[0].get("host_ip") == "127.0.0.1" and int(ports[0]["target"]) == 8080, "API must publish only port 8080 on loopback.")
require(not services["worker"].get("ports"), "Worker must not publish ports.")
for name in ("api", "worker"):
env = services[name]["environment"]
require(env.get("Mongo__Database") == "guestops" and "@mongo:27017/guestops?" in env.get("Mongo__ConnectionString", ""), "Backup supports only the dedicated Compose guestops database.")
require(str(env.get("Preview", "false")).lower() != "true", "Production preview is forbidden.")
require(env.get("Keys__Path") == "/var/lib/guestops/keys", "Unexpected key directory.")
require(services["api"]["environment"].get("ASPNETCORE_ENVIRONMENT") == "Production", "API must use Production environment.")
require(re.fullmatch(r"https://[A-Za-z0-9.-]+", services["api"]["environment"].get("PublicUrl", "")), "PublicUrl must be an HTTPS hostname without a path.")
persistence_layout(config)
return config
def wait_for(action, message, attempts=60):
for attempt in range(attempts):
try:
action()
return
except Exception:
if attempt == attempts - 1:
raise RuntimeError(message)
time.sleep(1)
def volume_identity(name):
require(not name.startswith("-"), "Invalid volume name.")
details = json.loads(run(["docker", "volume", "inspect", name]))
require(len(details) == 1 and details[0].get("Name") == name, "Named volume inspection failed.")
return {"name": name, "driver": details[0].get("Driver"), "scope": details[0].get("Scope")}
def readiness():
with urllib.request.urlopen("http://127.0.0.1:8080/health/ready", timeout=10) as response:
require(json.load(response) == {"status": "ready"}, "Loopback readiness failed.")
def persistence_drill(args):
require(args.confirm_restart, "Persistence drill requires --confirm-restart: all services will be restarted and application containers recreated.")
config = configuration()
layout = persistence_layout(config)
before_volumes = {purpose: volume_identity(name) for purpose, name in layout.items()}
before_inventory = json.loads(mongo(INVENTORY))
probe = compose("run", "--rm", "--no-deps", "-T", "-e", "Logging__LogLevel__Default=None", "api", "--backup-probe").decode().strip()
require(re.fullmatch(r"[A-Za-z0-9_-]{20,4096}", probe), "Key probe did not return a valid protected value.")
# Restart the database first, then its clients, so recovery is exercised in a known order.
compose("restart", "-t", "150", "mongo")
wait_for(lambda: mongo('const r=c.getDB("admin").runCommand({ping:1});if(!r.ok)quit(1);'), "MongoDB did not recover after restart.")
compose("restart", "-t", "150", "api", "worker")
wait_for(readiness, "API readiness did not recover after restart.")
# Recreate stateless containers as an image upgrade would, without rebuilding or changing data volumes.
compose("up", "-d", "--no-build", "--force-recreate", "api", "worker")
wait_for(readiness, "API readiness did not recover after container recreation.")
compose("run", "--rm", "--no-deps", "-T", "-e", "Logging__LogLevel__Default=None", "-e", "BACKUP_PROBE=" + probe, "api", "--verify-backup-probe")
after_volumes = {purpose: volume_identity(name) for purpose, name in layout.items()}
after_inventory = json.loads(mongo(INVENTORY))
require(after_volumes == before_volumes, "A persistent volume identity changed during the drill.")
require(after_inventory["collections"] == before_inventory["collections"], "Database collection counts or indexes changed during the drill.")
print("Persistence drill passed: named volumes, database counts/indexes, data-protection keys and readiness survived restart and container recreation.")
def preflight(args):
config = configuration()
env_file = ROOT / ".env"
require(env_file.is_file() and not env_file.is_symlink(), "Create a private .env file before deployment.")
require(stat.S_IMODE(env_file.stat().st_mode) & 0o077 == 0, ".env must not be accessible to group or other users.")
require(shutil.disk_usage(ROOT).free >= 8 * 1024**3, "Keep at least 8 GiB free before deployment; allow extra room for backups.")
for name, service in config["services"].items():
image_id(service["image"])
for volume in service.get("volumes", []):
if volume["type"] == "bind":
require(Path(volume["source"]).exists(), f"A required {name} bind mount is missing.")
if volume["target"] in ("/run/guestops/pms.json", "/run/guestops/payments.json"):
provider = Path(volume["source"])
if provider_configured(json.loads(provider.read_text()), volume["target"]):
require(stat.S_IMODE(provider.stat().st_mode) & 0o077 == 0, "Configured provider files must not be accessible to group or other users.")
if not args.offline:
url = config["services"]["api"]["environment"]["PublicUrl"]
require(re.fullmatch(r"https://[A-Za-z0-9.-]+", url), "PublicUrl must be an HTTPS hostname without a path.")
with urllib.request.urlopen(url + "/health/ready", timeout=15) as response:
require(response.url == url + "/health/ready" and json.load(response) == {"status": "ready"}, "Public HTTPS readiness failed.")
print("Preflight passed: private database, loopback API, production settings, images, mounts and disk headroom" + ("; HTTPS not checked." if args.offline else "; public HTTPS and database readiness checked."))
@contextlib.contextmanager
def maintenance_lock():
import fcntl
with (ROOT / ".guestops-maintenance.lock").open("a") as lock:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
yield
def backup(args):
require(args.confirm_maintenance, "Backup requires --confirm-maintenance: API and workers will briefly stop.")
require(re.fullmatch(r"[A-Fa-f0-9]{40}", args.recipient), "Use a verified full 40-character GPG recipient fingerprint.")
run(["gpg", "--batch", "--list-keys", args.recipient])
destination = Path(args.output).resolve()
require(not destination.exists(), "Backup output already exists; choose a new filename.")
require(destination.parent.is_dir(), "Create the private backup directory first.")
require(stat.S_IMODE(destination.parent.stat().st_mode) & 0o077 == 0, "Backup directory must be private (mode 700).")
config = configuration()
runtime = {}
for service in ("api", "worker", "mongo"):
cid = compose("ps", "--status", "running", "-q", service).decode().strip()
require(re.fullmatch(r"[a-f0-9]{12,64}", cid), f"Exactly one running {service} container is required.")
runtime[service] = json.loads(run(["docker", "inspect", cid]))[0]
require(shutil.disk_usage(destination.parent).free >= 3 * json.loads(mongo(INVENTORY))["bytes"] + 1024**3, "Insufficient free space for a consistent encrypted backup.")
partial = destination.with_name(destination.name + ".partial-" + uuid.uuid4().hex)
with maintenance_lock(), tempfile.TemporaryDirectory(prefix="guestops-backup-", dir=destination.parent) as folder:
folder = Path(folder)
stopped = False
ttl = None
try:
# Set before stopping so partial stop failures also attempt recovery.
stopped = True
compose("stop", "-t", "150", "api", "worker")
ttl = json.loads(mongo('print(JSON.stringify(c.getDB("admin").runCommand({getParameter:1,ttlMonitorEnabled:1}).ttlMonitorEnabled));'))
require(isinstance(ttl, bool), "Cannot determine MongoDB TTL monitor state.")
mongo('const r=c.getDB("admin").runCommand({setParameter:1,ttlMonitorEnabled:false});if(!r.ok)quit(1);')
inventory = json.loads(mongo(INVENTORY))
dump = 'set -eu; case "$MONGO_INITDB_ROOT_PASSWORD" in ""|*[!0-9a-fA-F]*) exit 1;; esac; umask 077; cfg=$(mktemp); trap \'rm -f "$cfg"\' EXIT; printf \'password: "%s"\\n\' "$MONGO_INITDB_ROOT_PASSWORD" > "$cfg"; mongodump --config "$cfg" --username "$MONGO_INITDB_ROOT_USERNAME" --authenticationDatabase admin --db guestops --archive --gzip'
with (folder / "mongo.archive.gz").open("wb") as output:
compose("exec", "-T", "mongo", "sh", "-c", dump, output=output)
probe = compose("run", "--rm", "--no-deps", "-T", "-e", "Logging__LogLevel__Default=None", "api", "--backup-probe").decode().strip()
require(re.fullmatch(r"[A-Za-z0-9_-]{20,4096}", probe), "Key probe did not return a valid protected value.")
with (folder / "keys.tar.gz").open("wb") as output:
compose("run", "--rm", "--no-deps", "-T", "--entrypoint", "tar", "api", "-C", "/var/lib/guestops/keys", "-czf", "-", ".", output=output)
mounted = {}
for volume in config["services"]["api"].get("volumes", []):
if volume["type"] == "bind" and volume["target"] in ("/run/guestops/pms.json", "/run/guestops/payments.json"):
mounted[volume["target"]] = Path(volume["source"]).read_text()
(folder / "configuration.json").write_text(json.dumps({"compose": config, "runtime": runtime, "providerFiles": mounted}))
manifest = {"format": 1, "createdAt": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "inventory": inventory, "probe": probe, "apiImageId": runtime["api"]["Image"], "mongoImageId": runtime["mongo"]["Image"], "sha256": {name: digest(folder / name) for name in FILES - {"manifest.json"}}}
(folder / "manifest.json").write_text(json.dumps(manifest))
finally:
try:
if ttl is not None:
mongo('const r=c.getDB("admin").runCommand({setParameter:1,ttlMonitorEnabled:' + str(ttl).lower() + '});if(!r.ok)quit(1);')
finally:
if stopped:
compose("start", "api", "worker")
try:
with tarfile.open(folder / "bundle.tar", "w") as archive:
for name in sorted(FILES):
archive.add(folder / name, arcname=name, recursive=False)
run(["gpg", "--batch", "--trust-model", "always", "--recipient", args.recipient, "--output", str(partial), "--encrypt", str(folder / "bundle.tar")])
os.link(partial, destination) # Atomic publication, never overwrite an existing backup.
finally:
partial.unlink(missing_ok=True)
print("Encrypted backup complete. Copy it off-server and perform a restore drill.")
def scheduled_backup(args):
require(args.confirm_maintenance, "Scheduled backup requires --confirm-maintenance because API and workers will briefly stop.")
recipient = os.environ.get("BACKUP_RECIPIENT", "")
directory_value = os.environ.get("BACKUP_DIRECTORY", "")
require(re.fullmatch(r"[A-Fa-f0-9]{40}", recipient), "BACKUP_RECIPIENT must be a verified full GPG fingerprint.")
require(directory_value != "", "BACKUP_DIRECTORY must name the private off-checkout staging directory.")
requested_directory = Path(directory_value)
require(requested_directory.is_absolute() and not requested_directory.is_symlink(), "BACKUP_DIRECTORY must be an absolute, non-symlink path.")
directory = requested_directory.resolve()
require(directory.is_dir(), "BACKUP_DIRECTORY must be an existing real directory.")
require(ROOT not in directory.parents and directory != ROOT, "Scheduled backups must be staged outside the application checkout.")
require(stat.S_IMODE(directory.stat().st_mode) & 0o077 == 0, "BACKUP_DIRECTORY must be private (mode 700).")
timestamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime())
destination = directory / f"guestops-{timestamp}.tar.gpg"
backup(argparse.Namespace(confirm_maintenance=True, recipient=recipient, output=str(destination)))
print(f"Scheduled backup staged as {destination.name}. Off-host transfer and alert verification remain required.")
def unpack(bundle, folder):
with tarfile.open(bundle, "r:") as archive:
members = archive.getmembers()
require(len(members) == len(FILES) and {m.name for m in members} == FILES, "Unexpected backup members.")
require(all(m.isfile() and 0 <= m.size <= LIMIT for m in members) and sum(m.size for m in members) <= LIMIT, "Invalid or oversized backup members.")
for member in members:
with archive.extractfile(member) as source, (folder / member.name).open("xb") as output:
shutil.copyfileobj(source, output)
manifest = json.loads((folder / "manifest.json").read_text())
require(manifest.get("format") == 1 and set(manifest.get("sha256", {})) == FILES - {"manifest.json"}, "Unsupported backup format.")
for name, expected in manifest["sha256"].items():
require(digest(folder / name) == expected, "Backup checksum verification failed.")
return manifest
def restore_drill(args):
backup_file = Path(args.backup).resolve()
require(backup_file.is_file(), "Backup file is missing.")
with tempfile.TemporaryDirectory(prefix="guestops-restore-", dir=ROOT) as temp:
folder = Path(temp)
run(["gpg", "--batch", "--max-output", str(LIMIT), "--output", str(folder / "bundle.tar"), "--decrypt", str(backup_file)])
require((folder / "bundle.tar").stat().st_size <= LIMIT, "Decrypted bundle exceeds the 8 GiB pilot limit.")
manifest = unpack(folder / "bundle.tar", folder)
require(image_id(args.api_image) == manifest["apiImageId"] and image_id(args.mongo_image) == manifest["mongoImageId"], "Load the exact trusted API and MongoDB images used for this backup.")
require(shutil.disk_usage(ROOT).free > 3 * manifest["inventory"]["bytes"] + 1024**3, "Insufficient restore drill space.")
keys = folder / "keys"
keys.mkdir(mode=0o700)
with tarfile.open(folder / "keys.tar.gz", "r:gz") as archive:
total = 0
for member in archive:
if member.name in (".", "./") and member.isdir():
continue
name = member.name.removeprefix("./")
total += member.size
require(member.isfile() and re.fullmatch(r"[A-Za-z0-9_-]+\.xml", name) and member.size < 1024**2 and total < 16 * 1024**2, "Invalid key archive.")
with archive.extractfile(member) as source, (keys / name).open("xb") as output:
shutil.copyfileobj(source, output)
run(["docker", "run", "--rm", "--pull", "never", "--network", "none", "--user", "0:0", "--read-only", "--mount", f"type=bind,src={keys},dst=/var/lib/guestops/keys,readonly", "-e", "Logging__LogLevel__Default=None", "-e", "BACKUP_PROBE=" + manifest["probe"], args.api_image, "--verify-backup-probe"])
cid = run(["docker", "run", "-d", "--pull", "never", "--network", "none", "--memory", "1g", "--label", "guestops.restore-drill=true", args.mongo_image, "--bind_ip", "127.0.0.1", "--setParameter", "ttlMonitorEnabled=false"]).decode().strip()
require(re.fullmatch(r"[a-f0-9]{64}", cid), "Unexpected restore container identifier.")
try:
for attempt in range(30):
try:
run(["docker", "exec", cid, "mongosh", "--quiet", "--eval", "db.adminCommand({ping:1})"], timeout=10)
break
except RuntimeError:
if attempt == 29:
raise
time.sleep(1)
with (folder / "mongo.archive.gz").open("rb") as source:
run(["docker", "exec", "-i", cid, "mongorestore", "--archive", "--gzip", "--nsInclude", "guestops.*"], input_file=source)
restored = json.loads(run(["docker", "exec", cid, "mongosh", "--quiet", "--nodb", "--eval", 'const c=new Mongo("mongodb://127.0.0.1");' + INVENTORY]))
require(restored["collections"] == manifest["inventory"]["collections"], "Restored collection counts or indexes differ.")
finally:
run(["docker", "rm", "-f", "-v", cid]) # Only the exact container created above and its anonymous volumes.
print("Restore drill passed: collection counts, indexes and actual key decryption verified in isolated containers. Production was not restored or modified.")
def main():
require(os.name == "posix", "Run deployment operations on Linux.")
os.umask(0o077)
parser = argparse.ArgumentParser(description=__doc__)
subs = parser.add_subparsers(dest="command", required=True)
check = subs.add_parser("preflight"); check.add_argument("--offline", action="store_true")
persistence = subs.add_parser("persistence-drill"); persistence.add_argument("--confirm-restart", action="store_true")
save = subs.add_parser("backup"); save.add_argument("--recipient", required=True); save.add_argument("--output", required=True); save.add_argument("--confirm-maintenance", action="store_true")
scheduled = subs.add_parser("scheduled-backup"); scheduled.add_argument("--confirm-maintenance", action="store_true")
drill = subs.add_parser("restore-drill"); drill.add_argument("backup"); drill.add_argument("--api-image", required=True); drill.add_argument("--mongo-image", default="mongo:8.0")
args = parser.parse_args()
{"preflight": preflight, "persistence-drill": persistence_drill, "backup": backup, "scheduled-backup": scheduled_backup, "restore-drill": restore_drill}[args.command](args)
if __name__ == "__main__":
try:
main()
except Exception as error:
# Never include subprocess output, config values or parsed guest data.
print(str(error) if isinstance(error, RuntimeError) else "Operation failed (" + type(error).__name__ + "). No sensitive details were logged.", file=sys.stderr)
sys.exit(1)
+8 -28
View File
@@ -5,7 +5,6 @@ from __future__ import annotations
import argparse
import gzip
import hashlib
import io
import json
from pathlib import Path
@@ -44,11 +43,7 @@ def versions(repository: Path, commit: str) -> tuple[str, str]:
return dotnet_version, web_version
def sha256_bytes(value: bytes) -> str:
return hashlib.sha256(value).hexdigest()
def build_package(repository: Path, ref: str, output_directory: Path) -> tuple[Path, Path, dict[str, object]]:
def build_package(repository: Path, ref: str, output_directory: Path) -> tuple[Path, dict[str, str]]:
repository = repository.resolve()
output_directory = output_directory.resolve()
if not (repository / ".git").exists():
@@ -66,7 +61,7 @@ def build_package(repository: Path, ref: str, output_directory: Path) -> tuple[P
if SEMVER.fullmatch(dotnet_version) is None:
raise ValueError("The committed version must use MAJOR.MINOR.PATCH.")
prefix = f"GuestOps-{dotnet_version}/"
prefix = f"guestops_{dotnet_version}/"
tar_bytes = bytes(
git(
repository,
@@ -82,28 +77,13 @@ def build_package(repository: Path, ref: str, output_directory: Path) -> tuple[P
stream.write(tar_bytes)
archive_bytes = compressed.getvalue()
stem = f"GuestOps-{dotnet_version}-{commit[:12]}"
archive = output_directory / f"{stem}.tar.gz"
record_path = output_directory / f"{stem}.source.json"
if archive.exists() or record_path.exists():
raise ValueError("The output package or source record already exists; releases are not overwritten.")
archive = output_directory / f"guestops_{dotnet_version}.tar.gz"
if archive.exists():
raise ValueError("The output package already exists; published versions are not overwritten.")
output_directory.mkdir(parents=True, exist_ok=True)
archive.write_bytes(archive_bytes)
record: dict[str, object] = {
"artifact": {
"name": archive.name,
"sha256": sha256_bytes(archive_bytes),
"size": len(archive_bytes),
},
"commit": commit,
"schemaVersion": 1,
"version": dotnet_version,
}
record_path.write_text(
json.dumps(record, indent=2, sort_keys=True) + "\n", encoding="utf-8"
)
return archive, record_path, record
return archive, {"commit": commit, "version": dotnet_version}
def main() -> None:
@@ -113,11 +93,11 @@ def main() -> None:
parser.add_argument("--output-directory", required=True, type=Path)
args = parser.parse_args()
try:
archive, record, details = build_package(args.repository, args.ref, args.output_directory)
archive, details = build_package(args.repository, args.ref, args.output_directory)
except (OSError, ValueError, ET.ParseError, json.JSONDecodeError, subprocess.SubprocessError) as error:
print(f"Source packaging failed: {error}", file=sys.stderr)
raise SystemExit(1)
print(json.dumps({"archive": str(archive), "record": str(record), **details}, indent=2))
print(json.dumps({"archive": str(archive), **details}, indent=2))
if __name__ == "__main__":
@@ -1,31 +0,0 @@
{
"schemaVersion": 1,
"system": "guestops-persistence",
"evidenceId": "persistence",
"releaseVersion": "0.2.1",
"releaseCommit": "0000000000000000000000000000000000000000",
"releaseRecordSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"archiveSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"environment": "https://sandbox-guestops.futuresens.co.uk",
"hostIdentifier": "guestops-sandbox-01",
"images": {
"api": {"reference": "guestops-api:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
"worker": {"reference": "guestops-worker:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"}
},
"drillCommand": "python3 deploy/ops.py persistence-drill --confirm-restart",
"operator": "REPLACE",
"reviewedBy": "REPLACE",
"startedAt": "2026-09-30T09:00:00Z",
"endedAt": "2026-09-30T10:00:00Z",
"reviewedAt": "2026-09-30T11:00:00Z",
"unresolvedCriticalFindings": 1,
"scenarios": [
{"id": "container-recreation", "status": "not-run", "evidence": []},
{"id": "data-protection-key", "status": "not-run", "evidence": []},
{"id": "database-inventory", "status": "not-run", "evidence": []},
{"id": "image-identity", "status": "not-run", "evidence": []},
{"id": "post-reboot-persistence", "status": "not-run", "evidence": []},
{"id": "separate-volume-layout", "status": "not-run", "evidence": []},
{"id": "service-restart", "status": "not-run", "evidence": []}
]
}
-76
View File
@@ -1,76 +0,0 @@
#!/usr/bin/env python3
"""Bind a reviewed GuestOps source package to its installed image identities."""
from __future__ import annotations
import argparse
import hashlib
import json
import re
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
def project_versions() -> tuple[str, str]:
props = (ROOT / "Directory.Build.props").read_text(encoding="utf-8")
match = re.search(r"<Version>([^<]+)</Version>", props)
if match is None:
raise ValueError("Directory.Build.props does not contain a Version element")
package = json.loads((ROOT / "web" / "package.json").read_text(encoding="utf-8"))
return match.group(1), str(package["version"])
def sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--artifact", required=True, type=Path)
parser.add_argument("--commit", required=True)
parser.add_argument("--api-image", required=True)
parser.add_argument("--api-id", required=True)
parser.add_argument("--worker-image", required=True)
parser.add_argument("--worker-id", required=True)
parser.add_argument("--output", required=True, type=Path)
args = parser.parse_args()
commit = args.commit.lower()
if re.fullmatch(r"[0-9a-f]{40}", commit) is None:
parser.error("--commit must be a full 40-character Git SHA")
if not args.artifact.is_file():
parser.error("--artifact must name an existing file")
dotnet_version, web_version = project_versions()
if dotnet_version != web_version:
parser.error(
f"release versions differ: .NET={dotnet_version}, web={web_version}"
)
record = {
"artifact": {
"name": args.artifact.name,
"sha256": sha256(args.artifact),
"size": args.artifact.stat().st_size,
},
"commit": commit,
"images": {
"api": {"id": args.api_id, "reference": args.api_image},
"worker": {"id": args.worker_id, "reference": args.worker_image},
},
"schemaVersion": 1,
"version": dotnet_version,
}
args.output.write_text(
json.dumps(record, indent=2, sort_keys=True) + "\n", encoding="utf-8"
)
if __name__ == "__main__":
main()
@@ -1,6 +0,0 @@
BACKUP_DIRECTORY=/var/backups/guestops
BACKUP_REMOTE_HOST=restricted-store.example.invalid
BACKUP_REMOTE_USER=guestops_upload
BACKUP_REMOTE_DIRECTORY=/restricted/guestops/backups
BACKUP_SSH_IDENTITY=/etc/guestops/backup-transfer.key
BACKUP_SSH_KNOWN_HOSTS=/etc/guestops/backup-known-hosts
-2
View File
@@ -1,2 +0,0 @@
BACKUP_RECIPIENT=0123456789ABCDEF0123456789ABCDEF01234567
BACKUP_DIRECTORY=/var/backups/guestops
@@ -1,19 +0,0 @@
[Unit]
Description=Transfer GuestOps encrypted backups to restricted storage
Wants=network-online.target
After=network-online.target guestops-backup.service
ConditionPathIsDirectory=/srv/guestops/current
ConditionPathIsDirectory=/var/backups/guestops
[Service]
Type=oneshot
WorkingDirectory=/srv/guestops/current
EnvironmentFile=/etc/guestops/backup-transfer.env
UMask=0077
ExecStart=/usr/bin/python3 /srv/guestops/current/deploy/backup_transfer.py --prune-verified --retention-days 7
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
ReadWritePaths=/var/backups/guestops
TimeoutStartSec=30min
@@ -1,11 +0,0 @@
[Unit]
Description=Retry GuestOps off-host backup transfer
[Timer]
OnBootSec=10min
OnUnitActiveSec=15min
Persistent=true
Unit=guestops-backup-transfer.service
[Install]
WantedBy=timers.target
-21
View File
@@ -1,21 +0,0 @@
[Unit]
Description=GuestOps encrypted maintenance backup
Requires=docker.service
After=docker.service
ConditionPathIsDirectory=/srv/guestops/current
ConditionPathIsDirectory=/var/backups/guestops
ConditionPathIsDirectory=/var/lib/guestops-backup/gnupg
[Service]
Type=oneshot
WorkingDirectory=/srv/guestops/current
EnvironmentFile=/etc/guestops/backup.env
Environment=GNUPGHOME=/var/lib/guestops-backup/gnupg
UMask=0077
ExecStart=/usr/bin/python3 /srv/guestops/current/deploy/ops.py scheduled-backup --confirm-maintenance
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
ReadWritePaths=/var/backups/guestops /var/lib/guestops-backup/gnupg
TimeoutStartSec=45min
-11
View File
@@ -1,11 +0,0 @@
[Unit]
Description=Run the GuestOps encrypted backup each day
[Timer]
OnCalendar=*-*-* 02:17:00 UTC
RandomizedDelaySec=30min
Persistent=true
Unit=guestops-backup.service
[Install]
WantedBy=timers.target
@@ -1,21 +0,0 @@
[Unit]
Description=Write non-sensitive GuestOps monitoring status
Requires=docker.service
After=docker.service network-online.target
ConditionPathIsDirectory=/srv/guestops/current
ConditionPathIsDirectory=/var/backups/guestops
[Service]
Type=oneshot
WorkingDirectory=/srv/guestops/current
UMask=0022
ExecStart=/usr/bin/python3 /srv/guestops/current/deploy/monitor_status.py
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
RuntimeDirectory=guestops-monitor
RuntimeDirectoryMode=0755
RuntimeDirectoryPreserve=yes
ReadWritePaths=/run/guestops-monitor
TimeoutStartSec=2min
@@ -1,11 +0,0 @@
[Unit]
Description=Refresh GuestOps monitoring status each minute
[Timer]
OnBootSec=2min
OnUnitActiveSec=1min
Persistent=true
Unit=guestops-monitor-status.service
[Install]
WantedBy=timers.target
@@ -1,2 +0,0 @@
# The privileged systemd probe owns Docker/database access. Zabbix reads only this aggregate JSON.
UserParameter=guestops.status,cat /run/guestops-monitor/status.json
-118
View File
@@ -1,118 +0,0 @@
#!/usr/bin/env python3
"""Verify a GuestOps source package and its immutable release record."""
from __future__ import annotations
import argparse
import hashlib
import json
from pathlib import Path
import re
import subprocess
SHA = re.compile(r"[0-9a-f]{40}")
DIGEST = re.compile(r"sha256:[0-9a-f]{64}")
def require(condition: bool, message: str) -> None:
if not condition:
raise ValueError(message)
def sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def loaded_image_id(reference: str) -> str:
result = subprocess.run(
["docker", "image", "inspect", "--format", "{{.Id}}", reference],
check=True,
capture_output=True,
text=True,
)
return result.stdout.strip()
def validate(
archive: Path,
record_path: Path,
expected_commit: str,
expected_version: str,
verify_loaded_images: bool = False,
) -> dict[str, object]:
require(archive.is_file(), "Release archive does not exist.")
require(record_path.is_file(), "Release record does not exist.")
require(SHA.fullmatch(expected_commit) is not None, "Expected commit must be a full lowercase Git SHA.")
require(re.fullmatch(r"[0-9]+\.[0-9]+\.[0-9]+", expected_version) is not None,
"Expected version must use MAJOR.MINOR.PATCH.")
record = json.loads(record_path.read_text(encoding="utf-8"))
require(isinstance(record, dict), "Release record must be a JSON object.")
require(record.get("schemaVersion") == 1, "Unsupported release-record schema.")
require(record.get("commit") == expected_commit, "Release-record commit does not match the approved candidate.")
require(record.get("version") == expected_version, "Release-record version does not match the approved version.")
artifact = record.get("artifact")
require(isinstance(artifact, dict), "Release record has no artifact object.")
require(artifact.get("name") == archive.name, "Release archive filename does not match the record.")
require(artifact.get("size") == archive.stat().st_size, "Release archive size does not match the record.")
archive_sha = sha256(archive)
require(artifact.get("sha256") == archive_sha, "Release archive SHA-256 does not match the record.")
images = record.get("images")
require(isinstance(images, dict) and set(images) == {"api", "worker"},
"Release record must contain exactly API and worker images.")
expected_references = {
"api": f"guestops-api:{expected_commit}",
"worker": f"guestops-worker:{expected_commit}",
}
verified_images: dict[str, dict[str, str]] = {}
for name, reference in expected_references.items():
image = images.get(name)
require(isinstance(image, dict), f"Release record has no {name} image object.")
require(image.get("reference") == reference, f"{name} image reference is not bound to the full candidate SHA.")
image_id = str(image.get("id", ""))
require(DIGEST.fullmatch(image_id) is not None, f"{name} image ID is not an immutable SHA-256 digest.")
if verify_loaded_images:
require(loaded_image_id(reference) == image_id, f"Loaded {name} image ID does not match the release record.")
verified_images[name] = {"reference": reference, "id": image_id}
return {
"schemaVersion": 1,
"verified": True,
"commit": expected_commit,
"version": expected_version,
"archive": {"name": archive.name, "size": archive.stat().st_size, "sha256": archive_sha},
"releaseRecord": {"name": record_path.name, "sha256": sha256(record_path)},
"images": verified_images,
"loadedImageIdsVerified": verify_loaded_images,
}
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--archive", required=True, type=Path)
parser.add_argument("--record", required=True, type=Path)
parser.add_argument("--commit", required=True)
parser.add_argument("--version", required=True)
parser.add_argument("--verify-loaded-images", action="store_true")
parser.add_argument("--output", type=Path, help="Optional path for the non-sensitive verification summary.")
args = parser.parse_args()
result = validate(args.archive, args.record, args.commit, args.version, args.verify_loaded_images)
rendered = json.dumps(result, indent=2, sort_keys=True) + "\n"
if args.output:
args.output.write_text(rendered, encoding="utf-8")
print(rendered, end="")
if __name__ == "__main__":
try:
main()
except (OSError, ValueError, json.JSONDecodeError, subprocess.SubprocessError) as error:
print(f"Release verification failed: {error}", file=__import__("sys").stderr)
raise SystemExit(1)
-104
View File
@@ -1,104 +0,0 @@
#!/usr/bin/env python3
"""Verify a GuestOps source archive against its deterministic source record."""
from __future__ import annotations
import argparse
import hashlib
import json
from pathlib import Path
import re
import sys
FULL_SHA = re.compile(r"[0-9a-f]{40}")
SHA256 = re.compile(r"[0-9a-f]{64}")
SEMVER = re.compile(r"[0-9]+\.[0-9]+\.[0-9]+")
def require(condition: bool, message: str) -> None:
if not condition:
raise ValueError(message)
def sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def validate(
archive: Path,
record_path: Path,
expected_commit: str | None = None,
expected_version: str | None = None,
) -> dict[str, object]:
require(archive.is_file(), "Source archive does not exist.")
require(record_path.is_file(), "Source record does not exist.")
record = json.loads(record_path.read_text(encoding="utf-8"))
require(isinstance(record, dict), "Source record must be a JSON object.")
require(record.get("schemaVersion") == 1, "Unsupported source-record schema.")
commit = record.get("commit")
version = record.get("version")
require(isinstance(commit, str) and FULL_SHA.fullmatch(commit) is not None,
"Source record commit must be a full lowercase Git SHA.")
require(isinstance(version, str) and SEMVER.fullmatch(version) is not None,
"Source record version must use MAJOR.MINOR.PATCH.")
if expected_commit is not None:
require(FULL_SHA.fullmatch(expected_commit) is not None,
"Expected commit must be a full lowercase Git SHA.")
require(commit == expected_commit, "Source-record commit does not match the selected release.")
if expected_version is not None:
require(SEMVER.fullmatch(expected_version) is not None,
"Expected version must use MAJOR.MINOR.PATCH.")
require(version == expected_version, "Source-record version does not match the selected release.")
artifact = record.get("artifact")
require(isinstance(artifact, dict), "Source record has no artifact object.")
require(artifact.get("name") == archive.name, "Source archive filename does not match the record.")
require(artifact.get("size") == archive.stat().st_size, "Source archive size does not match the record.")
recorded_sha = artifact.get("sha256")
require(isinstance(recorded_sha, str) and SHA256.fullmatch(recorded_sha) is not None,
"Source archive record must contain a lowercase SHA-256 digest.")
actual_sha = sha256(archive)
require(recorded_sha == actual_sha, "Source archive SHA-256 does not match the record.")
return {
"artifact": {"name": archive.name, "sha256": actual_sha, "size": archive.stat().st_size},
"commit": commit,
"schemaVersion": 1,
"sourceRecord": {"name": record_path.name, "sha256": sha256(record_path)},
"verified": True,
"version": version,
}
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--archive", required=True, type=Path)
parser.add_argument("--record", required=True, type=Path)
parser.add_argument("--expected-commit")
parser.add_argument("--expected-version")
parser.add_argument("--output", type=Path)
args = parser.parse_args()
try:
result = validate(
args.archive,
args.record,
args.expected_commit,
args.expected_version,
)
except (OSError, ValueError, json.JSONDecodeError) as error:
print(f"Source package verification failed: {error}", file=sys.stderr)
raise SystemExit(1)
rendered = json.dumps(result, indent=2, sort_keys=True) + "\n"
if args.output:
args.output.write_text(rendered, encoding="utf-8")
print(rendered, end="")
if __name__ == "__main__":
main()