Gate B release candidate

This commit is contained in:
wolf-demon 2026-09-30 15:17:34 +01:00 committed by mathew
parent a00fad5553
commit d7e19bd9da
21 changed files with 1573 additions and 2 deletions

View File

@ -1,10 +1,36 @@
# GuestOps Milestone Report
Version: **0.2.0 release candidate**
Last updated: **29 September 2026**
Last updated: **30 September 2026**
This is the working delivery tracker for GuestOps Web. Update a milestone when its state changes and link the pull request, release artifact, test run, or acceptance record that proves the change.
## Milestones at a glance
This summary explains what each milestone delivers and where it currently stands. The release-gate and delivery tables below contain the detailed evidence and exit conditions.
| # | Milestone | What it delivers | Current position |
| ---: | --- | --- | --- |
| 1 | Web foundation | The React application, ASP.NET Core API, tenant-isolated MongoDB storage, preview mode, and container foundation. | **Implemented.** The application foundation and automated tests are on `main`. |
| 2 | AI suggestions and reviewed Gmail sending | AI-assisted reply drafts and staff-reviewed Gmail delivery with safety and duplicate-send controls. | **Implemented; acceptance required.** Real Gmail threading, revocation, uncertain-send, and staff-review scenarios still need live evidence. |
| 3 | OHIP PMS workflow | Internal proposal, approval, and execution controls for PMS operations. | **Implemented; acceptance required.** The workflow exists, but provider-contract and sandbox acceptance remain outstanding. |
| 4 | NMI payment workflow | Internal payment proposal, approval, status, expiry, and reconciliation controls. | **Implemented; acceptance required.** The workflow exists, but real payment-provider sandbox acceptance remains outstanding. |
| 5 | FAQ automation | Knowledge-based FAQ drafting, test mode, approval controls, and guarded live automation. | **Implemented; acceptance required.** Live mode remains disabled pending quality, false-positive, monitoring, and rollback acceptance. |
| 6 | Team onboarding and account recovery | Staff invitations, password setup/reset, access disable/restore, and administrator recovery. | **Implemented; acceptance required.** Deployed link delivery, expiry, recovery, and administrator procedures still need operational evidence. |
| 7 | Google connection recovery | OAuth reconnect, checkpoint recovery, grant revocation handling, and worker restart safety. | **Implemented; acceptance required.** Dedicated Google-account and worker-restart exercises have not yet been accepted. |
| 8 | Operational readiness tooling | Release verification, diagnostics, encrypted backup, restore, preflight, and persistence tools. | **Implemented; acceptance required.** The tools must still be run against the exact release on the Debian host. |
| 9 | Gitea and reproducible releases | CI-built immutable images, checksummed release records, retained artifacts, and approval tagging. | **In progress.** The `0.2.0` candidate is frozen; successful exact-commit CI evidence and the post-Gate-B tag are still required. |
| 10 | Debian deployment and persistence | Secure Debian/Compose deployment, HTTPS, persistent database and key volumes, and reboot/recreation proof. | **In progress.** Acceptance tooling is ready, but Docker, correct HTTPS/network exposure, exact artifacts, privileged installation, and the supervised drills remain open. |
| 11 | Backups, monitoring, and recovery | Scheduled encrypted backups, verified off-host transfer, Zabbix monitoring, restore, and rollback rehearsal. | **In progress.** Repository tooling is ready; installation and timed operational evidence are blocked until Milestones 9 and 10 pass. |
| 12 | Google mailbox acceptance | End-to-end Gmail consent, import, recovery, reviewed sending, reconciliation, and revocation evidence. | **In progress.** The runbook and validator exist; the live synthetic-data exercise and independent review remain outstanding. |
| 13 | Rezlynx/Guestline adapter | The real PMS provider adapter, mappings, idempotency, reconciliation, and ambiguous-write handling. | **Planned.** Provider contract and sandbox access are still required before implementation and acceptance. |
| 14 | Payment links and status | The real payment-provider integration, webhooks, expiry, replay protection, and reconciliation. | **Planned.** The provider path and sandbox acceptance plan still need to be confirmed and completed. |
| 15 | Knowledge, AI, and FAQ activation | Supervised knowledge-quality, AI-draft, FAQ test-mode, staff-training, and stop-control acceptance. | **Implemented; acceptance required.** The evaluation tooling exists; the supervised evaluation and independent approval remain outstanding. |
| 16 | Identity, preferences, and privacy | Account/session controls, hotel preferences, privacy inventory, retention decisions, and audit review. | **Implemented; acceptance required.** Legal and operational decisions, identity checks, and independent review remain outstanding. |
| 17 | Inbox usability and desktop parity | Stable pagination, protected unsaved drafts, hotel-timezone display, and desktop workflow parity. | **Implemented; acceptance required.** Automated checks pass; the supervised desktop exercise and independent approval remain outstanding. |
| 18 | Pilot, capacity, and release approval | Capacity proof, incident exercise, five-business-day hotel pilot, findings closure, and Gate B approval. | **In progress.** Validators and targets exist; Gate A/B prerequisites, capacity evidence, incident rehearsal, pilot, and named approvals remain open. |
| 19 | Account security and self-service | TOTP MFA, recovery codes, transactional email, granular roles, preferences, and security notifications. | **Implemented on the development branch; acceptance required.** Keep it separate until `0.2.0` is approved and tagged, then review, merge, and version it as `0.3.0`. |
## Status key
- **Implemented** — present on `main` and supported by code or automated-test evidence.
@ -35,7 +61,7 @@ This is the working delivery tracker for GuestOps Web. Update a milestone when i
| 8 | Operational readiness tooling | A | Implemented / acceptance required | Backup, restore, release, and diagnostic tooling is promoted to local `main`; execute it on the actual Debian host and retain evidence. |
| 9 | Gitea and reproducible releases | A | In progress | The `0.2.0` candidate is versioned on `main`. CI records the full commit, matched application version, archive checksum and immutable image IDs, and the rollback procedure is documented. Retain the successful default-branch evidence off-host and create the immutable approval tag only after Gate B approval; the existing `0.1.0` tag remains attached to the foundation release. |
| 10 | Debian deployment and persistence | A | In progress | Compose uses separate named database and shared key volumes, private host configuration, loopback-only API access and bounded logs. The confirmation-gated persistence drill verifies restart and container-recreation behaviour. Run it on the provisioned Debian host, complete HTTPS and controlled-reboot acceptance, and retain the evidence. |
| 11 | Backups, monitoring, and recovery | A | In progress | Encrypted backup and isolated restore tooling now includes opt-in systemd scheduling without command-line secrets. Install and test it on Debian, configure monitored off-host transfer and durable logs, name alert/retention owners, and retain evidence from a timed restore and recovery drill. |
| 11 | Backups, monitoring, and recovery | A | In progress | Encrypted backup and isolated restore tooling now includes opt-in systemd scheduling, checksum-verified rsync transfer, a restricted Zabbix status boundary, guarded local retention and a release-bound acceptance validator. Install and test it on Debian, configure the restricted store and alerts, name operational/review owners, and retain independently reviewed evidence from the timed restore and rollback drill. |
| 12 | Google mailbox and reviewed-reply acceptance | B | In progress | The synthetic-data provider runbook, exact scenario set and restricted-record validator are implemented. Complete every scenario against the accepted Debian release and dedicated Google sandbox accounts, independently review the evidence, and retain the validated record. |
| 13 | Rezlynx/Guestline adapter | C | Planned | Obtain the provider contract and sandbox, implement the adapter and mapping, and accept idempotency, stale-data, ambiguous-write, and reconciliation paths. |
| 14 | Payment links and status | C | Planned | Select/confirm the payment-provider path, complete sandbox and webhook acceptance, and prove expiry, replay protection, reconciliation, and support recovery. |

View File

@ -0,0 +1,81 @@
{
"schemaVersion": 1,
"system": "guestops-backup-recovery",
"evidenceId": "backup-restore",
"dataClassification": "synthetic-only",
"releaseVersion": "0.2.0",
"releaseCommit": "0000000000000000000000000000000000000000",
"releaseRecordSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"archiveSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"environment": "https://sandbox-guestops.futuresens.co.uk",
"hostIdentifier": "guestops-sandbox-01",
"images": {
"api": {"reference": "guestops-api:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
"worker": {"reference": "guestops-worker:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
"mongo": {"reference": "mongo:8.0", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"}
},
"owners": {
"backup": "REPLACE",
"monitoring": "REPLACE",
"recoveryOperator": "REPLACE",
"technicalEscalation": "REPLACE",
"retention": "REPLACE",
"independentReviewer": "REPLACE"
},
"startedAt": "2026-10-01T09:00:00Z",
"endedAt": "2026-10-01T13:00:00Z",
"reviewedAt": "2026-10-01T14:00:00Z",
"recoveryObjectives": {
"targetRpoHours": 24,
"observedRpoHours": 25,
"targetRtoMinutes": 240,
"observedRtoMinutes": 241
},
"retention": {
"localVerifiedDays": 7,
"offHostDaily": 35,
"offHostMonthly": 12,
"legalHoldOverrideTested": false
},
"backup": {
"createdAt": "2026-10-01T08:00:00Z",
"sha256": "0000000000000000000000000000000000000000000000000000000000000000",
"transferredSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"privateKeyPresentOnHost": false
},
"rollback": {
"previousReleaseCommit": "1111111111111111111111111111111111111111",
"previousArchiveSha256": "1111111111111111111111111111111111111111111111111111111111111111",
"previousImages": {
"api": {"reference": "guestops-api:1111111111111111111111111111111111111111", "id": "sha256:1111111111111111111111111111111111111111111111111111111111111111"},
"worker": {"reference": "guestops-worker:1111111111111111111111111111111111111111", "id": "sha256:1111111111111111111111111111111111111111111111111111111111111111"}
},
"persistentVolumesReplaced": false,
"restoredReleaseCommit": "0000000000000000000000000000000000000000",
"unresolvedOperations": 1,
"finalControls": {
"googleSending": "disabled",
"faqLiveMode": "disabled",
"pmsWrites": "disabled",
"paymentCreation": "disabled"
}
},
"monitoringState": "not-configured",
"unresolvedCriticalFindings": 1,
"scenarios": [
{"id": "alert-escalation", "status": "not-run", "evidence": []},
{"id": "atomic-off-host-transfer", "status": "not-run", "evidence": []},
{"id": "controlled-return-to-service", "status": "not-run", "evidence": []},
{"id": "data-protection-recovery", "status": "not-run", "evidence": []},
{"id": "database-inventory", "status": "not-run", "evidence": []},
{"id": "durable-secret-free-logs", "status": "not-run", "evidence": []},
{"id": "encrypted-manual-backup", "status": "not-run", "evidence": []},
{"id": "image-rollback", "status": "not-run", "evidence": []},
{"id": "isolated-restore", "status": "not-run", "evidence": []},
{"id": "monitoring-coverage", "status": "not-run", "evidence": []},
{"id": "off-host-checksum", "status": "not-run", "evidence": []},
{"id": "production-service-recovery", "status": "not-run", "evidence": []},
{"id": "retention-and-legal-hold", "status": "not-run", "evidence": []},
{"id": "scheduled-backup", "status": "not-run", "evidence": []}
]
}

View File

@ -0,0 +1,220 @@
#!/usr/bin/env python3
"""Validate a restricted GuestOps backup, monitoring and recovery record."""
from __future__ import annotations
import argparse
import datetime as dt
import json
from pathlib import Path
import re
from urllib.parse import urlparse
VERSION = "0.2.0"
SCENARIOS = {
"encrypted-manual-backup",
"scheduled-backup",
"production-service-recovery",
"atomic-off-host-transfer",
"off-host-checksum",
"retention-and-legal-hold",
"monitoring-coverage",
"alert-escalation",
"durable-secret-free-logs",
"isolated-restore",
"data-protection-recovery",
"database-inventory",
"image-rollback",
"controlled-return-to-service",
}
SHA256 = re.compile(r"[0-9a-f]{64}")
GIT_SHA = re.compile(r"[0-9a-f]{40}")
def require(condition: bool, message: str) -> None:
if not condition:
raise ValueError(message)
def timestamp(value: object, field: str) -> dt.datetime:
require(isinstance(value, str) and value.endswith("Z"),
f"{field} must be a UTC timestamp ending in Z.")
try:
parsed = dt.datetime.fromisoformat(value.removesuffix("Z") + "+00:00")
except ValueError as error:
raise ValueError(f"{field} is not a valid timestamp.") from error
require(parsed.tzinfo == dt.timezone.utc, f"{field} must be UTC.")
return parsed
def safe_name(value: object, field: str) -> str:
name = str(value or "").strip()
require(2 <= len(name) <= 120 and "@" not in name and "/" not in name and "\\" not in name,
f"{field} requires a name without an email address or path.")
return name
def validate(record: object, expected_commit: str, expected_release_sha256: str) -> None:
require(isinstance(record, dict), "Acceptance record must be a JSON object.")
require(record.get("schemaVersion") == 1, "Unsupported backup/recovery schema.")
require(record.get("system") == "guestops-backup-recovery",
"system must be guestops-backup-recovery.")
require(record.get("evidenceId") == "backup-restore",
"evidenceId must be backup-restore.")
require(record.get("dataClassification") == "synthetic-only",
"Recovery acceptance must use synthetic data only.")
require(record.get("releaseVersion") == VERSION, f"releaseVersion must be {VERSION}.")
require(GIT_SHA.fullmatch(str(expected_commit)) is not None,
"Expected release commit must be a full lowercase Git SHA.")
require(SHA256.fullmatch(str(expected_release_sha256)) is not None,
"Expected release-record checksum must be a lowercase SHA-256 digest.")
require(record.get("releaseCommit") == expected_commit,
"releaseCommit does not match the approved candidate.")
require(record.get("releaseRecordSha256") == expected_release_sha256,
"releaseRecordSha256 does not match the retained release record.")
require(SHA256.fullmatch(str(record.get("archiveSha256", ""))) is not None,
"archiveSha256 must be a lowercase SHA-256 digest.")
origin = urlparse(str(record.get("environment", "")))
require(origin.scheme == "https" and origin.hostname and origin.path in ("", "/")
and not origin.query and not origin.fragment and origin.username is None
and origin.password is None,
"environment must be an HTTPS origin without credentials, path, query or fragment.")
safe_name(record.get("hostIdentifier"), "hostIdentifier")
images = record.get("images")
require(isinstance(images, dict) and set(images) == {"api", "worker", "mongo"},
"images must contain exactly api, worker and mongo.")
for name in ("api", "worker"):
image = images[name]
require(isinstance(image, dict) and set(image) == {"reference", "id"},
f"images.{name} must contain exactly reference and id.")
require(image["reference"] == f"guestops-{name}:{expected_commit}",
f"images.{name}.reference must use the full approved commit.")
require(re.fullmatch(r"sha256:[0-9a-f]{64}", str(image["id"])) is not None,
f"images.{name}.id must be immutable.")
mongo = images["mongo"]
require(isinstance(mongo, dict) and set(mongo) == {"reference", "id"}
and mongo["reference"] == "mongo:8.0"
and re.fullmatch(r"sha256:[0-9a-f]{64}", str(mongo["id"])) is not None,
"images.mongo must identify the immutable mongo:8.0 image.")
owners = record.get("owners")
owner_keys = {"backup", "monitoring", "recoveryOperator", "technicalEscalation",
"retention", "independentReviewer"}
require(isinstance(owners, dict) and set(owners) == owner_keys,
"owners must contain the exact operational and review roles.")
names = {key: safe_name(value, f"owners.{key}") for key, value in owners.items()}
reviewer = names["independentReviewer"].casefold()
require(reviewer not in {names[key].casefold() for key in owner_keys - {"independentReviewer"}},
"independentReviewer must be different from every operational owner.")
started = timestamp(record.get("startedAt"), "startedAt")
ended = timestamp(record.get("endedAt"), "endedAt")
reviewed = timestamp(record.get("reviewedAt"), "reviewedAt")
require(started <= ended <= reviewed, "Acceptance timestamps are out of order.")
recovery = record.get("recoveryObjectives")
require(isinstance(recovery, dict) and set(recovery) == {
"targetRpoHours", "observedRpoHours", "targetRtoMinutes", "observedRtoMinutes",
}, "recoveryObjectives must contain exact target and observed RPO/RTO values.")
for field in recovery:
require(isinstance(recovery[field], (int, float)) and not isinstance(recovery[field], bool)
and recovery[field] >= 0, f"recoveryObjectives.{field} must be non-negative.")
require(recovery["targetRpoHours"] == 24 and recovery["observedRpoHours"] <= 24,
"Observed RPO must meet the approved 24-hour target.")
require(recovery["targetRtoMinutes"] == 240 and recovery["observedRtoMinutes"] <= 240,
"Observed RTO must meet the approved four-hour target.")
retention = record.get("retention")
require(retention == {
"localVerifiedDays": 7,
"offHostDaily": 35,
"offHostMonthly": 12,
"legalHoldOverrideTested": True,
}, "Retention must record seven local days, 35 daily and 12 monthly off-host copies, and legal-hold testing.")
backup = record.get("backup")
require(isinstance(backup, dict) and set(backup) == {
"createdAt", "sha256", "transferredSha256", "privateKeyPresentOnHost",
}, "backup must contain exact creation, checksum, transfer and private-key fields.")
created = timestamp(backup["createdAt"], "backup.createdAt")
require(created <= started, "The accepted backup must exist when the timed exercise starts.")
require(abs(recovery["observedRpoHours"] - (started - created).total_seconds() / 3600) < 0.01,
"Observed RPO must match the backup and exercise timestamps.")
require(abs(recovery["observedRtoMinutes"] - (ended - started).total_seconds() / 60) < 0.01,
"Observed RTO must match the exercise timestamps.")
require(SHA256.fullmatch(str(backup["sha256"])) is not None
and backup["transferredSha256"] == backup["sha256"],
"Local and transferred backup checksums must match.")
require(backup["privateKeyPresentOnHost"] is False,
"The recovery private key must not be present on the Debian host.")
rollback = record.get("rollback")
require(isinstance(rollback, dict) and set(rollback) == {
"previousReleaseCommit", "previousArchiveSha256", "previousImages",
"persistentVolumesReplaced", "restoredReleaseCommit", "unresolvedOperations", "finalControls",
}, "rollback must contain the exact rehearsal and final-state fields.")
require(GIT_SHA.fullmatch(str(rollback["previousReleaseCommit"])) is not None
and rollback["previousReleaseCommit"] != expected_commit,
"Rollback must use a different retained previous release.")
require(SHA256.fullmatch(str(rollback["previousArchiveSha256"])) is not None,
"Rollback requires the previous archive checksum.")
previous_images = rollback["previousImages"]
require(isinstance(previous_images, dict) and set(previous_images) == {"api", "worker"},
"Rollback requires exact previous API and worker images.")
for name in ("api", "worker"):
image = previous_images[name]
require(isinstance(image, dict) and set(image) == {"reference", "id"}
and image["reference"] == f"guestops-{name}:{rollback['previousReleaseCommit']}"
and re.fullmatch(r"sha256:[0-9a-f]{64}", str(image["id"])) is not None,
f"rollback.previousImages.{name} must use the retained previous release identity.")
require(rollback["persistentVolumesReplaced"] is False,
"Image rollback must not replace persistent volumes.")
require(rollback["restoredReleaseCommit"] == expected_commit,
"The exercise must finish on the approved candidate.")
require(rollback["unresolvedOperations"] == 0,
"The exercise must finish without unresolved operations.")
require(rollback["finalControls"] == {
"googleSending": "disabled", "faqLiveMode": "disabled",
"pmsWrites": "disabled", "paymentCreation": "disabled",
}, "The exercise must finish with all unaccepted external writes disabled.")
scenarios = record.get("scenarios")
require(isinstance(scenarios, list), "scenarios must be a list.")
ids = [item.get("id") for item in scenarios if isinstance(item, dict)]
require(len(ids) == len(scenarios) and len(ids) == len(set(ids)) and set(ids) == SCENARIOS,
"Acceptance record requires the exact backup/recovery scenario set.")
for item in scenarios:
scenario_id = item["id"]
require(item.get("status") == "pass", f"Scenario {scenario_id} has not passed.")
evidence = item.get("evidence")
require(isinstance(evidence, list) and 1 <= len(evidence) <= 10 and all(
isinstance(value, str) and 3 <= len(value) <= 200 and "@" not in value
and "\\" not in value and not value.startswith("/") for value in evidence
), f"Scenario {scenario_id} requires safe opaque evidence references.")
require(record.get("monitoringState") == "healthy",
"Monitoring must be healthy at acceptance completion.")
require(record.get("unresolvedCriticalFindings") == 0,
"Acceptance cannot pass with unresolved critical findings.")
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("record", type=Path)
parser.add_argument("--expected-commit", required=True)
parser.add_argument("--expected-release-record-sha256", required=True)
args = parser.parse_args()
validate(json.loads(args.record.read_text(encoding="utf-8")),
args.expected_commit, args.expected_release_record_sha256)
print("Backup, monitoring and recovery acceptance record is structurally complete and passed. "
"This validates the record, not its restricted evidence.")
if __name__ == "__main__":
try:
main()
except (OSError, ValueError, json.JSONDecodeError) as error:
print(f"Backup/recovery acceptance record rejected: {error}", file=__import__("sys").stderr)
raise SystemExit(1)

238
deploy/backup_transfer.py Normal file
View File

@ -0,0 +1,238 @@
#!/usr/bin/env python3
"""Atomically transfer encrypted GuestOps backups to restricted storage."""
from __future__ import annotations
import argparse
import hashlib
import json
import os
from pathlib import Path
import re
import shlex
import shutil
import stat
import subprocess
import tempfile
import time
import uuid
BACKUP_NAME = re.compile(r"guestops-[0-9]{8}T[0-9]{6}Z\.tar\.gpg")
SAFE_HOST = re.compile(r"[A-Za-z0-9.-]{1,253}")
SAFE_USER = re.compile(r"[A-Za-z_][A-Za-z0-9_-]{0,31}")
SAFE_REMOTE_PATH = re.compile(r"/[A-Za-z0-9._/-]{1,500}")
SHA256 = re.compile(r"[0-9a-f]{64}")
def require(condition: bool, message: str) -> None:
if not condition:
raise RuntimeError(message)
def digest(path: Path) -> str:
with path.open("rb") as stream:
return hashlib.file_digest(stream, "sha256").hexdigest()
def private_directory(value: str) -> Path:
requested = Path(value)
require(requested.is_absolute() and not requested.is_symlink(),
"BACKUP_DIRECTORY must be an absolute, non-symlink path.")
directory = requested.resolve()
require(directory.is_dir(), "BACKUP_DIRECTORY must exist.")
require(stat.S_IMODE(directory.stat().st_mode) & 0o077 == 0,
"BACKUP_DIRECTORY must not be accessible to group or other users.")
return directory
def regular_file(value: str, field: str, *, private: bool) -> Path:
requested = Path(value)
require(requested.is_absolute() and not requested.is_symlink(),
f"{field} must be an absolute, non-symlink path.")
path = requested.resolve()
require(path.is_file(), f"{field} must be an existing regular file.")
if private:
require(stat.S_IMODE(path.stat().st_mode) & 0o077 == 0,
f"{field} must not be accessible to group or other users.")
return path
def configuration(environment: dict[str, str]) -> dict[str, object]:
directory = private_directory(environment.get("BACKUP_DIRECTORY", ""))
host = environment.get("BACKUP_REMOTE_HOST", "")
user = environment.get("BACKUP_REMOTE_USER", "")
remote = environment.get("BACKUP_REMOTE_DIRECTORY", "")
require(SAFE_HOST.fullmatch(host) is not None, "BACKUP_REMOTE_HOST is invalid.")
require(SAFE_USER.fullmatch(user) is not None, "BACKUP_REMOTE_USER is invalid.")
require(SAFE_REMOTE_PATH.fullmatch(remote) is not None and "//" not in remote
and "/../" not in remote + "/" and not remote.endswith("/.."),
"BACKUP_REMOTE_DIRECTORY must be a safe absolute path.")
identity = regular_file(environment.get("BACKUP_SSH_IDENTITY", ""),
"BACKUP_SSH_IDENTITY", private=True)
known_hosts = regular_file(environment.get("BACKUP_SSH_KNOWN_HOSTS", ""),
"BACKUP_SSH_KNOWN_HOSTS", private=False)
require(shutil.which("ssh") is not None and shutil.which("rsync") is not None,
"ssh and rsync are required.")
return {
"directory": directory,
"host": host,
"user": user,
"remote": remote.rstrip("/"),
"identity": identity,
"known_hosts": known_hosts,
}
def ssh_base(config: dict[str, object]) -> list[str]:
return [
"ssh", "-o", "BatchMode=yes", "-o", "IdentitiesOnly=yes",
"-o", "StrictHostKeyChecking=yes", "-o", "ConnectTimeout=15",
"-o", f"UserKnownHostsFile={config['known_hosts']}",
"-i", str(config["identity"]), f"{config['user']}@{config['host']}",
]
def run(args: list[str], *, environment: dict[str, str] | None = None) -> bytes:
result = subprocess.run(args, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, timeout=900, env=environment)
require(result.returncode == 0,
f"{Path(args[0]).name} step failed; review the restricted operator logs.")
return result.stdout
def remote_digest(config: dict[str, object], remote_path: str) -> str | None:
command = f"if test -f {remote_path} && test ! -L {remote_path}; then sha256sum -- {remote_path}; fi"
output = run([*ssh_base(config), command]).decode("utf-8", "strict").strip()
if not output:
return None
value = output.split()[0]
require(SHA256.fullmatch(value) is not None, "Remote checksum response was invalid.")
return value
def marker_path(backup: Path) -> Path:
return backup.with_name(backup.name + ".transferred.json")
def write_marker(backup: Path, checksum: str) -> None:
marker = marker_path(backup)
payload = json.dumps({
"schemaVersion": 1,
"backup": backup.name,
"sha256": checksum,
"verifiedAt": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
}, sort_keys=True) + "\n"
fd, temporary = tempfile.mkstemp(prefix=marker.name + ".", dir=marker.parent)
try:
os.fchmod(fd, 0o600)
with os.fdopen(fd, "w", encoding="utf-8") as stream:
stream.write(payload)
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary, marker)
finally:
try:
os.unlink(temporary)
except FileNotFoundError:
pass
def transfer_one(config: dict[str, object], backup: Path) -> None:
require(backup.is_file() and not backup.is_symlink()
and BACKUP_NAME.fullmatch(backup.name) is not None,
"Refusing to transfer an unexpected backup path.")
checksum = digest(backup)
remote_final = f"{config['remote']}/{backup.name}"
existing = remote_digest(config, remote_final)
if existing is not None:
require(existing == checksum, "A remote backup with this name has a different checksum.")
write_marker(backup, checksum)
return
remote_partial = f"{config['remote']}/.{backup.name}.partial-{uuid.uuid4().hex}"
rsh = shlex.join([
"ssh", "-o", "BatchMode=yes", "-o", "IdentitiesOnly=yes",
"-o", "StrictHostKeyChecking=yes", "-o", "ConnectTimeout=15",
"-o", f"UserKnownHostsFile={config['known_hosts']}",
"-i", str(config["identity"]),
])
rsync_environment = os.environ.copy()
rsync_environment["RSYNC_RSH"] = rsh
try:
run(["rsync", "--archive", "--chmod=F600", "--protect-args", "--",
str(backup), f"{config['user']}@{config['host']}:{remote_partial}"],
environment=rsync_environment)
require(remote_digest(config, remote_partial) == checksum,
"Transferred backup checksum does not match the local file.")
command = (f"test ! -e {remote_final} && mv -T -- {remote_partial} {remote_final} "
f"&& chmod 600 -- {remote_final}")
run([*ssh_base(config), command])
require(remote_digest(config, remote_final) == checksum,
"Final remote backup checksum does not match the local file.")
write_marker(backup, checksum)
except Exception:
# The name contains a fresh random suffix and is the only remote object this run may remove.
subprocess.run([*ssh_base(config), f"rm -f -- {remote_partial}"], stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=30)
raise
def transfer_pending(config: dict[str, object]) -> int:
directory = config["directory"]
backups = sorted(path for path in directory.iterdir()
if path.is_file() and not path.is_symlink()
and BACKUP_NAME.fullmatch(path.name) is not None)
for backup in backups:
marker = marker_path(backup)
if marker.is_file() and not marker.is_symlink():
try:
recorded = json.loads(marker.read_text(encoding="utf-8"))
if recorded.get("sha256") == digest(backup):
continue
except (OSError, ValueError, json.JSONDecodeError):
pass
transfer_one(config, backup)
return len(backups)
def prune_verified(config: dict[str, object], retention_days: int, now: float | None = None) -> int:
require(1 <= retention_days <= 365, "Local retention must be between 1 and 365 days.")
threshold = (time.time() if now is None else now) - retention_days * 86400
removed = 0
for backup in config["directory"].iterdir():
if not backup.is_file() or backup.is_symlink() or BACKUP_NAME.fullmatch(backup.name) is None:
continue
marker = marker_path(backup)
if backup.stat().st_mtime >= threshold or not marker.is_file() or marker.is_symlink():
continue
try:
recorded = json.loads(marker.read_text(encoding="utf-8"))
except (OSError, ValueError, json.JSONDecodeError):
continue
if recorded.get("backup") != backup.name or recorded.get("sha256") != digest(backup):
continue
backup.unlink()
marker.unlink()
removed += 1
return removed
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--prune-verified", action="store_true")
parser.add_argument("--retention-days", type=int, default=7)
args = parser.parse_args()
config = configuration(dict(os.environ))
observed = transfer_pending(config)
removed = prune_verified(config, args.retention_days) if args.prune_verified else 0
print(f"Backup transfer completed: {observed} encrypted backup(s) inspected; "
f"{removed} verified local backup(s) expired.")
if __name__ == "__main__":
try:
main()
except (OSError, RuntimeError, subprocess.SubprocessError, json.JSONDecodeError) as error:
print(f"Backup transfer failed: {error}", file=__import__("sys").stderr)
raise SystemExit(1)

View File

@ -0,0 +1,45 @@
{
"schemaVersion": 1,
"system": "guestops-debian-host",
"evidenceId": "debian-host",
"releaseVersion": "0.2.0",
"releaseCommit": "0000000000000000000000000000000000000000",
"releaseRecordSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"archiveSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"environment": "https://sandbox-guestops.futuresens.co.uk",
"hostIdentifier": "guestops-sandbox-01",
"images": {
"api": {"reference": "guestops-api:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
"worker": {"reference": "guestops-worker:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"}
},
"hostFacts": {
"debianMajor": 12,
"cpuCores": 4,
"memoryBytes": 8140382208,
"freeDiskBytes": 14275686400,
"publicTcpPorts": []
},
"featureControls": {
"googleSending": "disabled",
"faqLiveMode": "disabled",
"pmsWrites": "disabled",
"paymentCreation": "disabled"
},
"operator": "REPLACE",
"reviewedBy": "REPLACE",
"startedAt": "2026-09-30T09:00:00Z",
"endedAt": "2026-09-30T10:00:00Z",
"reviewedAt": "2026-09-30T11:00:00Z",
"unresolvedCriticalFindings": 1,
"scenarios": [
{"id": "boot-services", "status": "not-run", "evidence": []},
{"id": "controlled-reboot", "status": "not-run", "evidence": []},
{"id": "durable-log-retrieval", "status": "not-run", "evidence": []},
{"id": "host-baseline", "status": "not-run", "evidence": []},
{"id": "https-and-redirect", "status": "not-run", "evidence": []},
{"id": "network-exposure", "status": "not-run", "evidence": []},
{"id": "proxy-trust", "status": "not-run", "evidence": []},
{"id": "secret-free-logs", "status": "not-run", "evidence": []},
{"id": "workspace-health", "status": "not-run", "evidence": []}
]
}

206
deploy/debian_acceptance.py Normal file
View File

@ -0,0 +1,206 @@
#!/usr/bin/env python3
"""Validate restricted GuestOps Debian-host and persistence acceptance records."""
from __future__ import annotations
import argparse
import datetime as dt
import json
from pathlib import Path
import re
from urllib.parse import urlparse
VERSION = "0.2.0"
SYSTEMS = {
"guestops-debian-host": {
"evidenceId": "debian-host",
"scenarios": {
"host-baseline",
"network-exposure",
"https-and-redirect",
"proxy-trust",
"boot-services",
"controlled-reboot",
"workspace-health",
"durable-log-retrieval",
"secret-free-logs",
},
},
"guestops-persistence": {
"evidenceId": "persistence",
"scenarios": {
"separate-volume-layout",
"service-restart",
"container-recreation",
"database-inventory",
"data-protection-key",
"image-identity",
"post-reboot-persistence",
},
},
}
SHA256 = re.compile(r"[0-9a-f]{64}")
GIT_SHA = re.compile(r"[0-9a-f]{40}")
def require(condition: bool, message: str) -> None:
if not condition:
raise ValueError(message)
def timestamp(value: object, field: str) -> dt.datetime:
require(isinstance(value, str) and value.endswith("Z"),
f"{field} must be a UTC timestamp ending in Z.")
try:
parsed = dt.datetime.fromisoformat(value.removesuffix("Z") + "+00:00")
except ValueError as error:
raise ValueError(f"{field} is not a valid timestamp.") from error
require(parsed.tzinfo == dt.timezone.utc, f"{field} must be UTC.")
return parsed
def safe_text(value: object, field: str, minimum: int = 2, maximum: int = 160) -> str:
text = str(value or "").strip()
require(minimum <= len(text) <= maximum and "@" not in text and "\\" not in text,
f"{field} must be safe text without an email address or local path.")
return text
def validate_common(record: object, expected_commit: str, expected_release_sha256: str) -> str:
require(isinstance(record, dict), "Acceptance record must be a JSON object.")
require(record.get("schemaVersion") == 1, "Unsupported Debian acceptance schema.")
system = record.get("system")
require(system in SYSTEMS, "Unknown Debian acceptance record system.")
require(record.get("evidenceId") == SYSTEMS[system]["evidenceId"],
f"{system} has the wrong evidenceId.")
require(record.get("releaseVersion") == VERSION,
f"releaseVersion must be {VERSION}.")
require(GIT_SHA.fullmatch(str(expected_commit)) is not None,
"Expected release commit must be a full lowercase Git SHA.")
require(SHA256.fullmatch(str(expected_release_sha256)) is not None,
"Expected release-record checksum must be a lowercase SHA-256 digest.")
require(record.get("releaseCommit") == expected_commit,
"releaseCommit does not match the approved candidate.")
require(record.get("releaseRecordSha256") == expected_release_sha256,
"releaseRecordSha256 does not match the retained release record.")
require(SHA256.fullmatch(str(record.get("archiveSha256", ""))) is not None,
"archiveSha256 must be a lowercase SHA-256 digest.")
origin = urlparse(str(record.get("environment", "")))
require(origin.scheme == "https" and origin.hostname and origin.path in ("", "/")
and not origin.query and not origin.fragment and origin.username is None
and origin.password is None,
"environment must be an HTTPS origin without credentials, path, query or fragment.")
host = safe_text(record.get("hostIdentifier"), "hostIdentifier")
images = record.get("images")
require(isinstance(images, dict) and set(images) == {"api", "worker"},
"images must contain exactly api and worker.")
for name in ("api", "worker"):
image = images[name]
require(isinstance(image, dict) and set(image) == {"reference", "id"},
f"images.{name} must contain exactly reference and id.")
require(image["reference"] == f"guestops-{name}:{expected_commit}",
f"images.{name}.reference must use the full approved commit.")
require(re.fullmatch(r"sha256:[0-9a-f]{64}", str(image["id"])) is not None,
f"images.{name}.id must be an immutable image ID.")
operator = safe_text(record.get("operator"), "operator")
reviewer = safe_text(record.get("reviewedBy"), "reviewedBy")
require(operator.casefold() != reviewer.casefold(),
"operator and reviewedBy must be different people.")
started = timestamp(record.get("startedAt"), "startedAt")
ended = timestamp(record.get("endedAt"), "endedAt")
reviewed = timestamp(record.get("reviewedAt"), "reviewedAt")
require(started <= ended <= reviewed, "Acceptance timestamps are out of order.")
scenarios = record.get("scenarios")
require(isinstance(scenarios, list), "scenarios must be a list.")
ids = [item.get("id") for item in scenarios if isinstance(item, dict)]
required = SYSTEMS[system]["scenarios"]
require(len(ids) == len(scenarios) and len(ids) == len(set(ids)) and set(ids) == required,
f"{system} requires its exact acceptance scenario set.")
for item in scenarios:
scenario_id = item["id"]
require(item.get("status") == "pass", f"Scenario {scenario_id} has not passed.")
evidence = item.get("evidence")
require(isinstance(evidence, list) and 1 <= len(evidence) <= 10 and all(
isinstance(value, str) and 3 <= len(value) <= 200 and "@" not in value
and "\\" not in value and not value.startswith("/") for value in evidence
), f"Scenario {scenario_id} requires safe opaque evidence references.")
require(record.get("unresolvedCriticalFindings") == 0,
"Acceptance cannot pass with unresolved critical findings.")
return host
def validate_host(record: object, expected_commit: str, expected_release_sha256: str) -> str:
host = validate_common(record, expected_commit, expected_release_sha256)
require(record.get("system") == "guestops-debian-host",
"First record must be guestops-debian-host.")
facts = record.get("hostFacts")
require(isinstance(facts, dict) and set(facts) == {
"debianMajor", "cpuCores", "memoryBytes", "freeDiskBytes", "publicTcpPorts",
}, "hostFacts must contain the exact reviewed host facts.")
require(isinstance(facts["debianMajor"], int) and facts["debianMajor"] >= 12,
"Debian 12 or newer is required.")
require(isinstance(facts["cpuCores"], int) and facts["cpuCores"] >= 4,
"At least four CPU cores are required.")
require(isinstance(facts["memoryBytes"], int) and facts["memoryBytes"] >= 7_500_000_000,
"At least 7.5 GB of memory is required.")
require(isinstance(facts["freeDiskBytes"], int) and facts["freeDiskBytes"] >= 8 * 1024**3,
"At least 8 GiB of free disk space is required.")
require(facts["publicTcpPorts"] == [80, 443],
"Only TCP ports 80 and 443 may be public.")
require(record.get("featureControls") == {
"googleSending": "disabled",
"faqLiveMode": "disabled",
"pmsWrites": "disabled",
"paymentCreation": "disabled",
}, "Unaccepted external writes and FAQ live mode must remain disabled.")
return host
def validate_persistence(record: object, expected_commit: str, expected_release_sha256: str) -> str:
host = validate_common(record, expected_commit, expected_release_sha256)
require(record.get("system") == "guestops-persistence",
"Second record must be guestops-persistence.")
require(record.get("drillCommand") == "python3 deploy/ops.py persistence-drill --confirm-restart",
"drillCommand must identify the confirmation-gated persistence drill.")
return host
def validate_pair(host_record: object, persistence_record: object,
expected_commit: str, expected_release_sha256: str) -> None:
host = validate_host(host_record, expected_commit, expected_release_sha256)
persistence_host = validate_persistence(
persistence_record, expected_commit, expected_release_sha256)
require(host == persistence_host, "Both records must identify the same host.")
for field in ("environment", "releaseVersion", "releaseCommit", "releaseRecordSha256",
"archiveSha256", "images"):
require(host_record.get(field) == persistence_record.get(field),
f"Both records must use the same {field}.")
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("host_record", type=Path)
parser.add_argument("persistence_record", type=Path)
parser.add_argument("--expected-commit", required=True)
parser.add_argument("--expected-release-record-sha256", required=True)
args = parser.parse_args()
host_record = json.loads(args.host_record.read_text(encoding="utf-8"))
persistence_record = json.loads(args.persistence_record.read_text(encoding="utf-8"))
validate_pair(host_record, persistence_record,
args.expected_commit, args.expected_release_record_sha256)
print("Debian-host and persistence acceptance records are structurally complete and passed. "
"This validates the records, not their restricted evidence.")
if __name__ == "__main__":
try:
main()
except (OSError, ValueError, json.JSONDecodeError) as error:
print(f"Debian acceptance records rejected: {error}", file=__import__("sys").stderr)
raise SystemExit(1)

189
deploy/monitor_status.py Normal file
View File

@ -0,0 +1,189 @@
#!/usr/bin/env python3
"""Write non-sensitive GuestOps host status for a read-only monitoring agent."""
from __future__ import annotations
import argparse
import datetime as dt
import json
import os
from pathlib import Path
import re
import shutil
import socket
import ssl
import stat
import subprocess
import tempfile
import urllib.request
from urllib.parse import urlparse
BACKUP_NAME = re.compile(r"guestops-[0-9]{8}T[0-9]{6}Z\.tar\.gpg")
MARKER_NAME = re.compile(r"guestops-[0-9]{8}T[0-9]{6}Z\.tar\.gpg\.transferred\.json")
AUTH = ('const c=new Mongo("mongodb://127.0.0.1");'
'c.getDB("admin").auth(process.env.MONGO_INITDB_ROOT_USERNAME,'
'process.env.MONGO_INITDB_ROOT_PASSWORD);')
HEARTBEAT = ('const x=c.getDB("guestops").workerheartbeat.findOne({_id:"worker"});'
'print(JSON.stringify(x&&x.At?x.At:null));')
def require(condition: bool, message: str) -> None:
if not condition:
raise RuntimeError(message)
def run(args: list[str], root: Path, timeout: int = 30) -> bytes:
result = subprocess.run(args, cwd=root, stdin=subprocess.DEVNULL,
stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout)
require(result.returncode == 0, f"{Path(args[0]).name} probe failed.")
return result.stdout
def utc_now() -> dt.datetime:
return dt.datetime.now(dt.timezone.utc)
def age_seconds(path: Path, pattern: re.Pattern[str], now: dt.datetime) -> int | None:
if not path.is_dir() or path.is_symlink():
return None
times = [item.stat().st_mtime for item in path.iterdir()
if item.is_file() and not item.is_symlink() and pattern.fullmatch(item.name)]
return max(0, int(now.timestamp() - max(times))) if times else None
def https_status(origin: str, now: dt.datetime) -> dict[str, object]:
parsed = urlparse(origin)
require(parsed.scheme == "https" and parsed.hostname and parsed.port in (None, 443)
and parsed.path in ("", "/") and not parsed.query and not parsed.fragment
and parsed.username is None and parsed.password is None,
"GUESTOPS_ORIGIN must be an HTTPS origin without credentials or a path.")
context = ssl.create_default_context()
with socket.create_connection((parsed.hostname, 443), timeout=10) as connection:
with context.wrap_socket(connection, server_hostname=parsed.hostname) as secured:
certificate = secured.getpeercert()
expires = dt.datetime.strptime(certificate["notAfter"], "%b %d %H:%M:%S %Y %Z").replace(
tzinfo=dt.timezone.utc)
with urllib.request.urlopen(origin.rstrip("/") + "/health/ready", timeout=15,
context=context) as response:
ready = response.status == 200 and json.load(response) == {"status": "ready"}
return {"ready": ready, "certificateDaysRemaining": max(0, int((expires - now).total_seconds() // 86400))}
def parse_compose(value: bytes) -> dict[str, dict[str, str]]:
text = value.decode("utf-8", "strict").strip()
if not text:
return {}
try:
parsed = json.loads(text)
rows = parsed if isinstance(parsed, list) else [parsed]
except json.JSONDecodeError:
rows = [json.loads(line) for line in text.splitlines() if line.strip()]
result = {}
for row in rows:
if not isinstance(row, dict):
continue
service = str(row.get("Service", ""))
if service in {"api", "worker", "mongo"}:
result[service] = {
"state": str(row.get("State", "unknown")).lower(),
"health": str(row.get("Health", "none") or "none").lower(),
}
return result
def worker_heartbeat_age(root: Path, now: dt.datetime) -> int | None:
output = run(["docker", "compose", "exec", "-T", "mongo", "mongosh", "--quiet",
"--nodb", "--eval", AUTH + HEARTBEAT], root).decode("utf-8", "strict").strip()
value = json.loads(output)
if value is None:
return None
require(isinstance(value, str), "Worker heartbeat response was invalid.")
parsed = dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
require(parsed.tzinfo is not None, "Worker heartbeat must contain a timezone.")
return max(0, int((now - parsed.astimezone(dt.timezone.utc)).total_seconds()))
def build_status(root: Path, backup_directory: Path, origin: str,
now: dt.datetime | None = None) -> tuple[dict[str, object], list[str]]:
moment = now or utc_now()
errors: list[str] = []
try:
https = https_status(origin, moment)
except Exception:
https = {"ready": False, "certificateDaysRemaining": None}
errors.append("https-probe-failed")
try:
containers = parse_compose(run(["docker", "compose", "ps", "--format", "json"], root))
if set(containers) != {"api", "worker", "mongo"}:
errors.append("container-set-incomplete")
except Exception:
containers = {}
errors.append("container-probe-failed")
try:
heartbeat_age = worker_heartbeat_age(root, moment)
if heartbeat_age is None:
errors.append("worker-heartbeat-missing")
except Exception:
heartbeat_age = None
errors.append("worker-heartbeat-probe-failed")
disk = shutil.disk_usage(root)
status = {
"schemaVersion": 1,
"generatedAt": moment.isoformat().replace("+00:00", "Z"),
"https": https,
"containers": containers,
"workerHeartbeatAgeSeconds": heartbeat_age,
"backupAgeSeconds": age_seconds(backup_directory, BACKUP_NAME, moment),
"verifiedTransferAgeSeconds": age_seconds(backup_directory, MARKER_NAME, moment),
"diskFreePercent": round(disk.free * 100 / disk.total, 2),
"persistentJournal": Path("/var/log/journal").is_dir(),
"errors": sorted(set(errors)),
}
return status, errors
def write_status(path: Path, status: dict[str, object]) -> None:
require(path.is_absolute() and not path.is_symlink(),
"Status output must be an absolute, non-symlink path.")
parent = path.parent.resolve()
require(parent.is_dir() and not path.parent.is_symlink(), "Status output directory must exist.")
fd, temporary = tempfile.mkstemp(prefix=path.name + ".", dir=parent)
try:
os.fchmod(fd, 0o644)
with os.fdopen(fd, "w", encoding="utf-8") as stream:
json.dump(status, stream, sort_keys=True, separators=(",", ":"))
stream.write("\n")
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary, path)
finally:
try:
os.unlink(temporary)
except FileNotFoundError:
pass
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--root", type=Path, default=Path(__file__).resolve().parents[1])
parser.add_argument("--backup-directory", type=Path, default=Path("/var/backups/guestops"))
parser.add_argument("--origin", default="https://sandbox-guestops.futuresens.co.uk")
parser.add_argument("--output", type=Path, default=Path("/run/guestops-monitor/status.json"))
args = parser.parse_args()
root = args.root.resolve()
require(root.is_dir(), "GuestOps root must exist.")
backup_directory = args.backup_directory.resolve()
status, errors = build_status(root, backup_directory, args.origin)
write_status(args.output, status)
print("GuestOps monitoring status updated." if not errors
else "GuestOps monitoring status updated with failed probes.")
raise SystemExit(1 if errors else 0)
if __name__ == "__main__":
try:
main()
except (OSError, RuntimeError, ValueError, subprocess.SubprocessError, json.JSONDecodeError) as error:
print(f"GuestOps monitoring probe failed: {error}", file=__import__("sys").stderr)
raise SystemExit(1)

View File

@ -0,0 +1,31 @@
{
"schemaVersion": 1,
"system": "guestops-persistence",
"evidenceId": "persistence",
"releaseVersion": "0.2.0",
"releaseCommit": "0000000000000000000000000000000000000000",
"releaseRecordSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"archiveSha256": "0000000000000000000000000000000000000000000000000000000000000000",
"environment": "https://sandbox-guestops.futuresens.co.uk",
"hostIdentifier": "guestops-sandbox-01",
"images": {
"api": {"reference": "guestops-api:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
"worker": {"reference": "guestops-worker:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"}
},
"drillCommand": "python3 deploy/ops.py persistence-drill --confirm-restart",
"operator": "REPLACE",
"reviewedBy": "REPLACE",
"startedAt": "2026-09-30T09:00:00Z",
"endedAt": "2026-09-30T10:00:00Z",
"reviewedAt": "2026-09-30T11:00:00Z",
"unresolvedCriticalFindings": 1,
"scenarios": [
{"id": "container-recreation", "status": "not-run", "evidence": []},
{"id": "data-protection-key", "status": "not-run", "evidence": []},
{"id": "database-inventory", "status": "not-run", "evidence": []},
{"id": "image-identity", "status": "not-run", "evidence": []},
{"id": "post-reboot-persistence", "status": "not-run", "evidence": []},
{"id": "separate-volume-layout", "status": "not-run", "evidence": []},
{"id": "service-restart", "status": "not-run", "evidence": []}
]
}

View File

@ -0,0 +1,6 @@
BACKUP_DIRECTORY=/var/backups/guestops
BACKUP_REMOTE_HOST=restricted-store.example.invalid
BACKUP_REMOTE_USER=guestops_upload
BACKUP_REMOTE_DIRECTORY=/restricted/guestops/backups
BACKUP_SSH_IDENTITY=/etc/guestops/backup-transfer.key
BACKUP_SSH_KNOWN_HOSTS=/etc/guestops/backup-known-hosts

View File

@ -0,0 +1,2 @@
BACKUP_RECIPIENT=0123456789ABCDEF0123456789ABCDEF01234567
BACKUP_DIRECTORY=/var/backups/guestops

View File

@ -0,0 +1,19 @@
[Unit]
Description=Transfer GuestOps encrypted backups to restricted storage
Wants=network-online.target
After=network-online.target guestops-backup.service
ConditionPathIsDirectory=/srv/guestops
ConditionPathIsDirectory=/var/backups/guestops
[Service]
Type=oneshot
WorkingDirectory=/srv/guestops
EnvironmentFile=/etc/guestops/backup-transfer.env
UMask=0077
ExecStart=/usr/bin/python3 /srv/guestops/deploy/backup_transfer.py --prune-verified --retention-days 7
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=full
ReadWritePaths=/var/backups/guestops
TimeoutStartSec=30min

View File

@ -0,0 +1,11 @@
[Unit]
Description=Retry GuestOps off-host backup transfer
[Timer]
OnBootSec=10min
OnUnitActiveSec=15min
Persistent=true
Unit=guestops-backup-transfer.service
[Install]
WantedBy=timers.target

View File

@ -0,0 +1,21 @@
[Unit]
Description=Write non-sensitive GuestOps monitoring status
Requires=docker.service
After=docker.service network-online.target
ConditionPathIsDirectory=/srv/guestops
ConditionPathIsDirectory=/var/backups/guestops
[Service]
Type=oneshot
WorkingDirectory=/srv/guestops
UMask=0022
ExecStart=/usr/bin/python3 /srv/guestops/deploy/monitor_status.py
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
RuntimeDirectory=guestops-monitor
RuntimeDirectoryMode=0755
RuntimeDirectoryPreserve=yes
ReadWritePaths=/run/guestops-monitor
TimeoutStartSec=2min

View File

@ -0,0 +1,11 @@
[Unit]
Description=Refresh GuestOps monitoring status each minute
[Timer]
OnBootSec=2min
OnUnitActiveSec=1min
Persistent=true
Unit=guestops-monitor-status.service
[Install]
WantedBy=timers.target

View File

@ -0,0 +1,2 @@
# The privileged systemd probe owns Docker/database access. Zabbix reads only this aggregate JSON.
UserParameter=guestops.status,cat /run/guestops-monitor/status.json

View File

@ -86,4 +86,16 @@ The drill restarts MongoDB, API and worker, then force-recreates the stateless a
Record the host, operator, start/end time, release record checksum, resolved image IDs, preflight output and drill result in the deployment acceptance record. Also verify Docker starts at boot and perform a controlled Debian reboot before Gate A approval. After reboot, run the online preflight and inspect the Workspace health page; do not infer worker health solely from API readiness.
Keep the `debian-host` and `persistence` records in the restricted evidence store. Start from `deploy/debian-host-acceptance.example.json` and `deploy/persistence-acceptance.example.json`; the examples deliberately fail until every supervised scenario has passed. Bind both records to the expected release identifiers and validate them together:
```sh
python3 deploy/debian_acceptance.py \
/secure/acceptance/debian-host.json \
/secure/acceptance/persistence.json \
--expected-commit FULL_40_CHARACTER_SHA \
--expected-release-record-sha256 RELEASE_RECORD_SHA256
```
The validator requires matching archive and image identities, separate operator and reviewer names, the approved host capacity, only ports 80 and 443 recorded as publicly reachable, disabled unaccepted external writes, exact passed scenario sets and no unresolved critical findings. It validates record structure, not the restricted evidence itself. Retain the records, validator output and their checksums outside Git.
The supplied Docker `json-file` logs are size-capped to protect the small pilot disk, but container recreation removes that container's local log history. Before host acceptance, route GuestOps and Nginx logs to the site's durable restricted logging system, or use a reviewed Docker logging override backed by persistent systemd journal storage. Prove that operators can retrieve pre-recreation logs without exposing request credentials or OAuth callback query strings. Central retention and alerting are completed under milestone 11.

View File

@ -113,6 +113,43 @@ systemctl list-timers guestops-backup.timer
The timer deliberately causes the same brief maintenance interruption as a manual backup. `Persistent=true` runs a missed event after downtime, so choose and communicate the maintenance window. A successful unit only stages an encrypted file locally. Configure an independently monitored off-host transfer, verify the destination checksum, alert on both unit and transfer failure, and test the alert route. Do not add automatic deletion until retention, legal hold and recovery requirements have named owners.
### Restricted off-host transfer
The optional transfer service uses rsync over pinned-host SSH. Create a dedicated upload-only account at the restricted store, disable interactive login, agent/port forwarding and access outside the GuestOps backup directory, and keep its private key only in `/etc/guestops` with mode 600. Pin the reviewed server host key; never use `StrictHostKeyChecking=no`.
Start from `deploy/systemd/backup-transfer.env.example` and store the completed file as `/etc/guestops/backup-transfer.env` with mode 600. The local and remote directories must already exist and remain private. Install and verify the service and its 15-minute retry timer:
```sh
sudo install -m 600 deploy/systemd/backup-transfer.env.example /etc/guestops/backup-transfer.env
sudoedit /etc/guestops/backup-transfer.env
sudo install -m 644 deploy/systemd/guestops-backup-transfer.service /etc/systemd/system/
sudo install -m 644 deploy/systemd/guestops-backup-transfer.timer /etc/systemd/system/
sudo systemd-analyze verify /etc/systemd/system/guestops-backup-transfer.service /etc/systemd/system/guestops-backup-transfer.timer
sudo systemctl daemon-reload
sudo systemctl start guestops-backup-transfer.service
sudo systemctl enable --now guestops-backup-transfer.timer
```
Only files named `guestops-YYYYMMDDTHHMMSSZ.tar.gpg` are eligible. The transfer writes a unique remote partial file, compares the remote and local SHA-256 values, atomically publishes a previously unused final name, verifies it again, and then writes a non-sensitive local marker. An existing remote name is accepted only when its checksum matches. Failed or mismatched transfers are never marked. The service removes local backups older than seven days only when the marker still matches the local checksum; untransferred or changed files are never pruned.
The restricted store is the durable copy. Configure its independently reviewed retention policy for 35 daily and 12 monthly recovery points. Legal hold must override expiry. The host tool does not delete remote data or enforce remote retention.
### Zabbix status boundary
Do not give the Zabbix agent access to Docker, MongoDB credentials or the application owner session. A root-owned systemd probe reads those local sources and atomically publishes aggregate status under `/run/guestops-monitor/status.json`; the agent reads that file only.
```sh
sudo install -m 644 deploy/systemd/guestops-monitor-status.service /etc/systemd/system/
sudo install -m 644 deploy/systemd/guestops-monitor-status.timer /etc/systemd/system/
sudo install -m 644 deploy/systemd/zabbix-agent-guestops.conf.example /etc/zabbix/zabbix_agentd.d/guestops.conf
sudo systemd-analyze verify /etc/systemd/system/guestops-monitor-status.service /etc/systemd/system/guestops-monitor-status.timer
sudo systemctl daemon-reload
sudo systemctl enable --now guestops-monitor-status.timer
sudo systemctl restart zabbix-agent
```
Create dependent Zabbix items from `guestops.status` for HTTPS readiness, certificate days remaining, container state/health, worker heartbeat age, backup age, verified-transfer age, free-disk percentage, persistent journal availability and probe error categories. Alert when the heartbeat is older than three minutes; backup or transfer is older than 30 hours; free disk falls below 25% (warning) or 15% (critical); the certificate has fewer than 30 days (warning) or 14 days (critical); any required container is absent/unhealthy; or the probe/timers fail. Route alerts to the monitoring owner and escalate unacknowledged critical events to the technical owner after 15 minutes. Exercise every trigger and its recovery notification with synthetic conditions.
Temporary plaintext files are held in private directories and removed on normal completion or exceptions. Process termination or power loss can leave temporary data, stopped services or TTL expiry disabled. After an interrupted run, inspect the dedicated project and remove only its identified abandoned temporary directory after securing any recovery material. Restore the recorded TTL setting (normally true) and restart the services:
```sh
@ -146,4 +183,20 @@ Restore into new isolated MongoDB and key volumes; preserve the damaged original
**A restored database can predate emails, invoices and PMS changes that providers already completed.** Review pending, sending and uncertain records against provider evidence before enabling any worker, including automatic FAQ rules. Do not replay an older approval merely because the restored record says it is pending. Reconcile external effects, validate account sessions and mailbox authorization, and explicitly approve the cutover only after these checks. Rotate credentials if compromise prompted the recovery. Keep the old deployment stopped when enabling the replacement.
CI exercises a synthetic encrypted backup and isolated restore drill, including actual key decryption and database comparison. A successful CI drill is separate from the required rehearsal on the Debian server with its actual deployment configuration.
## Milestone 11 acceptance record
Keep raw backups, restored data, remote paths, host keys, monitoring recipients, screenshots and logs outside Git. Copy `deploy/backup-restore-acceptance.example.json` to the restricted evidence store and replace every placeholder only after completing the supervised exercises. The example deliberately fails.
The accepted record uses a 24-hour RPO, four-hour RTO, seven-day verified local staging window, 35 daily and 12 monthly off-host recovery points, and the opaque evidence ID `backup-restore`. Bind it to the same release identifiers as the Debian-host and persistence records:
```sh
python3 deploy/backup_restore_acceptance.py \
/secure/acceptance/backup-restore.json \
--expected-commit FULL_40_CHARACTER_SHA \
--expected-release-record-sha256 RELEASE_RECORD_SHA256
sha256sum /secure/acceptance/backup-restore.json
```
The validator requires matching local/remote backup checksums, exact immutable image identities, a timed isolated restore, tested retention/legal hold and alert escalation, an image rollback that preserves persistent volumes, return to the approved candidate with external writes disabled, separate independent review, healthy monitoring and no unresolved critical findings. Structural validation does not inspect the restricted evidence or authorize a release by itself.

View File

@ -0,0 +1,129 @@
import importlib.util
from pathlib import Path
import unittest
spec = importlib.util.spec_from_file_location(
"backup_restore_acceptance",
Path(__file__).resolve().parents[1] / "deploy" / "backup_restore_acceptance.py")
acceptance = importlib.util.module_from_spec(spec)
spec.loader.exec_module(acceptance)
COMMIT = "a" * 40
RELEASE_SHA = "b" * 64
def valid_record():
return {
"schemaVersion": 1,
"system": "guestops-backup-recovery",
"evidenceId": "backup-restore",
"dataClassification": "synthetic-only",
"releaseVersion": "0.2.0",
"releaseCommit": COMMIT,
"releaseRecordSha256": RELEASE_SHA,
"archiveSha256": "c" * 64,
"environment": "https://sandbox-guestops.futuresens.co.uk",
"hostIdentifier": "guestops-sandbox-01",
"images": {
"api": {"reference": f"guestops-api:{COMMIT}", "id": "sha256:" + "d" * 64},
"worker": {"reference": f"guestops-worker:{COMMIT}", "id": "sha256:" + "e" * 64},
"mongo": {"reference": "mongo:8.0", "id": "sha256:" + "f" * 64},
},
"owners": {
"backup": "Backup owner",
"monitoring": "Monitoring owner",
"recoveryOperator": "Recovery operator",
"technicalEscalation": "Technical owner",
"retention": "Retention owner",
"independentReviewer": "Independent reviewer",
},
"startedAt": "2026-10-01T09:00:00Z",
"endedAt": "2026-10-01T12:00:00Z",
"reviewedAt": "2026-10-01T13:00:00Z",
"recoveryObjectives": {
"targetRpoHours": 24, "observedRpoHours": 1,
"targetRtoMinutes": 240, "observedRtoMinutes": 180,
},
"retention": {
"localVerifiedDays": 7, "offHostDaily": 35,
"offHostMonthly": 12, "legalHoldOverrideTested": True,
},
"backup": {
"createdAt": "2026-10-01T08:00:00Z",
"sha256": "1" * 64,
"transferredSha256": "1" * 64,
"privateKeyPresentOnHost": False,
},
"rollback": {
"previousReleaseCommit": "2" * 40,
"previousArchiveSha256": "3" * 64,
"previousImages": {
"api": {"reference": "guestops-api:" + "2" * 40, "id": "sha256:" + "4" * 64},
"worker": {"reference": "guestops-worker:" + "2" * 40, "id": "sha256:" + "5" * 64},
},
"persistentVolumesReplaced": False,
"restoredReleaseCommit": COMMIT,
"unresolvedOperations": 0,
"finalControls": {
"googleSending": "disabled", "faqLiveMode": "disabled",
"pmsWrites": "disabled", "paymentCreation": "disabled",
},
},
"monitoringState": "healthy",
"unresolvedCriticalFindings": 0,
"scenarios": [
{"id": scenario, "status": "pass", "evidence": [f"restricted-{index}"]}
for index, scenario in enumerate(sorted(acceptance.SCENARIOS), 1)
],
}
class BackupRestoreAcceptanceTests(unittest.TestCase):
def test_complete_record_passes(self):
acceptance.validate(valid_record(), COMMIT, RELEASE_SHA)
def test_release_identity_and_images_are_bound(self):
record = valid_record(); record["releaseCommit"] = "9" * 40
with self.assertRaisesRegex(ValueError, "approved candidate"):
acceptance.validate(record, COMMIT, RELEASE_SHA)
record = valid_record(); record["images"]["api"]["reference"] = "guestops-api:latest"
with self.assertRaisesRegex(ValueError, "full approved commit"):
acceptance.validate(record, COMMIT, RELEASE_SHA)
record = valid_record(); record["rollback"]["previousImages"]["worker"]["id"] = "mutable"
with self.assertRaisesRegex(ValueError, "retained previous release identity"):
acceptance.validate(record, COMMIT, RELEASE_SHA)
def test_recovery_objectives_and_checksums_must_pass(self):
record = valid_record(); record["recoveryObjectives"]["observedRtoMinutes"] = 241
with self.assertRaisesRegex(ValueError, "four-hour"):
acceptance.validate(record, COMMIT, RELEASE_SHA)
record = valid_record(); record["backup"]["transferredSha256"] = "9" * 64
with self.assertRaisesRegex(ValueError, "checksums must match"):
acceptance.validate(record, COMMIT, RELEASE_SHA)
record = valid_record(); record["recoveryObjectives"]["observedRpoHours"] = 2
with self.assertRaisesRegex(ValueError, "match the backup"):
acceptance.validate(record, COMMIT, RELEASE_SHA)
def test_independent_review_retention_and_final_state_are_required(self):
record = valid_record(); record["owners"]["independentReviewer"] = record["owners"]["backup"]
with self.assertRaisesRegex(ValueError, "different"):
acceptance.validate(record, COMMIT, RELEASE_SHA)
record = valid_record(); record["retention"]["legalHoldOverrideTested"] = False
with self.assertRaisesRegex(ValueError, "Retention"):
acceptance.validate(record, COMMIT, RELEASE_SHA)
record = valid_record(); record["rollback"]["persistentVolumesReplaced"] = True
with self.assertRaisesRegex(ValueError, "must not replace"):
acceptance.validate(record, COMMIT, RELEASE_SHA)
def test_exact_passed_scenarios_and_monitoring_are_required(self):
record = valid_record(); record["scenarios"].pop()
with self.assertRaisesRegex(ValueError, "exact backup/recovery scenario"):
acceptance.validate(record, COMMIT, RELEASE_SHA)
record = valid_record(); record["monitoringState"] = "degraded"
with self.assertRaisesRegex(ValueError, "Monitoring must be healthy"):
acceptance.validate(record, COMMIT, RELEASE_SHA)
if __name__ == "__main__":
unittest.main()

View File

@ -0,0 +1,81 @@
import importlib.util
import json
import os
from pathlib import Path
import tempfile
import time
import unittest
from unittest.mock import patch
spec = importlib.util.spec_from_file_location(
"backup_transfer", Path(__file__).resolve().parents[1] / "deploy" / "backup_transfer.py")
transfer = importlib.util.module_from_spec(spec)
spec.loader.exec_module(transfer)
class BackupTransferTests(unittest.TestCase):
def backup(self, directory: Path, name="guestops-20261001T021700Z.tar.gpg") -> Path:
path = directory / name
path.write_bytes(b"encrypted-backup")
return path
def config(self, directory: Path):
return {
"directory": directory, "host": "store.example.invalid", "user": "guestops_upload",
"remote": "/restricted/guestops", "identity": Path("/safe/key"),
"known_hosts": Path("/safe/known_hosts"),
}
def test_existing_matching_remote_is_marked_without_upload(self):
with tempfile.TemporaryDirectory() as folder:
backup = self.backup(Path(folder)); checksum = transfer.digest(backup)
with patch.object(transfer, "remote_digest", return_value=checksum), \
patch.object(transfer, "run") as run:
transfer.transfer_one(self.config(Path(folder)), backup)
run.assert_not_called()
marker = json.loads(transfer.marker_path(backup).read_text(encoding="utf-8"))
self.assertEqual(marker["sha256"], checksum)
def test_existing_mismatched_remote_is_never_overwritten(self):
with tempfile.TemporaryDirectory() as folder:
backup = self.backup(Path(folder))
with patch.object(transfer, "remote_digest", return_value="9" * 64), \
patch.object(transfer, "run") as run:
with self.assertRaisesRegex(RuntimeError, "different checksum"):
transfer.transfer_one(self.config(Path(folder)), backup)
run.assert_not_called()
self.assertFalse(transfer.marker_path(backup).exists())
def test_new_remote_is_uploaded_verified_and_marked(self):
with tempfile.TemporaryDirectory() as folder:
backup = self.backup(Path(folder)); checksum = transfer.digest(backup)
with patch.object(transfer, "remote_digest", side_effect=[None, checksum, checksum]), \
patch.object(transfer, "run", return_value=b"") as run, \
patch.object(transfer.subprocess, "run"):
transfer.transfer_one(self.config(Path(folder)), backup)
self.assertTrue(any(call.args[0][0] == "rsync" for call in run.call_args_list))
self.assertTrue(transfer.marker_path(backup).exists())
def test_prune_removes_only_old_checksum_verified_backups(self):
with tempfile.TemporaryDirectory() as folder:
directory = Path(folder)
verified = self.backup(directory)
checksum = transfer.digest(verified)
transfer.write_marker(verified, checksum)
unverified = self.backup(directory, "guestops-20261002T021700Z.tar.gpg")
old = time.time() - 8 * 86400
os.utime(verified, (old, old)); os.utime(unverified, (old, old))
removed = transfer.prune_verified(self.config(directory), 7, now=time.time())
self.assertEqual(removed, 1)
self.assertFalse(verified.exists())
self.assertTrue(unverified.exists())
def test_retention_bounds_are_enforced(self):
with tempfile.TemporaryDirectory() as folder:
with self.assertRaisesRegex(RuntimeError, "between 1 and 365"):
transfer.prune_verified(self.config(Path(folder)), 0)
if __name__ == "__main__":
unittest.main()

View File

@ -0,0 +1,113 @@
import importlib.util
from pathlib import Path
import unittest
spec = importlib.util.spec_from_file_location(
"debian_acceptance", Path(__file__).resolve().parents[1] / "deploy" / "debian_acceptance.py")
acceptance = importlib.util.module_from_spec(spec)
spec.loader.exec_module(acceptance)
COMMIT = "a" * 40
RELEASE_SHA = "b" * 64
def common(system):
return {
"schemaVersion": 1,
"system": system,
"evidenceId": acceptance.SYSTEMS[system]["evidenceId"],
"releaseVersion": "0.2.0",
"releaseCommit": COMMIT,
"releaseRecordSha256": RELEASE_SHA,
"archiveSha256": "c" * 64,
"environment": "https://sandbox-guestops.futuresens.co.uk",
"hostIdentifier": "guestops-sandbox-01",
"images": {
"api": {"reference": f"guestops-api:{COMMIT}", "id": "sha256:" + "d" * 64},
"worker": {"reference": f"guestops-worker:{COMMIT}", "id": "sha256:" + "e" * 64},
},
"operator": "Deployment operator",
"reviewedBy": "Independent reviewer",
"startedAt": "2026-09-30T09:00:00Z",
"endedAt": "2026-09-30T10:00:00Z",
"reviewedAt": "2026-09-30T11:00:00Z",
"unresolvedCriticalFindings": 0,
"scenarios": [
{"id": scenario, "status": "pass", "evidence": [f"restricted-{index}"]}
for index, scenario in enumerate(sorted(acceptance.SYSTEMS[system]["scenarios"]), 1)
],
}
def valid_records():
host = common("guestops-debian-host")
host["hostFacts"] = {
"debianMajor": 12,
"cpuCores": 4,
"memoryBytes": 8_140_382_208,
"freeDiskBytes": 14_275_686_400,
"publicTcpPorts": [80, 443],
}
host["featureControls"] = {
"googleSending": "disabled",
"faqLiveMode": "disabled",
"pmsWrites": "disabled",
"paymentCreation": "disabled",
}
persistence = common("guestops-persistence")
persistence["drillCommand"] = "python3 deploy/ops.py persistence-drill --confirm-restart"
return host, persistence
class DebianAcceptanceTests(unittest.TestCase):
def test_complete_matching_records_pass(self):
acceptance.validate_pair(*valid_records(), COMMIT, RELEASE_SHA)
def test_expected_release_identity_is_required(self):
host, persistence = valid_records()
host["releaseCommit"] = "f" * 40
with self.assertRaisesRegex(ValueError, "approved candidate"):
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
host, persistence = valid_records()
persistence["releaseRecordSha256"] = "f" * 64
with self.assertRaisesRegex(ValueError, "retained release record"):
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
def test_exact_images_and_cross_record_identity_are_required(self):
host, persistence = valid_records()
host["images"]["api"]["reference"] = "guestops-api:latest"
with self.assertRaisesRegex(ValueError, "full approved commit"):
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
host, persistence = valid_records()
persistence["archiveSha256"] = "f" * 64
with self.assertRaisesRegex(ValueError, "same archiveSha256"):
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
def test_host_capacity_ports_and_disabled_controls_are_required(self):
host, persistence = valid_records()
host["hostFacts"]["publicTcpPorts"] = [80, 443, 8080]
with self.assertRaisesRegex(ValueError, "Only TCP ports"):
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
host, persistence = valid_records()
host["featureControls"]["googleSending"] = "enabled"
with self.assertRaisesRegex(ValueError, "must remain disabled"):
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
def test_independent_review_scenarios_and_findings_are_required(self):
host, persistence = valid_records()
host["reviewedBy"] = host["operator"]
with self.assertRaisesRegex(ValueError, "different people"):
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
host, persistence = valid_records()
persistence["scenarios"][0]["status"] = "not-run"
with self.assertRaisesRegex(ValueError, "has not passed"):
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
host, persistence = valid_records()
host["unresolvedCriticalFindings"] = 1
with self.assertRaisesRegex(ValueError, "critical findings"):
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
if __name__ == "__main__":
unittest.main()

View File

@ -0,0 +1,75 @@
import datetime as dt
import importlib.util
import json
import os
from pathlib import Path
import tempfile
import unittest
from unittest.mock import patch
spec = importlib.util.spec_from_file_location(
"monitor_status", Path(__file__).resolve().parents[1] / "deploy" / "monitor_status.py")
monitor = importlib.util.module_from_spec(spec)
spec.loader.exec_module(monitor)
class MonitorStatusTests(unittest.TestCase):
def test_compose_json_is_reduced_to_safe_state(self):
value = json.dumps([
{"Service": "api", "State": "running", "Health": "healthy", "Publishers": "secret"},
{"Service": "worker", "State": "running", "Health": ""},
{"Service": "mongo", "State": "running", "Health": "healthy"},
]).encode()
self.assertEqual(monitor.parse_compose(value)["api"], {"state": "running", "health": "healthy"})
self.assertNotIn("Publishers", monitor.parse_compose(value)["api"])
def test_age_ignores_symlinks_and_unexpected_files(self):
with tempfile.TemporaryDirectory() as folder:
directory = Path(folder)
backup = directory / "guestops-20261001T021700Z.tar.gpg"
backup.write_bytes(b"fixture")
moment = dt.datetime(2026, 10, 1, 3, 17, tzinfo=dt.timezone.utc)
os.utime(backup, (moment.timestamp() - 3600, moment.timestamp() - 3600))
(directory / "secret.txt").write_text("ignored", encoding="utf-8")
self.assertEqual(monitor.age_seconds(directory, monitor.BACKUP_NAME, moment), 3600)
def test_build_status_contains_only_aggregate_health(self):
now = dt.datetime(2026, 10, 1, 12, tzinfo=dt.timezone.utc)
compose = json.dumps([
{"Service": name, "State": "running", "Health": "healthy"}
for name in ("api", "worker", "mongo")
]).encode()
usage = type("Usage", (), {"total": 1000, "used": 500, "free": 500})()
with tempfile.TemporaryDirectory() as folder, \
patch.object(monitor, "https_status", return_value={"ready": True, "certificateDaysRemaining": 60}), \
patch.object(monitor, "run", return_value=compose), \
patch.object(monitor, "worker_heartbeat_age", return_value=30), \
patch.object(monitor.shutil, "disk_usage", return_value=usage):
status, errors = monitor.build_status(Path(folder), Path(folder), "https://example.invalid", now)
self.assertEqual(errors, [])
self.assertEqual(status["workerHeartbeatAgeSeconds"], 30)
self.assertEqual(status["diskFreePercent"], 50)
self.assertNotIn("credentials", json.dumps(status).lower())
def test_failed_probes_write_categories_not_exception_details(self):
now = dt.datetime(2026, 10, 1, 12, tzinfo=dt.timezone.utc)
usage = type("Usage", (), {"total": 1000, "used": 500, "free": 500})()
with tempfile.TemporaryDirectory() as folder, \
patch.object(monitor, "https_status", side_effect=RuntimeError("secret value")), \
patch.object(monitor, "run", side_effect=RuntimeError("secret value")), \
patch.object(monitor, "worker_heartbeat_age", side_effect=RuntimeError("secret value")), \
patch.object(monitor.shutil, "disk_usage", return_value=usage):
status, errors = monitor.build_status(Path(folder), Path(folder), "https://example.invalid", now)
self.assertGreaterEqual(len(errors), 3)
self.assertNotIn("secret value", json.dumps(status))
def test_status_output_is_atomic_json(self):
with tempfile.TemporaryDirectory() as folder:
path = Path(folder) / "status.json"
monitor.write_status(path, {"schemaVersion": 1, "errors": []})
self.assertEqual(json.loads(path.read_text(encoding="utf-8"))["schemaVersion"], 1)
if __name__ == "__main__":
unittest.main()