Gate B release candidate
Some checks failed
Build and verify web migration / verify (push) Has been cancelled
Some checks failed
Build and verify web migration / verify (push) Has been cancelled
This commit is contained in:
parent
f11a21aae8
commit
c41b937acb
@ -1,10 +1,36 @@
|
||||
# GuestOps Milestone Report
|
||||
|
||||
Version: **0.2.0 release candidate**
|
||||
Last updated: **29 September 2026**
|
||||
Last updated: **30 September 2026**
|
||||
|
||||
This is the working delivery tracker for GuestOps Web. Update a milestone when its state changes and link the pull request, release artifact, test run, or acceptance record that proves the change.
|
||||
|
||||
## Milestones at a glance
|
||||
|
||||
This summary explains what each milestone delivers and where it currently stands. The release-gate and delivery tables below contain the detailed evidence and exit conditions.
|
||||
|
||||
| # | Milestone | What it delivers | Current position |
|
||||
| ---: | --- | --- | --- |
|
||||
| 1 | Web foundation | The React application, ASP.NET Core API, tenant-isolated MongoDB storage, preview mode, and container foundation. | **Implemented.** The application foundation and automated tests are on `main`. |
|
||||
| 2 | AI suggestions and reviewed Gmail sending | AI-assisted reply drafts and staff-reviewed Gmail delivery with safety and duplicate-send controls. | **Implemented; acceptance required.** Real Gmail threading, revocation, uncertain-send, and staff-review scenarios still need live evidence. |
|
||||
| 3 | OHIP PMS workflow | Internal proposal, approval, and execution controls for PMS operations. | **Implemented; acceptance required.** The workflow exists, but provider-contract and sandbox acceptance remain outstanding. |
|
||||
| 4 | NMI payment workflow | Internal payment proposal, approval, status, expiry, and reconciliation controls. | **Implemented; acceptance required.** The workflow exists, but real payment-provider sandbox acceptance remains outstanding. |
|
||||
| 5 | FAQ automation | Knowledge-based FAQ drafting, test mode, approval controls, and guarded live automation. | **Implemented; acceptance required.** Live mode remains disabled pending quality, false-positive, monitoring, and rollback acceptance. |
|
||||
| 6 | Team onboarding and account recovery | Staff invitations, password setup/reset, access disable/restore, and administrator recovery. | **Implemented; acceptance required.** Deployed link delivery, expiry, recovery, and administrator procedures still need operational evidence. |
|
||||
| 7 | Google connection recovery | OAuth reconnect, checkpoint recovery, grant revocation handling, and worker restart safety. | **Implemented; acceptance required.** Dedicated Google-account and worker-restart exercises have not yet been accepted. |
|
||||
| 8 | Operational readiness tooling | Release verification, diagnostics, encrypted backup, restore, preflight, and persistence tools. | **Implemented; acceptance required.** The tools must still be run against the exact release on the Debian host. |
|
||||
| 9 | Gitea and reproducible releases | CI-built immutable images, checksummed release records, retained artifacts, and approval tagging. | **In progress.** The `0.2.0` candidate is frozen; successful exact-commit CI evidence and the post-Gate-B tag are still required. |
|
||||
| 10 | Debian deployment and persistence | Secure Debian/Compose deployment, HTTPS, persistent database and key volumes, and reboot/recreation proof. | **In progress.** Acceptance tooling is ready, but Docker, correct HTTPS/network exposure, exact artifacts, privileged installation, and the supervised drills remain open. |
|
||||
| 11 | Backups, monitoring, and recovery | Scheduled encrypted backups, verified off-host transfer, Zabbix monitoring, restore, and rollback rehearsal. | **In progress.** Repository tooling is ready; installation and timed operational evidence are blocked until Milestones 9 and 10 pass. |
|
||||
| 12 | Google mailbox acceptance | End-to-end Gmail consent, import, recovery, reviewed sending, reconciliation, and revocation evidence. | **In progress.** The runbook and validator exist; the live synthetic-data exercise and independent review remain outstanding. |
|
||||
| 13 | Rezlynx/Guestline adapter | The real PMS provider adapter, mappings, idempotency, reconciliation, and ambiguous-write handling. | **Planned.** Provider contract and sandbox access are still required before implementation and acceptance. |
|
||||
| 14 | Payment links and status | The real payment-provider integration, webhooks, expiry, replay protection, and reconciliation. | **Planned.** The provider path and sandbox acceptance plan still need to be confirmed and completed. |
|
||||
| 15 | Knowledge, AI, and FAQ activation | Supervised knowledge-quality, AI-draft, FAQ test-mode, staff-training, and stop-control acceptance. | **Implemented; acceptance required.** The evaluation tooling exists; the supervised evaluation and independent approval remain outstanding. |
|
||||
| 16 | Identity, preferences, and privacy | Account/session controls, hotel preferences, privacy inventory, retention decisions, and audit review. | **Implemented; acceptance required.** Legal and operational decisions, identity checks, and independent review remain outstanding. |
|
||||
| 17 | Inbox usability and desktop parity | Stable pagination, protected unsaved drafts, hotel-timezone display, and desktop workflow parity. | **Implemented; acceptance required.** Automated checks pass; the supervised desktop exercise and independent approval remain outstanding. |
|
||||
| 18 | Pilot, capacity, and release approval | Capacity proof, incident exercise, five-business-day hotel pilot, findings closure, and Gate B approval. | **In progress.** Validators and targets exist; Gate A/B prerequisites, capacity evidence, incident rehearsal, pilot, and named approvals remain open. |
|
||||
| 19 | Account security and self-service | TOTP MFA, recovery codes, transactional email, granular roles, preferences, and security notifications. | **Implemented on the development branch; acceptance required.** Keep it separate until `0.2.0` is approved and tagged, then review, merge, and version it as `0.3.0`. |
|
||||
|
||||
## Status key
|
||||
|
||||
- **Implemented** — present on `main` and supported by code or automated-test evidence.
|
||||
@ -35,7 +61,7 @@ This is the working delivery tracker for GuestOps Web. Update a milestone when i
|
||||
| 8 | Operational readiness tooling | A | Implemented / acceptance required | Backup, restore, release, and diagnostic tooling is promoted to local `main`; execute it on the actual Debian host and retain evidence. |
|
||||
| 9 | Gitea and reproducible releases | A | In progress | The `0.2.0` candidate is versioned on `main`. CI records the full commit, matched application version, archive checksum and immutable image IDs, and the rollback procedure is documented. Retain the successful default-branch evidence off-host and create the immutable approval tag only after Gate B approval; the existing `0.1.0` tag remains attached to the foundation release. |
|
||||
| 10 | Debian deployment and persistence | A | In progress | Compose uses separate named database and shared key volumes, private host configuration, loopback-only API access and bounded logs. The confirmation-gated persistence drill verifies restart and container-recreation behaviour. Run it on the provisioned Debian host, complete HTTPS and controlled-reboot acceptance, and retain the evidence. |
|
||||
| 11 | Backups, monitoring, and recovery | A | In progress | Encrypted backup and isolated restore tooling now includes opt-in systemd scheduling without command-line secrets. Install and test it on Debian, configure monitored off-host transfer and durable logs, name alert/retention owners, and retain evidence from a timed restore and recovery drill. |
|
||||
| 11 | Backups, monitoring, and recovery | A | In progress | Encrypted backup and isolated restore tooling now includes opt-in systemd scheduling, checksum-verified rsync transfer, a restricted Zabbix status boundary, guarded local retention and a release-bound acceptance validator. Install and test it on Debian, configure the restricted store and alerts, name operational/review owners, and retain independently reviewed evidence from the timed restore and rollback drill. |
|
||||
| 12 | Google mailbox and reviewed-reply acceptance | B | In progress | The synthetic-data provider runbook, exact scenario set and restricted-record validator are implemented. Complete every scenario against the accepted Debian release and dedicated Google sandbox accounts, independently review the evidence, and retain the validated record. |
|
||||
| 13 | Rezlynx/Guestline adapter | C | Planned | Obtain the provider contract and sandbox, implement the adapter and mapping, and accept idempotency, stale-data, ambiguous-write, and reconciliation paths. |
|
||||
| 14 | Payment links and status | C | Planned | Select/confirm the payment-provider path, complete sandbox and webhook acceptance, and prove expiry, replay protection, reconciliation, and support recovery. |
|
||||
|
||||
81
deploy/backup-restore-acceptance.example.json
Normal file
81
deploy/backup-restore-acceptance.example.json
Normal file
@ -0,0 +1,81 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"system": "guestops-backup-recovery",
|
||||
"evidenceId": "backup-restore",
|
||||
"dataClassification": "synthetic-only",
|
||||
"releaseVersion": "0.2.0",
|
||||
"releaseCommit": "0000000000000000000000000000000000000000",
|
||||
"releaseRecordSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"archiveSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"environment": "https://sandbox-guestops.futuresens.co.uk",
|
||||
"hostIdentifier": "guestops-sandbox-01",
|
||||
"images": {
|
||||
"api": {"reference": "guestops-api:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
|
||||
"worker": {"reference": "guestops-worker:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
|
||||
"mongo": {"reference": "mongo:8.0", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"}
|
||||
},
|
||||
"owners": {
|
||||
"backup": "REPLACE",
|
||||
"monitoring": "REPLACE",
|
||||
"recoveryOperator": "REPLACE",
|
||||
"technicalEscalation": "REPLACE",
|
||||
"retention": "REPLACE",
|
||||
"independentReviewer": "REPLACE"
|
||||
},
|
||||
"startedAt": "2026-10-01T09:00:00Z",
|
||||
"endedAt": "2026-10-01T13:00:00Z",
|
||||
"reviewedAt": "2026-10-01T14:00:00Z",
|
||||
"recoveryObjectives": {
|
||||
"targetRpoHours": 24,
|
||||
"observedRpoHours": 25,
|
||||
"targetRtoMinutes": 240,
|
||||
"observedRtoMinutes": 241
|
||||
},
|
||||
"retention": {
|
||||
"localVerifiedDays": 7,
|
||||
"offHostDaily": 35,
|
||||
"offHostMonthly": 12,
|
||||
"legalHoldOverrideTested": false
|
||||
},
|
||||
"backup": {
|
||||
"createdAt": "2026-10-01T08:00:00Z",
|
||||
"sha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"transferredSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"privateKeyPresentOnHost": false
|
||||
},
|
||||
"rollback": {
|
||||
"previousReleaseCommit": "1111111111111111111111111111111111111111",
|
||||
"previousArchiveSha256": "1111111111111111111111111111111111111111111111111111111111111111",
|
||||
"previousImages": {
|
||||
"api": {"reference": "guestops-api:1111111111111111111111111111111111111111", "id": "sha256:1111111111111111111111111111111111111111111111111111111111111111"},
|
||||
"worker": {"reference": "guestops-worker:1111111111111111111111111111111111111111", "id": "sha256:1111111111111111111111111111111111111111111111111111111111111111"}
|
||||
},
|
||||
"persistentVolumesReplaced": false,
|
||||
"restoredReleaseCommit": "0000000000000000000000000000000000000000",
|
||||
"unresolvedOperations": 1,
|
||||
"finalControls": {
|
||||
"googleSending": "disabled",
|
||||
"faqLiveMode": "disabled",
|
||||
"pmsWrites": "disabled",
|
||||
"paymentCreation": "disabled"
|
||||
}
|
||||
},
|
||||
"monitoringState": "not-configured",
|
||||
"unresolvedCriticalFindings": 1,
|
||||
"scenarios": [
|
||||
{"id": "alert-escalation", "status": "not-run", "evidence": []},
|
||||
{"id": "atomic-off-host-transfer", "status": "not-run", "evidence": []},
|
||||
{"id": "controlled-return-to-service", "status": "not-run", "evidence": []},
|
||||
{"id": "data-protection-recovery", "status": "not-run", "evidence": []},
|
||||
{"id": "database-inventory", "status": "not-run", "evidence": []},
|
||||
{"id": "durable-secret-free-logs", "status": "not-run", "evidence": []},
|
||||
{"id": "encrypted-manual-backup", "status": "not-run", "evidence": []},
|
||||
{"id": "image-rollback", "status": "not-run", "evidence": []},
|
||||
{"id": "isolated-restore", "status": "not-run", "evidence": []},
|
||||
{"id": "monitoring-coverage", "status": "not-run", "evidence": []},
|
||||
{"id": "off-host-checksum", "status": "not-run", "evidence": []},
|
||||
{"id": "production-service-recovery", "status": "not-run", "evidence": []},
|
||||
{"id": "retention-and-legal-hold", "status": "not-run", "evidence": []},
|
||||
{"id": "scheduled-backup", "status": "not-run", "evidence": []}
|
||||
]
|
||||
}
|
||||
220
deploy/backup_restore_acceptance.py
Normal file
220
deploy/backup_restore_acceptance.py
Normal file
@ -0,0 +1,220 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate a restricted GuestOps backup, monitoring and recovery record."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
VERSION = "0.2.0"
|
||||
SCENARIOS = {
|
||||
"encrypted-manual-backup",
|
||||
"scheduled-backup",
|
||||
"production-service-recovery",
|
||||
"atomic-off-host-transfer",
|
||||
"off-host-checksum",
|
||||
"retention-and-legal-hold",
|
||||
"monitoring-coverage",
|
||||
"alert-escalation",
|
||||
"durable-secret-free-logs",
|
||||
"isolated-restore",
|
||||
"data-protection-recovery",
|
||||
"database-inventory",
|
||||
"image-rollback",
|
||||
"controlled-return-to-service",
|
||||
}
|
||||
SHA256 = re.compile(r"[0-9a-f]{64}")
|
||||
GIT_SHA = re.compile(r"[0-9a-f]{40}")
|
||||
|
||||
|
||||
def require(condition: bool, message: str) -> None:
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def timestamp(value: object, field: str) -> dt.datetime:
|
||||
require(isinstance(value, str) and value.endswith("Z"),
|
||||
f"{field} must be a UTC timestamp ending in Z.")
|
||||
try:
|
||||
parsed = dt.datetime.fromisoformat(value.removesuffix("Z") + "+00:00")
|
||||
except ValueError as error:
|
||||
raise ValueError(f"{field} is not a valid timestamp.") from error
|
||||
require(parsed.tzinfo == dt.timezone.utc, f"{field} must be UTC.")
|
||||
return parsed
|
||||
|
||||
|
||||
def safe_name(value: object, field: str) -> str:
|
||||
name = str(value or "").strip()
|
||||
require(2 <= len(name) <= 120 and "@" not in name and "/" not in name and "\\" not in name,
|
||||
f"{field} requires a name without an email address or path.")
|
||||
return name
|
||||
|
||||
|
||||
def validate(record: object, expected_commit: str, expected_release_sha256: str) -> None:
|
||||
require(isinstance(record, dict), "Acceptance record must be a JSON object.")
|
||||
require(record.get("schemaVersion") == 1, "Unsupported backup/recovery schema.")
|
||||
require(record.get("system") == "guestops-backup-recovery",
|
||||
"system must be guestops-backup-recovery.")
|
||||
require(record.get("evidenceId") == "backup-restore",
|
||||
"evidenceId must be backup-restore.")
|
||||
require(record.get("dataClassification") == "synthetic-only",
|
||||
"Recovery acceptance must use synthetic data only.")
|
||||
require(record.get("releaseVersion") == VERSION, f"releaseVersion must be {VERSION}.")
|
||||
require(GIT_SHA.fullmatch(str(expected_commit)) is not None,
|
||||
"Expected release commit must be a full lowercase Git SHA.")
|
||||
require(SHA256.fullmatch(str(expected_release_sha256)) is not None,
|
||||
"Expected release-record checksum must be a lowercase SHA-256 digest.")
|
||||
require(record.get("releaseCommit") == expected_commit,
|
||||
"releaseCommit does not match the approved candidate.")
|
||||
require(record.get("releaseRecordSha256") == expected_release_sha256,
|
||||
"releaseRecordSha256 does not match the retained release record.")
|
||||
require(SHA256.fullmatch(str(record.get("archiveSha256", ""))) is not None,
|
||||
"archiveSha256 must be a lowercase SHA-256 digest.")
|
||||
|
||||
origin = urlparse(str(record.get("environment", "")))
|
||||
require(origin.scheme == "https" and origin.hostname and origin.path in ("", "/")
|
||||
and not origin.query and not origin.fragment and origin.username is None
|
||||
and origin.password is None,
|
||||
"environment must be an HTTPS origin without credentials, path, query or fragment.")
|
||||
safe_name(record.get("hostIdentifier"), "hostIdentifier")
|
||||
|
||||
images = record.get("images")
|
||||
require(isinstance(images, dict) and set(images) == {"api", "worker", "mongo"},
|
||||
"images must contain exactly api, worker and mongo.")
|
||||
for name in ("api", "worker"):
|
||||
image = images[name]
|
||||
require(isinstance(image, dict) and set(image) == {"reference", "id"},
|
||||
f"images.{name} must contain exactly reference and id.")
|
||||
require(image["reference"] == f"guestops-{name}:{expected_commit}",
|
||||
f"images.{name}.reference must use the full approved commit.")
|
||||
require(re.fullmatch(r"sha256:[0-9a-f]{64}", str(image["id"])) is not None,
|
||||
f"images.{name}.id must be immutable.")
|
||||
mongo = images["mongo"]
|
||||
require(isinstance(mongo, dict) and set(mongo) == {"reference", "id"}
|
||||
and mongo["reference"] == "mongo:8.0"
|
||||
and re.fullmatch(r"sha256:[0-9a-f]{64}", str(mongo["id"])) is not None,
|
||||
"images.mongo must identify the immutable mongo:8.0 image.")
|
||||
|
||||
owners = record.get("owners")
|
||||
owner_keys = {"backup", "monitoring", "recoveryOperator", "technicalEscalation",
|
||||
"retention", "independentReviewer"}
|
||||
require(isinstance(owners, dict) and set(owners) == owner_keys,
|
||||
"owners must contain the exact operational and review roles.")
|
||||
names = {key: safe_name(value, f"owners.{key}") for key, value in owners.items()}
|
||||
reviewer = names["independentReviewer"].casefold()
|
||||
require(reviewer not in {names[key].casefold() for key in owner_keys - {"independentReviewer"}},
|
||||
"independentReviewer must be different from every operational owner.")
|
||||
|
||||
started = timestamp(record.get("startedAt"), "startedAt")
|
||||
ended = timestamp(record.get("endedAt"), "endedAt")
|
||||
reviewed = timestamp(record.get("reviewedAt"), "reviewedAt")
|
||||
require(started <= ended <= reviewed, "Acceptance timestamps are out of order.")
|
||||
|
||||
recovery = record.get("recoveryObjectives")
|
||||
require(isinstance(recovery, dict) and set(recovery) == {
|
||||
"targetRpoHours", "observedRpoHours", "targetRtoMinutes", "observedRtoMinutes",
|
||||
}, "recoveryObjectives must contain exact target and observed RPO/RTO values.")
|
||||
for field in recovery:
|
||||
require(isinstance(recovery[field], (int, float)) and not isinstance(recovery[field], bool)
|
||||
and recovery[field] >= 0, f"recoveryObjectives.{field} must be non-negative.")
|
||||
require(recovery["targetRpoHours"] == 24 and recovery["observedRpoHours"] <= 24,
|
||||
"Observed RPO must meet the approved 24-hour target.")
|
||||
require(recovery["targetRtoMinutes"] == 240 and recovery["observedRtoMinutes"] <= 240,
|
||||
"Observed RTO must meet the approved four-hour target.")
|
||||
|
||||
retention = record.get("retention")
|
||||
require(retention == {
|
||||
"localVerifiedDays": 7,
|
||||
"offHostDaily": 35,
|
||||
"offHostMonthly": 12,
|
||||
"legalHoldOverrideTested": True,
|
||||
}, "Retention must record seven local days, 35 daily and 12 monthly off-host copies, and legal-hold testing.")
|
||||
|
||||
backup = record.get("backup")
|
||||
require(isinstance(backup, dict) and set(backup) == {
|
||||
"createdAt", "sha256", "transferredSha256", "privateKeyPresentOnHost",
|
||||
}, "backup must contain exact creation, checksum, transfer and private-key fields.")
|
||||
created = timestamp(backup["createdAt"], "backup.createdAt")
|
||||
require(created <= started, "The accepted backup must exist when the timed exercise starts.")
|
||||
require(abs(recovery["observedRpoHours"] - (started - created).total_seconds() / 3600) < 0.01,
|
||||
"Observed RPO must match the backup and exercise timestamps.")
|
||||
require(abs(recovery["observedRtoMinutes"] - (ended - started).total_seconds() / 60) < 0.01,
|
||||
"Observed RTO must match the exercise timestamps.")
|
||||
require(SHA256.fullmatch(str(backup["sha256"])) is not None
|
||||
and backup["transferredSha256"] == backup["sha256"],
|
||||
"Local and transferred backup checksums must match.")
|
||||
require(backup["privateKeyPresentOnHost"] is False,
|
||||
"The recovery private key must not be present on the Debian host.")
|
||||
|
||||
rollback = record.get("rollback")
|
||||
require(isinstance(rollback, dict) and set(rollback) == {
|
||||
"previousReleaseCommit", "previousArchiveSha256", "previousImages",
|
||||
"persistentVolumesReplaced", "restoredReleaseCommit", "unresolvedOperations", "finalControls",
|
||||
}, "rollback must contain the exact rehearsal and final-state fields.")
|
||||
require(GIT_SHA.fullmatch(str(rollback["previousReleaseCommit"])) is not None
|
||||
and rollback["previousReleaseCommit"] != expected_commit,
|
||||
"Rollback must use a different retained previous release.")
|
||||
require(SHA256.fullmatch(str(rollback["previousArchiveSha256"])) is not None,
|
||||
"Rollback requires the previous archive checksum.")
|
||||
previous_images = rollback["previousImages"]
|
||||
require(isinstance(previous_images, dict) and set(previous_images) == {"api", "worker"},
|
||||
"Rollback requires exact previous API and worker images.")
|
||||
for name in ("api", "worker"):
|
||||
image = previous_images[name]
|
||||
require(isinstance(image, dict) and set(image) == {"reference", "id"}
|
||||
and image["reference"] == f"guestops-{name}:{rollback['previousReleaseCommit']}"
|
||||
and re.fullmatch(r"sha256:[0-9a-f]{64}", str(image["id"])) is not None,
|
||||
f"rollback.previousImages.{name} must use the retained previous release identity.")
|
||||
require(rollback["persistentVolumesReplaced"] is False,
|
||||
"Image rollback must not replace persistent volumes.")
|
||||
require(rollback["restoredReleaseCommit"] == expected_commit,
|
||||
"The exercise must finish on the approved candidate.")
|
||||
require(rollback["unresolvedOperations"] == 0,
|
||||
"The exercise must finish without unresolved operations.")
|
||||
require(rollback["finalControls"] == {
|
||||
"googleSending": "disabled", "faqLiveMode": "disabled",
|
||||
"pmsWrites": "disabled", "paymentCreation": "disabled",
|
||||
}, "The exercise must finish with all unaccepted external writes disabled.")
|
||||
|
||||
scenarios = record.get("scenarios")
|
||||
require(isinstance(scenarios, list), "scenarios must be a list.")
|
||||
ids = [item.get("id") for item in scenarios if isinstance(item, dict)]
|
||||
require(len(ids) == len(scenarios) and len(ids) == len(set(ids)) and set(ids) == SCENARIOS,
|
||||
"Acceptance record requires the exact backup/recovery scenario set.")
|
||||
for item in scenarios:
|
||||
scenario_id = item["id"]
|
||||
require(item.get("status") == "pass", f"Scenario {scenario_id} has not passed.")
|
||||
evidence = item.get("evidence")
|
||||
require(isinstance(evidence, list) and 1 <= len(evidence) <= 10 and all(
|
||||
isinstance(value, str) and 3 <= len(value) <= 200 and "@" not in value
|
||||
and "\\" not in value and not value.startswith("/") for value in evidence
|
||||
), f"Scenario {scenario_id} requires safe opaque evidence references.")
|
||||
require(record.get("monitoringState") == "healthy",
|
||||
"Monitoring must be healthy at acceptance completion.")
|
||||
require(record.get("unresolvedCriticalFindings") == 0,
|
||||
"Acceptance cannot pass with unresolved critical findings.")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("record", type=Path)
|
||||
parser.add_argument("--expected-commit", required=True)
|
||||
parser.add_argument("--expected-release-record-sha256", required=True)
|
||||
args = parser.parse_args()
|
||||
validate(json.loads(args.record.read_text(encoding="utf-8")),
|
||||
args.expected_commit, args.expected_release_record_sha256)
|
||||
print("Backup, monitoring and recovery acceptance record is structurally complete and passed. "
|
||||
"This validates the record, not its restricted evidence.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (OSError, ValueError, json.JSONDecodeError) as error:
|
||||
print(f"Backup/recovery acceptance record rejected: {error}", file=__import__("sys").stderr)
|
||||
raise SystemExit(1)
|
||||
238
deploy/backup_transfer.py
Normal file
238
deploy/backup_transfer.py
Normal file
@ -0,0 +1,238 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Atomically transfer encrypted GuestOps backups to restricted storage."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shlex
|
||||
import shutil
|
||||
import stat
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
import uuid
|
||||
|
||||
|
||||
BACKUP_NAME = re.compile(r"guestops-[0-9]{8}T[0-9]{6}Z\.tar\.gpg")
|
||||
SAFE_HOST = re.compile(r"[A-Za-z0-9.-]{1,253}")
|
||||
SAFE_USER = re.compile(r"[A-Za-z_][A-Za-z0-9_-]{0,31}")
|
||||
SAFE_REMOTE_PATH = re.compile(r"/[A-Za-z0-9._/-]{1,500}")
|
||||
SHA256 = re.compile(r"[0-9a-f]{64}")
|
||||
|
||||
|
||||
def require(condition: bool, message: str) -> None:
|
||||
if not condition:
|
||||
raise RuntimeError(message)
|
||||
|
||||
|
||||
def digest(path: Path) -> str:
|
||||
with path.open("rb") as stream:
|
||||
return hashlib.file_digest(stream, "sha256").hexdigest()
|
||||
|
||||
|
||||
def private_directory(value: str) -> Path:
|
||||
requested = Path(value)
|
||||
require(requested.is_absolute() and not requested.is_symlink(),
|
||||
"BACKUP_DIRECTORY must be an absolute, non-symlink path.")
|
||||
directory = requested.resolve()
|
||||
require(directory.is_dir(), "BACKUP_DIRECTORY must exist.")
|
||||
require(stat.S_IMODE(directory.stat().st_mode) & 0o077 == 0,
|
||||
"BACKUP_DIRECTORY must not be accessible to group or other users.")
|
||||
return directory
|
||||
|
||||
|
||||
def regular_file(value: str, field: str, *, private: bool) -> Path:
|
||||
requested = Path(value)
|
||||
require(requested.is_absolute() and not requested.is_symlink(),
|
||||
f"{field} must be an absolute, non-symlink path.")
|
||||
path = requested.resolve()
|
||||
require(path.is_file(), f"{field} must be an existing regular file.")
|
||||
if private:
|
||||
require(stat.S_IMODE(path.stat().st_mode) & 0o077 == 0,
|
||||
f"{field} must not be accessible to group or other users.")
|
||||
return path
|
||||
|
||||
|
||||
def configuration(environment: dict[str, str]) -> dict[str, object]:
|
||||
directory = private_directory(environment.get("BACKUP_DIRECTORY", ""))
|
||||
host = environment.get("BACKUP_REMOTE_HOST", "")
|
||||
user = environment.get("BACKUP_REMOTE_USER", "")
|
||||
remote = environment.get("BACKUP_REMOTE_DIRECTORY", "")
|
||||
require(SAFE_HOST.fullmatch(host) is not None, "BACKUP_REMOTE_HOST is invalid.")
|
||||
require(SAFE_USER.fullmatch(user) is not None, "BACKUP_REMOTE_USER is invalid.")
|
||||
require(SAFE_REMOTE_PATH.fullmatch(remote) is not None and "//" not in remote
|
||||
and "/../" not in remote + "/" and not remote.endswith("/.."),
|
||||
"BACKUP_REMOTE_DIRECTORY must be a safe absolute path.")
|
||||
identity = regular_file(environment.get("BACKUP_SSH_IDENTITY", ""),
|
||||
"BACKUP_SSH_IDENTITY", private=True)
|
||||
known_hosts = regular_file(environment.get("BACKUP_SSH_KNOWN_HOSTS", ""),
|
||||
"BACKUP_SSH_KNOWN_HOSTS", private=False)
|
||||
require(shutil.which("ssh") is not None and shutil.which("rsync") is not None,
|
||||
"ssh and rsync are required.")
|
||||
return {
|
||||
"directory": directory,
|
||||
"host": host,
|
||||
"user": user,
|
||||
"remote": remote.rstrip("/"),
|
||||
"identity": identity,
|
||||
"known_hosts": known_hosts,
|
||||
}
|
||||
|
||||
|
||||
def ssh_base(config: dict[str, object]) -> list[str]:
|
||||
return [
|
||||
"ssh", "-o", "BatchMode=yes", "-o", "IdentitiesOnly=yes",
|
||||
"-o", "StrictHostKeyChecking=yes", "-o", "ConnectTimeout=15",
|
||||
"-o", f"UserKnownHostsFile={config['known_hosts']}",
|
||||
"-i", str(config["identity"]), f"{config['user']}@{config['host']}",
|
||||
]
|
||||
|
||||
|
||||
def run(args: list[str], *, environment: dict[str, str] | None = None) -> bytes:
|
||||
result = subprocess.run(args, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE, timeout=900, env=environment)
|
||||
require(result.returncode == 0,
|
||||
f"{Path(args[0]).name} step failed; review the restricted operator logs.")
|
||||
return result.stdout
|
||||
|
||||
|
||||
def remote_digest(config: dict[str, object], remote_path: str) -> str | None:
|
||||
command = f"if test -f {remote_path} && test ! -L {remote_path}; then sha256sum -- {remote_path}; fi"
|
||||
output = run([*ssh_base(config), command]).decode("utf-8", "strict").strip()
|
||||
if not output:
|
||||
return None
|
||||
value = output.split()[0]
|
||||
require(SHA256.fullmatch(value) is not None, "Remote checksum response was invalid.")
|
||||
return value
|
||||
|
||||
|
||||
def marker_path(backup: Path) -> Path:
|
||||
return backup.with_name(backup.name + ".transferred.json")
|
||||
|
||||
|
||||
def write_marker(backup: Path, checksum: str) -> None:
|
||||
marker = marker_path(backup)
|
||||
payload = json.dumps({
|
||||
"schemaVersion": 1,
|
||||
"backup": backup.name,
|
||||
"sha256": checksum,
|
||||
"verifiedAt": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
}, sort_keys=True) + "\n"
|
||||
fd, temporary = tempfile.mkstemp(prefix=marker.name + ".", dir=marker.parent)
|
||||
try:
|
||||
os.fchmod(fd, 0o600)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as stream:
|
||||
stream.write(payload)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temporary, marker)
|
||||
finally:
|
||||
try:
|
||||
os.unlink(temporary)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
|
||||
def transfer_one(config: dict[str, object], backup: Path) -> None:
|
||||
require(backup.is_file() and not backup.is_symlink()
|
||||
and BACKUP_NAME.fullmatch(backup.name) is not None,
|
||||
"Refusing to transfer an unexpected backup path.")
|
||||
checksum = digest(backup)
|
||||
remote_final = f"{config['remote']}/{backup.name}"
|
||||
existing = remote_digest(config, remote_final)
|
||||
if existing is not None:
|
||||
require(existing == checksum, "A remote backup with this name has a different checksum.")
|
||||
write_marker(backup, checksum)
|
||||
return
|
||||
|
||||
remote_partial = f"{config['remote']}/.{backup.name}.partial-{uuid.uuid4().hex}"
|
||||
rsh = shlex.join([
|
||||
"ssh", "-o", "BatchMode=yes", "-o", "IdentitiesOnly=yes",
|
||||
"-o", "StrictHostKeyChecking=yes", "-o", "ConnectTimeout=15",
|
||||
"-o", f"UserKnownHostsFile={config['known_hosts']}",
|
||||
"-i", str(config["identity"]),
|
||||
])
|
||||
rsync_environment = os.environ.copy()
|
||||
rsync_environment["RSYNC_RSH"] = rsh
|
||||
try:
|
||||
run(["rsync", "--archive", "--chmod=F600", "--protect-args", "--",
|
||||
str(backup), f"{config['user']}@{config['host']}:{remote_partial}"],
|
||||
environment=rsync_environment)
|
||||
require(remote_digest(config, remote_partial) == checksum,
|
||||
"Transferred backup checksum does not match the local file.")
|
||||
command = (f"test ! -e {remote_final} && mv -T -- {remote_partial} {remote_final} "
|
||||
f"&& chmod 600 -- {remote_final}")
|
||||
run([*ssh_base(config), command])
|
||||
require(remote_digest(config, remote_final) == checksum,
|
||||
"Final remote backup checksum does not match the local file.")
|
||||
write_marker(backup, checksum)
|
||||
except Exception:
|
||||
# The name contains a fresh random suffix and is the only remote object this run may remove.
|
||||
subprocess.run([*ssh_base(config), f"rm -f -- {remote_partial}"], stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=30)
|
||||
raise
|
||||
|
||||
|
||||
def transfer_pending(config: dict[str, object]) -> int:
|
||||
directory = config["directory"]
|
||||
backups = sorted(path for path in directory.iterdir()
|
||||
if path.is_file() and not path.is_symlink()
|
||||
and BACKUP_NAME.fullmatch(path.name) is not None)
|
||||
for backup in backups:
|
||||
marker = marker_path(backup)
|
||||
if marker.is_file() and not marker.is_symlink():
|
||||
try:
|
||||
recorded = json.loads(marker.read_text(encoding="utf-8"))
|
||||
if recorded.get("sha256") == digest(backup):
|
||||
continue
|
||||
except (OSError, ValueError, json.JSONDecodeError):
|
||||
pass
|
||||
transfer_one(config, backup)
|
||||
return len(backups)
|
||||
|
||||
|
||||
def prune_verified(config: dict[str, object], retention_days: int, now: float | None = None) -> int:
|
||||
require(1 <= retention_days <= 365, "Local retention must be between 1 and 365 days.")
|
||||
threshold = (time.time() if now is None else now) - retention_days * 86400
|
||||
removed = 0
|
||||
for backup in config["directory"].iterdir():
|
||||
if not backup.is_file() or backup.is_symlink() or BACKUP_NAME.fullmatch(backup.name) is None:
|
||||
continue
|
||||
marker = marker_path(backup)
|
||||
if backup.stat().st_mtime >= threshold or not marker.is_file() or marker.is_symlink():
|
||||
continue
|
||||
try:
|
||||
recorded = json.loads(marker.read_text(encoding="utf-8"))
|
||||
except (OSError, ValueError, json.JSONDecodeError):
|
||||
continue
|
||||
if recorded.get("backup") != backup.name or recorded.get("sha256") != digest(backup):
|
||||
continue
|
||||
backup.unlink()
|
||||
marker.unlink()
|
||||
removed += 1
|
||||
return removed
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--prune-verified", action="store_true")
|
||||
parser.add_argument("--retention-days", type=int, default=7)
|
||||
args = parser.parse_args()
|
||||
config = configuration(dict(os.environ))
|
||||
observed = transfer_pending(config)
|
||||
removed = prune_verified(config, args.retention_days) if args.prune_verified else 0
|
||||
print(f"Backup transfer completed: {observed} encrypted backup(s) inspected; "
|
||||
f"{removed} verified local backup(s) expired.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (OSError, RuntimeError, subprocess.SubprocessError, json.JSONDecodeError) as error:
|
||||
print(f"Backup transfer failed: {error}", file=__import__("sys").stderr)
|
||||
raise SystemExit(1)
|
||||
45
deploy/debian-host-acceptance.example.json
Normal file
45
deploy/debian-host-acceptance.example.json
Normal file
@ -0,0 +1,45 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"system": "guestops-debian-host",
|
||||
"evidenceId": "debian-host",
|
||||
"releaseVersion": "0.2.0",
|
||||
"releaseCommit": "0000000000000000000000000000000000000000",
|
||||
"releaseRecordSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"archiveSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"environment": "https://sandbox-guestops.futuresens.co.uk",
|
||||
"hostIdentifier": "guestops-sandbox-01",
|
||||
"images": {
|
||||
"api": {"reference": "guestops-api:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
|
||||
"worker": {"reference": "guestops-worker:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"}
|
||||
},
|
||||
"hostFacts": {
|
||||
"debianMajor": 12,
|
||||
"cpuCores": 4,
|
||||
"memoryBytes": 8140382208,
|
||||
"freeDiskBytes": 14275686400,
|
||||
"publicTcpPorts": []
|
||||
},
|
||||
"featureControls": {
|
||||
"googleSending": "disabled",
|
||||
"faqLiveMode": "disabled",
|
||||
"pmsWrites": "disabled",
|
||||
"paymentCreation": "disabled"
|
||||
},
|
||||
"operator": "REPLACE",
|
||||
"reviewedBy": "REPLACE",
|
||||
"startedAt": "2026-09-30T09:00:00Z",
|
||||
"endedAt": "2026-09-30T10:00:00Z",
|
||||
"reviewedAt": "2026-09-30T11:00:00Z",
|
||||
"unresolvedCriticalFindings": 1,
|
||||
"scenarios": [
|
||||
{"id": "boot-services", "status": "not-run", "evidence": []},
|
||||
{"id": "controlled-reboot", "status": "not-run", "evidence": []},
|
||||
{"id": "durable-log-retrieval", "status": "not-run", "evidence": []},
|
||||
{"id": "host-baseline", "status": "not-run", "evidence": []},
|
||||
{"id": "https-and-redirect", "status": "not-run", "evidence": []},
|
||||
{"id": "network-exposure", "status": "not-run", "evidence": []},
|
||||
{"id": "proxy-trust", "status": "not-run", "evidence": []},
|
||||
{"id": "secret-free-logs", "status": "not-run", "evidence": []},
|
||||
{"id": "workspace-health", "status": "not-run", "evidence": []}
|
||||
]
|
||||
}
|
||||
206
deploy/debian_acceptance.py
Normal file
206
deploy/debian_acceptance.py
Normal file
@ -0,0 +1,206 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate restricted GuestOps Debian-host and persistence acceptance records."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
VERSION = "0.2.0"
|
||||
SYSTEMS = {
|
||||
"guestops-debian-host": {
|
||||
"evidenceId": "debian-host",
|
||||
"scenarios": {
|
||||
"host-baseline",
|
||||
"network-exposure",
|
||||
"https-and-redirect",
|
||||
"proxy-trust",
|
||||
"boot-services",
|
||||
"controlled-reboot",
|
||||
"workspace-health",
|
||||
"durable-log-retrieval",
|
||||
"secret-free-logs",
|
||||
},
|
||||
},
|
||||
"guestops-persistence": {
|
||||
"evidenceId": "persistence",
|
||||
"scenarios": {
|
||||
"separate-volume-layout",
|
||||
"service-restart",
|
||||
"container-recreation",
|
||||
"database-inventory",
|
||||
"data-protection-key",
|
||||
"image-identity",
|
||||
"post-reboot-persistence",
|
||||
},
|
||||
},
|
||||
}
|
||||
SHA256 = re.compile(r"[0-9a-f]{64}")
|
||||
GIT_SHA = re.compile(r"[0-9a-f]{40}")
|
||||
|
||||
|
||||
def require(condition: bool, message: str) -> None:
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def timestamp(value: object, field: str) -> dt.datetime:
|
||||
require(isinstance(value, str) and value.endswith("Z"),
|
||||
f"{field} must be a UTC timestamp ending in Z.")
|
||||
try:
|
||||
parsed = dt.datetime.fromisoformat(value.removesuffix("Z") + "+00:00")
|
||||
except ValueError as error:
|
||||
raise ValueError(f"{field} is not a valid timestamp.") from error
|
||||
require(parsed.tzinfo == dt.timezone.utc, f"{field} must be UTC.")
|
||||
return parsed
|
||||
|
||||
|
||||
def safe_text(value: object, field: str, minimum: int = 2, maximum: int = 160) -> str:
|
||||
text = str(value or "").strip()
|
||||
require(minimum <= len(text) <= maximum and "@" not in text and "\\" not in text,
|
||||
f"{field} must be safe text without an email address or local path.")
|
||||
return text
|
||||
|
||||
|
||||
def validate_common(record: object, expected_commit: str, expected_release_sha256: str) -> str:
|
||||
require(isinstance(record, dict), "Acceptance record must be a JSON object.")
|
||||
require(record.get("schemaVersion") == 1, "Unsupported Debian acceptance schema.")
|
||||
system = record.get("system")
|
||||
require(system in SYSTEMS, "Unknown Debian acceptance record system.")
|
||||
require(record.get("evidenceId") == SYSTEMS[system]["evidenceId"],
|
||||
f"{system} has the wrong evidenceId.")
|
||||
require(record.get("releaseVersion") == VERSION,
|
||||
f"releaseVersion must be {VERSION}.")
|
||||
require(GIT_SHA.fullmatch(str(expected_commit)) is not None,
|
||||
"Expected release commit must be a full lowercase Git SHA.")
|
||||
require(SHA256.fullmatch(str(expected_release_sha256)) is not None,
|
||||
"Expected release-record checksum must be a lowercase SHA-256 digest.")
|
||||
require(record.get("releaseCommit") == expected_commit,
|
||||
"releaseCommit does not match the approved candidate.")
|
||||
require(record.get("releaseRecordSha256") == expected_release_sha256,
|
||||
"releaseRecordSha256 does not match the retained release record.")
|
||||
require(SHA256.fullmatch(str(record.get("archiveSha256", ""))) is not None,
|
||||
"archiveSha256 must be a lowercase SHA-256 digest.")
|
||||
|
||||
origin = urlparse(str(record.get("environment", "")))
|
||||
require(origin.scheme == "https" and origin.hostname and origin.path in ("", "/")
|
||||
and not origin.query and not origin.fragment and origin.username is None
|
||||
and origin.password is None,
|
||||
"environment must be an HTTPS origin without credentials, path, query or fragment.")
|
||||
host = safe_text(record.get("hostIdentifier"), "hostIdentifier")
|
||||
|
||||
images = record.get("images")
|
||||
require(isinstance(images, dict) and set(images) == {"api", "worker"},
|
||||
"images must contain exactly api and worker.")
|
||||
for name in ("api", "worker"):
|
||||
image = images[name]
|
||||
require(isinstance(image, dict) and set(image) == {"reference", "id"},
|
||||
f"images.{name} must contain exactly reference and id.")
|
||||
require(image["reference"] == f"guestops-{name}:{expected_commit}",
|
||||
f"images.{name}.reference must use the full approved commit.")
|
||||
require(re.fullmatch(r"sha256:[0-9a-f]{64}", str(image["id"])) is not None,
|
||||
f"images.{name}.id must be an immutable image ID.")
|
||||
|
||||
operator = safe_text(record.get("operator"), "operator")
|
||||
reviewer = safe_text(record.get("reviewedBy"), "reviewedBy")
|
||||
require(operator.casefold() != reviewer.casefold(),
|
||||
"operator and reviewedBy must be different people.")
|
||||
started = timestamp(record.get("startedAt"), "startedAt")
|
||||
ended = timestamp(record.get("endedAt"), "endedAt")
|
||||
reviewed = timestamp(record.get("reviewedAt"), "reviewedAt")
|
||||
require(started <= ended <= reviewed, "Acceptance timestamps are out of order.")
|
||||
|
||||
scenarios = record.get("scenarios")
|
||||
require(isinstance(scenarios, list), "scenarios must be a list.")
|
||||
ids = [item.get("id") for item in scenarios if isinstance(item, dict)]
|
||||
required = SYSTEMS[system]["scenarios"]
|
||||
require(len(ids) == len(scenarios) and len(ids) == len(set(ids)) and set(ids) == required,
|
||||
f"{system} requires its exact acceptance scenario set.")
|
||||
for item in scenarios:
|
||||
scenario_id = item["id"]
|
||||
require(item.get("status") == "pass", f"Scenario {scenario_id} has not passed.")
|
||||
evidence = item.get("evidence")
|
||||
require(isinstance(evidence, list) and 1 <= len(evidence) <= 10 and all(
|
||||
isinstance(value, str) and 3 <= len(value) <= 200 and "@" not in value
|
||||
and "\\" not in value and not value.startswith("/") for value in evidence
|
||||
), f"Scenario {scenario_id} requires safe opaque evidence references.")
|
||||
|
||||
require(record.get("unresolvedCriticalFindings") == 0,
|
||||
"Acceptance cannot pass with unresolved critical findings.")
|
||||
return host
|
||||
|
||||
|
||||
def validate_host(record: object, expected_commit: str, expected_release_sha256: str) -> str:
|
||||
host = validate_common(record, expected_commit, expected_release_sha256)
|
||||
require(record.get("system") == "guestops-debian-host",
|
||||
"First record must be guestops-debian-host.")
|
||||
facts = record.get("hostFacts")
|
||||
require(isinstance(facts, dict) and set(facts) == {
|
||||
"debianMajor", "cpuCores", "memoryBytes", "freeDiskBytes", "publicTcpPorts",
|
||||
}, "hostFacts must contain the exact reviewed host facts.")
|
||||
require(isinstance(facts["debianMajor"], int) and facts["debianMajor"] >= 12,
|
||||
"Debian 12 or newer is required.")
|
||||
require(isinstance(facts["cpuCores"], int) and facts["cpuCores"] >= 4,
|
||||
"At least four CPU cores are required.")
|
||||
require(isinstance(facts["memoryBytes"], int) and facts["memoryBytes"] >= 7_500_000_000,
|
||||
"At least 7.5 GB of memory is required.")
|
||||
require(isinstance(facts["freeDiskBytes"], int) and facts["freeDiskBytes"] >= 8 * 1024**3,
|
||||
"At least 8 GiB of free disk space is required.")
|
||||
require(facts["publicTcpPorts"] == [80, 443],
|
||||
"Only TCP ports 80 and 443 may be public.")
|
||||
require(record.get("featureControls") == {
|
||||
"googleSending": "disabled",
|
||||
"faqLiveMode": "disabled",
|
||||
"pmsWrites": "disabled",
|
||||
"paymentCreation": "disabled",
|
||||
}, "Unaccepted external writes and FAQ live mode must remain disabled.")
|
||||
return host
|
||||
|
||||
|
||||
def validate_persistence(record: object, expected_commit: str, expected_release_sha256: str) -> str:
|
||||
host = validate_common(record, expected_commit, expected_release_sha256)
|
||||
require(record.get("system") == "guestops-persistence",
|
||||
"Second record must be guestops-persistence.")
|
||||
require(record.get("drillCommand") == "python3 deploy/ops.py persistence-drill --confirm-restart",
|
||||
"drillCommand must identify the confirmation-gated persistence drill.")
|
||||
return host
|
||||
|
||||
|
||||
def validate_pair(host_record: object, persistence_record: object,
|
||||
expected_commit: str, expected_release_sha256: str) -> None:
|
||||
host = validate_host(host_record, expected_commit, expected_release_sha256)
|
||||
persistence_host = validate_persistence(
|
||||
persistence_record, expected_commit, expected_release_sha256)
|
||||
require(host == persistence_host, "Both records must identify the same host.")
|
||||
for field in ("environment", "releaseVersion", "releaseCommit", "releaseRecordSha256",
|
||||
"archiveSha256", "images"):
|
||||
require(host_record.get(field) == persistence_record.get(field),
|
||||
f"Both records must use the same {field}.")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("host_record", type=Path)
|
||||
parser.add_argument("persistence_record", type=Path)
|
||||
parser.add_argument("--expected-commit", required=True)
|
||||
parser.add_argument("--expected-release-record-sha256", required=True)
|
||||
args = parser.parse_args()
|
||||
host_record = json.loads(args.host_record.read_text(encoding="utf-8"))
|
||||
persistence_record = json.loads(args.persistence_record.read_text(encoding="utf-8"))
|
||||
validate_pair(host_record, persistence_record,
|
||||
args.expected_commit, args.expected_release_record_sha256)
|
||||
print("Debian-host and persistence acceptance records are structurally complete and passed. "
|
||||
"This validates the records, not their restricted evidence.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (OSError, ValueError, json.JSONDecodeError) as error:
|
||||
print(f"Debian acceptance records rejected: {error}", file=__import__("sys").stderr)
|
||||
raise SystemExit(1)
|
||||
189
deploy/monitor_status.py
Normal file
189
deploy/monitor_status.py
Normal file
@ -0,0 +1,189 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Write non-sensitive GuestOps host status for a read-only monitoring agent."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import socket
|
||||
import ssl
|
||||
import stat
|
||||
import subprocess
|
||||
import tempfile
|
||||
import urllib.request
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
BACKUP_NAME = re.compile(r"guestops-[0-9]{8}T[0-9]{6}Z\.tar\.gpg")
|
||||
MARKER_NAME = re.compile(r"guestops-[0-9]{8}T[0-9]{6}Z\.tar\.gpg\.transferred\.json")
|
||||
AUTH = ('const c=new Mongo("mongodb://127.0.0.1");'
|
||||
'c.getDB("admin").auth(process.env.MONGO_INITDB_ROOT_USERNAME,'
|
||||
'process.env.MONGO_INITDB_ROOT_PASSWORD);')
|
||||
HEARTBEAT = ('const x=c.getDB("guestops").workerheartbeat.findOne({_id:"worker"});'
|
||||
'print(JSON.stringify(x&&x.At?x.At:null));')
|
||||
|
||||
|
||||
def require(condition: bool, message: str) -> None:
|
||||
if not condition:
|
||||
raise RuntimeError(message)
|
||||
|
||||
|
||||
def run(args: list[str], root: Path, timeout: int = 30) -> bytes:
|
||||
result = subprocess.run(args, cwd=root, stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout)
|
||||
require(result.returncode == 0, f"{Path(args[0]).name} probe failed.")
|
||||
return result.stdout
|
||||
|
||||
|
||||
def utc_now() -> dt.datetime:
|
||||
return dt.datetime.now(dt.timezone.utc)
|
||||
|
||||
|
||||
def age_seconds(path: Path, pattern: re.Pattern[str], now: dt.datetime) -> int | None:
|
||||
if not path.is_dir() or path.is_symlink():
|
||||
return None
|
||||
times = [item.stat().st_mtime for item in path.iterdir()
|
||||
if item.is_file() and not item.is_symlink() and pattern.fullmatch(item.name)]
|
||||
return max(0, int(now.timestamp() - max(times))) if times else None
|
||||
|
||||
|
||||
def https_status(origin: str, now: dt.datetime) -> dict[str, object]:
|
||||
parsed = urlparse(origin)
|
||||
require(parsed.scheme == "https" and parsed.hostname and parsed.port in (None, 443)
|
||||
and parsed.path in ("", "/") and not parsed.query and not parsed.fragment
|
||||
and parsed.username is None and parsed.password is None,
|
||||
"GUESTOPS_ORIGIN must be an HTTPS origin without credentials or a path.")
|
||||
context = ssl.create_default_context()
|
||||
with socket.create_connection((parsed.hostname, 443), timeout=10) as connection:
|
||||
with context.wrap_socket(connection, server_hostname=parsed.hostname) as secured:
|
||||
certificate = secured.getpeercert()
|
||||
expires = dt.datetime.strptime(certificate["notAfter"], "%b %d %H:%M:%S %Y %Z").replace(
|
||||
tzinfo=dt.timezone.utc)
|
||||
with urllib.request.urlopen(origin.rstrip("/") + "/health/ready", timeout=15,
|
||||
context=context) as response:
|
||||
ready = response.status == 200 and json.load(response) == {"status": "ready"}
|
||||
return {"ready": ready, "certificateDaysRemaining": max(0, int((expires - now).total_seconds() // 86400))}
|
||||
|
||||
|
||||
def parse_compose(value: bytes) -> dict[str, dict[str, str]]:
|
||||
text = value.decode("utf-8", "strict").strip()
|
||||
if not text:
|
||||
return {}
|
||||
try:
|
||||
parsed = json.loads(text)
|
||||
rows = parsed if isinstance(parsed, list) else [parsed]
|
||||
except json.JSONDecodeError:
|
||||
rows = [json.loads(line) for line in text.splitlines() if line.strip()]
|
||||
result = {}
|
||||
for row in rows:
|
||||
if not isinstance(row, dict):
|
||||
continue
|
||||
service = str(row.get("Service", ""))
|
||||
if service in {"api", "worker", "mongo"}:
|
||||
result[service] = {
|
||||
"state": str(row.get("State", "unknown")).lower(),
|
||||
"health": str(row.get("Health", "none") or "none").lower(),
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def worker_heartbeat_age(root: Path, now: dt.datetime) -> int | None:
|
||||
output = run(["docker", "compose", "exec", "-T", "mongo", "mongosh", "--quiet",
|
||||
"--nodb", "--eval", AUTH + HEARTBEAT], root).decode("utf-8", "strict").strip()
|
||||
value = json.loads(output)
|
||||
if value is None:
|
||||
return None
|
||||
require(isinstance(value, str), "Worker heartbeat response was invalid.")
|
||||
parsed = dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
require(parsed.tzinfo is not None, "Worker heartbeat must contain a timezone.")
|
||||
return max(0, int((now - parsed.astimezone(dt.timezone.utc)).total_seconds()))
|
||||
|
||||
|
||||
def build_status(root: Path, backup_directory: Path, origin: str,
|
||||
now: dt.datetime | None = None) -> tuple[dict[str, object], list[str]]:
|
||||
moment = now or utc_now()
|
||||
errors: list[str] = []
|
||||
try:
|
||||
https = https_status(origin, moment)
|
||||
except Exception:
|
||||
https = {"ready": False, "certificateDaysRemaining": None}
|
||||
errors.append("https-probe-failed")
|
||||
try:
|
||||
containers = parse_compose(run(["docker", "compose", "ps", "--format", "json"], root))
|
||||
if set(containers) != {"api", "worker", "mongo"}:
|
||||
errors.append("container-set-incomplete")
|
||||
except Exception:
|
||||
containers = {}
|
||||
errors.append("container-probe-failed")
|
||||
try:
|
||||
heartbeat_age = worker_heartbeat_age(root, moment)
|
||||
if heartbeat_age is None:
|
||||
errors.append("worker-heartbeat-missing")
|
||||
except Exception:
|
||||
heartbeat_age = None
|
||||
errors.append("worker-heartbeat-probe-failed")
|
||||
disk = shutil.disk_usage(root)
|
||||
status = {
|
||||
"schemaVersion": 1,
|
||||
"generatedAt": moment.isoformat().replace("+00:00", "Z"),
|
||||
"https": https,
|
||||
"containers": containers,
|
||||
"workerHeartbeatAgeSeconds": heartbeat_age,
|
||||
"backupAgeSeconds": age_seconds(backup_directory, BACKUP_NAME, moment),
|
||||
"verifiedTransferAgeSeconds": age_seconds(backup_directory, MARKER_NAME, moment),
|
||||
"diskFreePercent": round(disk.free * 100 / disk.total, 2),
|
||||
"persistentJournal": Path("/var/log/journal").is_dir(),
|
||||
"errors": sorted(set(errors)),
|
||||
}
|
||||
return status, errors
|
||||
|
||||
|
||||
def write_status(path: Path, status: dict[str, object]) -> None:
|
||||
require(path.is_absolute() and not path.is_symlink(),
|
||||
"Status output must be an absolute, non-symlink path.")
|
||||
parent = path.parent.resolve()
|
||||
require(parent.is_dir() and not path.parent.is_symlink(), "Status output directory must exist.")
|
||||
fd, temporary = tempfile.mkstemp(prefix=path.name + ".", dir=parent)
|
||||
try:
|
||||
os.fchmod(fd, 0o644)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as stream:
|
||||
json.dump(status, stream, sort_keys=True, separators=(",", ":"))
|
||||
stream.write("\n")
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temporary, path)
|
||||
finally:
|
||||
try:
|
||||
os.unlink(temporary)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--root", type=Path, default=Path(__file__).resolve().parents[1])
|
||||
parser.add_argument("--backup-directory", type=Path, default=Path("/var/backups/guestops"))
|
||||
parser.add_argument("--origin", default="https://sandbox-guestops.futuresens.co.uk")
|
||||
parser.add_argument("--output", type=Path, default=Path("/run/guestops-monitor/status.json"))
|
||||
args = parser.parse_args()
|
||||
root = args.root.resolve()
|
||||
require(root.is_dir(), "GuestOps root must exist.")
|
||||
backup_directory = args.backup_directory.resolve()
|
||||
status, errors = build_status(root, backup_directory, args.origin)
|
||||
write_status(args.output, status)
|
||||
print("GuestOps monitoring status updated." if not errors
|
||||
else "GuestOps monitoring status updated with failed probes.")
|
||||
raise SystemExit(1 if errors else 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (OSError, RuntimeError, ValueError, subprocess.SubprocessError, json.JSONDecodeError) as error:
|
||||
print(f"GuestOps monitoring probe failed: {error}", file=__import__("sys").stderr)
|
||||
raise SystemExit(1)
|
||||
31
deploy/persistence-acceptance.example.json
Normal file
31
deploy/persistence-acceptance.example.json
Normal file
@ -0,0 +1,31 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"system": "guestops-persistence",
|
||||
"evidenceId": "persistence",
|
||||
"releaseVersion": "0.2.0",
|
||||
"releaseCommit": "0000000000000000000000000000000000000000",
|
||||
"releaseRecordSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"archiveSha256": "0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"environment": "https://sandbox-guestops.futuresens.co.uk",
|
||||
"hostIdentifier": "guestops-sandbox-01",
|
||||
"images": {
|
||||
"api": {"reference": "guestops-api:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"},
|
||||
"worker": {"reference": "guestops-worker:0000000000000000000000000000000000000000", "id": "sha256:0000000000000000000000000000000000000000000000000000000000000000"}
|
||||
},
|
||||
"drillCommand": "python3 deploy/ops.py persistence-drill --confirm-restart",
|
||||
"operator": "REPLACE",
|
||||
"reviewedBy": "REPLACE",
|
||||
"startedAt": "2026-09-30T09:00:00Z",
|
||||
"endedAt": "2026-09-30T10:00:00Z",
|
||||
"reviewedAt": "2026-09-30T11:00:00Z",
|
||||
"unresolvedCriticalFindings": 1,
|
||||
"scenarios": [
|
||||
{"id": "container-recreation", "status": "not-run", "evidence": []},
|
||||
{"id": "data-protection-key", "status": "not-run", "evidence": []},
|
||||
{"id": "database-inventory", "status": "not-run", "evidence": []},
|
||||
{"id": "image-identity", "status": "not-run", "evidence": []},
|
||||
{"id": "post-reboot-persistence", "status": "not-run", "evidence": []},
|
||||
{"id": "separate-volume-layout", "status": "not-run", "evidence": []},
|
||||
{"id": "service-restart", "status": "not-run", "evidence": []}
|
||||
]
|
||||
}
|
||||
6
deploy/systemd/backup-transfer.env.example
Normal file
6
deploy/systemd/backup-transfer.env.example
Normal file
@ -0,0 +1,6 @@
|
||||
BACKUP_DIRECTORY=/var/backups/guestops
|
||||
BACKUP_REMOTE_HOST=restricted-store.example.invalid
|
||||
BACKUP_REMOTE_USER=guestops_upload
|
||||
BACKUP_REMOTE_DIRECTORY=/restricted/guestops/backups
|
||||
BACKUP_SSH_IDENTITY=/etc/guestops/backup-transfer.key
|
||||
BACKUP_SSH_KNOWN_HOSTS=/etc/guestops/backup-known-hosts
|
||||
2
deploy/systemd/backup.env.example
Normal file
2
deploy/systemd/backup.env.example
Normal file
@ -0,0 +1,2 @@
|
||||
BACKUP_RECIPIENT=0123456789ABCDEF0123456789ABCDEF01234567
|
||||
BACKUP_DIRECTORY=/var/backups/guestops
|
||||
19
deploy/systemd/guestops-backup-transfer.service
Normal file
19
deploy/systemd/guestops-backup-transfer.service
Normal file
@ -0,0 +1,19 @@
|
||||
[Unit]
|
||||
Description=Transfer GuestOps encrypted backups to restricted storage
|
||||
Wants=network-online.target
|
||||
After=network-online.target guestops-backup.service
|
||||
ConditionPathIsDirectory=/srv/guestops
|
||||
ConditionPathIsDirectory=/var/backups/guestops
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=/srv/guestops
|
||||
EnvironmentFile=/etc/guestops/backup-transfer.env
|
||||
UMask=0077
|
||||
ExecStart=/usr/bin/python3 /srv/guestops/deploy/backup_transfer.py --prune-verified --retention-days 7
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectHome=true
|
||||
ProtectSystem=full
|
||||
ReadWritePaths=/var/backups/guestops
|
||||
TimeoutStartSec=30min
|
||||
11
deploy/systemd/guestops-backup-transfer.timer
Normal file
11
deploy/systemd/guestops-backup-transfer.timer
Normal file
@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=Retry GuestOps off-host backup transfer
|
||||
|
||||
[Timer]
|
||||
OnBootSec=10min
|
||||
OnUnitActiveSec=15min
|
||||
Persistent=true
|
||||
Unit=guestops-backup-transfer.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
21
deploy/systemd/guestops-monitor-status.service
Normal file
21
deploy/systemd/guestops-monitor-status.service
Normal file
@ -0,0 +1,21 @@
|
||||
[Unit]
|
||||
Description=Write non-sensitive GuestOps monitoring status
|
||||
Requires=docker.service
|
||||
After=docker.service network-online.target
|
||||
ConditionPathIsDirectory=/srv/guestops
|
||||
ConditionPathIsDirectory=/var/backups/guestops
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=/srv/guestops
|
||||
UMask=0022
|
||||
ExecStart=/usr/bin/python3 /srv/guestops/deploy/monitor_status.py
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectHome=true
|
||||
ProtectSystem=strict
|
||||
RuntimeDirectory=guestops-monitor
|
||||
RuntimeDirectoryMode=0755
|
||||
RuntimeDirectoryPreserve=yes
|
||||
ReadWritePaths=/run/guestops-monitor
|
||||
TimeoutStartSec=2min
|
||||
11
deploy/systemd/guestops-monitor-status.timer
Normal file
11
deploy/systemd/guestops-monitor-status.timer
Normal file
@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=Refresh GuestOps monitoring status each minute
|
||||
|
||||
[Timer]
|
||||
OnBootSec=2min
|
||||
OnUnitActiveSec=1min
|
||||
Persistent=true
|
||||
Unit=guestops-monitor-status.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
2
deploy/systemd/zabbix-agent-guestops.conf.example
Normal file
2
deploy/systemd/zabbix-agent-guestops.conf.example
Normal file
@ -0,0 +1,2 @@
|
||||
# The privileged systemd probe owns Docker/database access. Zabbix reads only this aggregate JSON.
|
||||
UserParameter=guestops.status,cat /run/guestops-monitor/status.json
|
||||
@ -86,4 +86,16 @@ The drill restarts MongoDB, API and worker, then force-recreates the stateless a
|
||||
|
||||
Record the host, operator, start/end time, release record checksum, resolved image IDs, preflight output and drill result in the deployment acceptance record. Also verify Docker starts at boot and perform a controlled Debian reboot before Gate A approval. After reboot, run the online preflight and inspect the Workspace health page; do not infer worker health solely from API readiness.
|
||||
|
||||
Keep the `debian-host` and `persistence` records in the restricted evidence store. Start from `deploy/debian-host-acceptance.example.json` and `deploy/persistence-acceptance.example.json`; the examples deliberately fail until every supervised scenario has passed. Bind both records to the expected release identifiers and validate them together:
|
||||
|
||||
```sh
|
||||
python3 deploy/debian_acceptance.py \
|
||||
/secure/acceptance/debian-host.json \
|
||||
/secure/acceptance/persistence.json \
|
||||
--expected-commit FULL_40_CHARACTER_SHA \
|
||||
--expected-release-record-sha256 RELEASE_RECORD_SHA256
|
||||
```
|
||||
|
||||
The validator requires matching archive and image identities, separate operator and reviewer names, the approved host capacity, only ports 80 and 443 recorded as publicly reachable, disabled unaccepted external writes, exact passed scenario sets and no unresolved critical findings. It validates record structure, not the restricted evidence itself. Retain the records, validator output and their checksums outside Git.
|
||||
|
||||
The supplied Docker `json-file` logs are size-capped to protect the small pilot disk, but container recreation removes that container's local log history. Before host acceptance, route GuestOps and Nginx logs to the site's durable restricted logging system, or use a reviewed Docker logging override backed by persistent systemd journal storage. Prove that operators can retrieve pre-recreation logs without exposing request credentials or OAuth callback query strings. Central retention and alerting are completed under milestone 11.
|
||||
|
||||
@ -113,6 +113,43 @@ systemctl list-timers guestops-backup.timer
|
||||
|
||||
The timer deliberately causes the same brief maintenance interruption as a manual backup. `Persistent=true` runs a missed event after downtime, so choose and communicate the maintenance window. A successful unit only stages an encrypted file locally. Configure an independently monitored off-host transfer, verify the destination checksum, alert on both unit and transfer failure, and test the alert route. Do not add automatic deletion until retention, legal hold and recovery requirements have named owners.
|
||||
|
||||
### Restricted off-host transfer
|
||||
|
||||
The optional transfer service uses rsync over pinned-host SSH. Create a dedicated upload-only account at the restricted store, disable interactive login, agent/port forwarding and access outside the GuestOps backup directory, and keep its private key only in `/etc/guestops` with mode 600. Pin the reviewed server host key; never use `StrictHostKeyChecking=no`.
|
||||
|
||||
Start from `deploy/systemd/backup-transfer.env.example` and store the completed file as `/etc/guestops/backup-transfer.env` with mode 600. The local and remote directories must already exist and remain private. Install and verify the service and its 15-minute retry timer:
|
||||
|
||||
```sh
|
||||
sudo install -m 600 deploy/systemd/backup-transfer.env.example /etc/guestops/backup-transfer.env
|
||||
sudoedit /etc/guestops/backup-transfer.env
|
||||
sudo install -m 644 deploy/systemd/guestops-backup-transfer.service /etc/systemd/system/
|
||||
sudo install -m 644 deploy/systemd/guestops-backup-transfer.timer /etc/systemd/system/
|
||||
sudo systemd-analyze verify /etc/systemd/system/guestops-backup-transfer.service /etc/systemd/system/guestops-backup-transfer.timer
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl start guestops-backup-transfer.service
|
||||
sudo systemctl enable --now guestops-backup-transfer.timer
|
||||
```
|
||||
|
||||
Only files named `guestops-YYYYMMDDTHHMMSSZ.tar.gpg` are eligible. The transfer writes a unique remote partial file, compares the remote and local SHA-256 values, atomically publishes a previously unused final name, verifies it again, and then writes a non-sensitive local marker. An existing remote name is accepted only when its checksum matches. Failed or mismatched transfers are never marked. The service removes local backups older than seven days only when the marker still matches the local checksum; untransferred or changed files are never pruned.
|
||||
|
||||
The restricted store is the durable copy. Configure its independently reviewed retention policy for 35 daily and 12 monthly recovery points. Legal hold must override expiry. The host tool does not delete remote data or enforce remote retention.
|
||||
|
||||
### Zabbix status boundary
|
||||
|
||||
Do not give the Zabbix agent access to Docker, MongoDB credentials or the application owner session. A root-owned systemd probe reads those local sources and atomically publishes aggregate status under `/run/guestops-monitor/status.json`; the agent reads that file only.
|
||||
|
||||
```sh
|
||||
sudo install -m 644 deploy/systemd/guestops-monitor-status.service /etc/systemd/system/
|
||||
sudo install -m 644 deploy/systemd/guestops-monitor-status.timer /etc/systemd/system/
|
||||
sudo install -m 644 deploy/systemd/zabbix-agent-guestops.conf.example /etc/zabbix/zabbix_agentd.d/guestops.conf
|
||||
sudo systemd-analyze verify /etc/systemd/system/guestops-monitor-status.service /etc/systemd/system/guestops-monitor-status.timer
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now guestops-monitor-status.timer
|
||||
sudo systemctl restart zabbix-agent
|
||||
```
|
||||
|
||||
Create dependent Zabbix items from `guestops.status` for HTTPS readiness, certificate days remaining, container state/health, worker heartbeat age, backup age, verified-transfer age, free-disk percentage, persistent journal availability and probe error categories. Alert when the heartbeat is older than three minutes; backup or transfer is older than 30 hours; free disk falls below 25% (warning) or 15% (critical); the certificate has fewer than 30 days (warning) or 14 days (critical); any required container is absent/unhealthy; or the probe/timers fail. Route alerts to the monitoring owner and escalate unacknowledged critical events to the technical owner after 15 minutes. Exercise every trigger and its recovery notification with synthetic conditions.
|
||||
|
||||
Temporary plaintext files are held in private directories and removed on normal completion or exceptions. Process termination or power loss can leave temporary data, stopped services or TTL expiry disabled. After an interrupted run, inspect the dedicated project and remove only its identified abandoned temporary directory after securing any recovery material. Restore the recorded TTL setting (normally true) and restart the services:
|
||||
|
||||
```sh
|
||||
@ -146,4 +183,20 @@ Restore into new isolated MongoDB and key volumes; preserve the damaged original
|
||||
**A restored database can predate emails, invoices and PMS changes that providers already completed.** Review pending, sending and uncertain records against provider evidence before enabling any worker, including automatic FAQ rules. Do not replay an older approval merely because the restored record says it is pending. Reconcile external effects, validate account sessions and mailbox authorization, and explicitly approve the cutover only after these checks. Rotate credentials if compromise prompted the recovery. Keep the old deployment stopped when enabling the replacement.
|
||||
|
||||
CI exercises a synthetic encrypted backup and isolated restore drill, including actual key decryption and database comparison. A successful CI drill is separate from the required rehearsal on the Debian server with its actual deployment configuration.
|
||||
|
||||
## Milestone 11 acceptance record
|
||||
|
||||
Keep raw backups, restored data, remote paths, host keys, monitoring recipients, screenshots and logs outside Git. Copy `deploy/backup-restore-acceptance.example.json` to the restricted evidence store and replace every placeholder only after completing the supervised exercises. The example deliberately fails.
|
||||
|
||||
The accepted record uses a 24-hour RPO, four-hour RTO, seven-day verified local staging window, 35 daily and 12 monthly off-host recovery points, and the opaque evidence ID `backup-restore`. Bind it to the same release identifiers as the Debian-host and persistence records:
|
||||
|
||||
```sh
|
||||
python3 deploy/backup_restore_acceptance.py \
|
||||
/secure/acceptance/backup-restore.json \
|
||||
--expected-commit FULL_40_CHARACTER_SHA \
|
||||
--expected-release-record-sha256 RELEASE_RECORD_SHA256
|
||||
sha256sum /secure/acceptance/backup-restore.json
|
||||
```
|
||||
|
||||
The validator requires matching local/remote backup checksums, exact immutable image identities, a timed isolated restore, tested retention/legal hold and alert escalation, an image rollback that preserves persistent volumes, return to the approved candidate with external writes disabled, separate independent review, healthy monitoring and no unresolved critical findings. Structural validation does not inspect the restricted evidence or authorize a release by itself.
|
||||
|
||||
|
||||
129
tests/test_backup_restore_acceptance.py
Normal file
129
tests/test_backup_restore_acceptance.py
Normal file
@ -0,0 +1,129 @@
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import unittest
|
||||
|
||||
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"backup_restore_acceptance",
|
||||
Path(__file__).resolve().parents[1] / "deploy" / "backup_restore_acceptance.py")
|
||||
acceptance = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(acceptance)
|
||||
|
||||
COMMIT = "a" * 40
|
||||
RELEASE_SHA = "b" * 64
|
||||
|
||||
|
||||
def valid_record():
|
||||
return {
|
||||
"schemaVersion": 1,
|
||||
"system": "guestops-backup-recovery",
|
||||
"evidenceId": "backup-restore",
|
||||
"dataClassification": "synthetic-only",
|
||||
"releaseVersion": "0.2.0",
|
||||
"releaseCommit": COMMIT,
|
||||
"releaseRecordSha256": RELEASE_SHA,
|
||||
"archiveSha256": "c" * 64,
|
||||
"environment": "https://sandbox-guestops.futuresens.co.uk",
|
||||
"hostIdentifier": "guestops-sandbox-01",
|
||||
"images": {
|
||||
"api": {"reference": f"guestops-api:{COMMIT}", "id": "sha256:" + "d" * 64},
|
||||
"worker": {"reference": f"guestops-worker:{COMMIT}", "id": "sha256:" + "e" * 64},
|
||||
"mongo": {"reference": "mongo:8.0", "id": "sha256:" + "f" * 64},
|
||||
},
|
||||
"owners": {
|
||||
"backup": "Backup owner",
|
||||
"monitoring": "Monitoring owner",
|
||||
"recoveryOperator": "Recovery operator",
|
||||
"technicalEscalation": "Technical owner",
|
||||
"retention": "Retention owner",
|
||||
"independentReviewer": "Independent reviewer",
|
||||
},
|
||||
"startedAt": "2026-10-01T09:00:00Z",
|
||||
"endedAt": "2026-10-01T12:00:00Z",
|
||||
"reviewedAt": "2026-10-01T13:00:00Z",
|
||||
"recoveryObjectives": {
|
||||
"targetRpoHours": 24, "observedRpoHours": 1,
|
||||
"targetRtoMinutes": 240, "observedRtoMinutes": 180,
|
||||
},
|
||||
"retention": {
|
||||
"localVerifiedDays": 7, "offHostDaily": 35,
|
||||
"offHostMonthly": 12, "legalHoldOverrideTested": True,
|
||||
},
|
||||
"backup": {
|
||||
"createdAt": "2026-10-01T08:00:00Z",
|
||||
"sha256": "1" * 64,
|
||||
"transferredSha256": "1" * 64,
|
||||
"privateKeyPresentOnHost": False,
|
||||
},
|
||||
"rollback": {
|
||||
"previousReleaseCommit": "2" * 40,
|
||||
"previousArchiveSha256": "3" * 64,
|
||||
"previousImages": {
|
||||
"api": {"reference": "guestops-api:" + "2" * 40, "id": "sha256:" + "4" * 64},
|
||||
"worker": {"reference": "guestops-worker:" + "2" * 40, "id": "sha256:" + "5" * 64},
|
||||
},
|
||||
"persistentVolumesReplaced": False,
|
||||
"restoredReleaseCommit": COMMIT,
|
||||
"unresolvedOperations": 0,
|
||||
"finalControls": {
|
||||
"googleSending": "disabled", "faqLiveMode": "disabled",
|
||||
"pmsWrites": "disabled", "paymentCreation": "disabled",
|
||||
},
|
||||
},
|
||||
"monitoringState": "healthy",
|
||||
"unresolvedCriticalFindings": 0,
|
||||
"scenarios": [
|
||||
{"id": scenario, "status": "pass", "evidence": [f"restricted-{index}"]}
|
||||
for index, scenario in enumerate(sorted(acceptance.SCENARIOS), 1)
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class BackupRestoreAcceptanceTests(unittest.TestCase):
|
||||
def test_complete_record_passes(self):
|
||||
acceptance.validate(valid_record(), COMMIT, RELEASE_SHA)
|
||||
|
||||
def test_release_identity_and_images_are_bound(self):
|
||||
record = valid_record(); record["releaseCommit"] = "9" * 40
|
||||
with self.assertRaisesRegex(ValueError, "approved candidate"):
|
||||
acceptance.validate(record, COMMIT, RELEASE_SHA)
|
||||
record = valid_record(); record["images"]["api"]["reference"] = "guestops-api:latest"
|
||||
with self.assertRaisesRegex(ValueError, "full approved commit"):
|
||||
acceptance.validate(record, COMMIT, RELEASE_SHA)
|
||||
record = valid_record(); record["rollback"]["previousImages"]["worker"]["id"] = "mutable"
|
||||
with self.assertRaisesRegex(ValueError, "retained previous release identity"):
|
||||
acceptance.validate(record, COMMIT, RELEASE_SHA)
|
||||
|
||||
def test_recovery_objectives_and_checksums_must_pass(self):
|
||||
record = valid_record(); record["recoveryObjectives"]["observedRtoMinutes"] = 241
|
||||
with self.assertRaisesRegex(ValueError, "four-hour"):
|
||||
acceptance.validate(record, COMMIT, RELEASE_SHA)
|
||||
record = valid_record(); record["backup"]["transferredSha256"] = "9" * 64
|
||||
with self.assertRaisesRegex(ValueError, "checksums must match"):
|
||||
acceptance.validate(record, COMMIT, RELEASE_SHA)
|
||||
record = valid_record(); record["recoveryObjectives"]["observedRpoHours"] = 2
|
||||
with self.assertRaisesRegex(ValueError, "match the backup"):
|
||||
acceptance.validate(record, COMMIT, RELEASE_SHA)
|
||||
|
||||
def test_independent_review_retention_and_final_state_are_required(self):
|
||||
record = valid_record(); record["owners"]["independentReviewer"] = record["owners"]["backup"]
|
||||
with self.assertRaisesRegex(ValueError, "different"):
|
||||
acceptance.validate(record, COMMIT, RELEASE_SHA)
|
||||
record = valid_record(); record["retention"]["legalHoldOverrideTested"] = False
|
||||
with self.assertRaisesRegex(ValueError, "Retention"):
|
||||
acceptance.validate(record, COMMIT, RELEASE_SHA)
|
||||
record = valid_record(); record["rollback"]["persistentVolumesReplaced"] = True
|
||||
with self.assertRaisesRegex(ValueError, "must not replace"):
|
||||
acceptance.validate(record, COMMIT, RELEASE_SHA)
|
||||
|
||||
def test_exact_passed_scenarios_and_monitoring_are_required(self):
|
||||
record = valid_record(); record["scenarios"].pop()
|
||||
with self.assertRaisesRegex(ValueError, "exact backup/recovery scenario"):
|
||||
acceptance.validate(record, COMMIT, RELEASE_SHA)
|
||||
record = valid_record(); record["monitoringState"] = "degraded"
|
||||
with self.assertRaisesRegex(ValueError, "Monitoring must be healthy"):
|
||||
acceptance.validate(record, COMMIT, RELEASE_SHA)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
81
tests/test_backup_transfer.py
Normal file
81
tests/test_backup_transfer.py
Normal file
@ -0,0 +1,81 @@
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"backup_transfer", Path(__file__).resolve().parents[1] / "deploy" / "backup_transfer.py")
|
||||
transfer = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(transfer)
|
||||
|
||||
|
||||
class BackupTransferTests(unittest.TestCase):
|
||||
def backup(self, directory: Path, name="guestops-20261001T021700Z.tar.gpg") -> Path:
|
||||
path = directory / name
|
||||
path.write_bytes(b"encrypted-backup")
|
||||
return path
|
||||
|
||||
def config(self, directory: Path):
|
||||
return {
|
||||
"directory": directory, "host": "store.example.invalid", "user": "guestops_upload",
|
||||
"remote": "/restricted/guestops", "identity": Path("/safe/key"),
|
||||
"known_hosts": Path("/safe/known_hosts"),
|
||||
}
|
||||
|
||||
def test_existing_matching_remote_is_marked_without_upload(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
backup = self.backup(Path(folder)); checksum = transfer.digest(backup)
|
||||
with patch.object(transfer, "remote_digest", return_value=checksum), \
|
||||
patch.object(transfer, "run") as run:
|
||||
transfer.transfer_one(self.config(Path(folder)), backup)
|
||||
run.assert_not_called()
|
||||
marker = json.loads(transfer.marker_path(backup).read_text(encoding="utf-8"))
|
||||
self.assertEqual(marker["sha256"], checksum)
|
||||
|
||||
def test_existing_mismatched_remote_is_never_overwritten(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
backup = self.backup(Path(folder))
|
||||
with patch.object(transfer, "remote_digest", return_value="9" * 64), \
|
||||
patch.object(transfer, "run") as run:
|
||||
with self.assertRaisesRegex(RuntimeError, "different checksum"):
|
||||
transfer.transfer_one(self.config(Path(folder)), backup)
|
||||
run.assert_not_called()
|
||||
self.assertFalse(transfer.marker_path(backup).exists())
|
||||
|
||||
def test_new_remote_is_uploaded_verified_and_marked(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
backup = self.backup(Path(folder)); checksum = transfer.digest(backup)
|
||||
with patch.object(transfer, "remote_digest", side_effect=[None, checksum, checksum]), \
|
||||
patch.object(transfer, "run", return_value=b"") as run, \
|
||||
patch.object(transfer.subprocess, "run"):
|
||||
transfer.transfer_one(self.config(Path(folder)), backup)
|
||||
self.assertTrue(any(call.args[0][0] == "rsync" for call in run.call_args_list))
|
||||
self.assertTrue(transfer.marker_path(backup).exists())
|
||||
|
||||
def test_prune_removes_only_old_checksum_verified_backups(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
directory = Path(folder)
|
||||
verified = self.backup(directory)
|
||||
checksum = transfer.digest(verified)
|
||||
transfer.write_marker(verified, checksum)
|
||||
unverified = self.backup(directory, "guestops-20261002T021700Z.tar.gpg")
|
||||
old = time.time() - 8 * 86400
|
||||
os.utime(verified, (old, old)); os.utime(unverified, (old, old))
|
||||
removed = transfer.prune_verified(self.config(directory), 7, now=time.time())
|
||||
self.assertEqual(removed, 1)
|
||||
self.assertFalse(verified.exists())
|
||||
self.assertTrue(unverified.exists())
|
||||
|
||||
def test_retention_bounds_are_enforced(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
with self.assertRaisesRegex(RuntimeError, "between 1 and 365"):
|
||||
transfer.prune_verified(self.config(Path(folder)), 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
113
tests/test_debian_acceptance.py
Normal file
113
tests/test_debian_acceptance.py
Normal file
@ -0,0 +1,113 @@
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import unittest
|
||||
|
||||
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"debian_acceptance", Path(__file__).resolve().parents[1] / "deploy" / "debian_acceptance.py")
|
||||
acceptance = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(acceptance)
|
||||
|
||||
COMMIT = "a" * 40
|
||||
RELEASE_SHA = "b" * 64
|
||||
|
||||
|
||||
def common(system):
|
||||
return {
|
||||
"schemaVersion": 1,
|
||||
"system": system,
|
||||
"evidenceId": acceptance.SYSTEMS[system]["evidenceId"],
|
||||
"releaseVersion": "0.2.0",
|
||||
"releaseCommit": COMMIT,
|
||||
"releaseRecordSha256": RELEASE_SHA,
|
||||
"archiveSha256": "c" * 64,
|
||||
"environment": "https://sandbox-guestops.futuresens.co.uk",
|
||||
"hostIdentifier": "guestops-sandbox-01",
|
||||
"images": {
|
||||
"api": {"reference": f"guestops-api:{COMMIT}", "id": "sha256:" + "d" * 64},
|
||||
"worker": {"reference": f"guestops-worker:{COMMIT}", "id": "sha256:" + "e" * 64},
|
||||
},
|
||||
"operator": "Deployment operator",
|
||||
"reviewedBy": "Independent reviewer",
|
||||
"startedAt": "2026-09-30T09:00:00Z",
|
||||
"endedAt": "2026-09-30T10:00:00Z",
|
||||
"reviewedAt": "2026-09-30T11:00:00Z",
|
||||
"unresolvedCriticalFindings": 0,
|
||||
"scenarios": [
|
||||
{"id": scenario, "status": "pass", "evidence": [f"restricted-{index}"]}
|
||||
for index, scenario in enumerate(sorted(acceptance.SYSTEMS[system]["scenarios"]), 1)
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def valid_records():
|
||||
host = common("guestops-debian-host")
|
||||
host["hostFacts"] = {
|
||||
"debianMajor": 12,
|
||||
"cpuCores": 4,
|
||||
"memoryBytes": 8_140_382_208,
|
||||
"freeDiskBytes": 14_275_686_400,
|
||||
"publicTcpPorts": [80, 443],
|
||||
}
|
||||
host["featureControls"] = {
|
||||
"googleSending": "disabled",
|
||||
"faqLiveMode": "disabled",
|
||||
"pmsWrites": "disabled",
|
||||
"paymentCreation": "disabled",
|
||||
}
|
||||
persistence = common("guestops-persistence")
|
||||
persistence["drillCommand"] = "python3 deploy/ops.py persistence-drill --confirm-restart"
|
||||
return host, persistence
|
||||
|
||||
|
||||
class DebianAcceptanceTests(unittest.TestCase):
|
||||
def test_complete_matching_records_pass(self):
|
||||
acceptance.validate_pair(*valid_records(), COMMIT, RELEASE_SHA)
|
||||
|
||||
def test_expected_release_identity_is_required(self):
|
||||
host, persistence = valid_records()
|
||||
host["releaseCommit"] = "f" * 40
|
||||
with self.assertRaisesRegex(ValueError, "approved candidate"):
|
||||
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
|
||||
host, persistence = valid_records()
|
||||
persistence["releaseRecordSha256"] = "f" * 64
|
||||
with self.assertRaisesRegex(ValueError, "retained release record"):
|
||||
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
|
||||
|
||||
def test_exact_images_and_cross_record_identity_are_required(self):
|
||||
host, persistence = valid_records()
|
||||
host["images"]["api"]["reference"] = "guestops-api:latest"
|
||||
with self.assertRaisesRegex(ValueError, "full approved commit"):
|
||||
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
|
||||
host, persistence = valid_records()
|
||||
persistence["archiveSha256"] = "f" * 64
|
||||
with self.assertRaisesRegex(ValueError, "same archiveSha256"):
|
||||
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
|
||||
|
||||
def test_host_capacity_ports_and_disabled_controls_are_required(self):
|
||||
host, persistence = valid_records()
|
||||
host["hostFacts"]["publicTcpPorts"] = [80, 443, 8080]
|
||||
with self.assertRaisesRegex(ValueError, "Only TCP ports"):
|
||||
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
|
||||
host, persistence = valid_records()
|
||||
host["featureControls"]["googleSending"] = "enabled"
|
||||
with self.assertRaisesRegex(ValueError, "must remain disabled"):
|
||||
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
|
||||
|
||||
def test_independent_review_scenarios_and_findings_are_required(self):
|
||||
host, persistence = valid_records()
|
||||
host["reviewedBy"] = host["operator"]
|
||||
with self.assertRaisesRegex(ValueError, "different people"):
|
||||
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
|
||||
host, persistence = valid_records()
|
||||
persistence["scenarios"][0]["status"] = "not-run"
|
||||
with self.assertRaisesRegex(ValueError, "has not passed"):
|
||||
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
|
||||
host, persistence = valid_records()
|
||||
host["unresolvedCriticalFindings"] = 1
|
||||
with self.assertRaisesRegex(ValueError, "critical findings"):
|
||||
acceptance.validate_pair(host, persistence, COMMIT, RELEASE_SHA)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
75
tests/test_monitor_status.py
Normal file
75
tests/test_monitor_status.py
Normal file
@ -0,0 +1,75 @@
|
||||
import datetime as dt
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"monitor_status", Path(__file__).resolve().parents[1] / "deploy" / "monitor_status.py")
|
||||
monitor = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(monitor)
|
||||
|
||||
|
||||
class MonitorStatusTests(unittest.TestCase):
|
||||
def test_compose_json_is_reduced_to_safe_state(self):
|
||||
value = json.dumps([
|
||||
{"Service": "api", "State": "running", "Health": "healthy", "Publishers": "secret"},
|
||||
{"Service": "worker", "State": "running", "Health": ""},
|
||||
{"Service": "mongo", "State": "running", "Health": "healthy"},
|
||||
]).encode()
|
||||
self.assertEqual(monitor.parse_compose(value)["api"], {"state": "running", "health": "healthy"})
|
||||
self.assertNotIn("Publishers", monitor.parse_compose(value)["api"])
|
||||
|
||||
def test_age_ignores_symlinks_and_unexpected_files(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
directory = Path(folder)
|
||||
backup = directory / "guestops-20261001T021700Z.tar.gpg"
|
||||
backup.write_bytes(b"fixture")
|
||||
moment = dt.datetime(2026, 10, 1, 3, 17, tzinfo=dt.timezone.utc)
|
||||
os.utime(backup, (moment.timestamp() - 3600, moment.timestamp() - 3600))
|
||||
(directory / "secret.txt").write_text("ignored", encoding="utf-8")
|
||||
self.assertEqual(monitor.age_seconds(directory, monitor.BACKUP_NAME, moment), 3600)
|
||||
|
||||
def test_build_status_contains_only_aggregate_health(self):
|
||||
now = dt.datetime(2026, 10, 1, 12, tzinfo=dt.timezone.utc)
|
||||
compose = json.dumps([
|
||||
{"Service": name, "State": "running", "Health": "healthy"}
|
||||
for name in ("api", "worker", "mongo")
|
||||
]).encode()
|
||||
usage = type("Usage", (), {"total": 1000, "used": 500, "free": 500})()
|
||||
with tempfile.TemporaryDirectory() as folder, \
|
||||
patch.object(monitor, "https_status", return_value={"ready": True, "certificateDaysRemaining": 60}), \
|
||||
patch.object(monitor, "run", return_value=compose), \
|
||||
patch.object(monitor, "worker_heartbeat_age", return_value=30), \
|
||||
patch.object(monitor.shutil, "disk_usage", return_value=usage):
|
||||
status, errors = monitor.build_status(Path(folder), Path(folder), "https://example.invalid", now)
|
||||
self.assertEqual(errors, [])
|
||||
self.assertEqual(status["workerHeartbeatAgeSeconds"], 30)
|
||||
self.assertEqual(status["diskFreePercent"], 50)
|
||||
self.assertNotIn("credentials", json.dumps(status).lower())
|
||||
|
||||
def test_failed_probes_write_categories_not_exception_details(self):
|
||||
now = dt.datetime(2026, 10, 1, 12, tzinfo=dt.timezone.utc)
|
||||
usage = type("Usage", (), {"total": 1000, "used": 500, "free": 500})()
|
||||
with tempfile.TemporaryDirectory() as folder, \
|
||||
patch.object(monitor, "https_status", side_effect=RuntimeError("secret value")), \
|
||||
patch.object(monitor, "run", side_effect=RuntimeError("secret value")), \
|
||||
patch.object(monitor, "worker_heartbeat_age", side_effect=RuntimeError("secret value")), \
|
||||
patch.object(monitor.shutil, "disk_usage", return_value=usage):
|
||||
status, errors = monitor.build_status(Path(folder), Path(folder), "https://example.invalid", now)
|
||||
self.assertGreaterEqual(len(errors), 3)
|
||||
self.assertNotIn("secret value", json.dumps(status))
|
||||
|
||||
def test_status_output_is_atomic_json(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
path = Path(folder) / "status.json"
|
||||
monitor.write_status(path, {"schemaVersion": 1, "errors": []})
|
||||
self.assertEqual(json.loads(path.read_text(encoding="utf-8"))["schemaVersion"], 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Loading…
x
Reference in New Issue
Block a user