GuestOps asible playbook

This commit is contained in:
2026-10-21 11:55:33 +01:00
parent c5c6ec1453
commit db8422afda
61 changed files with 108 additions and 4417 deletions
+1 -1
View File
@@ -27,7 +27,7 @@ unset RECOVERY_EMAIL
The command uses the configured database, generates a private link, prints it to the administrator's terminal and exits. Share it only with the verified owner. Do not paste it into tickets, chat logs, build logs or source control; avoid running this command in a recorded terminal. It expires in 30 minutes and can be consumed once. Running the command again invalidates the earlier link. Bootstrap remains a create-only command and cannot reset passwords.
`PublicUrl` must be the configured HTTPS origin, normally `https://sandbox-guestops.futuresens.co.uk`. Request Host headers never determine recovery-link origins. The Development-only preview uses `http://127.0.0.1:5173`.
`PublicUrl` must be the configured HTTPS origin, such as `https://guestops-sandbox.futuresens.co.uk`. Request Host headers never determine recovery-link origins. The Development-only preview uses `http://127.0.0.1:5173`.
## Storage and session behavior
+45 -100
View File
@@ -1,120 +1,65 @@
# Debian sandbox deployment
# GuestOps deployment
Target: `https://sandbox-guestops.futuresens.co.uk`. These are reviewable deployment instructions; creating these files does not deploy or change the server.
GuestOps is deployed by the central Futuresens Ansible repository. Sandbox is
`https://guestops-sandbox.futuresens.co.uk` and production is
`https://guestops.futuresens.co.uk`.
## 1. Prepare the host
## Prepare a version
Check the existing Nginx, Docker, MongoDB and firewall configuration before installation. Preserve existing services. The provided Compose file creates a dedicated MongoDB instance for GuestOps with no host port; if MongoDB already exists on the host, use a reviewed private connection and application-scoped user instead of starting a conflicting instance.
Install Docker Engine/Compose and Nginx using their official Debian instructions. Keep SSH access unchanged. Only HTTPS/HTTP for this hostname need public access; port 8080 is loopback-only and MongoDB has no published port.
Install the versioned source package through the Futuresens Ansible repository. Do not clone GuestOps from the target server and do not place Gitea credentials on it. The playbook should extract the package into a dedicated application directory. Copy `.env.example` to `.env`, set permissions to 600, and fill two different MongoDB passwords generated with `openssl rand -hex 32`. Use hex values so they are safe in the MongoDB URI. Store real values only on the server or in its secret-management system.
The MongoDB initialization script runs only on a new volume. Changing `.env` later does not rotate existing database users. Rotate those credentials through MongoDB administration and update application configuration together.
## 2. Package and install with Ansible
GuestOps follows the CMS/CMSFront deployment pattern: Gitea stores the application source, a specific committed version is compressed, and a version-selected Ansible playbook installs it. No Gitea Actions runner is required.
On the trusted packaging machine, fetch the current remote refs, run the automated checks, select the full commit SHA, and create the archive from that committed tree rather than from a working directory:
The .NET version in `Directory.Build.props` and the frontend version in
`web/package.json` must match. Test the committed revision, then package it:
```sh
git fetch origin --prune --tags
git rev-parse origin/main
dotnet run --project tests/GuestOps.Tests
python3 -m unittest discover -s tests -p "test_*.py"
cd web && npm ci && npm run build && cd ..
python3 deploy/package_source.py \
--ref FULL_40_CHARACTER_SHA \
--output-directory /secure/release-staging
--ref FULL_COMMIT_OR_TAG \
--output-directory ../ansible/playbooks/files/guestops
```
The packaging command resolves the ref to a full commit, reads both application versions from that committed tree, requires them to match, creates deterministic gzip output, and writes a `.source.json` record containing the version, commit, filename, size and SHA-256. Existing package files are never overwritten. Independently confirm that the selected commit is the intended release line before transferring it.
For version `0.2.1`, this creates `guestops_0.2.1.tar.gz`. Published version
files are immutable; change the application version before making another
archive.
Store the archive and source record under a versioned GuestOps files directory in the private Ansible repository. The GuestOps playbook and environment variables should select that version, verify the archive against the source record, extract it into the application directory, preserve the private `.env` and provider configuration, and build images tagged with the full source commit:
## Configure Ansible
The application-owned handoff playbook and required variables are documented in [`deploy/ansible/README.md`](../deploy/ansible/README.md). Import or copy that playbook into the central private Ansible repository, bind its `guestops` inventory group, and keep inventory and secret values there.
The server group needs only the application switch, version and subdomain:
The playbook requires Debian 12 or newer with the approved CPU/RAM baseline, enables Docker and Nginx at boot, installs the reviewed site only after loopback readiness, runs `nginx -t`, rejects a host MongoDB listener or non-loopback API listener, and verifies the public redirect and trusted HTTPS readiness before selecting the release. Firewall reachability, certificate renewal, controlled reboot, durable logs and the persistence exercise still require the supervised checks below.
```yaml
hasguestops: true
guestopsversion: "0.2.1"
guestopssubdomain: "guestops-sandbox"
```
Production uses `guestopssubdomain: "guestops"`. Runtime credentials and the
MongoDB connection string belong in encrypted Ansible Vault data. The MongoDB
URI must use a least-privilege `guestops` database user and an address reachable
from Docker containers.
## Deploy
Deploy sandbox first:
```sh
docker build --target api -t guestops-api:FULL_40_CHARACTER_SHA .
docker build --target worker -t guestops-worker:FULL_40_CHARACTER_SHA .
ansible-playbook playbooks/guestops.yml -K --ask-vault-pass \
--extra-vars "thehost=sandbox"
```
Record the resulting immutable image IDs and bind them to the source archive with `deploy/release_record.py`. Retain the source archive, checksum, release record, its checksum, build output, commit and Ansible run result in the restricted release store. Set `GUESTOPS_API_IMAGE=guestops-api:FULL_40_CHARACTER_SHA` and `GUESTOPS_WORKER_IMAGE=guestops-worker:FULL_40_CHARACTER_SHA` in `.env`. Then the playbook runs:
After acceptance, deploy the same versioned archive to production:
```sh
docker compose config --quiet
docker compose up -d --no-build
curl --fail http://127.0.0.1:8080/health
ansible-playbook playbooks/guestops.yml -K --ask-vault-pass \
--extra-vars "thehost=dbserver"
```
Do not run `docker compose config` without `--quiet` in shared logs: expanded configuration contains secrets. Do not store `.env`, provider credentials, host inventory secrets, or restricted evidence in either application repository. Ansible must stop on a checksum, commit, version, build, or health-check mismatch.
The playbook stops the existing Compose project without deleting volumes,
replaces `/srv/guestops`, renders private configuration, builds and starts the
API and worker, configures Nginx, and verifies local and public readiness. Port
8080 remains loopback-only and the `guestops_app-keys` volume survives
redeployment.
## 3. Provision the first hotel owner
The API has an administrative bootstrap command. It creates one hotel and one owner, refuses an existing email address, and does not reset passwords. Run it with environment variables passed from a private terminal; do not put the password directly in shell history or a repository file.
```sh
read -r -p 'Owner email: ' BOOTSTRAP_EMAIL
read -r -p 'Hotel name: ' BOOTSTRAP_HOTEL
read -r -s -p 'Owner password (14+ characters): ' BOOTSTRAP_PASSWORD
printf '\n'
export BOOTSTRAP_EMAIL BOOTSTRAP_HOTEL BOOTSTRAP_PASSWORD
docker compose run --rm --no-deps -e BOOTSTRAP_EMAIL -e BOOTSTRAP_HOTEL -e BOOTSTRAP_PASSWORD api --bootstrap
unset BOOTSTRAP_EMAIL BOOTSTRAP_HOTEL BOOTSTRAP_PASSWORD
```
There is no development/demo account in production. Owners can issue staff invitation and assisted recovery links through Team; see [account setup](accounts.md). Passwords must be 14–128 characters. Public self-registration is not enabled.
## 4. Configure HTTPS
Verify the hostname's DNS resolves to this server. Obtain a valid certificate for it using your existing ACME/Certbot process. The Nginx example expects a Let's Encrypt certificate. For first issuance, configure the port-80 ACME location before enabling the port-443 block; do not point Nginx at missing certificate files.
Merge `deploy/nginx.conf` into the existing host configuration, check with `nginx -t`, then reload Nginx. Confirm HTTPS serves the login page. API cookies are always Secure outside preview mode; logging in through plain HTTP is intentionally unsupported.
Compose reserves the private bridge `172.30.87.0/24`, with gateway `172.30.87.1`. The API trusts the host gateway's `X-Forwarded-Proto` header so Nginx's HTTPS connections receive secure session and CSRF cookies. Check for an existing network using that range. If it conflicts, set both `GUESTOPS_SUBNET` and `GUESTOPS_GATEWAY` in `.env` to a free matching subnet and gateway. Do not replace this with unrestricted forwarded-header trust.
The login rate limiter uses the client address forwarded by the explicitly configured host proxy (10 attempts per client address per minute). ASP.NET accepts one forwarding hop only from `Proxy__KnownAddress`; arbitrary client-supplied forwarding headers are not trusted. Keep the API port loopback-only and update the known address together with any reviewed Compose subnet change.
## 5. Configure Google
Create or select your Google Cloud project, enable Gmail API, and configure a Web application OAuth client. Register this exact redirect URI:
`https://sandbox-guestops.futuresens.co.uk/api/integrations/google/callback`
Configure `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET` in the server's `.env` and recreate the API/worker services. Sign in as hotel owner, open Settings and connect a dedicated test mailbox. By default the only requested Gmail scope is `gmail.readonly`. Optional staff-approved sending adds `gmail.send` after server configuration and reconnection; see [AI drafts and reply delivery](replies.md).
OAuth requests expire after ten minutes, are bound to the signed-in user and hotel, and can be consumed once. Refresh tokens are protected with ASP.NET Data Protection. API and worker share the private persistent key volume; back it up securely with the database. Losing it prevents existing mailbox tokens and sessions from being decrypted. Filesystem protection for the key volume is required; it is separate from MongoDB and must not be publicly served or committed.
A multi-hotel production launch using Gmail restricted scopes requires planning for Google's verification/security-assessment requirements. This implementation does not bypass those requirements. See [Google restricted-scope verification](https://developers.google.com/identity/protocols/oauth2/production-readiness/restricted-scope-verification).
## 6. Acceptance and rollback
Verify separate hotels cannot read or edit each other's records; save and reload settings; restart services and confirm persistence; import test messages twice without duplicates; check the worker resumes a paginated import; confirm no mail is sent without explicit staff approval and that default-disabled sending remains blocked. Review the activity log and Google account used by the connection.
Keep reviewed image IDs and release images for rollback and backups of both MongoDB and the key volume. Do not remove named volumes to fix application errors. The initial release has no automatic schema migration that destroys data. Use the [operational preflight, encrypted backup and isolated restore drill](operations.md), and establish retention and off-server copies before importing real guest data. `/health/ready` checks database reachability; the owner's Workspace health page also reports worker heartbeat and mailbox/reply exceptions.
Before accepting the host, run the confirmation-gated persistence drill during an announced maintenance window:
```sh
python3 deploy/ops.py preflight
python3 deploy/ops.py persistence-drill --confirm-restart
python3 deploy/ops.py preflight
```
The drill restarts MongoDB, API and worker, then force-recreates the stateless application containers using the already selected images. It verifies that the resolved database and key volumes keep the same identities, MongoDB collection counts and indexes remain unchanged, a value protected before restart can still be decrypted, and loopback readiness recovers. It does not alter provider feature flags, upgrade images, validate public TLS, or replace the separate backup/restore drill.
Record the host, operator, start/end time, release record checksum, resolved image IDs, preflight output and drill result in the deployment acceptance record. Also verify Docker starts at boot and perform a controlled Debian reboot before Gate A approval. After reboot, run the online preflight and inspect the Workspace health page; do not infer worker health solely from API readiness.
Keep the `debian-host` and `persistence` records in the restricted evidence store. Start from `deploy/debian-host-acceptance.example.json` and `deploy/persistence-acceptance.example.json`; the examples deliberately fail until every supervised scenario has passed. Bind both records to the expected release identifiers and validate them together:
```sh
python3 deploy/debian_acceptance.py \
/secure/acceptance/debian-host.json \
/secure/acceptance/persistence.json \
--expected-commit FULL_40_CHARACTER_SHA \
--expected-release-record-sha256 RELEASE_RECORD_SHA256
```
The validator requires matching archive and image identities, separate operator and reviewer names, the approved host capacity, only ports 80 and 443 recorded as publicly reachable, disabled unaccepted external writes, exact passed scenario sets and no unresolved critical findings. It validates record structure, not the restricted evidence itself. Retain the records, validator output and their checksums outside Git.
The supplied Docker `json-file` logs are size-capped to protect the small pilot disk, but container recreation removes that container's local log history. Before host acceptance, route GuestOps and Nginx logs to the site's durable restricted logging system, or use a reviewed Docker logging override backed by persistent systemd journal storage. Prove that operators can retrieve pre-recreation logs without exposing request credentials or OAuth callback query strings. Central retention and alerting are completed under milestone 11.
To roll back, select an older retained `guestopsversion` in group vars and run
the same playbook. MongoDB backup, Filebeat, certificates, and host monitoring
are provided by the existing central infrastructure.
+1 -1
View File
@@ -6,7 +6,7 @@ Keep screenshots, Gmail message source, provider console records and request dia
## Preparation and stop conditions
Record the release commit, release-record SHA-256, image IDs, environment, operator, approver and maintenance window. Use two dedicated Google test accounts so the different-account rejection can be exercised without a personal account. Send only clearly synthetic messages between controlled recipients.
Record the deployed GuestOps version, environment, operator, approver and maintenance window. Use two dedicated Google test accounts so the different-account rejection can be exercised without a personal account. Send only clearly synthetic messages between controlled recipients.
Start with AI drafts, staff sending, FAQ live mode, PMS writes and payment creation disabled. Confirm the OAuth redirect URI exactly matches the HTTPS sandbox. Stop immediately if a mailbox appears under the wrong hotel, a recipient differs from the review screen, an unapproved message is sent, a duplicate is observed, provider output exposes credentials, or an uncertain outcome is about to be blindly retried. Preserve evidence and follow the incident process.
+1 -1
View File
@@ -17,7 +17,7 @@ For Gate C, also exercise `pms-ambiguous-write` and `payment-ambiguous-create`.
## Record and validation
Copy `deploy/incident-exercise.example.json` into the restricted acceptance store. Bind it to the same full release commit and release-record SHA-256 used by the deployment and pilot decision. Replace the example timestamps, people, targets and evidence references. Change a scenario to `pass` only after its evidence has been independently reviewed. The example is intentionally invalid because its scenarios are `not-run`.
Copy `deploy/incident-exercise.example.json` into the restricted acceptance store and record the deployed GuestOps version and environment. Replace the example timestamps, people, targets and evidence references. Change a scenario to `pass` only after its evidence has been independently reviewed. The example is intentionally invalid because its scenarios are `not-run`.
For a Gate C exercise, set `targetGate` to `C` and add the PMS and payment scenario records. Validate the completed record with:
+2 -2
View File
@@ -42,11 +42,11 @@ Implemented owner-only disconnect/reconnect and import restart controls, synchro
## Milestone 8: operational readiness
Implemented owner-only workspace health, database readiness, worker heartbeat, Linux deployment preflight, maintenance-window encrypted database/key/configuration backup and an isolated restore drill. The drill verifies collection counts, indexes and actual key decryption. Scheduling, off-server copies and production disaster cutover remain operator tasks; the Debian server rehearsal is still required. See [operations and recovery](operations.md).
The application provides owner-only workspace health, database readiness and worker heartbeat information. Docker, Filebeat, MongoDB backup, host monitoring and disaster recovery are owned by the existing central infrastructure rather than duplicated inside GuestOps.
## Remaining milestones
1. Run dedicated Google test-mailbox acceptance, add full thread aggregation and history repair, and rehearse server backup/restore before the first hotel pilot.
1. Run dedicated Google test-mailbox acceptance, add full thread aggregation and history repair, and confirm the central server backup/restore process before the first hotel pilot.
2. Add verified transactional email for invitations and recovery notifications, MFA, granular roles and user preferences.
3. Run a representative live AI draft evaluation, improve retrieval and evidence presentation, and approve the selected provider data-processing arrangements.
4. Extend the implemented durable reply queue with operator recovery tooling and broaden the controlled FAQ rules only after live acceptance. Preserve the rule that uncertain sends are never blindly replayed.
-208
View File
@@ -1,208 +0,0 @@
# Operations and recovery
The owner-only **Workspace health** page reports database reachability, the worker's last heartbeat, mailbox recovery counts and reply exceptions. A heartbeat older than three minutes is marked stale. It proves that the worker process recently reached MongoDB, not that every provider or job succeeded. Reply totals cover at most the latest 500 conversations in this hotel. The page does not verify backups. `/health/ready` returns only readiness status and HTTP 503 when the database cannot be reached.
## Release evidence and rollback
Create the release source archive from an exact committed Gitea tree with `deploy/package_source.py`, then verify its generated source record before Ansible installs it. The tool packages only the selected commit, produces deterministic gzip output, rejects version disagreement, and refuses to overwrite an existing release. Ansible builds the API and worker images under the full Git commit SHA. The post-build `release-record.json` binds the source-archive checksum, application version, commit, image references and immutable Docker image IDs. Retain the archive, source record, release record and Ansible result together in restricted off-host release storage.
```sh
git fetch origin --prune --tags
python3 deploy/package_source.py \
--ref FULL_40_CHARACTER_SHA \
--output-directory /secure/release-staging
```
Before deployment, verify the archive against its source record:
```sh
python3 - <<'PY'
import hashlib, json, pathlib
r = json.load(open('GuestOps-0.2.1-COMMIT.source.json', encoding='utf-8'))
p = pathlib.Path(r['artifact']['name'])
assert hashlib.sha256(p.read_bytes()).hexdigest() == r['artifact']['sha256']
print(r['commit'], r['version'], r['artifact'])
PY
```
After Ansible has built the commit-tagged images, the repository verifier performs the same checks strictly, calculates the release-record checksum used by later acceptance records, and compares the record with the installed Docker images:
```sh
python3 deploy/verify_release.py \
--archive GuestOps-0.2.1.tar.gz \
--record release-record.json \
--commit FULL_40_CHARACTER_SHA \
--version 0.2.1 \
--output release-verification.json
python3 deploy/verify_release.py \
--archive GuestOps-0.2.1.tar.gz \
--record release-record.json \
--commit FULL_40_CHARACTER_SHA \
--version 0.2.1 \
--verify-loaded-images \
--output loaded-image-verification.json
```
Retain both verification summaries with the untouched source archive, its checksum, the release record, its SHA-256, and the exact Ansible run metadata. A package from an uncommitted working tree, a mismatched checksum, or a failed Ansible run is not release evidence. Do not silently replace an approved package or rebuild under the same release identity.
Verify each installed image ID matches the record, set `GUESTOPS_API_IMAGE` and `GUESTOPS_WORKER_IMAGE` to the recorded full-SHA references, and run the deployment preflight. Record the Ansible run, commit, archive checksum and operator in the change ticket. A release tag is an approval marker; do not move or reuse an existing tag. The application and web versions must match before the package is accepted.
For rollback, first disable worker-driven external writes and reconcile any sending, payment or PMS operation that may have completed since the prior release. Confirm the previous release archive and record are retained, verify its checksum and image IDs, take an encrypted backup, then select the previous recorded image references in `.env` and recreate only the API and worker. Do not roll back MongoDB or the key volume merely to change application images. Run the online preflight, readiness check and read-only smoke test before re-enabling the worker or provider writes. If a release introduced an incompatible data change, follow its release-specific recovery plan rather than starting an older image against newer data.
## Deployment preflight
Run from the dedicated GuestOps checkout on Linux with Python 3.11 or later, Docker with Compose, and GnuPG installed. The tool supports the supplied three-service Compose deployment and the `guestops` database only. Docker access is administrator-equivalent; use the designated server operator account. Load the reviewed API, worker and MongoDB images first, configure `.env` with mode 600, and follow [deployment](deployment.md).
```sh
python3 deploy/ops.py preflight --offline
# After Nginx, DNS, TLS and services are running:
python3 deploy/ops.py preflight
```
Preflight checks production settings, shared persistent keys, unpublished MongoDB/worker ports, a loopback API port, required images and mounts, private secret files, and at least 8 GiB free disk. The online check also verifies the configured HTTPS readiness URL. This is not a firewall, capacity or provider acceptance test. Allow additional disk space for the database, images and temporary backup/restore files; the supplied server initially had 18 GiB free.
Configured provider files must be readable by the API container's `app` user while inaccessible to other users. Set their ownership to that image's application UID and mode 600, and run the backup as an authorized operator able to read them (for example root with its designated public GPG keyring). Keep their parent directory private. Empty shipped example files contain no credentials and do not need this ownership change. Check the UID in the reviewed image rather than assuming it matches your host login.
## Encrypted backup
Keep the recovery private key on an administrator-controlled recovery machine, with a securely stored passphrase and a second protected recovery copy. Import only its public key on the server and verify its full 40-character fingerprint through a trusted channel. The tool selects that exact fingerprint; it does not establish who owns the key. Do not put private keys, decrypted backups or provider secrets in GitHub.
```sh
gpg --import /secure/path/recovery-public.asc
gpg --fingerprint
install -d -m 700 "$HOME/guestops-backups"
read -r -p 'Verified recovery key fingerprint: ' BACKUP_RECIPIENT
python3 deploy/ops.py backup --recipient "$BACKUP_RECIPIENT" \
--output "$HOME/guestops-backups/guestops-$(date -u +%Y%m%dT%H%M%SZ).tar.gpg" \
--confirm-maintenance
unset BACKUP_RECIPIENT
```
This command causes a maintenance interruption. It stops the API and worker, temporarily pauses MongoDB's TTL expiry monitor, dumps MongoDB, and saves the shared Data Protection keys and deployed configuration. The configuration includes credentials and provider files, so the entire bundle is encrypted. Services and the previous TTL setting are restored in a `finally` block before encryption finishes. A successful backup message does not prove that restarted services are healthy; run the online preflight afterwards.
No other application or administrator may write to this database during the snapshot. Standalone MongoDB dumps need coordinated writes for consistency; see the [MongoDB backup guidance](https://www.mongodb.com/docs/v8.0/tutorial/backup-and-restore-tools/). The tool is intentionally limited to the dedicated stack. It requires the hexadecimal MongoDB root password generated in the deployment guide.
Copy the encrypted file off-server to restricted storage after every successful backup. Retain the exact API, worker and MongoDB images with the release: an image tag alone can change, and the drill requires matching image IDs. Agree the backup schedule, retention and tolerated data loss before a hotel pilot. Scheduling, off-server transfer, deletion and alerting are operator responsibilities in this milestone; none is silently installed.
### Optional systemd schedule
The repository includes an opt-in daily systemd service and timer. The service follows the Ansible-selected release at `/srv/guestops/current` and stages encrypted files in `/var/backups/guestops`. The application-owned `deploy/ansible/guestops-operations.yml` installs and verifies these units after the exact release is selected; keep the backup timer disabled until the supervised manual backup passes.
Create the staging directory and environment file without storing a private key or passphrase on the server:
```sh
sudo install -d -m 700 /var/backups/guestops /var/lib/guestops-backup/gnupg /etc/guestops
sudo install -m 600 /dev/null /etc/guestops/backup.env
sudoedit /etc/guestops/backup.env
sudo env GNUPGHOME=/var/lib/guestops-backup/gnupg gpg --import /secure/path/recovery-public.asc
sudo env GNUPGHOME=/var/lib/guestops-backup/gnupg gpg --fingerprint
```
The file contains only these two settings. `BACKUP_RECIPIENT` is the verified 40-character public-key fingerprint already imported into the service account's GPG keyring.
```text
BACKUP_RECIPIENT=0123456789ABCDEF0123456789ABCDEF01234567
BACKUP_DIRECTORY=/var/backups/guestops
```
Review the unit sandbox against the host, copy `deploy/systemd/guestops-backup.service` and `.timer` to `/etc/systemd/system`, then validate and test before enabling:
```sh
sudo systemd-analyze verify /etc/systemd/system/guestops-backup.service /etc/systemd/system/guestops-backup.timer
sudo systemctl daemon-reload
sudo systemctl start guestops-backup.service
sudo systemctl status guestops-backup.service
sudo systemctl enable --now guestops-backup.timer
systemctl list-timers guestops-backup.timer
```
The timer deliberately causes the same brief maintenance interruption as a manual backup. `Persistent=true` runs a missed event after downtime, so choose and communicate the maintenance window. A successful unit only stages an encrypted file locally. Configure an independently monitored off-host transfer, verify the destination checksum, alert on both unit and transfer failure, and test the alert route. Do not add automatic deletion until retention, legal hold and recovery requirements have named owners.
### Restricted off-host transfer
The optional transfer service uses rsync over pinned-host SSH. Create a dedicated upload-only account at the restricted store, disable interactive login, agent/port forwarding and access outside the GuestOps backup directory, and keep its private key only in `/etc/guestops` with mode 600. Pin the reviewed server host key; never use `StrictHostKeyChecking=no`.
Start from `deploy/systemd/backup-transfer.env.example` and store the completed file as `/etc/guestops/backup-transfer.env` with mode 600. The local and remote directories must already exist and remain private. Install and verify the service and its 15-minute retry timer:
```sh
sudo install -m 600 deploy/systemd/backup-transfer.env.example /etc/guestops/backup-transfer.env
sudoedit /etc/guestops/backup-transfer.env
sudo install -m 644 deploy/systemd/guestops-backup-transfer.service /etc/systemd/system/
sudo install -m 644 deploy/systemd/guestops-backup-transfer.timer /etc/systemd/system/
sudo systemd-analyze verify /etc/systemd/system/guestops-backup-transfer.service /etc/systemd/system/guestops-backup-transfer.timer
sudo systemctl daemon-reload
sudo systemctl start guestops-backup-transfer.service
sudo systemctl enable --now guestops-backup-transfer.timer
```
Only files named `guestops-YYYYMMDDTHHMMSSZ.tar.gpg` are eligible. The transfer writes a unique remote partial file, compares the remote and local SHA-256 values, atomically publishes a previously unused final name, verifies it again, and then writes a non-sensitive local marker. An existing remote name is accepted only when its checksum matches. Failed or mismatched transfers are never marked. The service removes local backups older than seven days only when the marker still matches the local checksum; untransferred or changed files are never pruned.
The restricted store is the durable copy. Configure its independently reviewed retention policy for 35 daily and 12 monthly recovery points. Legal hold must override expiry. The host tool does not delete remote data or enforce remote retention.
### Zabbix status boundary
Do not give the Zabbix agent access to Docker, MongoDB credentials or the application owner session. A root-owned systemd probe reads those local sources and atomically publishes aggregate status under `/run/guestops-monitor/status.json`; the agent reads that file only.
```sh
sudo install -m 644 deploy/systemd/guestops-monitor-status.service /etc/systemd/system/
sudo install -m 644 deploy/systemd/guestops-monitor-status.timer /etc/systemd/system/
sudo install -m 644 deploy/systemd/zabbix-agent-guestops.conf.example /etc/zabbix/zabbix_agentd.d/guestops.conf
sudo systemd-analyze verify /etc/systemd/system/guestops-monitor-status.service /etc/systemd/system/guestops-monitor-status.timer
sudo systemctl daemon-reload
sudo systemctl enable --now guestops-monitor-status.timer
sudo systemctl restart zabbix-agent
```
Create dependent Zabbix items from `guestops.status` for HTTPS readiness, certificate days remaining, container state/health, worker heartbeat age, backup age, verified-transfer age, free-disk percentage, persistent journal availability and probe error categories. Alert when the heartbeat is older than three minutes; backup or transfer is older than 30 hours; free disk falls below 25% (warning) or 15% (critical); the certificate has fewer than 30 days (warning) or 14 days (critical); any required container is absent/unhealthy; or the probe/timers fail. Route alerts to the monitoring owner and escalate unacknowledged critical events to the technical owner after 15 minutes. Exercise every trigger and its recovery notification with synthetic conditions.
Temporary plaintext files are held in private directories and removed on normal completion or exceptions. Process termination or power loss can leave temporary data, stopped services or TTL expiry disabled. After an interrupted run, inspect the dedicated project and remove only its identified abandoned temporary directory after securing any recovery material. Restore the recorded TTL setting (normally true) and restart the services:
```sh
docker compose exec -T mongo mongosh --quiet --nodb --eval 'const c=new Mongo("mongodb://127.0.0.1"); c.getDB("admin").auth(process.env.MONGO_INITDB_ROOT_USERNAME,process.env.MONGO_INITDB_ROOT_PASSWORD); const r=c.getDB("admin").runCommand({setParameter:1,ttlMonitorEnabled:true}); if(!r.ok)quit(1);'
docker compose start api worker
python3 deploy/ops.py preflight
```
Use the previous value instead of true if TTL expiry was deliberately disabled beforehand. Do not remove production volumes to recover from a failed backup.
## Isolated restore drill
Use a trusted encrypted backup and the recovery key on a Linux recovery machine. Unlock the key through the local GPG agent before running the batch command; never pass its passphrase on the command line. Load the exact reviewed images from the backed-up release first.
```sh
python3 deploy/ops.py restore-drill /secure/path/guestops-backup.tar.gpg \
--api-image guestops-api:REVIEWED_RELEASE \
--mongo-image mongo:8.0
```
The drill decrypts into a private temporary directory, checks the four expected files and their hashes, rejects unsafe archive entries, and verifies that the restored keys decrypt a protected test value. It then restores MongoDB into a newly created container with no external network or published ports and compares every restored collection's count and indexes. TTL expiry is disabled in this disposable database so expired token records do not disappear during comparison. The exact temporary container and its volumes are removed afterwards. No worker or production application is started.
The decrypted bundle is capped at 8 GiB for this pilot tool. Leave room for the encrypted file, decrypted archive, extracted files and restored database. GPG's [output limit](https://www.gnupg.org/documentation/manuals/gnupg/GPG-Input-and-Output.html) bounds decrypted output; checksums detect corruption, not the identity of the backup creator. Use backups from the trusted operator only. A drill checks counts, indexes and key decryption, not every document's business meaning or live provider connectivity.
## Actual disaster recovery
This milestone implements a rehearsal, not an automatic production restore or cutover command. Before real guest data is accepted, rehearse a separate recovery deployment and document its precise image IDs, volume names, secret locations and operator responsibilities.
Restore into new isolated MongoDB and key volumes; preserve the damaged original for investigation. Reconstruct reviewed configuration from the encrypted bundle without copying old Docker container IDs or blindly executing archived configuration. Restore the `guestops` database with the compatible MongoDB tools and restore the matching key files with the application's required ownership. Check hotel/account records and saved settings before exposing the recovered API. Keep the worker stopped, provider writes disabled and external network access restricted throughout this process.
**A restored database can predate emails, invoices and PMS changes that providers already completed.** Review pending, sending and uncertain records against provider evidence before enabling any worker, including automatic FAQ rules. Do not replay an older approval merely because the restored record says it is pending. Reconcile external effects, validate account sessions and mailbox authorization, and explicitly approve the cutover only after these checks. Rotate credentials if compromise prompted the recovery. Keep the old deployment stopped when enabling the replacement.
Run the synthetic encrypted backup and isolated restore drill in a controlled test environment, including actual key decryption and database comparison. This automated check is separate from the required rehearsal on the Debian server with its actual deployment configuration.
## Milestone 11 acceptance record
Keep raw backups, restored data, remote paths, host keys, monitoring recipients, screenshots and logs outside Git. Copy `deploy/backup-restore-acceptance.example.json` to the restricted evidence store and replace every placeholder only after completing the supervised exercises. The example deliberately fails.
The accepted record uses a 24-hour RPO, four-hour RTO, seven-day verified local staging window, 35 daily and 12 monthly off-host recovery points, and the opaque evidence ID `backup-restore`. Bind it to the same release identifiers as the Debian-host and persistence records:
```sh
python3 deploy/backup_restore_acceptance.py \
/secure/acceptance/backup-restore.json \
--expected-commit FULL_40_CHARACTER_SHA \
--expected-release-record-sha256 RELEASE_RECORD_SHA256
sha256sum /secure/acceptance/backup-restore.json
```
The validator requires matching local/remote backup checksums, exact immutable image identities, a timed isolated restore, tested retention/legal hold and alert escalation, an image rollback that preserves persistent volumes, return to the approved candidate with external writes disabled, separate independent review, healthy monitoring and no unresolved critical findings. Structural validation does not inspect the restricted evidence or authorize a release by itself.
-93
View File
@@ -1,93 +0,0 @@
# Supervised pilot, capacity and release approval
Milestone 18 is an evidence exercise against the exact approved release, not a feature toggle. Use synthetic data for capacity work and a separately approved, tightly supervised hotel cohort for the pilot. Keep provider writes and FAQ live mode disabled until their individual acceptance records are approved.
Before the pilot, complete the [Gate B automation, identity and privacy acceptance](gate-b-prerequisites.md) as well as the Google and desktop exercises. These reviews must use the same release identifiers as the final decision.
## Read-only capacity probe
The capacity probe logs in once with a dedicated sandbox staff account and sends bounded concurrent GET requests to readiness, hotel settings and cursor-paginated inbox endpoints. It never calls provider integrations, creates records, edits drafts or retains response bodies. Run it only during an approved sandbox window and monitor CPU, memory, MongoDB latency, disk, Nginx and application errors independently.
```sh
read -r -p 'Capacity account email: ' CAPACITY_EMAIL
read -r -s -p 'Capacity account password: ' CAPACITY_PASSWORD
export CAPACITY_EMAIL CAPACITY_PASSWORD
python3 deploy/capacity_probe.py \
--origin https://sandbox-guestops.futuresens.co.uk \
--requests 500 --concurrency 10 \
--release-commit FULL_40_CHARACTER_SHA \
--release-record-sha256 RELEASE_RECORD_SHA256 \
--output /secure/acceptance/capacity.json \
--confirm-sandbox
unset CAPACITY_EMAIL CAPACITY_PASSWORD
```
For the `0.2.1` Gate B candidate, the approved targets are concurrency 10, p95 latency at or below 500 ms, error rate at or below 1%, and at least 25% CPU and memory headroom on the documented four-core, 7.6 GiB host. The generated result reports HTTP observations, not a pass/fail claim. Repeat after a warm-up, investigate every error, and retain independently captured host metrics with the report. Hash both retained files for the approval record. Do not point the probe at a live hotel or increase its built-in bounds to simulate a denial of service.
## Five-business-day supervised pilot
Use one approved hotel and named hotel, technical and rollback owners. Start only after the prerequisite evidence below has passed. Keep PMS writes, payment creation and FAQ live mode disabled. Complete one daily review on each of five business days and stop for tenant leakage, credential exposure, data loss, an unapproved or duplicate send, an unreconciled uncertain send, failed rollback or loss of monitoring.
Copy `deploy/pilot-run.example.json` to the restricted evidence store and replace all placeholders. Resolve critical and high findings; lower-severity findings may be contained only with an owner, expiry and objective rollback trigger. Validate and hash the final record:
```sh
python3 deploy/pilot_run.py /secure/acceptance/pilot-run.json
sha256sum /secure/acceptance/pilot-run.json
```
The example deliberately fails while daily reviews are `not-run`. A structurally valid record does not substitute for the five elapsed business days or independent evidence review.
## Pilot exit record
The go/no-go record must bind all evidence to the same release commit and release-record checksum. Record named owners, dates, evidence locations, findings and explicit dispositions for:
- exact-commit source package, checksum, recorded image IDs, and successful Ansible installation;
- Debian preflight, HTTPS, persistence and controlled reboot;
- encrypted off-host backup and timed isolated restore;
- Google mailbox and reviewed-send acceptance;
- knowledge, AI and FAQ test-mode evaluation;
- identity, privacy, retention and audit review;
- inbox pagination, draft protection, timezone and desktop-parity acceptance;
- capacity targets, observed host headroom and expected pilot workload;
- support, rollback, provider-reconciliation and incident exercises;
- every pilot usability or security finding.
Approval requires separate named decisions from the hotel pilot owner and technical release owner. Gate C additionally requires independently accepted PMS and payment-provider evidence. A conditional approval must identify the containment, owner, expiry and rollback trigger; an unresolved finding is not silently converted into acceptance. Retain the signed decision with the release rather than committing guest, credential or incident data to this repository.
Copy `deploy/pilot-approval.example.json` into the restricted release store and complete it only after reviewing the referenced evidence. Approval schema version 2 binds the five-day pilot record and both the capacity report and independently captured host metrics. The example is intentionally invalid while its decision is `pending`. For a contained finding, record its named owner, future expiry and objective rollback trigger. Validate the completed record with:
```sh
python3 deploy/pilot_approval.py /secure/acceptance/pilot-approval.json
```
The validator requires the exact Gate B evidence set, or that set plus independently accepted PMS and payment-provider evidence for Gate C. It also verifies that observed concurrency meets the pre-agreed target and that p95 latency and error rate remain within their pre-agreed bounds. Structural validation does not inspect evidence or authorize rollout by itself.
## Integrated Gate B bundle validation
After every individual record passes independent review, validate the complete bundle together. This prevents records from different commits, release records, archives, image builds or sandbox environments from being combined into one approval. It also binds the final decision to the retained capacity report, host metrics and five-day pilot record by checksum.
```sh
python3 deploy/gate_b_bundle.py \
--source-archive /secure/releases/GuestOps-0.2.1-COMMIT.tar.gz \
--source-record /secure/releases/GuestOps-0.2.1-COMMIT.source.json \
--release-record /secure/releases/release-record.json \
--host-metrics /secure/acceptance/capacity-host-metrics.json \
--debian-host /secure/acceptance/debian-host.json \
--persistence /secure/acceptance/persistence.json \
--backup-restore /secure/acceptance/backup-restore.json \
--google-mailbox /secure/acceptance/google-mailbox.json \
--automation /secure/acceptance/automation.json \
--identity-privacy /secure/acceptance/identity-privacy.json \
--inbox-usability /secure/acceptance/inbox-usability.json \
--capacity /secure/acceptance/capacity.json \
--incident-support /secure/acceptance/incident-support.json \
--pilot-findings /secure/acceptance/pilot-run.json \
--pilot-approval /secure/acceptance/pilot-approval.json \
--output /secure/acceptance/gate-b-bundle-summary.json
```
The output contains only release identity and file checksums, uses mode `0600`, and refuses to overwrite an existing summary. Retain it with the individual validator outputs. A passing bundle proves structural consistency, not that screenshots, provider activity, approvals, findings or other referenced evidence are genuine.
Complete the [incident and rollback exercise](incident-exercise.md) before marking `incident-support` as passed. Its record must use the same release identifiers and target gate as this decision. Reference the retained exercise record and validator output; do not substitute a local automated-test result for the supervised exercise.
Complete the [desktop-parity acceptance exercise](desktop-acceptance.md) before marking `inbox-usability` as passed. Bind it to the same release identifiers and retain its independently reviewed record outside the repository.