futuresens/GuestOps
6
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add deterministic source release packaging

Browse Source
This commit is contained in:
mathew 2026-09-30 21:03:16 +01:00
parent 8588b058c5
commit 9c29bfcd86
6 changed files with 238 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
MILESTONES.md
View File

@ -19,7 +19,7 @@ This summary explains what each milestone delivers and where it currently stands
| 6 | Team onboarding and account recovery | Staff invitations, password setup/reset, access disable/restore, and administrator recovery. | **Implemented; acceptance required.** Deployed link delivery, expiry, recovery, and administrator procedures still need operational evidence. |
| 7 | Google connection recovery | OAuth reconnect, checkpoint recovery, grant revocation handling, and worker restart safety. | **Implemented; acceptance required.** Dedicated Google-account and worker-restart exercises have not yet been accepted. |
| 8 | Operational readiness tooling | Release verification, diagnostics, encrypted backup, restore, preflight, and persistence tools. | **Implemented; acceptance required.** The tools must still be run against the exact release on the Debian host. |
| 9 | Gitea and reproducible releases | A checksummed source package, Ansible-controlled installation, recorded image identities, retained artifacts, and approval tagging. | **In progress.** The Gitea Action has been removed to match the CMS/CMSFront deployment model; the exact-commit package, Ansible playbook evidence, and post-Gate-B tag are still required. |
| 9 | Gitea and reproducible releases | A checksummed source package, Ansible-controlled installation, recorded image identities, retained artifacts, and approval tagging. | **In progress.** The runner-free deterministic packager is implemented and the Gitea Action is removed; the release-line identity must be confirmed, then the package, Ansible installation evidence, and approval record must be retained. |
| 10 | Debian deployment and persistence | Secure Debian/Compose deployment, HTTPS, persistent database and key volumes, and reboot/recreation proof. | **In progress.** Acceptance tooling is ready, but Docker, correct HTTPS/network exposure, exact artifacts, privileged installation, and the supervised drills remain open. |
| 11 | Backups, monitoring, and recovery | Scheduled encrypted backups, verified off-host transfer, Zabbix monitoring, restore, and rollback rehearsal. | **In progress.** Repository tooling is ready; installation and timed operational evidence are blocked until Milestones 9 and 10 pass. |
| 12 | Google mailbox acceptance | End-to-end Gmail consent, import, recovery, reviewed sending, reconciliation, and revocation evidence. | **In progress.** The runbook and validator exist; the live synthetic-data exercise and independent review remain outstanding. |
@ -59,7 +59,7 @@ This summary explains what each milestone delivers and where it currently stands
| 6 | Team onboarding and account recovery | B | Implemented / acceptance required | Invitation, password reset, and recovery flows are promoted to local `main`; verify deployed links, mail delivery, token expiry, and administrator recovery procedures. |
| 7 | Google connection recovery | B | Implemented / acceptance required | Connection epochs, checkpoint recovery, and revocation handling are promoted to local `main`; complete real Google acceptance and worker-restart exercises. |
| 8 | Operational readiness tooling | A | Implemented / acceptance required | Backup, restore, release, and diagnostic tooling is promoted to local `main`; execute it on the actual Debian host and retain evidence. |
| 9 | Gitea and reproducible releases | A | In progress | Package the exact versioned `main` commit as a checksummed source archive and hand it to a version-selected Ansible playbook, following the CMS/CMSFront pattern. Ansible installs the archive, builds commit-tagged images, records their immutable IDs, and deploys without using a Gitea runner. Retain the package/install evidence off-host and create the immutable approval tag only after Gate B approval; the existing `0.1.0` tag remains attached to the foundation release. |
| 9 | Gitea and reproducible releases | A | In progress | `deploy/package_source.py` now packages only an explicit committed ref, verifies matched application versions, produces deterministic gzip output and a SHA-256 source record, and refuses overwrite. Hand that package to a version-selected Ansible playbook following the CMS/CMSFront pattern. Ansible must verify and install it, build commit-tagged images, record their immutable IDs, and deploy without a Gitea runner. Retain the package/install evidence off-host and resolve the release-line/tag identity before approval; the existing `0.1.0` tag remains attached to the foundation release. |
| 10 | Debian deployment and persistence | A | In progress | Compose uses separate named database and shared key volumes, private host configuration, loopback-only API access and bounded logs. The confirmation-gated persistence drill verifies restart and container-recreation behaviour. Run it on the provisioned Debian host, complete HTTPS and controlled-reboot acceptance, and retain the evidence. |
| 11 | Backups, monitoring, and recovery | A | In progress | Encrypted backup and isolated restore tooling now includes opt-in systemd scheduling, checksum-verified rsync transfer, a restricted Zabbix status boundary, guarded local retention and a release-bound acceptance validator. Install and test it on Debian, configure the restricted store and alerts, name operational/review owners, and retain independently reviewed evidence from the timed restore and rollback drill. |
| 12 | Google mailbox and reviewed-reply acceptance | B | In progress | The synthetic-data provider runbook, exact scenario set and restricted-record validator are implemented. Complete every scenario against the accepted Debian release and dedicated Google sandbox accounts, independently review the evidence, and retain the validated record. |
@ -81,7 +81,8 @@ Milestones 13 (Guestline/Rezlynx) and 14 (payments) can progress as parallel pro
## Next actions
- [ ] Merge the Action removal and release-process update to the intended default branch; that resulting commit becomes the new package candidate.
- [ ] Create and checksum the `0.2.0` source archive, add/select it in the GuestOps Ansible playbook, and retain the package and installation evidence.
- [ ] Confirm the intended release commit/version because remote `main` and the published `0.2.0` tag currently identify different histories; do not move or reuse the published tag.
- [ ] Create and checksum the approved source archive with `deploy/package_source.py`, add/select it in the GuestOps Ansible playbook, and retain the package and installation evidence.
- [ ] Create and archive the immutable `0.2.0` approval tag only after Gate B approval.
- [ ] Deploy to the target Debian environment with persistent MongoDB and data-protection keys.
- [ ] Run and record backup, restore, restart, monitoring, and rollback exercises.

2
README.md
View File

@ -59,6 +59,8 @@ For MongoDB-backed operation, disable Preview and set `Mongo__ConnectionString`,
Operational tooling includes an owner-only Workspace health page and Linux deployment preflight, encrypted backup and isolated restore drill. See [operations and recovery](docs/operations.md) before the hotel pilot.
Release packaging is runner-free. `deploy/package_source.py` creates a deterministic archive and checksum record from one explicit committed Git ref for handoff to the Futuresens Ansible deployment. It never packages uncommitted working-tree files.
## Verification
```sh

124
deploy/package_source.py Normal file
View File

@ -0,0 +1,124 @@
#!/usr/bin/env python3
"""Create a deterministic GuestOps source package from one committed Git tree."""
from __future__ import annotations
import argparse
import gzip
import hashlib
import io
import json
from pathlib import Path
import re
import subprocess
import sys
import xml.etree.ElementTree as ET
SEMVER = re.compile(r"[0-9]+\.[0-9]+\.[0-9]+")
FULL_SHA = re.compile(r"[0-9a-f]{40}")
def git(repository: Path, *arguments: str, binary: bool = False) -> bytes | str:
result = subprocess.run(
["git", "-C", str(repository), *arguments],
check=True,
capture_output=True,
text=not binary,
)
return result.stdout if binary else result.stdout.strip()
def committed_text(repository: Path, commit: str, path: str) -> str:
return str(git(repository, "show", f"{commit}:{path}"))
def versions(repository: Path, commit: str) -> tuple[str, str]:
props = ET.fromstring(committed_text(repository, commit, "Directory.Build.props"))
version_node = props.find(".//Version")
if version_node is None or not version_node.text:
raise ValueError("The committed Directory.Build.props has no Version element.")
dotnet_version = version_node.text.strip()
package = json.loads(committed_text(repository, commit, "web/package.json"))
web_version = str(package.get("version", ""))
return dotnet_version, web_version
def sha256_bytes(value: bytes) -> str:
return hashlib.sha256(value).hexdigest()
def build_package(repository: Path, ref: str, output_directory: Path) -> tuple[Path, Path, dict[str, object]]:
repository = repository.resolve()
output_directory = output_directory.resolve()
if not (repository / ".git").exists():
raise ValueError("--repository must be a Git working tree.")
commit = str(git(repository, "rev-parse", "--verify", f"{ref}^{{commit}}")).lower()
if FULL_SHA.fullmatch(commit) is None:
raise ValueError("The selected ref did not resolve to a full Git commit SHA.")
dotnet_version, web_version = versions(repository, commit)
if dotnet_version != web_version:
raise ValueError(
f"Committed release versions differ: .NET={dotnet_version}, web={web_version}."
)
if SEMVER.fullmatch(dotnet_version) is None:
raise ValueError("The committed version must use MAJOR.MINOR.PATCH.")
prefix = f"GuestOps-{dotnet_version}/"
tar_bytes = bytes(
git(
repository,
"archive",
"--format=tar",
f"--prefix={prefix}",
commit,
binary=True,
)
)
compressed = io.BytesIO()
with gzip.GzipFile(fileobj=compressed, mode="wb", filename="", mtime=0) as stream:
stream.write(tar_bytes)
archive_bytes = compressed.getvalue()
stem = f"GuestOps-{dotnet_version}-{commit[:12]}"
archive = output_directory / f"{stem}.tar.gz"
record_path = output_directory / f"{stem}.source.json"
if archive.exists() or record_path.exists():
raise ValueError("The output package or source record already exists; releases are not overwritten.")
output_directory.mkdir(parents=True, exist_ok=True)
archive.write_bytes(archive_bytes)
record: dict[str, object] = {
"artifact": {
"name": archive.name,
"sha256": sha256_bytes(archive_bytes),
"size": len(archive_bytes),
},
"commit": commit,
"schemaVersion": 1,
"version": dotnet_version,
}
record_path.write_text(
json.dumps(record, indent=2, sort_keys=True) + "\n", encoding="utf-8"
)
return archive, record_path, record
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--ref", required=True, help="Exact tag, branch, or full commit to package.")
parser.add_argument("--repository", type=Path, default=Path(__file__).resolve().parents[1])
parser.add_argument("--output-directory", required=True, type=Path)
args = parser.parse_args()
try:
archive, record, details = build_package(args.repository, args.ref, args.output_directory)
except (OSError, ValueError, ET.ParseError, json.JSONDecodeError, subprocess.SubprocessError) as error:
print(f"Source packaging failed: {error}", file=sys.stderr)
raise SystemExit(1)
print(json.dumps({"archive": str(archive), "record": str(record), **details}, indent=2))
if __name__ == "__main__":
main()

14
docs/deployment.md
View File

@ -16,15 +16,19 @@ The MongoDB initialization script runs only on a new volume. Changing `.env` lat
GuestOps follows the CMS/CMSFront deployment pattern: Gitea stores the application source, a specific committed version is compressed, and a version-selected Ansible playbook installs it. No Gitea Actions runner is required.
On the trusted packaging machine, run the automated checks, select the full commit SHA, and create the archive from that committed tree rather than from a working directory:
On the trusted packaging machine, fetch the current remote refs, run the automated checks, select the full commit SHA, and create the archive from that committed tree rather than from a working directory:
```sh
git archive --format=tar.gz --prefix=GuestOps-0.2.0/ \
--output GuestOps-0.2.0.tar.gz FULL_40_CHARACTER_SHA
sha256sum GuestOps-0.2.0.tar.gz > GuestOps-0.2.0.tar.gz.sha256
git fetch origin --prune --tags
git rev-parse origin/main
python3 deploy/package_source.py \
--ref FULL_40_CHARACTER_SHA \
--output-directory /secure/release-staging
```
Store the archive and checksum under a versioned GuestOps files directory in the private Ansible repository. The GuestOps playbook and environment variables should select that version, copy and verify the archive, extract it into the application directory, preserve the private `.env` and provider configuration, and build images tagged with the full source commit:
The packaging command resolves the ref to a full commit, reads both application versions from that committed tree, requires them to match, creates deterministic gzip output, and writes a `.source.json` record containing the version, commit, filename, size and SHA-256. Existing package files are never overwritten. Independently confirm that the selected commit is the intended release line before transferring it.
Store the archive and source record under a versioned GuestOps files directory in the private Ansible repository. The GuestOps playbook and environment variables should select that version, verify the archive against the source record, extract it into the application directory, preserve the private `.env` and provider configuration, and build images tagged with the full source commit:
```sh
docker build --target api -t guestops-api:FULL_40_CHARACTER_SHA .

15
docs/operations.md
View File

@ -4,17 +4,24 @@ The owner-only **Workspace health** page reports database reachability, the work
## Release evidence and rollback
Create the release source archive from an exact committed Gitea tree with `git archive`, then verify its SHA-256 before Ansible installs it. Ansible builds the API and worker images under the full Git commit SHA. The accompanying `release-record.json` binds the source-archive checksum, application version, commit, image references and immutable Docker image IDs. Retain the archive, checksum, record and Ansible result together in restricted off-host release storage.
Create the release source archive from an exact committed Gitea tree with `deploy/package_source.py`, then verify its generated source record before Ansible installs it. The tool packages only the selected commit, produces deterministic gzip output, rejects version disagreement, and refuses to overwrite an existing release. Ansible builds the API and worker images under the full Git commit SHA. The post-build `release-record.json` binds the source-archive checksum, application version, commit, image references and immutable Docker image IDs. Retain the archive, source record, release record and Ansible result together in restricted off-host release storage.
Before deployment, verify the archive against its record without loading it:
```sh
git fetch origin --prune --tags
python3 deploy/package_source.py \
--ref FULL_40_CHARACTER_SHA \
--output-directory /secure/release-staging
```
Before deployment, verify the archive against its source record:
```sh
python3 - <<'PY'
import hashlib, json, pathlib
r = json.load(open('release-record.json', encoding='utf-8'))
r = json.load(open('GuestOps-0.2.0-COMMIT.source.json', encoding='utf-8'))
p = pathlib.Path(r['artifact']['name'])
assert hashlib.sha256(p.read_bytes()).hexdigest() == r['artifact']['sha256']
print(r['commit'], r['version'], r['images'])
print(r['commit'], r['version'], r['artifact'])
PY
```

88
tests/test_package_source.py Normal file
View File

@ -0,0 +1,88 @@
import hashlib
import importlib.util
import json
from pathlib import Path
import subprocess
import tarfile
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
spec = importlib.util.spec_from_file_location("package_source", ROOT / "deploy" / "package_source.py")
package_source = importlib.util.module_from_spec(spec)
spec.loader.exec_module(package_source)
def run(repository: Path, *arguments: str) -> str:
result = subprocess.run(
["git", "-C", str(repository), *arguments],
check=True,
capture_output=True,
text=True,
)
return result.stdout.strip()
class PackageSourceTests(unittest.TestCase):
def repository(self, root: Path, dotnet_version: str = "1.2.3", web_version: str = "1.2.3") -> tuple[Path, str]:
repository = root / "repository"
(repository / "web").mkdir(parents=True)
run(repository, "init")
run(repository, "config", "user.name", "GuestOps Test")
run(repository, "config", "user.email", "guestops@example.invalid")
run(repository, "config", "core.autocrlf", "false")
(repository / "Directory.Build.props").write_text(
f"<Project><PropertyGroup><Version>{dotnet_version}</Version></PropertyGroup></Project>\n",
encoding="utf-8",
)
(repository / "web" / "package.json").write_text(
json.dumps({"version": web_version}) + "\n", encoding="utf-8"
)
(repository / "application.txt").write_text("committed application\n", encoding="utf-8")
run(repository, "add", ".")
run(repository, "commit", "-m", "fixture")
return repository, run(repository, "rev-parse", "HEAD")
def test_packages_only_the_selected_commit_and_records_identity(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
repository, commit = self.repository(root)
(repository / "application.txt").write_text("uncommitted change\n", encoding="utf-8")
archive, record_path, record = package_source.build_package(repository, commit, root / "output")
self.assertEqual(record["commit"], commit)
self.assertEqual(record["version"], "1.2.3")
self.assertEqual(record["artifact"]["sha256"], hashlib.sha256(archive.read_bytes()).hexdigest())
self.assertEqual(json.loads(record_path.read_text(encoding="utf-8")), record)
with tarfile.open(archive, "r:gz") as package:
member = package.extractfile("GuestOps-1.2.3/application.txt")
self.assertIsNotNone(member)
self.assertEqual(member.read().decode("utf-8").strip(), "committed application")
def test_same_commit_produces_identical_package(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
repository, commit = self.repository(root)
first, _, _ = package_source.build_package(repository, commit, root / "first")
second, _, _ = package_source.build_package(repository, commit, root / "second")
self.assertEqual(first.read_bytes(), second.read_bytes())
def test_rejects_mismatched_versions_and_existing_output(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
repository, commit = self.repository(root, web_version="1.2.4")
with self.assertRaisesRegex(ValueError, "versions differ"):
package_source.build_package(repository, commit, root / "output")
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
repository, commit = self.repository(root)
package_source.build_package(repository, commit, root / "output")
with self.assertRaisesRegex(ValueError, "not overwritten"):
package_source.build_package(repository, commit, root / "output")
if __name__ == "__main__":
unittest.main()