Skip to content

VM Testing and Release Validation

Objective

Validate release artifacts across target OSes from a clean VM state.

VM validation pipeline

Diagram source: vm-validation-pipeline.mmd

Components

  • Host orchestrator: scripts/run_automated_vm_tests.sh
  • Artifact downloader: scripts/download_release_artifacts.sh
  • Linux in-VM runner: scripts/vm_test_runner.sh
  • Windows in-VM runner: scripts/vm_test_runner.ps1
  • VM definitions: Vagrantfile

Current VM matrix

  • ubuntu18
  • ubuntu22
  • ubuntu24
  • debian12
  • almalinux
  • opensuse
  • windows

(Deprecated/removed: Debian 11 from Vagrant + orchestrator)

Execution

./scripts/run_automated_vm_tests.sh v1.1.2

The orchestrator: 1. Downloads release assets to vms/artifacts 2. Runs VM tests with bounded concurrency (default: 3 at a time) 3. Logs per VM to vms/artifacts/logs/<vm>_run.log 4. Collects reports under vms/artifacts/reports/

Override concurrency when needed:

MAX_PARALLEL_VM_TESTS=2 ./scripts/run_automated_vm_tests.sh v1.1.2

What each VM runner validates

Linux runner

  1. Detect OS family and distro/version
  2. Install DEB or fallback AppImage
  3. Verify --version
  4. Install selected tools (lynis, ciscat, conditional openscap, conditional usg)
  5. Execute scans and export PDF outputs
  6. Validate server start/status/stop and print server logs if available
  7. Copy reports
  8. Cleanup

Windows runner

  1. Install windows artifact
  2. Verify --version
  3. Install hardeningkitty, ciscat
  4. Run scans and export PDFs
  5. Validate server start/status/stop
  6. Collect reports and cleanup

Practical debugging checklist

  • Tail each log:
  • tail -n 100 vms/artifacts/logs/<vm>_run.log
  • For running jobs, follow specific VM log in real-time:
  • tail -f vms/artifacts/logs/ubuntu22_run.log
  • Validate artifacts downloaded:
  • ls -la vms/artifacts

Retagging workflow (when a release needs to be reissued)

  1. Commit required fixes.
  2. Delete/recreate tag on latest commit.
  3. Push branch + tag.
  4. Confirm new Build and Release run is triggered for the tag.

Example sequence:

git tag -d v1.1.2
git push origin :refs/tags/v1.1.2
git tag v1.1.2
git push origin main
git push origin v1.1.2

Common non-code failures

  • Interrupted/corrupt Vagrant box download (ubuntu24)
  • Host resource pressure during concurrent provisioning
  • Port conflicts on host for server checks