tests: Rework tests/installed → tests/kola

Previously we made an effort to use the [Fedora Standard Test Interface](https://docs.fedoraproject.org/en-US/ci/standard-test-interface/).
This effort was not very successful; the primary thing that
it really died on is Ansible just didn't support rebooting
very well.  I think that's since gotten better, but even
then, Ansible wasn't the best thing for a test framework
for us anyways.

In the meantime Fedora CoreOS happened emphasizing Ignition
and not "post-hoc reconciliation" models like Ansible over
ssh.

And, [coreos-assembler](https://github.com/coreos/coreos-assembler) happened too.

Furthermore, we really need to test OSTree's interaction
with Ignition as we've invented several special things there.

Then most recently, I've been working on having
cosa/kola support running externally defined tests:
https://github.com/coreos/coreos-assembler/pull/1215

There's a lot of things to clean up after this but at least this
works for me:

```
$ cd /srv/fcos
$ cosa kola run -- --parallel 4 --output-dir tmp/kola -E ~/src/github/ostreedev/ostree/ 'ext.ostree.*'
```

NOTE: This *does not* drop ostree binaries into the target.  See:
https://github.com/coreos/coreos-assembler/pull/1252#issuecomment-600623315

This drops our dependency on Python in the installed tests, and
also fixes a few bugs that came up.

I disabled the `itest-bare-user-root.sh` one because it's
entangled with the shell script infrastructure for the unit tests.

tests/installed/README.md

@@ -1,15 +0,0 @@
-This directory holds tests that use the
-[Fedora Standard Test Interface](https://fedoraproject.org/wiki/CI/Standard_Test_Interface).
-
-The high level structure is that we take a qcow2 file, inject
-built RPMs into it, and then use Ansible to run tests.
-
-See `.papr.yml` for canonical usage.
-
-For local development, you should cache the qcow2 somewhere
-stable (outside of this git repo).  Also note that `../ci/build-rpms.sh`
-does *not* pick up uncommitted changes!  Stated more strongly, you
-currently need to run `build-rpms.sh` after every change.
-
-To run just a specific test, use e.g.:
-`env TEST_SUBJECTS=/path/to/qcow2 ./playbook-run.sh -e tests=.*pull nondestructive.yml`

tests/installed/destructive-ansible.yml

@@ -1,16 +0,0 @@
-# Ansible-based tests.
----
-- hosts: localhost
-  tags:
-  - atomic
-  remote_user: root
-  vars:
-    use_git_build: True
-    tests: "."
-  tasks:
-    - import_tasks: tasks/disable-all-rpmmd-repos.yml
-    - import_tasks: tasks/install-git.yml
-      when: use_git_build
-    - import_tasks: tasks/query-host.yml
-    - import_tasks: destructive/staged-deploy.yml
-    - import_tasks: destructive/var-mount.yml

tests/installed/destructive-unit.yml

@@ -1,42 +0,0 @@
-# This entrypoint right now just runs shell-script based tests
-# from destructive/.  Note that we `rpm-ostree usroverlay` git
-# builds.  So it's not supported to reboot in these tests.
-# These tests will be run serially, and can e.g. change deployments.
----
-- hosts: localhost
-  tags:
-  - atomic
-  remote_user: root
-  vars:
-    use_git_build: True
-    tests: "."
-  tasks:
-    - import_tasks: tasks/query-host.yml
-    - set_fact:
-        rpmostree_initial_deployment: "{{ rpmostree_status[\"deployments\"][0] }}"
-    - import_tasks: tasks/overlay-git.yml
-      when: use_git_build
-    # Next copy all of the tests/ directory
-    - name: Copy test data
-      synchronize: src=../../ dest=/root/tests/ archive=yes
-
-    - find:
-        paths: /root/tests/installed/destructive
-        patterns: "itest-*.sh"
-      register: all_tests
-    - set_fact:
-        selected_tests: "{{ all_tests.files|map(attribute='path') | select('match', tests) | list }}"
-    - assert:
-        that:
-          - "{{ selected_tests|length }} != 0"
-    - file: path=/root/logs state=directory
-    - block:
-        - name: Run destructive tests
-          shell: "{{ item }} &> /root/logs/$(basename {{ item }}).log"
-          with_items:
-            - "{{ selected_tests }}"
-      always:
-        - synchronize:
-            src: /root/logs/
-            dest: artifacts/installed-destructive
-            mode: pull

tests/installed/destructive/README.md

@@ -1,5 +0,0 @@
-This suite of tests is run from PAPR. Everything in here is destructive; it's
-recommended to only run them in disposable virtual machines.  This is done
-in `tests/fedora-str/sysinstalled-tests.yml`, which currently uses a single VM
-and runs the tests serially.  It's likely in the future this will be changed
-to do one VM per test.

tests/installed/destructive/var-mount.yml

@@ -1,13 +0,0 @@
-# https://github.com/ostreedev/ostree/issues/1667
-- name: Set up /var as a mountpoint
-  shell: |
-    set -xeuo pipefail
-    cp -a /var /sysroot/myvar
-    touch /sysroot/myvar/somenewfile
-    echo '/sysroot/myvar /var none bind 0 0' >> /etc/fstab
-- include_tasks: ../tasks/reboot.yml
-- name: Check that /var mountpoint worked
-  shell: |
-    set -xeuo pipefail
-    systemctl status var.mount
-    test -f /var/somenewfile

tests/installed/execute_batch.yml

@@ -1,23 +0,0 @@
-#####################
-# execute_batch.yml
-#####################
-- name: Begin async command execution
-  shell: "{{ async_item }} &> {{ logdir }}/{{ async_item|basename }}.log"
-  # 10 minutes; the PAPR tester generally times out before that
-  async: 600
-  poll: 0
-  with_items: "{{ async_commands }}"
-  loop_control:
-    loop_var: "async_item"
-  register: async_results
-
-- name: Check async command status
-  async_status:
-    jid: "{{ async_result_item.ansible_job_id }}"
-  with_items: "{{ async_results.results }}"
-  loop_control:
-    loop_var: "async_result_item"
-  register: async_poll_results
-  until: async_poll_results.finished
-  retries: 500
-  retry_pause: 5

tests/installed/libinsttest.sh

@@ -1,101 +0,0 @@
-# Common definitions for installed, privileged tests
-#
-# Copyright (C) 2017 Colin Walters <walters@verbum.org>
-#
-# SPDX-License-Identifier: LGPL-2.0+
-#
-# This library is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Lesser General Public
-# License as published by the Free Software Foundation; either
-# version 2 of the License, or (at your option) any later version.
-#
-# This library is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-# Lesser General Public License for more details.
-#
-# You should have received a copy of the GNU Lesser General Public
-# License along with this library; if not, write to the
-# Free Software Foundation, Inc., 59 Temple Place - Suite 330,
-# Boston, MA 02111-1307, USA.
-
-dn=$(dirname $0)
-. ${dn}/../libtest-core.sh
-
-# Copy of bits from tap-test
-test_tmpdir=
-function _tmpdir_cleanup () {
-    if test -z "${TEST_SKIP_CLEANUP:-}" &&
-       test -n "${test_tmpdir}" && test -f ${test_tmpdir}/.testtmp; then
-	      rm "${test_tmpdir}" -rf
-    fi
-}
-prepare_tmpdir() {
-    local tmpdir=${1:-/tmp}
-    test_tmpdir=$(mktemp -p ${tmpdir} -d ostree-insttest.XXXXXXXX)
-    touch ${test_tmpdir}/.testtmp
-    cd ${test_tmpdir}
-}
-
-if test -x /usr/bin/python3; then
-    export PYTHON=/usr/bin/python3
-    export PYTHONHTTPSERVER=http.server
-elif test -x /usr/bin/python; then
-    export PYTHON=/usr/bin/python
-    export PYTHONHTTPSERVER=SimpleHTTPServer
-else
-    fatal "no python found"
-fi
-
-# This is copied from flatpak/flatpak/tests/test-webserver.sh
-run_tmp_webserver() {
-    dir=$1
-
-    test -n ${test_tmpdir}
-
-    cd ${dir}
-    env PYTHONUNBUFFERED=1 setsid $PYTHON -m $PYTHONHTTPSERVER 0 &>${test_tmpdir}/httpd-output &
-    cd -
-    child_pid=$!
-
-    for x in $(seq 60); do
-        echo "Waiting for web server ($x/60)..." >&2
-        # Snapshot the output
-        cp ${test_tmpdir}/httpd-output{,.tmp}
-        # If it's non-empty, see whether it matches our regexp
-        if test -s ${test_tmpdir}/httpd-output.tmp; then       # py3's http.server prints the http:// address also
-          sed -e 's,Serving HTTP on 0.0.0.0 port \([0-9]*\)\( (http://[^)]*)\)\? \.\.\.,\1,' < ${test_tmpdir}/httpd-output.tmp > ${test_tmpdir}/httpd-port
-            if ! cmp ${test_tmpdir}/httpd-output.tmp ${test_tmpdir}/httpd-port 1>/dev/null; then
-                # If so, we've successfully extracted the port
-                break
-            fi
-        fi
-        sleep 1
-    done
-
-    if [ ! -f ${test_tmpdir}/httpd-port ]; then
-      cat ${test_tmpdir}/httpd-output
-      fatal "can't start up httpd"
-    fi
-
-    port=$(cat ${test_tmpdir}/httpd-port)
-    echo "http://127.0.0.1:${port}" > ${test_tmpdir}/httpd-address
-    echo "$child_pid" > ${test_tmpdir}/httpd-pid
-}
-
-# Determine our origin refspec - we'll use this as a test base
-rpmostree=$(which rpm-ostree 2>/dev/null)
-if test -z "${rpmostree}"; then
-    skip "no rpm-ostree, at some point point this to raw ostree too"
-fi
-
-# We need to be root
-assert_streq $(id -u) 0
-
-rpmostree_query_json() {
-    query=$1
-    rpm-ostree status --json | $PYTHON -c 'import json,sys; v=json.load(sys.stdin); print(v'${query}')'
-}
-host_refspec=$(rpmostree_query_json '["deployments"][0]["origin"]')
-host_commit=$(rpmostree_query_json '["deployments"][0]["checksum"]')
-host_osname=$(rpmostree_query_json '["deployments"][0]["osname"]')

tests/installed/nondestructive.yml

@@ -1,41 +0,0 @@
-# Nondestructive sysinstalled tests, run in parallel.
----
-- hosts: localhost
-  tags:
-  - atomic
-  remote_user: root
-  vars:
-    use_git_build: True
-    tests: "."
-    # Arbitrary...we want some parallelism
-    batching_factor: 4
-  tasks:
-    - import_tasks: tasks/query-host.yml
-    - import_tasks: tasks/overlay-git.yml
-      when: use_git_build
-    # Next copy all of the tests/ directory
-    - name: Copy test data
-      synchronize: src=../../ dest=/root/tests/ archive=yes
-    - find:
-        paths: /root/tests/installed/nondestructive
-        patterns: "itest-*.sh"
-      register: all_tests
-    - set_fact:
-        selected_tests: "{{ all_tests.files|map(attribute='path') | select('match', tests) | list }}"
-    - assert:
-        that:
-          - "{{ selected_tests|length }} != 0"
-    - file: path=/root/logs state=directory
-    - block:
-        - name: Run nondestructive tests
-          vars:
-            logdir: /root/logs
-          async_commands: "{{ item }}"
-          include_tasks: execute_batch.yml
-          with_items:
-            - "{{ selected_tests | batch('{{ batching_factor }}') | list }}"
-      always:
-        - synchronize:
-            src: /root/logs
-            dest: artifacts/installed-nondestructive
-            mode: pull

tests/installed/nondestructive/libtest-core.sh

@@ -1 +0,0 @@
-../libtest-core.sh

tests/installed/playbook-run.sh

@@ -1,41 +0,0 @@
-#!/usr/bin/bash
-# A thin wrapper for ansible-playbook which has a nice check for
-# TEST_SUBJECTS being set.
-set -xeuo pipefail
-
-dn=$(cd $(dirname $0) && pwd)
-if ! test -d build; then
-    mkdir -p build
-    (cd build && ${dn}/../../ci/build-rpm.sh)
-else
-    # XXX: we should invalidate based on `git describe` or something
-    echo "NOTE: Re-using prebuilt RPMs:"
-    find build -name '*.rpm'
-fi
-
-# https://fedoraproject.org/wiki/CI/Tests
-if test -z "${TEST_SUBJECTS:-}"; then
-    cat <<EOF
-
-error: TEST_SUBJECTS must be set; e.g.:
-
-  curl -Lo fedora-atomic-host.qcow2 'https://getfedora.org/atomic_qcow2_latest'
-  export TEST_SUBJECTS=\$(pwd)/fedora-atomic-host.qcow2
-
-If you're doing interactive development, we recommend caching the qcow2
-somewhere persistent.
-EOF
-    exit 1
-fi
-for subj in ${TEST_SUBJECTS}; do
-    ls -al ${subj} && file ${subj}
-done
-
-# This is required
-rpm -q standard-test-roles
-
-export ANSIBLE_INVENTORY=${ANSIBLE_INVENTORY:-$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)}
-ls -al /dev/kvm
-# Sadly having this on makes the reboot playbook break
-export ANSIBLE_SSH_ARGS='-o ControlMaster=no'
-exec ansible-playbook --tags=atomic "$@"

tests/installed/provision.sh

@@ -1,8 +0,0 @@
-#!/usr/bin/bash
-set -xeuo pipefail
-
-dn=$(dirname $0)
-. ${dn}/../../ci/libpaprci/libbuild.sh
-
-pkg_upgrade
-pkg_install git rsync openssh-clients ansible standard-test-roles

tests/installed/run.sh

@@ -1,12 +0,0 @@
-#!/usr/bin/bash
-# Run all installed tests; see README.md in this directory for more
-# information.
-set -xeuo pipefail
-
-dn=$(cd $(dirname $0) && pwd)
-
-# TODO: parallelize this
-PLAYBOOKS=${PLAYBOOKS:-nondestructive.yml destructive-ansible.yml destructive-unit.yml}
-for playbook in $PLAYBOOKS; do
-    time ${dn}/playbook-run.sh -v ${dn}/${playbook}
-done

tests/installed/tasks/disable-all-rpmmd-repos.yml

@@ -1,5 +0,0 @@
-- name: Disable all rpmmd repos
-  shell: |
-    for x in /etc/yum.repos.d/*.repo; do
-      sed -i -e 's,^enabled=,enabled=0,g' $x
-    done

tests/installed/tasks/install-git.yml

@@ -1,23 +0,0 @@
-# Use package layering to install git builds.
-- command: ostree --version
-  changed_when: False
-  register: ostree_orig_version
-- set_fact:
-    ostree_orig_version_yaml: "{{ ostree_orig_version.stdout | from_yaml }}"
-- name: Copy locally built RPMs
-  synchronize: src=build/x86_64/ dest=/root/x86_64/ archive=yes
-- name: Install RPMs
-  shell: rpm-ostree override replace /root/x86_64/*.rpm
-# Regenerate to make sure new ostree binaries also make it to the initrd
-- name: Regenerate initramfs
-  shell: rpm-ostree initramfs --enable
-- import_tasks: ../tasks/reboot.yml
-- import_tasks: ../tasks/query-host.yml
-- command: ostree --version
-  register: ostree_new_version
-- set_fact:
-    ostree_new_version_yaml: "{{ ostree_new_version.stdout | from_yaml }}"
-- name: "Fail if we didn't change the ostree version"
-  when: ostree_orig_version_yaml['libostree']['Git'] == ostree_new_version_yaml['libostree']['Git']
-  fail:
-    msg: "Failed to change ostree version"

tests/installed/tasks/overlay-git.yml

@@ -1,23 +0,0 @@
-# Run "admin unlock" and add locally built RPMs, then
-# copy the whole tests/ directory into the VM.
-- command: ostree --version
-  changed_when: False
-  register: ostree_orig_version
-- set_fact:
-    ostree_orig_version_yaml: "{{ ostree_orig_version.stdout | from_yaml }}"
-- name: Copy locally built RPMs
-  synchronize: src=build/x86_64/ dest=/root/x86_64/ archive=yes
-- shell: ostree admin unlock || true
-# Install the RPMs we already have.  For the test suite we use rpm2cpio
-# since it depends on libsoup, but we're not using that yet for the sysinstalled tests
-- shell: >
-    /usr/bin/rpm -Fvh /root/x86_64/*.rpm && \
-    cd / && rpm2cpio /root/x86_64/ostree-tests-2*.rpm | cpio -div
-- command: ostree --version
-  register: ostree_new_version
-- set_fact:
-    ostree_new_version_yaml: "{{ ostree_new_version.stdout | from_yaml }}"
-- name: "Fail if we didn't change the ostree version"
-  when: ostree_orig_version_yaml['libostree']['Git'] == ostree_new_version_yaml['libostree']['Git']
-  fail:
-    msg: "Failed to change ostree version"

tests/installed/tasks/query-host.yml

@@ -1,6 +0,0 @@
-- name: Load status json
-  command: rpm-ostree status --json
-  changed_when: False
-  register: rpmostree_status_json
-- set_fact:
-    rpmostree_status: "{{ rpmostree_status_json.stdout | from_json }}"

tests/installed/tasks/reboot.yml

@@ -1,72 +0,0 @@
-# This file is copied from atomic-host-tests
-
-# vim: set ft=ansible:
-# There is no clean way to restart hosts in ansible. The general issue is that
-# the shutdown command may close sshd before ansible has time to "return" from
-# the task, even with async & poll. This is due to the fact that asynchronous
-# tasks still require a small synchronous bootstrapping script which takes 1 sec
-# to complete, during which it is vulnerable to erroring out if sshd dies.
-#       To mitigate this, we prefix a sleep command before the shutdown so
-# ansible has time to move on. For more info on this issue, see:
-# https://github.com/ansible/ansible/issues/10616
-#
-# The Ansible docs now recommend this combination of tasks to handle reboots
-# https://support.ansible.com/hc/en-us/articles/201958037-Reboot-a-server-and-wait-for-it-to-come-back
-
-# remember the real ansible_host for following local actions
-# (otherwise ansible will target the localhost)
-- set_fact:
-    real_ansible_host: "{{ ansible_host }}"
-    timeout: "{{ cli_reboot_timeout | default('120') }}"
-
-# Have to account for both because Fedora STR uses the old version of these
-# inventory values for some reason.
-- when: ansible_port is defined
-  set_fact:
-    real_ansible_port: "{{ ansible_port }}"
-
-- when: ansible_ssh_port is defined
-  set_fact:
-    real_ansible_port: "{{ ansible_ssh_port }}"
-
-- name: Get original bootid
-  command: cat /proc/sys/kernel/random/boot_id
-  register: orig_bootid
-
-# Stop sshd (thus preventing new connections) and kill our current user's
-# connection so that we can't race to get back in to the system while it's
-# shutting down
-- name: restart hosts
-  when: (not skip_shutdown is defined) or (not skip_shutdown)
-  shell: |
-    systemctl stop sshd
-    systemd-run --on-active=5 systemctl reboot
-  async: 1
-  poll: 0
-  ignore_errors: true
-
-# NB: The wait_for is executed locally and doesn't require privs, so avoid sudo
-- debug:
-    msg: "Waiting for reboot: {{ ansible_date_time.iso8601 }}"
-- wait_for_connection:
-    delay: 5
-    timeout: 120
-    search_regex: "OpenSSH"
-- debug:
-    msg: "SSH port is up {{ ansible_date_time.iso8601 }}"
-
-- name: Assert that the bootid changed
-  command: cat /proc/sys/kernel/random/boot_id
-  register: new_bootid
-  until: new_bootid.stdout != orig_bootid.stdout
-- assert:
-    that:
-      - new_bootid.stdout != orig_bootid.stdout
-
-# provide an empty iterator when a list is not provided
-# http://docs.ansible.com/ansible/playbooks_conditionals.html#loops-and-conditionals
-- name: check services have started
-  service:
-    name: "{{ item }}"
-    state: started
-  with_items: "{{ wait_for_services|default([]) }}"

tests/kola/README.md

@@ -0,0 +1 @@
+This uses https://github.com/coreos/coreos-assembler/pull/1252

tests/kola/destructive/data

@@ -0,0 +1 @@
+..

tests/kola/destructive/itest-bare-root.sh

@@ -4,12 +4,13 @@
 
 set -xeuo pipefail
 
-dn=$(dirname $0)
+. ${KOLA_EXT_DATA}/libinsttest.sh
-. ${dn}/../libinsttest.sh
 
 echo "1..2"
 date
 
+require_writable_sysroot
+
 cd /ostree/repo/tmp
 rm co -rf
 rm co-testref -rf

tests/kola/destructive/itest-deploy-selinux.sh

@@ -4,8 +4,9 @@
 
 set -xeuo pipefail
 
-dn=$(dirname $0)
+. ${KOLA_EXT_DATA}/libinsttest.sh
-. ${dn}/../libinsttest.sh
+
+require_writable_sysroot
 
 date
 # Create a new deployment
@@ -35,8 +36,8 @@ ostree admin undeploy 0
 
 cd /ostree/repo/tmp
 ostree checkout --fsync=0 -H ${host_commit} test-label
-rm test-label/usr/lib/ostree-boot/vmlinuz*
+rm -vf test-label/usr/lib/ostree-boot/vmlinuz*
-rm test-label/usr/lib/ostree-boot/initramfs*
+rm -vf test-label/usr/lib/ostree-boot/initramfs*
 cd test-label/usr/lib/modules/*
 rm initramfs.img
 echo new initramfs > initramfs.img

tests/kola/destructive/itest-label-selinux.sh

@@ -4,8 +4,7 @@
 
 set -xeuo pipefail
 
-dn=$(dirname $0)
+. ${KOLA_EXT_DATA}/libinsttest.sh
-. ${dn}/../libinsttest.sh
 
 date
 cd /ostree/repo/tmp

tests/kola/destructive/staged-deploy.sh

@@ -1,14 +1,21 @@
+#!/bin/bash
+set -xeuo pipefail
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+
+require_writable_sysroot
+prepare_tmpdir
+
+n=$(nth_boot)
+case "${n}" in
+  1)
+    commit=${host_commit}
   # Test the deploy --stage functionality; first, we stage a deployment
   # reboot, and validate that it worked.
-
   # for now, until the preset propagates down
-- name: Start up path unit
+  # Start up path unit
-  shell: |
-    set -xeuo pipefail
     systemctl enable --now ostree-finalize-staged.path
-- name: Write staged-deploy commit
+  # Write staged-deploy commit
-  shell: |
-    set -xeuo pipefail
     export OSTREE_SYSROOT_DEBUG="test-staged-path"
     cd /ostree/repo/tmp
     # https://github.com/ostreedev/ostree/issues/1569
@@ -20,10 +27,7 @@
     systemctl show -p ActiveState ostree-finalize-staged.service | grep -q inactive
     systemctl show -p TriggeredBy ostree-finalize-staged.service | grep -q path
     ostree admin deploy --stage staged-deploy | tee out.txt
-    if ! grep -q 'test-staged-path: Not running' out.txt; then
+    assert_file_has_content out.txt 'test-staged-path: Not running'
-      cat out.txt
-      exit 1
-    fi
     systemctl show -p SubState ostree-finalize-staged.path | grep running
     systemctl show -p ActiveState ostree-finalize-staged.service | grep active
     new_mtime=$(stat -c '%.Y' /sysroot/ostree/deploy)
@@ -37,24 +41,22 @@
     test -f deployment-ref-found
     rm deployment-ref-found
     if ostree admin pin 0 2>err.txt; then
-      echo "Pinned staged deployment"; exit 1
+      fatal "Pinned staged deployment"
     fi
-    grep -qFe 'Cannot pin staged deployment' err.txt
+    assert_file_has_content err.txt 'Cannot pin staged deployment'
-  environment:
+    kola_reboot
-    commit: "{{ rpmostree_status['deployments'][0]['checksum'] }}"
+    ;;
-- include_tasks: ../tasks/reboot.yml
+  2) 
-- name: Check that deploy-staged service worked
+    # Check that deploy-staged service worked
-  shell: |
-    set -xeuo pipefail
     rpm-ostree status
     # Assert that the previous boot had a journal entry for it
-    journalctl -b "-1" -u ostree-finalize-staged.service | grep -q -e 'Transaction complete'
+    journalctl -b "-1" -u ostree-finalize-staged.service > svc.txt
+    assert_file_has_content svc.txt 'Bootloader updated; bootconfig swap: yes; deployment count change: 1'
+    rm -f svc.txt
     # And there should not be a staged deployment
     test '!' -f /run/ostree/staged-deployment
 
-- name: Upgrade with staging
+    # Upgrade with staging
-  shell: |
-    set -xeuo pipefail
     test '!' -f /run/ostree/staged-deployment
     ostree admin deploy --stage staged-deploy
     test -f /run/ostree/staged-deployment
@@ -67,27 +69,25 @@
     test -f /run/ostree/staged-deployment
     # Debating bouncing back out to Ansible for this
     firstdeploycommit=$(rpm-ostree status |grep 'Commit:' |head -1|sed -e 's,^ *Commit: *,,')
-    test "${firstdeploycommit}" = "${newcommit}"
+    assert_streq "${firstdeploycommit}" "${newcommit}"
     # Cleanup
     rpm-ostree cleanup -rp
-- import_tasks: ../tasks/query-host.yml
+    echo "ok upgrade with staging"
 
     # Ensure we can unstage
-- name: Write staged-deploy commit, then unstage
+    # Write staged-deploy commit, then unstage
-  shell: |
-    set -xeuo pipefail
     ostree admin deploy --stage staged-deploy
     ostree admin status > status.txt
-    grep -qFe '(staged)' status.txt
+    assert_file_has_content_literal status.txt '(staged)'
     test -f /run/ostree/staged-deployment
     ostree admin undeploy 0
     ostree admin status > status.txt
     grep -vqFe '(staged)' status.txt
     test '!' -f /run/ostree/staged-deployment
+    echo "ok unstage"
 
-- name: Staged should be overwritten by non-staged as first
+   # Staged should be overwritten by non-staged as first
-  shell: |
+    commit=$(rpmostree_query_json '.deployments[0].checksum')
-    set -xeuo pipefail
     ostree admin deploy --stage staged-deploy
     test -f /run/ostree/staged-deployment
     ostree --repo=/ostree/repo refs --create nonstaged-deploy "${commit}"
@@ -96,22 +96,23 @@
     grep -vqFe '(staged)' status.txt
     test '!' -f /run/ostree/staged-deployment
     ostree admin undeploy 0
-  environment:
+    echo "ok staged overwritten by non-staged"
-    commit: "{{ rpmostree_status['deployments'][0]['checksum'] }}"
 
-- name: Staged is retained when pushing rollback
+  # Staged is retained when pushing rollback
-  shell: |
+    commit=$(rpmostree_query_json '.deployments[0].checksum')
-    set -xeuo pipefail
     ostree admin deploy --stage staged-deploy
     test -f /run/ostree/staged-deployment
     ostree admin deploy --retain-pending --not-as-default nonstaged-deploy
     test -f /run/ostree/staged-deployment
     ostree admin status > status.txt
-    grep -qFe '(staged)' status.txt
+    assert_file_has_content_literal status.txt '(staged)'
     ostree admin undeploy 0
     ostree admin undeploy 1
-  environment:
+    echo "ok staged retained"
-    commit: "{{ rpmostree_status['deployments'][0]['checksum'] }}"
 
-- name: Cleanup refs
+    # Cleanup refs
-  shell: ostree refs --delete staged-deploy nonstaged-deploy 
+    ostree refs --delete staged-deploy nonstaged-deploy
+    echo "ok cleanup refs"
+    ;;
+  *) fatal "Unexpected boot count" ;;
+esac

tests/kola/destructive/var-mount.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+# https://github.com/ostreedev/ostree/issues/1667
+set -xeuo pipefail
+
+. ${KOLA_EXT_DATA}/libinsttest.sh
+
+n=$(nth_boot)
+case "${n}" in
+  1) 
+    require_writable_sysroot
+    # Hack this off for now
+    chattr -i /sysroot
+    cp -a /var /sysroot/myvar
+    touch /sysroot/myvar/somenewfile
+    echo '/sysroot/myvar /var none bind 0 0' >> /etc/fstab
+    kola_reboot
+    ;;
+  2)
+    systemctl status var.mount
+    test -f /var/somenewfile
+    ;;
+  *) fatal "Unexpected boot count $n"
+esac

tests/kola/libinsttest.sh

@@ -0,0 +1,91 @@
+# Common definitions for installed, privileged tests
+#
+# Copyright (C) 2017 Colin Walters <walters@verbum.org>
+#
+# SPDX-License-Identifier: LGPL-2.0+
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the
+# Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+. ${KOLA_EXT_DATA}/libtest-core.sh
+
+# Copy of bits from tap-test
+test_tmpdir=
+function _tmpdir_cleanup () {
+    if test -z "${TEST_SKIP_CLEANUP:-}" &&
+       test -n "${test_tmpdir}" && test -f ${test_tmpdir}/.testtmp; then
+	      rm "${test_tmpdir}" -rf
+    fi
+}
+prepare_tmpdir() {
+    local tmpdir=${1:-/var/tmp}
+    test_tmpdir=$(mktemp -p ${tmpdir} -d ostree-insttest.XXXXXXXX)
+    touch ${test_tmpdir}/.testtmp
+    cd ${test_tmpdir}
+}
+
+# This is copied from flatpak/flatpak/tests/test-webserver.sh
+run_tmp_webserver() {
+    dir=$1
+
+    port=8000
+    podman create --name ostree-httpd --privileged  -ti --net=host -v "${dir}":/srv --workdir /srv \
+        registry.svc.ci.openshift.org/coreos/fedora:31 python3 -m http.server "${port}"
+    podman generate systemd ostree-httpd > /etc/systemd/system/ostree-httpd.service
+    systemctl daemon-reload
+    systemctl start ostree-httpd.service
+
+    address="http://127.0.0.1:${port}"
+    while ! curl --head "${address}" &>/dev/null; do sleep 1; done
+    echo "${address}" > ${test_tmpdir}/httpd-address
+}
+
+# Yeah this is a hack.  Doing this better requires more first class support
+# for creating derived commits in ostree potentially.  Or barring that,
+# just recommend to people to use `unshare -m` or equivalent and
+# mount -o remount,rw /sysroot in their code.
+require_writable_sysroot() {
+    if ! test -w /sysroot; then
+        mount -o remount,rw /sysroot
+    fi
+}
+
+nth_boot() {
+    journalctl --list-boots | wc -l
+}
+
+kola_reboot() {
+    kill -TERM $$
+    sleep 2m
+    echo "failed to reboot?" 1>&2
+    exit 1
+}
+
+# Determine our origin refspec - we'll use this as a test base
+rpmostree=$(which rpm-ostree 2>/dev/null)
+if test -z "${rpmostree}"; then
+    skip "no rpm-ostree, at some point point this to raw ostree too"
+fi
+
+# We need to be root
+assert_streq $(id -u) 0
+
+rpmostree_query_json() {
+    query=$1
+    rpm-ostree status --json | jq -r "${query}"
+}
+host_refspec=$(rpmostree_query_json '.deployments[0].origin')
+host_commit=$(rpmostree_query_json '.deployments[0].checksum')
+host_osname=$(rpmostree_query_json '.deployments[0].osname')

tests/kola/libtest-core.sh

@@ -0,0 +1,153 @@
+# Core source library for shell script tests; the
+# canonical version lives in:
+#
+#   https://github.com/ostreedev/ostree
+#
+# Known copies are in the following repos:
+#
+# - https://github.com/projectatomic/rpm-ostree
+#
+# Copyright (C) 2017 Colin Walters <walters@verbum.org>
+#
+# SPDX-License-Identifier: LGPL-2.0+
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the
+# Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+# Boston, MA 02111-1307, USA.
+
+fatal() {
+    echo $@ 1>&2; exit 1
+}
+# fatal() is shorter to type, but retain this alias
+assert_not_reached () {
+    fatal "$@"
+}
+
+# Some tests look for specific English strings. Use a UTF-8 version
+# of the C (POSIX) locale if we have one, or fall back to en_US.UTF-8
+# (https://sourceware.org/glibc/wiki/Proposals/C.UTF-8)
+#
+# If we can't find the locale command assume we have support for C.UTF-8
+# (e.g. musl based systems)
+if type -p locale >/dev/null; then
+    export LC_ALL=$(locale -a | grep -iEe '^(C|en_US)\.(UTF-8|utf8)$' | head -n1 || true)
+    if [ -z "${LC_ALL}" ]; then fatal "Can't find suitable UTF-8 locale"; fi
+else
+    export LC_ALL=C.UTF-8
+fi
+# A GNU extension, used whenever LC_ALL is not C
+unset LANGUAGE
+
+# This should really be the default IMO
+export G_DEBUG=fatal-warnings
+
+assert_streq () {
+    test "$1" = "$2" || fatal "$1 != $2"
+}
+
+assert_str_match () {
+    if ! echo "$1" | grep -E -q "$2"; then
+	      fatal "$1 does not match regexp $2"
+    fi
+}
+
+assert_not_streq () {
+    (! test "$1" = "$2") || fatal "$1 == $2"
+}
+
+assert_has_file () {
+    test -f "$1" || fatal "Couldn't find '$1'"
+}
+
+assert_has_dir () {
+    test -d "$1" || fatal "Couldn't find '$1'"
+}
+
+# Dump ls -al + file contents to stderr, then fatal()
+_fatal_print_file() {
+    file="$1"
+    shift
+    ls -al "$file" >&2
+    sed -e 's/^/# /' < "$file" >&2
+    fatal "$@"
+}
+
+assert_not_has_file () {
+    if test -f "$1"; then
+        _fatal_print_file "$1" "File '$1' exists"
+    fi
+}
+
+assert_not_file_has_content () {
+    fpath=$1
+    shift
+    for re in "$@"; do
+        if grep -q -e "$re" "$fpath"; then
+            _fatal_print_file "$fpath" "File '$fpath' matches regexp '$re'"
+        fi
+    done
+}
+
+assert_not_has_dir () {
+    if test -d "$1"; then
+	      fatal "Directory '$1' exists"
+    fi
+}
+
+assert_file_has_content () {
+    fpath=$1
+    shift
+    for re in "$@"; do
+        if ! grep -q -e "$re" "$fpath"; then
+            _fatal_print_file "$fpath" "File '$fpath' doesn't match regexp '$re'"
+        fi
+    done
+}
+
+assert_file_has_content_literal () {
+    fpath=$1; shift
+    for s in "$@"; do
+        if ! grep -q -F -e "$s" "$fpath"; then
+            _fatal_print_file "$fpath" "File '$fpath' doesn't match fixed string list '$s'"
+        fi
+    done
+}
+
+assert_file_has_mode () {
+    mode=$(stat -c '%a' $1)
+    if [ "$mode" != "$2" ]; then
+        fatal "File '$1' has wrong mode: expected $2, but got $mode"
+    fi
+}
+
+assert_symlink_has_content () {
+    if ! test -L "$1"; then
+        fatal "File '$1' is not a symbolic link"
+    fi
+    if ! readlink "$1" | grep -q -e "$2"; then
+        _fatal_print_file "$1" "Symbolic link '$1' doesn't match regexp '$2'"
+    fi
+}
+
+assert_file_empty() {
+    if test -s "$1"; then
+        _fatal_print_file "$1" "File '$1' is not empty"
+    fi
+}
+
+# Use to skip all of these tests
+skip() {
+    echo "1..0 # SKIP" "$@"
+    exit 0
+}

tests/kola/nondestructive/data

@@ -0,0 +1 @@
+..

tests/kola/nondestructive/itest-bare-unit.sh

@@ -5,8 +5,9 @@
 
 set -xeuo pipefail
 
-dn=$(dirname $0)
+. ${KOLA_EXT_DATA}/libinsttest.sh
-. ${dn}/../libinsttest.sh
+
+fatal "FIXME - need to also sync over the installed tests"
 
 date
 # These tests sort of bypass the installed-tests spec;

tests/kola/nondestructive/itest-bare-user-root.sh

@@ -4,8 +4,7 @@
 
 set -xeuo pipefail
 
-dn=$(dirname $0)
+. ${KOLA_EXT_DATA}/libinsttest.sh
-. ${dn}/../libinsttest.sh
 
 echo "1..1"
 date

tests/kola/nondestructive/itest-bareuser-nouserxattrs.sh

@@ -7,8 +7,7 @@
 
 set -xeuo pipefail
 
-dn=$(dirname $0)
+. ${KOLA_EXT_DATA}/libinsttest.sh
-. ${dn}/../libinsttest.sh
 
 prepare_tmpdir
 trap _tmpdir_cleanup EXIT

tests/kola/nondestructive/itest-payload-link.sh

@@ -21,8 +21,7 @@
 
 set -xeuo pipefail
 
-dn=$(dirname $0)
+. ${KOLA_EXT_DATA}/libinsttest.sh
-. ${dn}/../libinsttest.sh
 
 echo "1..1"
 date

tests/kola/nondestructive/itest-pull-space.sh

@@ -3,14 +3,10 @@
 
 set -xeuo pipefail
 
-dn=$(dirname $0)
+. ${KOLA_EXT_DATA}/libinsttest.sh
-. ${dn}/../libinsttest.sh
 date
 
 prepare_tmpdir
-trap _tmpdir_cleanup EXIT
-
-cd ${test_tmpdir}
 truncate -s 20MB testblk.img
 blkdev=$(losetup --find --show $(pwd)/testblk.img)
 mkfs.xfs ${blkdev}
@@ -51,9 +47,7 @@ rm -rf mnt/repo
 ostree --repo=mnt/repo init --mode=bare-user
 echo 'fsync=false' >> mnt/repo/config
 echo 'min-free-space-size=10MB' >> mnt/repo/config
-if ostree --repo=mnt/repo pull-local --commit-metadata-only /ostree/repo ${host_commit} 2>err.txt; then
+ostree --repo=mnt/repo pull-local --commit-metadata-only /ostree/repo ${host_commit}
-    fatal "could not write metadata objects even when min-free-space value should allow it"
-fi
 echo "ok metadata write even when free space is lower than min-free-space value"
 
 rm -rf mnt/repo

tests/kola/nondestructive/itest-pull.sh

@@ -7,8 +7,7 @@ set -xeuo pipefail
 # FIXME: https://github.com/ostreedev/ostree/pull/1548
 exit 0
 
-dn=$(dirname $0)
+. ${KOLA_EXT_DATA}/libinsttest.sh
-. ${dn}/../libinsttest.sh
 date
 
 prepare_tmpdir /var/tmp

tests/kola/nondestructive/itest-remotes.sh

@@ -4,8 +4,7 @@
 
 set -xeuo pipefail
 
-dn=$(dirname $0)
+. ${KOLA_EXT_DATA}/libinsttest.sh
-. ${dn}/../libinsttest.sh
 date
 
 prepare_tmpdir

tests/libtest-core.sh

@@ -1,153 +0,0 @@
-# Core source library for shell script tests; the
-# canonical version lives in:
-#
-#   https://github.com/ostreedev/ostree
-#
-# Known copies are in the following repos:
-#
-# - https://github.com/projectatomic/rpm-ostree
-#
-# Copyright (C) 2017 Colin Walters <walters@verbum.org>
-#
-# SPDX-License-Identifier: LGPL-2.0+
-#
-# This library is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Lesser General Public
-# License as published by the Free Software Foundation; either
-# version 2 of the License, or (at your option) any later version.
-#
-# This library is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-# Lesser General Public License for more details.
-#
-# You should have received a copy of the GNU Lesser General Public
-# License along with this library; if not, write to the
-# Free Software Foundation, Inc., 59 Temple Place - Suite 330,
-# Boston, MA 02111-1307, USA.
-
-fatal() {
-    echo $@ 1>&2; exit 1
-}
-# fatal() is shorter to type, but retain this alias
-assert_not_reached () {
-    fatal "$@"
-}
-
-# Some tests look for specific English strings. Use a UTF-8 version
-# of the C (POSIX) locale if we have one, or fall back to en_US.UTF-8
-# (https://sourceware.org/glibc/wiki/Proposals/C.UTF-8)
-#
-# If we can't find the locale command assume we have support for C.UTF-8
-# (e.g. musl based systems)
-if type -p locale >/dev/null; then
-    export LC_ALL=$(locale -a | grep -iEe '^(C|en_US)\.(UTF-8|utf8)$' | head -n1 || true)
-    if [ -z "${LC_ALL}" ]; then fatal "Can't find suitable UTF-8 locale"; fi
-else
-    export LC_ALL=C.UTF-8
-fi
-# A GNU extension, used whenever LC_ALL is not C
-unset LANGUAGE
-
-# This should really be the default IMO
-export G_DEBUG=fatal-warnings
-
-assert_streq () {
-    test "$1" = "$2" || fatal "$1 != $2"
-}
-
-assert_str_match () {
-    if ! echo "$1" | grep -E -q "$2"; then
-	      fatal "$1 does not match regexp $2"
-    fi
-}
-
-assert_not_streq () {
-    (! test "$1" = "$2") || fatal "$1 == $2"
-}
-
-assert_has_file () {
-    test -f "$1" || fatal "Couldn't find '$1'"
-}
-
-assert_has_dir () {
-    test -d "$1" || fatal "Couldn't find '$1'"
-}
-
-# Dump ls -al + file contents to stderr, then fatal()
-_fatal_print_file() {
-    file="$1"
-    shift
-    ls -al "$file" >&2
-    sed -e 's/^/# /' < "$file" >&2
-    fatal "$@"
-}
-
-assert_not_has_file () {
-    if test -f "$1"; then
-        _fatal_print_file "$1" "File '$1' exists"
-    fi
-}
-
-assert_not_file_has_content () {
-    fpath=$1
-    shift
-    for re in "$@"; do
-        if grep -q -e "$re" "$fpath"; then
-            _fatal_print_file "$fpath" "File '$fpath' matches regexp '$re'"
-        fi
-    done
-}
-
-assert_not_has_dir () {
-    if test -d "$1"; then
-	      fatal "Directory '$1' exists"
-    fi
-}
-
-assert_file_has_content () {
-    fpath=$1
-    shift
-    for re in "$@"; do
-        if ! grep -q -e "$re" "$fpath"; then
-            _fatal_print_file "$fpath" "File '$fpath' doesn't match regexp '$re'"
-        fi
-    done
-}
-
-assert_file_has_content_literal () {
-    fpath=$1; shift
-    for s in "$@"; do
-        if ! grep -q -F -e "$s" "$fpath"; then
-            _fatal_print_file "$fpath" "File '$fpath' doesn't match fixed string list '$s'"
-        fi
-    done
-}
-
-assert_file_has_mode () {
-    mode=$(stat -c '%a' $1)
-    if [ "$mode" != "$2" ]; then
-        fatal "File '$1' has wrong mode: expected $2, but got $mode"
-    fi
-}
-
-assert_symlink_has_content () {
-    if ! test -L "$1"; then
-        fatal "File '$1' is not a symbolic link"
-    fi
-    if ! readlink "$1" | grep -q -e "$2"; then
-        _fatal_print_file "$1" "Symbolic link '$1' doesn't match regexp '$2'"
-    fi
-}
-
-assert_file_empty() {
-    if test -s "$1"; then
-        _fatal_print_file "$1" "File '$1' is not empty"
-    fi
-}
-
-# Use to skip all of these tests
-skip() {
-    echo "1..0 # SKIP" "$@"
-    exit 0
-}

tests/libtest-core.sh

@@ -0,0 +1 @@
+kola/libtest-core.sh