[tests] Start using avocado for test suite

This commit represents the start of an overhaul of the test suite used
by sos. Note that several more commits to follow will be required in
order for the test suite to be considered stable.

The new test suite will use the avocado-framework to build out new
tests.

This first part adopts a new 'stageX' naming scheme for our tests as
follows:

	stage0 -> Unittests
	stage1 -> Basic function tests, no mocking allowed
	stage2 -> Mocked tests for specific scenarios/regressions
	stage3 -> Complex setups for layered products/environments

At the moment, these unittests are not updated for avocado, though most
should still work with `nosetest` directly.

A new set of base classes is defined in tests/sos_tests.py which provide
the foundation for actual tests cases. This approach entails new test
cases subclassing a base class, such as the new `StageOneReportTest`,
and setting the `sos_cmd` class attr to the _options_ for an sos report
run. By default `sos report --batch` will be run, and targeted to the
test job's directory as a tmpdir.

Each sos command will be executed once, and all test_* methods within a
test case that subclasses `StageOneReportTest` will be checked against
the output of that execution. Note that this diverges from avocado's
typical approach where each test_* method is performed against a brand
new instance of the class (thus meaning any setup including our sos
report run would normally be run fresh). However, after speaking with
the avocado devel team, this is still seen as a valid pattern for the
framework.

The current organizational approach is to separate the tests by
component rather than stage. For example. `tests/report_tests/` should
hold any report-centric tests, and the `plugin_tests` directory therein
should be used for plugin-specific tests. As of this commit, there are
basic functionality tests under `tests/report_tests/` and a single
plugin test under `tests/report_tests/plugin_tests/` to act as a POC.

Further, there is a `tests/vendor_tests/` directory for organizing
vendor-specific bug/feature tests that are not covered by the generic
project-wide tests. A POC test from RHBZ1928628 is available with this
commit.

Note that in order for these tests to be run properly _without_
installing the current branch to the local system, you will need to run
the following command:

`PYTHONPATH=tests/ avocado run -t stageone tests/`

Related: #2431

Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>

5 files changed, 181 insertions, 0 deletions

diff --git a/tests/report_tests/__init__.py b/tests/report_tests/__init__.py
new file mode 100644
index 00000000..8b137891
--- /dev/null
+++ b/tests/report_tests/__init__.py
@@ -0,0 +1 @@
+
diff --git a/tests/report_tests/basic_report_tests.py b/tests/report_tests/basic_report_tests.py
new file mode 100644
index 00000000..0cf5bd4e
--- /dev/null
+++ b/tests/report_tests/basic_report_tests.py
@@ -0,0 +1,80 @@
+# This file is part of the sos project: https://github.com/sosreport/sos
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# version 2 of the GNU General Public License.
+#
+# See the LICENSE file in the source distribution for further information.
+
+
+from sos_tests import StageOneReportTest
+
+
+class NormalSoSReport(StageOneReportTest):
+    """
+    :avocado: tags=stageone
+    """
+
+    sos_cmd = '-vvv --label thisismylabel'
+
+    def test_debug_in_logs_verbose(self):
+        self.assertSosLogContains('DEBUG')
+
+    def test_postproc_called(self):
+        self.assertSosLogContains('substituting scrpath')
+
+    def test_label_applied_to_archive(self):
+        self.assertTrue('thisismylabel' in self.archive)
+
+    def test_free_symlink_created(self):
+        self.assertFileCollected('free')
+
+
+class RestrictedSoSReport(StageOneReportTest):
+    """
+    :avocado: tags=stageone
+    """
+
+    sos_cmd = '-o kernel,host,sudo,hardware,dbus,x11 --no-env-var --no-report -t1 --no-postproc'
+
+    def test_no_env_vars_collected(self):
+        self.assertFileNotCollected('environment')
+
+    def test_no_reports_generated(self):
+        self.assertFileNotCollected('sos_reports/sos.html')
+        self.assertFileNotCollected('sos_reports/sos.json')
+        self.assertFileNotCollected('sos_reports/sos.txt')
+
+    def test_was_single_threaded_run(self):
+        self.assertOutputNotContains('Finishing plugins')
+
+    def test_postproc_not_called(self):
+        self.assertOutputNotContains('substituting')
+
+    def test_only_selected_plugins_run(self):
+        self.assertOnlyPluginsIncluded(['kernel', 'host', 'sudo', 'hardware', 'dbus', 'x11'])
+
+
+class DisabledCollectionsReport(StageOneReportTest):
+    """
+    :avocado: tags=stageone
+    """
+
+    sos_cmd = "-n networking,system,logs --skip-files=/etc/fstab --skip-commands='journalctl*'"
+
+    def test_plugins_disabled(self):
+        self.assertPluginNotIncluded('networking')
+        self.assertPluginNotIncluded('system')
+        self.assertPluginNotIncluded('logs')
+
+    def test_skipped_plugins_have_no_dir(self):
+        self.assertFileNotCollected('sos_commands/networking/')
+        self.assertFileNotCollected('sos_commands/system/')
+        self.assertFileNotCollected('sos_commands/logs/')
+
+    def test_skip_files_working(self):
+        self.assertFileNotCollected('/etc/fstab')
+
+    def test_skip_commands_working(self):
+        self.assertFileGlobNotInArchive('sos_commands/*/journalctl*')
+
diff --git a/tests/report_tests/plugin_tests/__init__.py b/tests/report_tests/plugin_tests/__init__.py
new file mode 100644
index 00000000..e69de29b
--- /dev/null
+++ b/tests/report_tests/plugin_tests/__init__.py
diff --git a/tests/report_tests/plugin_tests/networking.py b/tests/report_tests/plugin_tests/networking.py
new file mode 100644
index 00000000..f6cfa2d8
--- /dev/null
+++ b/tests/report_tests/plugin_tests/networking.py
@@ -0,0 +1,31 @@
+# This file is part of the sos project: https://github.com/sosreport/sos
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# version 2 of the GNU General Public License.
+#
+# See the LICENSE file in the source distribution for further information.
+
+
+from sos_tests import StageOneReportTest
+
+
+class NetworkingPluginTest(StageOneReportTest):
+    """
+    Basic tests to ensure proper collection from the networking plugins
+
+    :avocado: tags=stageone
+    """
+
+    sos_cmd = '-o networking'
+
+    def test_common_files_collected(self):
+        self.assertFileCollected('/etc/resolv.conf')
+        self.assertFileCollected('/etc/hosts')
+
+    def test_ip_addr_symlink_created(self):
+        self.assertFileCollected('ip_addr')
+
+    def test_forbidden_globs_skipped(self):
+        self.assertFileGlobNotInArchive('/proc/net/rpc/*/channel')
+        self.assertFileGlobNotInArchive('/proc/net/rpc/*/flush')
diff --git a/tests/report_tests/report_with_mask.py b/tests/report_tests/report_with_mask.py
new file mode 100644
index 00000000..a62888ae
--- /dev/null
+++ b/tests/report_tests/report_with_mask.py
@@ -0,0 +1,69 @@
+# This file is part of the sos project: https://github.com/sosreport/sos
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# version 2 of the GNU General Public License.
+#
+# See the LICENSE file in the source distribution for further information.
+
+from sos_tests import StageOneReportTest
+
+import re
+
+
+class ReportWithMask(StageOneReportTest):
+    """Testing around basic --clean/--mask usage and expectations
+
+    :avocado: tags=stageone
+    """
+
+    sos_cmd = '--mask -o host,networking'
+
+    def test_mask_was_run(self):
+        self.assertOutputContains('Beginning obfuscation')
+        self.assertOutputContains('Obfuscation completed')
+
+    def test_private_map_was_generated(self):
+        self.assertOutputContains('A mapping of obfuscated elements is available at')
+        map_file = re.findall('/.*sosreport-.*-private_map', self.cmd_output.stdout)[-1]
+        self.assertFileExists(map_file)
+
+    def test_tarball_named_obfuscated(self):
+        self.assertTrue('obfuscated' in self.archive)
+
+    def test_localhost_was_obfuscated(self):
+        self.assertFileHasContent('/etc/hostname', 'host0')
+
+    def test_ip_address_was_obfuscated(self):
+        # Note: do not test for starting with the 100.* block here, as test
+        # machines may have /32 addresses. Instead, test that the actual
+        # IP address is not present
+        self.assertFileNotHasContent('ip_addr', self.sysinfo['pre']['networking']['ip_addr'])
+
+    def test_loopback_was_not_obfuscated(self):
+        self.assertFileHasContent('ip_addr', '127.0.0.1/8')
+
+    def test_mac_addrs_were_obfuscated(self):
+        content = self.get_file_content('sos_commands/networking/ip_maddr_show')
+        for line in content.splitlines():
+            if line.strip().startswith('link'):
+                mac = line.strip().split()[1]
+                assert mac.startswith('53:4f:53'), "Found unobfuscated mac addr %s" % mac
+
+
+class ReportWithCleanedKeywords(StageOneReportTest):
+    """Testing for obfuscated keywords provided by the user
+
+    :avocado: tags=stageone
+    """
+
+    sos_cmd = '--clean -o filesys,kernel --keywords=fstab,Linux'
+
+    # Ok, sort of cheesy here but this does actually test filename changes on
+    # a file common to all distros
+    def test_filename_obfuscated(self):
+        self.assertFileNotCollected('/etc/fstab')
+        self.assertFileGlobInArchive('/etc/obfuscatedword*')
+
+    def test_keyword_obfuscated_in_file(self):
+        self.assertFileNotHasContent('sos_commands/kernel/uname_-a', 'Linux')