| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Extra changes required due to module and function changes in new
avocado
* CentOS Stream is now seen as centos-stream in avocado instead of
centos
* Fine tune tox for stage tests
Closes: #3588
Resolves: #3594
Co-authored-by: Jake Hunsaker <jacob.r.hunsaker@gmail.com>
Signed-off-by: Arif Ali <arif.ali@canonical.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Consolidate testing, so that things can be run from CLI, CI and
various other means in a consistent way by using tox.
* Updated all the flake8 tests in the tests folder.
* Added pylint as a neutral test so that we can work on this in stages
and have some collaboration on what we test and don't
* The tox tests for unit, stageone, stagetwo testing makes is easier
for users to know how to run tests, and not have to do things
manually
* Using tox for CI doesn't make sense, as that will create virtual
envs and will disregard system/snap based python modules so may not
work
Signed-off-by: Arif Ali <arif.ali@canonical.com>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Fix the issues reported by the tests on the new version of ubuntu
Resolves: #3587
Closes: #3588
Related: SET-594,SET-595
Co-authored-by: David Negreira <david.negreira@canonical.com>
Signed-off-by: Arif Ali <arif.ali@canonical.com>
Signed-off-by: David Negreira <david.negreira@canonical.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Change grep_for_content to grep for a fixed string by default.
The reason is tests would match 1a2b3c4 as IP address 1.2.3.4 (which it
is not).
Add regexp option to grep_for_content, to allow the default "grep" search.
Resolves: #3320
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a new `arch` class attr that allows test classes to specify which
architecture(s) they are allowed/designed to execute on. If the test is
attempted to be executed on an architecture that is not in the specified
list, we will skip the test before any setup or sos executions are done.
By default, the `arch` class attr is an empty list, which implies that
the test can run on _any_ architecture. Tests should specify `arch` as a
list even for single-arch tests.
Included in this change is restricting the Foreman tests to x86_64 only,
as that is the only architecture on which that product is supported.
Closes: #3186
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, mocked files were kept under the `tests/test_data/`
directory and generally mimic'd the file location they would be
temporarily copied to during the execution of their relevant tests.
This has a few maintainability drawbacks, and the handling of the
`files` attribute for test cases as either strings or tuples is at best
confusing.
Improve on this by instead making the `files` references relative to
where the test case file is written. This enables easier maintenance by
keeping all test requirements closer together, rather than spread across
the repo. As such, the `files` attribute now requires a list of tuples,
taking the form `(relative_src, absolute_dest)`. Additionally, fake
plugins for tests that need them to artificially test a specific
criteria should also be included in the test's subdir now.
Along with this change, move several StageTwo tests to their own subdirs
that now contain both the test cases and the needed files for mocking.
This should be the new design pattern going forward - if a test needs to
mock files of any kind, put it in a new subdirectory (and if it doesn't
need to mock files, continue to keep it in the relevant directory within
the test suite).
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
| |
Resolves: #3109
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Up until now, the test suite was forced to be run using the git
checkout. While this is useful for on-the-fly testing, it does miss an
important use case of building a test package from the checkout, and
running it using that as the system installation of sos.
This commit allows the use of an installed version of sos to test
against. This may be leveraged by adding `-p TESTLOCAL=true` in the
`avocado run` command used to launch the test suite. Setting this
parameter to any other value, or omitting it entirely, will continue the
current behavior or using the git checkout for running tests.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a new test case for the `teamd` plugin that also inherently acts to
test team device enumeration.
Included with this test case is an addition to the base test classes
that allows for tests to define a `post_test_tear_down()` method that
will be run at the end of each test execution to allow for manual
cleanup - in this case deleting a 'fake' team device created for the
test.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For our CI tests, CPU resources are not guaranteed which in turn can
cause longer-than-expected run times for test executions. In practice,
this is only seen occasionally but it requires manual intervention when
the timeouts are hit (and thus far it has been seen that the changes
these timeouts are hit on are not actually causing the timeouts).
Previous conversations have revolved around improving test efficiency,
however this seems unlikely given the nature of some of the test setup
and further, even with the most efficienct approach possible we would
still be at the whim of resource availability.
As such, increase the default timeout to account for this resource
consideration.
Closes: #2700
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
In the event a CI test execution fails before the archive is created,
for any reason, we want to preserve the temp directory as the logs there
are still useful.
As such, add a specific env var to our test runs via avocado. Then, when
we detect a failure check for that env var to determine if we should in
fact do our cleanup or not.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
The debug level messages gated by `-v` are very helpful for diagnosing
test failures, but currently not all tests specify the use of verbosity.
Make use of verobsity a default parameter for all test runs to address
this.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Adds failure handling to cirrus tasks that actually run sos so that if
an error is encountered in the test suite, the logs from the tests are
uploaded to the GCE cloud storage bucket associated with the GCE sos
project.
This will also make those logs available from the cirrus task page.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Up until now the `files` attr for Stage Two tests was simply a list that
dropped files of the same name from our `tests/test_data/` directory
into the relevant paths on the test system. This however limited us to
single copies of files within this directory.
Instead, also allow `files` to take a list of tuples in the form of
`(source_path, dest_path)` so that we may have multiple copies of say
`sos.conf` within `tests/test_data` for different test cases while still
being able to temporarily override the test system's actual
`/etc/sos/sos.conf`.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously new test cases had to be written in such a way that they
could be run on any supported distribution, even if their individual
tests were all marked for specific distributions. Use of the distro-only
decorators would fail when applied to test cases as a whole due to how
test instantiation errors are handled.
To address this, provide a distro check within `BaseSoSTest` so that
tests can be made distro-specific and we can signal to avocado to skip
all tests within that test case without failing the test suite entirely.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a test that ensures we are properly extracting and cleaning an
already existing archive (one is being included under tests/test_data/
now), much the same as we test for a proper `sos report --clean` run.
As part of this, some aspects of the `full_report_run.py` test are moved
into the base test classes, and the new `existing_archive.py` test will
explicitly run `sos clean` instead of `sos report`.
The included archive for obfuscation testing is taken from a stock
Fedora 34 VM with a known configuration which is reflected in the items
being tested for.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
| |
If we get an exception we aren't expecting and can't investigate due to
a missing `CmdResult` object being attached to the raised exception,
simply re-raise that exception rather than failing on trying to inspect
a non-existing object.
|
| |
|
|
|
|
|
|
| |
Allows test cases to set their own timeout for sos command executions,
rather than applying a 300 second timeout to all test cases. 300 seconds
remains the default, however.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
In the past, there have been otherwise trivial typos and the like that
have caused `--help` output to be unreliable. In the case of "help
options" such as `--list-plugins` this also includes potentially
unavailable detailed information about whatever is being listed.
These tests are sanity checks to ensure that changes aren't regressing
this behavior and that the informational output options, and `--help`
directly, continue to provide the right output.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Moves several aspects of test setup and design from `BaseSoSReportTest`
into the lower-level `BaseSoSTest`, so that adding new test classes that
are not report-centric will be easier going forward. Where needed,
overrides in `BaseSoSReportTest` have been added.
In general, this means that sos command construction, execution, tmpdir
management, and the methods for mocking and other pre-execution tasks
are now in `BaseSoSTest` and the additions in `BaseSoSReportTest` are
now centered around handling a generated archive.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds automatic collection of journals for any service defined in a
plugin's `services` tuple, if that service exists on the system.
Updates several plugins to define a `services` tuple in place of
manually calling `add_journal()` (and as a bonus, removing redundant
`add_service_status()` calls that are already performed based on the
tuple).
Additionally, add an appropriate tag to each `add_journal` call for
easier collection lookup in the manifest.
Resolves: #2579
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
Adds two test cases for using `--encrypt-pass`; one with just the bare
option and another in conjunction with `--clean` to ensure both can be
used in the same execution.
Resolves: #2566
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Improves error reporting for failed sos commands by logging stderr (or
stdout if stderr is not populated) to the console, which was previously
being truncated by the builtin error handling of avocado. Printed output
is limited to the last 8k to avoid dumping several MBs at a time for
scenarios such as timeouts where command failure may generate
significant logging prior to failing.
Included with this are 2 minor changes to existing tests. First, remove
verbose output from the expected plugins test to reduce otherwise
irrelevant output for command failures. Second limit the number of
plugins run for the LogLevelTest, both to reduce overall run time for a
test where we aren't testing specific plugins and to improve readability
of failures for such a test.
Resolves: #2556
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
Adds a test that checks that the new `priority` parameter is working as
expected for well known and common collections.
Resolves: #2553
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds an integration test for the upstream Foreman project. After
stageone tests have passed, and if the PR has changes relevant to the
Foreman plugins (or the cirrus config), launch a pre-built GCE image
that has a Foreman deployment on it.
Per request of the Foreman devel team, these tests will be run against
both a CentOS and Debian based Foreman image. Currently, testing is
performed against Foreman 2.4.
Resolves: #2531
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
As sos is now python3-only, we can avoid shelling-out to compression
utilities like `xz` or `gzip`, and instead use the method provided by
the built-in `tarfile` module.
Resolves: #2523
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Like `assertFileCollected`, `assertFileGlobInArchive` should only fail
if the referenced collection is not in the archive *and* it is present
on the test system. If it is not present on the test system, sos cannot
collect it and it is thus not an error to be missing from the archive.
Related: #2499
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
First, correct the logs plugin stagetwo test to specify the needed
package in a list, and second move the import to within the test class
that needs it so that we don't fail on the stageone tests if the package
is not present.
Second, correct the journal injections that this plugin test does to use
`pre_sos_setup()` instead of overriding `setup_mocking()`, and call the
former after the mocking has been setup.
Finally, add the resulting size of the collected journal if we fail the
size requirements of the test.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a new `StageOneReportExceptionTest` class that is to be used for
tests that should ensure an exception IS hit, for example when testing
plugin enablement or invalid options.
Included with this commit is a small set of plugin enablement tests.
Future tests for component and plugin specific exception handling should
be built using this class and design approach.
Resolves: #2518
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Standardizes the `assertOutputContains` and `assertOutputNotContains`
methods to use `re.search()`, instead of separate parsing methods.
`search()` replaces `match()` for efficiency purposes against the
collected output.
Related: #2507
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
| |
Updates `assertPlugin(Not)Included` to allow for lists as well as single
strings.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds helper decorators to define specific test methods for use on
specific distributions only. Currently two decorators are available:
@redhat_only Only run on fedora, centos, or rhel
@ubuntu_only Only run on ubuntu or debian
Note that these decorators are only intended for individual `test_*`
methods, and will not function to define distro-specific test classes.
These should make it easier to write plugin test cases where packaging
differences between distributions otherwise makes plugin tests either
impossible needlessly complex to write generically.
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Moves the detection of the local system's distribution to StageOne, and
defines two constants to reference RH and Ubuntu distribution lists to
allow for distro-specific tests.
Further, make the expected_warnings_displayed test skipped if the local
test system is not a RH-family distro, as those warnings are not
expected on non RH-family systems.
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
First, add a "pre-setup setup" method, in the form of `pre_sos_setup()`,
that can be used in the way the traditional `setUp()` method would be
used (but can't because that's our entry point for executing our sos
runs). This method will be executed _prior_ to any mocking.
Second, add a smoke test that enables all plugins that exist in the
local branch being tested. This will test that doing so does not
generate any exceptions and that some expected warnings from select
plugins are displayed.
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commits adds a `StageTwoReportTest` from which tests that need to
do "light" mocking should be defined. As the name implies, this is the
base class for our Stage 2 tests.
"Light" mocking for our purposes is currently defined as dropping files
in locations on the test system's filesystem, which may include
replacing binaries, and installing packages for the durtation of the
test case. At the end of the test, packages are uninstalled and mocked
files are removed. In the event that a file already exists at a given
location on the test system, that file is temporarily renamed with a
'.sostesting' extension. On test clean up, once mocked files are
removed, any files renamed previously are moved back to their original
location/name.
Mocking is controlled automatically during test setup, and is based upon
test class attributes, similar to how plugin enablement is dependent
upon plugin class attributes. For Stage 2 tests cases, this includes:
`files` - a list of file names. These names should represent the
"real" location on the filesystem where the mocked
files will be placed. The mocked content should be
placed under a similar file path under tests/test_data
E.G. using `files = ['/etc/foo/bar']` will place a
copy of `tests/test_data/etc/foo/bar` under
`/etc/foo/bar` on the test system.
`packages` - a dict of packages to install. The dict keys map to
the distribution the package names are used for,
e.g. `rhel` or `ubuntu`. The values are lists of
package names, optionally with version included.
This commit includes just the framework for these tests, and no actual
Stage 2 tests are included as of yet.
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
This commit represents the start of an overhaul of the test suite used
by sos. Note that several more commits to follow will be required in
order for the test suite to be considered stable.
The new test suite will use the avocado-framework to build out new
tests.
This first part adopts a new 'stageX' naming scheme for our tests as
follows:
stage0 -> Unittests
stage1 -> Basic function tests, no mocking allowed
stage2 -> Mocked tests for specific scenarios/regressions
stage3 -> Complex setups for layered products/environments
At the moment, these unittests are not updated for avocado, though most
should still work with `nosetest` directly.
A new set of base classes is defined in tests/sos_tests.py which provide
the foundation for actual tests cases. This approach entails new test
cases subclassing a base class, such as the new `StageOneReportTest`,
and setting the `sos_cmd` class attr to the _options_ for an sos report
run. By default `sos report --batch` will be run, and targeted to the
test job's directory as a tmpdir.
Each sos command will be executed once, and all test_* methods within a
test case that subclasses `StageOneReportTest` will be checked against
the output of that execution. Note that this diverges from avocado's
typical approach where each test_* method is performed against a brand
new instance of the class (thus meaning any setup including our sos
report run would normally be run fresh). However, after speaking with
the avocado devel team, this is still seen as a valid pattern for the
framework.
The current organizational approach is to separate the tests by
component rather than stage. For example. `tests/report_tests/` should
hold any report-centric tests, and the `plugin_tests` directory therein
should be used for plugin-specific tests. As of this commit, there are
basic functionality tests under `tests/report_tests/` and a single
plugin test under `tests/report_tests/plugin_tests/` to act as a POC.
Further, there is a `tests/vendor_tests/` directory for organizing
vendor-specific bug/feature tests that are not covered by the generic
project-wide tests. A POC test from RHBZ1928628 is available with this
commit.
Note that in order for these tests to be run properly _without_
installing the current branch to the local system, you will need to run
the following command:
`PYTHONPATH=tests/ avocado run -t stageone tests/`
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
