| Commit message (Collapse) | Author | Age | Files | Lines |
... | |
|
|
|
|
|
|
|
|
|
|
| |
Standardizes the `assertOutputContains` and `assertOutputNotContains`
methods to use `re.search()`, instead of separate parsing methods.
`search()` replaces `match()` for efficiency purposes against the
collected output.
Related: #2507
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes an issue where archive debug logging was controlled by the use of
`--debug` rather than `--verbose`. Removes a superfluous log in
`FileCacheArchive.add_link()`.
Also, adjusts the different verbosity levels to be more meaninful. Now,
the use of `-v` will enable debug logging but will not print those
messages to console, `-vv` will print debug logging to console, while
`-vvv` will enable archive debug logging which is expected to be
significant due to most file operations being logging at some point
with archive debug logging.
Resolves: #2507
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds collection of per-process files from /proc/$pid, limited to a
number of pids controlled by the new `process.numprocs` option. This
option will default to the first 2048 processes. Using a value of `0`
for this option will result in all pids being collected.
In testing, the initial limit of 2048 process did not significantly
raise collection time or archive size. Note however, that collection of
20k or more processes does show a significant increase in plugin
execution time and archive size. This is however tempered, by the
whole-plugin timeout that sos imposes.
Related: #436
Closes: #542
Resolves: #1783
Resolves: #2498
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a test case to ensure that the plugins we can expect to always be
enabled when a "normal" `sos report` command is run are in fact enabled.
First, test the distro-independent plugins that do not have specific
enablement triggers (and thus should always be enabled).
Second, have a distro-specific test for distro-specific plugins that are
also expected to always run on those systems.
Closes: #2365
Resolves: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
| |
Updates `assertPlugin(Not)Included` to allow for lists as well as single
strings.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds helper decorators to define specific test methods for use on
specific distributions only. Currently two decorators are available:
@redhat_only Only run on fedora, centos, or rhel
@ubuntu_only Only run on ubuntu or debian
Note that these decorators are only intended for individual `test_*`
methods, and will not function to define distro-specific test classes.
These should make it easier to write plugin test cases where packaging
differences between distributions otherwise makes plugin tests either
impossible needlessly complex to write generically.
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Moves the detection of the local system's distribution to StageOne, and
defines two constants to reference RH and Ubuntu distribution lists to
allow for distro-specific tests.
Further, make the expected_warnings_displayed test skipped if the local
test system is not a RH-family distro, as those warnings are not
expected on non RH-family systems.
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
First, add a "pre-setup setup" method, in the form of `pre_sos_setup()`,
that can be used in the way the traditional `setUp()` method would be
used (but can't because that's our entry point for executing our sos
runs). This method will be executed _prior_ to any mocking.
Second, add a smoke test that enables all plugins that exist in the
local branch being tested. This will test that doing so does not
generate any exceptions and that some expected warnings from select
plugins are displayed.
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds several plugin tests for plugins that can be expected to run on all
distributions, in particular these tests are including mostly Stage Two
tests.
As such, these tests are only recommended to be run on test machines.
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
| |
Adds a test to ensure our timeout control is working properly.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commits adds a `StageTwoReportTest` from which tests that need to
do "light" mocking should be defined. As the name implies, this is the
base class for our Stage 2 tests.
"Light" mocking for our purposes is currently defined as dropping files
in locations on the test system's filesystem, which may include
replacing binaries, and installing packages for the durtation of the
test case. At the end of the test, packages are uninstalled and mocked
files are removed. In the event that a file already exists at a given
location on the test system, that file is temporarily renamed with a
'.sostesting' extension. On test clean up, once mocked files are
removed, any files renamed previously are moved back to their original
location/name.
Mocking is controlled automatically during test setup, and is based upon
test class attributes, similar to how plugin enablement is dependent
upon plugin class attributes. For Stage 2 tests cases, this includes:
`files` - a list of file names. These names should represent the
"real" location on the filesystem where the mocked
files will be placed. The mocked content should be
placed under a similar file path under tests/test_data
E.G. using `files = ['/etc/foo/bar']` will place a
copy of `tests/test_data/etc/foo/bar` under
`/etc/foo/bar` on the test system.
`packages` - a dict of packages to install. The dict keys map to
the distribution the package names are used for,
e.g. `rhel` or `ubuntu`. The values are lists of
package names, optionally with version included.
This commit includes just the framework for these tests, and no actual
Stage 2 tests are included as of yet.
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
| |
Updates the filepaths in our unittests for the temp data files to
reflect their new location in the updated directory layout.
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit represents the start of an overhaul of the test suite used
by sos. Note that several more commits to follow will be required in
order for the test suite to be considered stable.
The new test suite will use the avocado-framework to build out new
tests.
This first part adopts a new 'stageX' naming scheme for our tests as
follows:
stage0 -> Unittests
stage1 -> Basic function tests, no mocking allowed
stage2 -> Mocked tests for specific scenarios/regressions
stage3 -> Complex setups for layered products/environments
At the moment, these unittests are not updated for avocado, though most
should still work with `nosetest` directly.
A new set of base classes is defined in tests/sos_tests.py which provide
the foundation for actual tests cases. This approach entails new test
cases subclassing a base class, such as the new `StageOneReportTest`,
and setting the `sos_cmd` class attr to the _options_ for an sos report
run. By default `sos report --batch` will be run, and targeted to the
test job's directory as a tmpdir.
Each sos command will be executed once, and all test_* methods within a
test case that subclasses `StageOneReportTest` will be checked against
the output of that execution. Note that this diverges from avocado's
typical approach where each test_* method is performed against a brand
new instance of the class (thus meaning any setup including our sos
report run would normally be run fresh). However, after speaking with
the avocado devel team, this is still seen as a valid pattern for the
framework.
The current organizational approach is to separate the tests by
component rather than stage. For example. `tests/report_tests/` should
hold any report-centric tests, and the `plugin_tests` directory therein
should be used for plugin-specific tests. As of this commit, there are
basic functionality tests under `tests/report_tests/` and a single
plugin test under `tests/report_tests/plugin_tests/` to act as a POC.
Further, there is a `tests/vendor_tests/` directory for organizing
vendor-specific bug/feature tests that are not covered by the generic
project-wide tests. A POC test from RHBZ1928628 is available with this
commit.
Note that in order for these tests to be run properly _without_
installing the current branch to the local system, you will need to run
the following command:
`PYTHONPATH=tests/ avocado run -t stageone tests/`
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
/var/tmp/sosreport_test/ instead of
/var/tmp/sosreport/_test/
Resolves: #2457
Signed-off-by: Eric Desrochers <eric.desrochers@canonical.com>
Signed-off-by: Bryan Quigley <code@bryanquigley.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It was discovered that if a report was generated without the host plugin
being run, then the hostname parser would not obfuscate the localhost
hostname or domain at all. This was due to the fact that while the
default_mapping values were being read into the parser's map's dataset
correctly, they weren't being loaded into the 'intermediary' dicts that
the parser uses to separate short names from domain names.
Fix this by reading the contents of the map's dataset dict (which is
populated by the map file) into those intermediary dicts that the parser
uses.
Additionally, safeguard against overloading domain and host obfuscation
numbers when loading this way by setting the respective counts to the
next-highest number after the map loading is complete.
Resolves: #2406
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Move cleaner to ignore both sha256/md5s
Move collector to use sha256 as well.
Move to get hash from stdout
Resolves: #2351
Closes: #1204
Signed-off-by: Bryan Quigley <code@bryanquigley.com>
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit alters several option long-form names or destination names
to align those values in a sensible way. This serves to not only remove
some abiguity in option naming in code, but also to make it so that the
"effective options" line logged in every sos execution can be direction
copy-pasted as a working command.
Closes: #2288
Resolves: #2398
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds policy-controlled forbidden path checking, which
should be the final part of implementing "global" forbidden paths. With
this commit, policies may now add paths and glob matches for paths which
should never be collected in any plugin.
Combined with plugin-defined paths and user-defined paths already
available, plugins should now be able to be properly restricted from
sensitive collections.
Note that the way this is implemented is that policies that define the
`set_forbidden_paths()` classmethod *extend* this forbidden list as it
is built from the subclass(es) that also define one. This way,
"top-level" policies do not need to maintain independent copies of
entire trees of paths just to add a few specific additional ones that
are not forbidden within other policies.
This initial commit adds paths that are either very well-known to be
ones we should avoid, or are paths that have previously been part of
reported issues where these paths/files should not be collected.
Closes: #316
Closes: #796
Closes: #919
Closes: #1316
Resolves: #2360
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Moves `PackageManager` out from `sos/policies/__init__.py` into a new
`sos/policies/package_managers` subdir.
Future commits will aim to canonicalize package manager subclasses for
policies to use, and ease the creation of new reusable package managers.
Related: #2349
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
| |
Moves the actual policy files for supported distributions out of
`sos/policies` and into a new subdir `sos/policies/distros`. Note that
`Policy()` still lives in the former, while `LinuxPolicy()` has been
moved into `sos/policies/distros/__init__.py`
Related: #2349
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a test to `simple.sh` to check if we loaded any new kernel modules
during a normal exeuction. This test will be somewhat limited in the
fact that this currently only tests the plugins that are loaded by
default on each tested distribution.
Also included is a minor typo fix for reporting where IP addresses were
not obfuscated, if any. This typo did not affect the actual test
however.
Resolves: #2326
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch overhauls tests/simple.sh to provide more granular details on
why a test run may have failed, beyond checking for a non-zero exit code
or if output was written to stderr.
This should also serve as another step towards more easily extendible
tests for our automated processes.
Resolves: #2305
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds two new options, `--skip-commands` and `--skip-files`, that allow
users to selectively skip specific command or file collection instead of
having to disable whole plugins to skip those collections.
These options are also exposed via `sos collect`, being gated by a
version of 4.1 since that is the next scheduled release where we can
guarantee this functionality will be present.
Closes: #2203
Resolves: #2271
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
| |
Signed-off-by: Bryan Quigley <code@bryanquigley.com>
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
| |
Signed-off-by: Bryan Quigley <code@bryanquigley.com>
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
| |
Updates the check for sos.conf with the new location
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
| |
Adds a new parser and map to allow user defined keyword obfuscation.
Users may now use the `--keywords` option to have `SoSCleaner` scan
lines for matching keywords, and replace them in place like we do for
other parsers.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a hostname parser to `sos clean` that will attempt to obfuscate
FQDNs matching the hostname of the system that generated the sosreport,
as found in sos_commands/host/hostname.
Additionally, any domains added via the `--domains` option will also be
obfuscated, including any subdomains of the domain(s) specified by the
option.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
| |
Adds tests for the mappings and parsers used by SoSCleaner
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This just leaves xz with gzip as the automatic backup.
Bzip2 took the most time by far and only does marginally
better than gzip.
time 27 Size 10.72MiB --batch -z xz --log-size 1
time 27 Size 16.32MiB --batch -z gzip --log-size 1
time 40 Size 14.17MiB --batch -z bzip2 --log-size 1
time 43 Size 105.21MiB --batch -z xz --all-logs
time 64 Size 187.38MiB --batch -z gzip --all-logs
time 135 Size 182.86MiB --batch -z bzip2 --all-logs
Resolves: #2053
Signed-off-by: Bryan Quigley <bryan.quigley@canonical.com>
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Timing - see how long each test takes
Summary - output the key results at the end
Use /dev/shm to store temporary files
Make since command actually meaningful
Resolves: #2039
Signed-off-by: Bryan Quigley <bryan.quigley@canonical.com>
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
Converts all current plugin docstrings into a `short_desc` attribute,
that is now referenced by `Plugin.get_description()`.
Closes: #1960
Resolves: #2036
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
Fixes a unittest failure due to an attempt to pass print() a `file`
keyword for Red Hat, which is no longer accepted in python3
Additionally fixes 2 string related errors in the test suite since
changing from python2 to python3.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
| |
Resolves: #2006
Signed-off-by: Bryan Quigley <bryan.quigley@canonical.com>
|
|
|
|
|
|
|
| |
Updates the test script `simple.sh` used by Travis to use the new `sos`
binary with the `report` subcommand.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
| |
Moves the HTML/JSON reporting bits under sos/report.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
| |
Updates the uses of 'from sos.plugins' to 'from sos.report.plugins' in
order to fix imports across the project with the new tree organization.
Additionally, the legacy `sosreport` binary now injects the user's
current working directory into the path the python interpreter uses in
order to allow local execution from a git checkout.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a new mechanism for iterating over a list of devices with a given
command or list of commands. This is to allow us to stop doing ad-hoc
device enumeration in multiple plugins and instead provide a consistent
behavior for iterating over devices such as disks.
Before setting up plugins, sos will now enumerate a list of block
devices and fibre devices that is made available to the plugins. From
there the new 'add_blockdev_cmd()' method may be passed a command or
list of commands that will be iterated over for every device. Commands
passed need to include '%(dev)s' in the appropriate place for device
name substitution. By default this will iterate over all discovered
block devices. The 'devices' parameter may be used to change this,
either by passing 'fibre' to instead iterate over fibre devices, or by
passing a list of devices to use directly.
To filter out or restrict commands to running on specific types of
devices, the blacklist and whitelist parameters can be used respectively
and can accept either a single regex string or a list of regexes.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a `ContainerRuntime()` class to allow policies to specify a
container runtime to allow plugins to utilize.
The `ContainerRuntime` is intended to allow for the discovery of
containers, including specific ones by name, and for the execution of
commands inside those containers.
This is meant to remove the overhead of manually defining ways to
determine an active runtime and if a component is containerized within
plugins.
Related: #1866
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
| |
We no longer have to workaround an odd LXD/Travis interaction
so make simple.sh not treat Travis special.
Closes: #1921
Resolves: #1983
Signed-off-by: Bryan Quigley <bryan.quigley@canonical.com>
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
| |
Resolves: #1958
Signed-off-by: Bryn M. Reeves <bmr@redhat.com>
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Various unit test source files in tests/ have unnecessary shell
"shebang" notation specifying a particular Python interpereter.
Since these files are executed under the test suite runner there
is no need for them to include these lines.
Related: #1949
Signed-off-by: Bryn M. Reeves <bmr@redhat.com>
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
| |
Related: #1949
Signed-off-by: Bryn M. Reeves <bmr@redhat.com>
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
| |
Related: #1949
Signed-off-by: Bryn M. Reeves <bmr@redhat.com>
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
| |
Use a byte string instead of a regular string.
Related: #1949
Signed-off-by: Bryn M. Reeves <bmr@redhat.com>
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
Use the executable 'true' when testing is_executable() on a
non-absolute path.
Related: #1949
Signed-off-by: Bryn M. Reeves <bmr@redhat.com>
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Commit 285873a introduces a regression in the performance of
LinuxProfile class initialisation:
commit 285873a4f753822a88d475a1b030ab622bf4c72e
Author: Bryan Quigley <bryan.quigley@canonical.com>
Date: Tue Feb 11 15:03:16 2020 -0800
[policies] Detect systemd use instead of hardcoding it
All the patch does is to switch from testing self.init to
determine whether to use the SystemdInit() class or the
basic InitSystem() one, to looking for '/run/systemd/system'
in the file system.
This has more broad side effects than it might seem since
the test suite uses LinuxPolicy() as a mock policy object
for a large number of tests. Since SystemdInit() calls out
to systemctl to obtain the state of the init system this
both increases the run time for the tests and causes high
resource use in systemd itself:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 177196 13504 4512 R 92.7 0.1 5:24.04 systemd
This causes the run time for the Python2 and 3 test suite
(plus pep8/pycodestyle) to grow from ~5s to over 1m20s on
my test system:
285873a~1 real 0m5.683s
285873a real 1m20.353s
Allow direct users of LinuxPolicy to override the init
system detection by specifying an init= kwarg initialised
to an InitSystem-like object directly, and use this in the
test suite to avoid the cost of initialising the full
SystemdInit() class.
Resolves: #1953
Signed-off-by: Bryn M. Reeves <bmr@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Move all the sosreport runs to a simple bash script:
tests/simple.sh so it can be called multiple times.
This also makes it possible to run locally.
The travis job matrix let's us have 2 Ubuntu native sos
runs against installed python, while letting us to also
continue to do the old style python runs. Ideally,
we can utilize this to do Fedora/CentOS/RHEL jobs in the
future.
Python 3.4 testing was also dropped - as it seems to not
be as well used (many test failures due to not being available).
Python 3.8 testing was added.
pycodestyle will only run once with the bionic version.
Resolves: #1896
Signed-off-by: Bryan Quigley <bryan.quigley@canonical.com>
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds two mechanisms by which users can choose to disable postprocessing
of collected information.
First, is a global method exposed via the `--no-postproc` option. Using
this option will skip postprocessing for all plugins.
Second, is a per-plugin option exposed via a new 'postproc' plugin
option. This is set to _True_ by default (meaning yes, perform
postprocessing), which users can set to False or off to disable
postprocessing for that plugin only; e.g. `-k podman.postproc=off`
Closes: #286
Resolves: #1862
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
Rather than call just Plugin.setup() and Plugin._do_copy_path(),
add an add_copy_spec() call to the mock plugin setup() method,
and invoke copying by calling the Plugin.collect() method.
Related: #1845
Signed-off-by: Bryn M. Reeves <bmr@redhat.com>
|