| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| ... | |
| |
|
|
|
|
|
|
|
|
|
| |
Adds the ability for plugins to define a default set of environment vars
to pass to all commands executed by the plugin. This may be done either
via the new `set_default_cmd_environment()` or
`add_default_cmd_environment()` methods. The former will override any
previously set values, whereas the latter will add/update/modify any
existing values.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
First, update the regexes to account for possible quotes wrapping the
mac address to match.
Second, fix an edge case with these quoted mac addresses in our check
for avoiding duplicating obfuscations of already obfuscated addresses by
checking the stripped mac address instead of the raw one.
Closes: #2873
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There was a gap in our testing that allows #2863 to escape our notice -
that a `Plugin()`'s `plugin_timeout` attribute would be ignored if it
wasn't set to `TIMEOUT_DEFAULT`.
As that was resolved by #2864, add a test to ensure it remains working
as expected. The expected resolution order for a plugin's whole timeout
is as follows:
1. The value set by `-k plugin.timeout`
2. The value set by `--plugin-timeout`
3. The value hardcoded in the plugin via the `plugin_timeout` attr
4. `TIMEOUT_DEFAULT`
Related: #2863
Related: #2864
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For parsers such as the username and keyword parsers, we don't discover
new items through parsing archives - these parsers use static lists
determined before we begin the actual obfuscation process.
As such, we can build a list of regexes for these static items once, and
then reference those regexes during execution, rather than rebuilding
the regex for each of these items for every obfuscation.
For use cases where hundreds of items, e.g. hundreds of usernames, are
being obfuscated this results in a significant performance increase.
Individual per-file gains are minor - fractions of a second - however
these gains build up over the course of the hundreds to thousands of
files a typical archive can be expected to contain.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
In the event a CI test execution fails before the archive is created,
for any reason, we want to preserve the temp directory as the logs there
are still useful.
As such, add a specific env var to our test runs via avocado. Then, when
we detect a failure check for that env var to determine if we should in
fact do our cleanup or not.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
| |
It was discovered that our extra handling for shortnames was
unintentionally case sensitive. Fix this to ensure that shortnames are
obfuscated regardless of case in all collected text.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It was discovered that setting a specific plugin timeout via the `-k
$plugin.timeout` option could influence the timeout setting for other
plugins that are not also having their timeout explicitly set. Fix this
by moving the default plugin opts into `Plugin.__init__()` so that each
plugin is ensured a private copy of these default plugin options.
Additionally, add more timeout data to plugin manifest entries to allow
for better tracking of this setting.
Adds a test case for this scenario.
Closes: #2744
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
As an interim stopgap measure, increase the timeout for the stagetwo
`logs` test to allow for more time for handling random data generation
and logging, until we're able to define a better/more efficient way to
generate this data within the test suite.
Related: #2700
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
The debug level messages gated by `-v` are very helpful for diagnosing
test failures, but currently not all tests specify the use of verbosity.
Make use of verobsity a default parameter for all test runs to address
this.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`nose` is no longer maintained, and as of python-3.10 is functionally
broken. As such, instead transition to running those tests via avocado,
like we do with our integration test suite.
The tests themselves do not need much modification, however due to the
isolation provided for executing the tests we do need to explicitly set
a new PYTHONPATH env var for those executions. This means we still need
to run the unit tests as a separate step from the stageone tests.
The changes needed are mostly around file paths relative to the pwd
where the tests are executed from originally.
Additionally, remove the sosreport_pexpect unit test as it is no longer
useful in its own right, would need more significant changes to run
properly with avocado, and the integration test suite provides better
coverage for what it was testing.
Closes: #2716
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit allows plugins that call `add_string_as_file` to specify if
the string should be written to `sos_strings/$name/` (current behavior)
or if it should be written to `sos_commands/$name/` which may be
desireable for organizational purposes for plugin collections.
`add_string_as_file()` has also been updated to write to a plugin's
manifest section for any files written this way. Accordingly, the method
will now accept a `tags` parameter to add specified tags to the manifest
entry.
Certain plugins directly calling this method have been updated, but the
existing logic to write truncated data to `sos_strings/` remains
untouched, and will not generate manifest entries (as those should
already be handled by the method that trigged the truncated collection).
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit removes the restriction imposed on `sos clean` since its
introduction in sos-4.0 to only work against known sos report archives
or build directories. This is because there has been interest in using
the obfuscation bits of sos in other data-collector projects.
The `SoSObfuscationArchive()` class has been revamped to now be an
abstraction for different types of archives, and the cleaner logic has
been updated to leverage this new abstraction rather than assuming we're
working on an sos archive.
Abstractions are added for our own native use cases - that being `sos
report` and `sos collect` for at-runtime obfuscation, as well as
standalone archives previously generated. Further generic abstractions
are available for plain directories and tarballs however these will not
provide the same level of coverage as fully supported archive types, as
is noted in the manpage for sos-clean.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Adds failure handling to cirrus tasks that actually run sos so that if
an error is encountered in the test suite, the logs from the tests are
uploaded to the GCE cloud storage bucket associated with the GCE sos
project.
This will also make those logs available from the cirrus task page.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
| |
Adds a new report test that ensures that options set in `sos.conf` are
picked up, handled properly, and logged as we expect them to be.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Up until now the `files` attr for Stage Two tests was simply a list that
dropped files of the same name from our `tests/test_data/` directory
into the relevant paths on the test system. This however limited us to
single copies of files within this directory.
Instead, also allow `files` to take a list of tuples in the form of
`(source_path, dest_path)` so that we may have multiple copies of say
`sos.conf` within `tests/test_data` for different test cases while still
being able to temporarily override the test system's actual
`/etc/sos/sos.conf`.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Up until now plugin options were defined via tuples, with positional
significance of the tuple elements. This made plugin option creation
fairly easy, but option handling could easily become confusing.
Instead, create a new `PluginOpt` class that plugin options from here on
out will need to build from in order to function. These will still be
applied to plugins by inserting them into the `option_list` class attr
in order to retain an easy way to expand plugin options for authors.
Internally, options are now assigned to a dict which is then directly
accessed for plugin option manipulations. PluginOpt default values are
retained separate from their current value, and elements are assigned
directly to meaningful identifiers within the class. This should
alleviate some of the overhead when handling plugin options within sos.
Not all current tuple elements have been carried over into the new
`PluginOpt` class - for example, the 'speed' attribute has been dropped
as it does not have a current function. In the planned `sos info`
component, the time effects of a plugin option should be documented in
the `long_desc` attribute instead.
Additionally, the `Plugin.get_option_as_list()` method has been removed
as it was not being used anywhere.
Note that this particular commit only introduces the new class, and the
loading options used by `SoSReport()`. As such, plugins using options
currently will report errors during test runs. A commit following this
one will shuffle existing plugin options into the new class structure
and allow the plugins to execute normally.
Resolves: #274
Resolves: #452
Resolves: #1597
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit follows up on the previous changes from #2555 by fully
dropping the `--master` option in favor of `--primary`, `--manager`, and
`--controller`.
Internal references have been updated as well, using the term `primary`
across all instances. Note that within OCP cluster profile, 'primary'
labeling and option setting still relies on the 'master' label/role
reported by the environment, as that remains the term used to identify
such nodes.
Resolves: #2329
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When a `report` or `collect` run would use `--clean` or `--mask` to do
in-line obfuscation of collected reports, sos would not read the config
section for clean - it would only be read if `sos clean` was called
directly. As such, users would need to manually specify config file
values for each run.
Alleviate this gap by reading the config section for `clean` if either
of the cleaner options are used. Do this before we apply cmdline options
so that we maintain our order of precedence.
Related: RHBZ#1950350
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously new test cases had to be written in such a way that they
could be run on any supported distribution, even if their individual
tests were all marked for specific distributions. Use of the distro-only
decorators would fail when applied to test cases as a whole due to how
test instantiation errors are handled.
To address this, provide a distro check within `BaseSoSTest` so that
tests can be made distro-specific and we can signal to avocado to skip
all tests within that test case without failing the test suite entirely.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a test that ensures we are properly extracting and cleaning an
already existing archive (one is being included under tests/test_data/
now), much the same as we test for a proper `sos report --clean` run.
As part of this, some aspects of the `full_report_run.py` test are moved
into the base test classes, and the new `existing_archive.py` test will
explicitly run `sos clean` instead of `sos report`.
The included archive for obfuscation testing is taken from a stock
Fedora 34 VM with a known configuration which is reflected in the items
being tested for.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a global `--namespaces` option that can be used to limit the
number of namespaces all plugins will iterate over. If a plugin provides
a specific plugin option, such as the `networking.namespaces` option,
then if set that plugin option will override the global option value.
The global option defaults to not limiting namespaces, matching current
behavior.
Closes: #2092
Resolves: #2547
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
Instead of re-reading the mapping file for each parser, read it once and
store the contents, then hand those contents over to each parser.
This allows us to side-step handling the same exception for malformed
config files over and over for each parser loaded.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Avocado will, by default, use the "recursive" behavior when looking
for tests. It means that the class hierarchy will be crawled
recursively, until, for "avocado-instrumented" tests, the top-most
parent "avocado.Test" is found.
When the "enable" behavior is activated, it forces the class to be
considered one containing avocado-instrumented tests, but, it disables
the recursive behavior and only the tests local to that specific class
are found.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Add tests checking independence of foreman_* plugins on the
"main" foreman plugin.
Add the two plugins to the list of expectedly enabled on a tfm instance.
Relevant: #2546
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds wrapper methods to `sos.utilities` that allows the common `os`
methods, such as `os.path.exists()` to account for the setting of
`sysroot`. This will allow sos, and particularly plugins, to more
easily and reliably function as expected when sos is run within a
container.
Helpers have been added directly to `Plugin` that automatically pass the
set `sysroot` option, so plugin authors do not need to handle that
option or directly import these path functions from `sos.utilities`.
Closes: #494
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
| |
If we get an exception we aren't expecting and can't investigate due to
a missing `CmdResult` object being attached to the raised exception,
simply re-raise that exception rather than failing on trying to inspect
a non-existing object.
|
| |
|
|
|
|
|
|
|
|
|
| |
Adds a new test case that captures a full, unrestricted, sos report and
runs it through `sos clean` to enhance our confidence in `clean`
performance.
This is a StageTwo test as it will temporarily overwrite any existing
default_mapping.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
| |
Allows test cases to set their own timeout for sos command executions,
rather than applying a 300 second timeout to all test cases. 300 seconds
remains the default, however.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
| |
Moves the report_with_mask tests to be under the cleaner_tests/
directory. No changes are actually being made to the tests, this is an
organization change.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
In the past, there have been otherwise trivial typos and the like that
have caused `--help` output to be unreliable. In the case of "help
options" such as `--list-plugins` this also includes potentially
unavailable detailed information about whatever is being listed.
These tests are sanity checks to ensure that changes aren't regressing
this behavior and that the informational output options, and `--help`
directly, continue to provide the right output.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Moves several aspects of test setup and design from `BaseSoSReportTest`
into the lower-level `BaseSoSTest`, so that adding new test classes that
are not report-centric will be easier going forward. Where needed,
overrides in `BaseSoSReportTest` have been added.
In general, this means that sos command construction, execution, tmpdir
management, and the methods for mocking and other pre-execution tasks
are now in `BaseSoSTest` and the additions in `BaseSoSReportTest` are
now centered around handling a generated archive.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Adds an enablement trigger tuple, `containers`, that allows for enabling
a plugin based on the presence of a _running_ container matching a given
name or regex. Note that this is only functional for the default runtime
discovered by a policy during initialization.
Resolves: #2561
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Binary files generally speaking cannot be obfuscated, and as such we
should remove them from archives being obfuscated by default so that
sensitive data is not mistakenly included in an obfuscated archive.
This commits adds a new `--keep-binary-files` option that if used will
keep any encountered binary files in the final archive. The default
option of `false` will ensure that encountered binary files are removed.
The number of removed binary files per archive is reported when
obfuscation is completed for that archive.
Closes: #2478
Resolves: #2524
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds automatic collection of journals for any service defined in a
plugin's `services` tuple, if that service exists on the system.
Updates several plugins to define a `services` tuple in place of
manually calling `add_journal()` (and as a bonus, removing redundant
`add_service_status()` calls that are already performed based on the
tuple).
Additionally, add an appropriate tag to each `add_journal` call for
easier collection lookup in the manifest.
Resolves: #2579
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
Adds two test cases for using `--encrypt-pass`; one with just the bare
option and another in conjunction with `--clean` to ensure both can be
used in the same execution.
Resolves: #2566
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
A supplement of #1399 fix, now also for adding strings or special
device files.
Also adding a (vendor) test case for it.
Resolves: #2560
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Improves error reporting for failed sos commands by logging stderr (or
stdout if stderr is not populated) to the console, which was previously
being truncated by the builtin error handling of avocado. Printed output
is limited to the last 8k to avoid dumping several MBs at a time for
scenarios such as timeouts where command failure may generate
significant logging prior to failing.
Included with this are 2 minor changes to existing tests. First, remove
verbose output from the expected plugins test to reduce otherwise
irrelevant output for command failures. Second limit the number of
plugins run for the LogLevelTest, both to reduce overall run time for a
test where we aren't testing specific plugins and to improve readability
of failures for such a test.
Resolves: #2556
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
Adds a test that checks that the new `priority` parameter is working as
expected for well known and common collections.
Resolves: #2553
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
Adds a unittest that ensures conformance for plugin design, e.g.
ensuring that enablement triggers are proper tuples and not
inadvertent strings due to a missing trailing comma for single-tuples.
Resolves: #2549
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Specifying a non-existing plugin to skip is no longer considered a fatal
error. This was previously in place, in part, for third party utilities
calling sos to ensure adherence to known plugins. However, since
`collect` has been integrated into sos natively, this is far less of a
concern.
Specfying non-existing plugins will now only generate a warning message
for plugins wanting to be skipped (`-n`). If non-existing plugins are
specified for enablement via `-e` or `-o`, that will still generate a
fatal error.
Closes: #537
Closes: #1723
Resolves: #2517
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds an integration test for the upstream Foreman project. After
stageone tests have passed, and if the PR has changes relevant to the
Foreman plugins (or the cirrus config), launch a pre-built GCE image
that has a Foreman deployment on it.
Per request of the Foreman devel team, these tests will be run against
both a CentOS and Debian based Foreman image. Currently, testing is
performed against Foreman 2.4.
Resolves: #2531
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
As sos is now python3-only, we can avoid shelling-out to compression
utilities like `xz` or `gzip`, and instead use the method provided by
the built-in `tarfile` module.
Resolves: #2523
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Changes the logs pre_sos_setup() to write to the journal using a stream
rather than writing several MB directly all at once. This fixes an issue
where using the `journal.send()` method from the systemd module would
result in no data being written to the journal during the test setup.
Second, switch over to inspecting the full journal rather than the most
recent boot journal.
Related: #2499
|
| |
|
|
|
|
|
|
|
|
|
| |
Like `assertFileCollected`, `assertFileGlobInArchive` should only fail
if the referenced collection is not in the archive *and* it is present
on the test system. If it is not present on the test system, sos cannot
collect it and it is thus not an error to be missing from the archive.
Related: #2499
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
First, correct the logs plugin stagetwo test to specify the needed
package in a list, and second move the import to within the test class
that needs it so that we don't fail on the stageone tests if the package
is not present.
Second, correct the journal injections that this plugin test does to use
`pre_sos_setup()` instead of overriding `setup_mocking()`, and call the
former after the mocking has been setup.
Finally, add the resulting size of the collected journal if we fail the
size requirements of the test.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a new `StageOneReportExceptionTest` class that is to be used for
tests that should ensure an exception IS hit, for example when testing
plugin enablement or invalid options.
Included with this commit is a small set of plugin enablement tests.
Future tests for component and plugin specific exception handling should
be built using this class and design approach.
Resolves: #2518
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When command output is size limited it should be made obvious that it is
not the whole output of the command, beyond the collected output (most
likely) starting in the middle of a string.
To do this, we add a 'truncated' key to the results dict returned by
`sos_get_command_output()`, which is set based on if the `AsyncReader`
was filled during command execution.
From that, first log if a command was truncated. Second, instead of
dropping the collected output in the normal plugin directory, instead
drop it into `sos_strings` with a `.tailed` extention to mimic how we
handled tailed file collections. Finally, add a symlink from the plugin
dir with the original filename that points to the `sos_strings` path.
Closes: #1332
Resolves: #2506
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Standardizes the `assertOutputContains` and `assertOutputNotContains`
methods to use `re.search()`, instead of separate parsing methods.
`search()` replaces `match()` for efficiency purposes against the
collected output.
Related: #2507
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes an issue where archive debug logging was controlled by the use of
`--debug` rather than `--verbose`. Removes a superfluous log in
`FileCacheArchive.add_link()`.
Also, adjusts the different verbosity levels to be more meaninful. Now,
the use of `-v` will enable debug logging but will not print those
messages to console, `-vv` will print debug logging to console, while
`-vvv` will enable archive debug logging which is expected to be
significant due to most file operations being logging at some point
with archive debug logging.
Resolves: #2507
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds collection of per-process files from /proc/$pid, limited to a
number of pids controlled by the new `process.numprocs` option. This
option will default to the first 2048 processes. Using a value of `0`
for this option will result in all pids being collected.
In testing, the initial limit of 2048 process did not significantly
raise collection time or archive size. Note however, that collection of
20k or more processes does show a significant increase in plugin
execution time and archive size. This is however tempered, by the
whole-plugin timeout that sos imposes.
Related: #436
Closes: #542
Resolves: #1783
Resolves: #2498
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
