| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Hostname parser treats strings like 'host.domain.com' with precompiled
domain 'domain.com' in a wrong way. It first obfuscates the domain while
subsequent _parse_line skips host obfuscation.
Calling _parse_line before _parse_line_with_compiled_regexes does clean
both the host name and the domain name well.
Adding a unittest with a reproducer.
Resolves: #3054
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
After all plugins have run, sos will now generate a "tag summary" and
add it to the report section of the manifest. This summary will be a
json-formatted entry that details all collections within the report that
have any tag associated with them at all. This should allow for easier
parsing of the manifest by analyzers such as Red Hat Insights.
As part of this change, commands will no longer be automatically tagged
with the name of the binary used in the command collection.
Additionally, manual collections performed by a plugin's `collect()`
method will now be recorded in the manifest in the same manner as file
and command output collections.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
It was found that the `krb5` plugin was not executing on Ubuntu/Debian
systems due to a legacy change that inadvertently removed support for
those distributions.
Re-enable support by defining a new plugin class for those distros,
setting the kdc directory appropriately. Additionally, expand the
package tuple to include newer package names.
Closes: #3041
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Adds a new manual collection to the `process` plugin, that tries to
compile a mapping of running processes' binaries to an owning package
via the package manager. As such, package managers now have a new
`pkg_by_path()` method that serves this purpose.
Closes: #1350
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Historically, sos has limited collections to existing files and command
output. While there have been many reasons for this, there have also
been several exceptions made to current plugins as well as ongoing
requests for data that is not currently provided via existing commands.
As sos evolves, it should in turn be more capable to provide diagnostic
data beyond what is strictly available via command outputs.
As such, add a new step to the collection phase that allows plugins to
perform these manual data collections. Plugins may now define their own
`collect()` method to do so, thus moving the existing exceptions out of
`setup()` phase execution.
To aide in writing these collections to the plugin directory, a new
`collection_file()` generator has been added which will handle creating,
managing, and closing the new file so that plugins only need to be
concerned with the content being written to such files.
Plugin contributors should note that these manual collections are
executed at the end of the collection phase - meaning they are more
likely to be skipped or interrupted due to plugin timeouts.
Closes: #2992
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Foreman has since made several releases since we initially added
integration testing for it. Rebuilding the images is maintenance
overhead we can reduce by automating the installation of foreman on
images.
Provide this scripted installation, and expand our testing matrix to
additional versions of Foreman. As of this commit, we will be testing
versions 2.5 and 3.1 on CentOS Stream 8 to cover Red Hat Satellite
features, as well as Foreman version 3.4 (the latest current upstream)
for CentOS 8 and Debian 11.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A scenario was found that if a file had encoding issues when `sos clean`
went to open the file for obfuscation, we would stop processing the file
but leave it in the archive, which had the potential to leave
unobfuscated information in that file in the archive.
Fix this, by using the `errors='replace'` parameter when opening archive
files. This allows us to continue parsing the file normally, while
replacing the problematic characters with `?`s.
Closes: #3015
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
- test for #3022 .
- fixup of opt_parser being a tuple with a string
Relates: #3022
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
| |
|
|
|
|
|
|
| |
The Fedora 36 image is now available on GCP, so update our testing
matrix to use it. Accordingly, this means we no longer need to build or
maintain our own Fedora images going forward.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Up until now, the test suite was forced to be run using the git
checkout. While this is useful for on-the-fly testing, it does miss an
important use case of building a test package from the checkout, and
running it using that as the system installation of sos.
This commit allows the use of an installed version of sos to test
against. This may be leveraged by adding `-p TESTLOCAL=true` in the
`avocado run` command used to launch the test suite. Setting this
parameter to any other value, or omitting it entirely, will continue the
current behavior or using the git checkout for running tests.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a new test case for the `teamd` plugin that also inherently acts to
test team device enumeration.
Included with this test case is an addition to the base test classes
that allows for tests to define a `post_test_tear_down()` method that
will be run at the end of each test execution to allow for manual
cleanup - in this case deleting a 'fake' team device created for the
test.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
| |
Adds a new test to the networking plugin test to ensure we iterate
correctly over network devices.
|
| |
|
|
|
|
|
|
|
|
|
| |
For files that can be considered obvious version files - those that end
specifically with either `version` or `version.txt` - skip processing
via the IP parser, as this may lead to improper obfuscation of certain
version strings. This is also applied to files ending in `release`.
Closes: #2962
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
Removes catalog entries from the journal collection for `this` and
`last` boot collections.
Closes: #2132
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For our CI tests, CPU resources are not guaranteed which in turn can
cause longer-than-expected run times for test executions. In practice,
this is only seen occasionally but it requires manual intervention when
the timeouts are hit (and thus far it has been seen that the changes
these timeouts are hit on are not actually causing the timeouts).
Previous conversations have revolved around improving test efficiency,
however this seems unlikely given the nature of some of the test setup
and further, even with the most efficienct approach possible we would
still be at the whim of resource availability.
As such, increase the default timeout to account for this resource
consideration.
Closes: #2700
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As highlighted in #1817, `pkg_by_name()` could provide unpredictable
results, when using wildcards. As such, limited this method to only
returning package info for exact package name matches. In turn, change
`Plugin.is_installed()` to leverage `PackageManager.all_pkgs_by_name()`
which does explicitly support wildcards and returns information on _all_
matching packages, not just the last one found.
In so doing, clean up the `PackageManager` design to use a new
`packages` property for these lookups, and update the former usage of
`all_pkgs()` accordingly. Similarly, signal `get_pkg_list()` should be
private (in any sense that a python method can be) by renaming to
`_get_pkg_list()` and update the single Plugin (`etcd`) referencing this
method.
Closes: #1817
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Adds a new `--disable-parsers` option that allows users to selectively
disable parsers for a given execution of `sos clean`.
This may be useful in specific scenarios where obfuscation is not
strictly needed for all the types of data we obfuscate, and where the
user trusts whomever may be receiving the archive for review.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
When a file collection will reach its sizelimit, don't tail the file if
it has binary content, as this will be useless collection.
Closes: #2851
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Adds the ability for plugins to define a default set of environment vars
to pass to all commands executed by the plugin. This may be done either
via the new `set_default_cmd_environment()` or
`add_default_cmd_environment()` methods. The former will override any
previously set values, whereas the latter will add/update/modify any
existing values.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
First, update the regexes to account for possible quotes wrapping the
mac address to match.
Second, fix an edge case with these quoted mac addresses in our check
for avoiding duplicating obfuscations of already obfuscated addresses by
checking the stripped mac address instead of the raw one.
Closes: #2873
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There was a gap in our testing that allows #2863 to escape our notice -
that a `Plugin()`'s `plugin_timeout` attribute would be ignored if it
wasn't set to `TIMEOUT_DEFAULT`.
As that was resolved by #2864, add a test to ensure it remains working
as expected. The expected resolution order for a plugin's whole timeout
is as follows:
1. The value set by `-k plugin.timeout`
2. The value set by `--plugin-timeout`
3. The value hardcoded in the plugin via the `plugin_timeout` attr
4. `TIMEOUT_DEFAULT`
Related: #2863
Related: #2864
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For parsers such as the username and keyword parsers, we don't discover
new items through parsing archives - these parsers use static lists
determined before we begin the actual obfuscation process.
As such, we can build a list of regexes for these static items once, and
then reference those regexes during execution, rather than rebuilding
the regex for each of these items for every obfuscation.
For use cases where hundreds of items, e.g. hundreds of usernames, are
being obfuscated this results in a significant performance increase.
Individual per-file gains are minor - fractions of a second - however
these gains build up over the course of the hundreds to thousands of
files a typical archive can be expected to contain.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
In the event a CI test execution fails before the archive is created,
for any reason, we want to preserve the temp directory as the logs there
are still useful.
As such, add a specific env var to our test runs via avocado. Then, when
we detect a failure check for that env var to determine if we should in
fact do our cleanup or not.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
| |
It was discovered that our extra handling for shortnames was
unintentionally case sensitive. Fix this to ensure that shortnames are
obfuscated regardless of case in all collected text.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It was discovered that setting a specific plugin timeout via the `-k
$plugin.timeout` option could influence the timeout setting for other
plugins that are not also having their timeout explicitly set. Fix this
by moving the default plugin opts into `Plugin.__init__()` so that each
plugin is ensured a private copy of these default plugin options.
Additionally, add more timeout data to plugin manifest entries to allow
for better tracking of this setting.
Adds a test case for this scenario.
Closes: #2744
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
As an interim stopgap measure, increase the timeout for the stagetwo
`logs` test to allow for more time for handling random data generation
and logging, until we're able to define a better/more efficient way to
generate this data within the test suite.
Related: #2700
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
The debug level messages gated by `-v` are very helpful for diagnosing
test failures, but currently not all tests specify the use of verbosity.
Make use of verobsity a default parameter for all test runs to address
this.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`nose` is no longer maintained, and as of python-3.10 is functionally
broken. As such, instead transition to running those tests via avocado,
like we do with our integration test suite.
The tests themselves do not need much modification, however due to the
isolation provided for executing the tests we do need to explicitly set
a new PYTHONPATH env var for those executions. This means we still need
to run the unit tests as a separate step from the stageone tests.
The changes needed are mostly around file paths relative to the pwd
where the tests are executed from originally.
Additionally, remove the sosreport_pexpect unit test as it is no longer
useful in its own right, would need more significant changes to run
properly with avocado, and the integration test suite provides better
coverage for what it was testing.
Closes: #2716
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit allows plugins that call `add_string_as_file` to specify if
the string should be written to `sos_strings/$name/` (current behavior)
or if it should be written to `sos_commands/$name/` which may be
desireable for organizational purposes for plugin collections.
`add_string_as_file()` has also been updated to write to a plugin's
manifest section for any files written this way. Accordingly, the method
will now accept a `tags` parameter to add specified tags to the manifest
entry.
Certain plugins directly calling this method have been updated, but the
existing logic to write truncated data to `sos_strings/` remains
untouched, and will not generate manifest entries (as those should
already be handled by the method that trigged the truncated collection).
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit removes the restriction imposed on `sos clean` since its
introduction in sos-4.0 to only work against known sos report archives
or build directories. This is because there has been interest in using
the obfuscation bits of sos in other data-collector projects.
The `SoSObfuscationArchive()` class has been revamped to now be an
abstraction for different types of archives, and the cleaner logic has
been updated to leverage this new abstraction rather than assuming we're
working on an sos archive.
Abstractions are added for our own native use cases - that being `sos
report` and `sos collect` for at-runtime obfuscation, as well as
standalone archives previously generated. Further generic abstractions
are available for plain directories and tarballs however these will not
provide the same level of coverage as fully supported archive types, as
is noted in the manpage for sos-clean.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Adds failure handling to cirrus tasks that actually run sos so that if
an error is encountered in the test suite, the logs from the tests are
uploaded to the GCE cloud storage bucket associated with the GCE sos
project.
This will also make those logs available from the cirrus task page.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
| |
Adds a new report test that ensures that options set in `sos.conf` are
picked up, handled properly, and logged as we expect them to be.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Up until now the `files` attr for Stage Two tests was simply a list that
dropped files of the same name from our `tests/test_data/` directory
into the relevant paths on the test system. This however limited us to
single copies of files within this directory.
Instead, also allow `files` to take a list of tuples in the form of
`(source_path, dest_path)` so that we may have multiple copies of say
`sos.conf` within `tests/test_data` for different test cases while still
being able to temporarily override the test system's actual
`/etc/sos/sos.conf`.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Up until now plugin options were defined via tuples, with positional
significance of the tuple elements. This made plugin option creation
fairly easy, but option handling could easily become confusing.
Instead, create a new `PluginOpt` class that plugin options from here on
out will need to build from in order to function. These will still be
applied to plugins by inserting them into the `option_list` class attr
in order to retain an easy way to expand plugin options for authors.
Internally, options are now assigned to a dict which is then directly
accessed for plugin option manipulations. PluginOpt default values are
retained separate from their current value, and elements are assigned
directly to meaningful identifiers within the class. This should
alleviate some of the overhead when handling plugin options within sos.
Not all current tuple elements have been carried over into the new
`PluginOpt` class - for example, the 'speed' attribute has been dropped
as it does not have a current function. In the planned `sos info`
component, the time effects of a plugin option should be documented in
the `long_desc` attribute instead.
Additionally, the `Plugin.get_option_as_list()` method has been removed
as it was not being used anywhere.
Note that this particular commit only introduces the new class, and the
loading options used by `SoSReport()`. As such, plugins using options
currently will report errors during test runs. A commit following this
one will shuffle existing plugin options into the new class structure
and allow the plugins to execute normally.
Resolves: #274
Resolves: #452
Resolves: #1597
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit follows up on the previous changes from #2555 by fully
dropping the `--master` option in favor of `--primary`, `--manager`, and
`--controller`.
Internal references have been updated as well, using the term `primary`
across all instances. Note that within OCP cluster profile, 'primary'
labeling and option setting still relies on the 'master' label/role
reported by the environment, as that remains the term used to identify
such nodes.
Resolves: #2329
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When a `report` or `collect` run would use `--clean` or `--mask` to do
in-line obfuscation of collected reports, sos would not read the config
section for clean - it would only be read if `sos clean` was called
directly. As such, users would need to manually specify config file
values for each run.
Alleviate this gap by reading the config section for `clean` if either
of the cleaner options are used. Do this before we apply cmdline options
so that we maintain our order of precedence.
Related: RHBZ#1950350
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously new test cases had to be written in such a way that they
could be run on any supported distribution, even if their individual
tests were all marked for specific distributions. Use of the distro-only
decorators would fail when applied to test cases as a whole due to how
test instantiation errors are handled.
To address this, provide a distro check within `BaseSoSTest` so that
tests can be made distro-specific and we can signal to avocado to skip
all tests within that test case without failing the test suite entirely.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a test that ensures we are properly extracting and cleaning an
already existing archive (one is being included under tests/test_data/
now), much the same as we test for a proper `sos report --clean` run.
As part of this, some aspects of the `full_report_run.py` test are moved
into the base test classes, and the new `existing_archive.py` test will
explicitly run `sos clean` instead of `sos report`.
The included archive for obfuscation testing is taken from a stock
Fedora 34 VM with a known configuration which is reflected in the items
being tested for.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a global `--namespaces` option that can be used to limit the
number of namespaces all plugins will iterate over. If a plugin provides
a specific plugin option, such as the `networking.namespaces` option,
then if set that plugin option will override the global option value.
The global option defaults to not limiting namespaces, matching current
behavior.
Closes: #2092
Resolves: #2547
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
Instead of re-reading the mapping file for each parser, read it once and
store the contents, then hand those contents over to each parser.
This allows us to side-step handling the same exception for malformed
config files over and over for each parser loaded.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Avocado will, by default, use the "recursive" behavior when looking
for tests. It means that the class hierarchy will be crawled
recursively, until, for "avocado-instrumented" tests, the top-most
parent "avocado.Test" is found.
When the "enable" behavior is activated, it forces the class to be
considered one containing avocado-instrumented tests, but, it disables
the recursive behavior and only the tests local to that specific class
are found.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Add tests checking independence of foreman_* plugins on the
"main" foreman plugin.
Add the two plugins to the list of expectedly enabled on a tfm instance.
Relevant: #2546
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds wrapper methods to `sos.utilities` that allows the common `os`
methods, such as `os.path.exists()` to account for the setting of
`sysroot`. This will allow sos, and particularly plugins, to more
easily and reliably function as expected when sos is run within a
container.
Helpers have been added directly to `Plugin` that automatically pass the
set `sysroot` option, so plugin authors do not need to handle that
option or directly import these path functions from `sos.utilities`.
Closes: #494
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
| |
If we get an exception we aren't expecting and can't investigate due to
a missing `CmdResult` object being attached to the raised exception,
simply re-raise that exception rather than failing on trying to inspect
a non-existing object.
|
| |
|
|
|
|
|
|
|
|
|
| |
Adds a new test case that captures a full, unrestricted, sos report and
runs it through `sos clean` to enhance our confidence in `clean`
performance.
This is a StageTwo test as it will temporarily overwrite any existing
default_mapping.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
| |
Allows test cases to set their own timeout for sos command executions,
rather than applying a 300 second timeout to all test cases. 300 seconds
remains the default, however.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
| |
Moves the report_with_mask tests to be under the cleaner_tests/
directory. No changes are actually being made to the tests, this is an
organization change.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
In the past, there have been otherwise trivial typos and the like that
have caused `--help` output to be unreliable. In the case of "help
options" such as `--list-plugins` this also includes potentially
unavailable detailed information about whatever is being listed.
These tests are sanity checks to ensure that changes aren't regressing
this behavior and that the informational output options, and `--help`
directly, continue to provide the right output.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Moves several aspects of test setup and design from `BaseSoSReportTest`
into the lower-level `BaseSoSTest`, so that adding new test classes that
are not report-centric will be easier going forward. Where needed,
overrides in `BaseSoSReportTest` have been added.
In general, this means that sos command construction, execution, tmpdir
management, and the methods for mocking and other pre-execution tasks
are now in `BaseSoSTest` and the additions in `BaseSoSReportTest` are
now centered around handling a generated archive.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Adds an enablement trigger tuple, `containers`, that allows for enabling
a plugin based on the presence of a _running_ container matching a given
name or regex. Note that this is only functional for the default runtime
discovered by a policy during initialization.
Resolves: #2561
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
