| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
| |
Related: Discussion #3513
Signed-off-by: Arif Ali <arif.ali@canonical.com>
|
| |
|
|
|
|
|
|
|
| |
Some parsers should skip strings <=3 length and some mappings should
obfuscate just full words.
Relevant: #3403
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
`unittest.TestCase.assertEquals()` was deprecated in Python 3 [1], and
finally dropped in Python 3.12. This now causes the unit tests to fail [2].
[1] https://docs.python.org/3.5/library/unittest.html#deprecated-aliases
[2] https://bugs.debian.org/1058214
Signed-off-by: Martin Pitt <mpitt@debian.org>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
New version of juju uses colorisation, and therefore juju status
and json.loads doesn't load the juju status correctly. By using
--no-color based on the version of juju this should fix this
particular use-case
Resolves: #3399
Resolves: SET-339
Signed-off-by: Arif Ali <arif.ali@canonical.com>
|
| |
|
|
|
|
|
|
|
| |
Adds a new Prepper for handling hostname determination for preparing the
mapping and parser.
As part of this new prepper, pass the CLI options to each prepper for use.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Up until now, the archive abstractions have defined what files `sos
clean` will use to prepare the mappings for obfuscation before entering
the normal obfuscation loop over every file in every archive.
While this is straight forward enough, it is not particularly flexible,
and prevents us from easily using other approaches for preparing the
mappings beyond what is directly obtained via the parsers (which in some
cases need special handling to be prepared at all).
Change this by introducing `SoSPrepper`s which will be used to determine
now only what files to pass to which parsers on an archive-by-archive
basis, but will also allow for manually retrieving items from disaparate
sources within the archive(s) and handing those directly to the
mappings, without the need for those items to first pass the parser
check.
Related: RH: SUPDEV-135
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When collecting an sos report from witin a container, and intending to
collect host-level information, symlink collection would previously
collect the container's version of the symlink target due to how they
would resolve within the `Archive.add_file()` flow.
Fix this by simply forcing the follow up collection of the sysroot
location of the symlink target, which would previously be aborted due to
the fact that sos had already collected the container's matching file.
This introduces a certain level of inefficiency by way of doubling the
copy operations, however ensuring that we only make a single copy
operation originally involves a quite fragile set of path mangling that
depends heavily upon how the symlinks are created on the host's
filesystem. For example a symlink using a relative path could be
resolved somewhat easily, but a symlink using an absolute path
introduces a certain amount of indeterminate resolution, which we cannot
allow for the vast majority of use cases that are performed outside of a
container.
Resolves: rhbz#2075720
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
| |
Adds basic unit tests for the fitness of `PackageManager`
implementations for rpm and dpkg, as well as the new
`MultiPackageManager` using both of those.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
Fix below two scenarios:
1. If subordinate's parent is missing.
2. If subordinate's parent's units is missing.
Closes: #3213
Signed-off-by: jneo8 <james.lin@canonical.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add a new cluster profile and transport called "juju" for `sos collect`.
Both the profile and transport are intended to be used on juju managed
environments which assumes that `juju` is installed on the machine where
`sos collect` is called, and that the juju user has superuser privilege
to the current controller.
When using the "juju" cluster profile, the sos reports will be collected
from all the applications within the current model by default. If
necessary, one can filter the nodes by models / applications / units /
machines with cluster options. For example `-c "juju.models=sos" -c
"juju.apps=a,b,c"`. Moreover, transport will also be dynamically changed
to "juju" when cluster type is juju.
If not using "juju" cluster profile, one can still choose to use the
"juju" transport by specifying --transport option. However, not that the
--nodes option will be expected to be a comma separated machine IDs ,
**not** IP addr, since `juju ssh` identifies the ssh target by machine
ID. For example, `sos collect --nodes 0,1,2`
Closes: #2668
Signed-off-by: Chi Wai Chan <chiwai.chan@canonical.com>
Signed-off-by: jneo8 <james.lin@canonical.com>
Co-authored-by: Chi Wai Chan <chiwai.chan@canonical.com>
Co-authored-by: jneo8 <james.lin@canonical.com>
|
| |
|
|
|
|
|
|
|
| |
Add unit test for #3030.
Relevant: #3030
Resolves: #3135
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit adds a new parser and accompanying map for obfuscating IPv6
addresses.
This new parser will attempt to capture valid IPv6 networks and
addresses, and produce a mostly-randomized obfuscated pair. Due to the
multiple formats an IPv6 address can take, some identifiers are
necessary to preserve relevant information while still obfuscating
actual addresses and networks.
For example, global unicast addresses that have more than one defined
hextet (greater than /16 prefix) will always generate an obfuscated
address starting with `534f` (or 'so', continuing the style of our mac
address handling that uses 'sos' as an identifier). Addresses with a /16
prefix or less, will start with simply '53'. Private addresses, which
start with `fd` will generate an obfuscated address starting with
`fd53`, so that the contextual understanding that it is a private
network/address can remain. Link-local addresses which start with
`fe80::` will remain that way, only having the device hextets obfuscated
- again, keeping the contextual information that it is a link-local
interface intact, as otherwise these obfuscations may confuse end
users reviewing an sos report for problems.
Note that the address `::1` and `::/0` are explicitly skipped and never
obfuscated, for the same reasons given above.
Additionally, this parser/map will write data to the default map (and
any per-run private maps) differently than previous parsers. Rather than
simply dumping the obfuscation pairs into the map, it is broken up via
network, with hosts belonging to that network nested inside those
network entries (still being json-formatted). Users will also note that
the ipv6 entries in the map also have a `version` key, which is intended
to be used for handling future updates to the parser/map when upgrading
from an older sos version to a newer one. This may or may not be carried
over to future updates to other parsers.
Closes: #3008
Related: RHBZ#2134906
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Hostname parser treats strings like 'host.domain.com' with precompiled
domain 'domain.com' in a wrong way. It first obfuscates the domain while
subsequent _parse_line skips host obfuscation.
Calling _parse_line before _parse_line_with_compiled_regexes does clean
both the host name and the domain name well.
Adding a unittest with a reproducer.
Resolves: #3054
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Historically, sos has limited collections to existing files and command
output. While there have been many reasons for this, there have also
been several exceptions made to current plugins as well as ongoing
requests for data that is not currently provided via existing commands.
As sos evolves, it should in turn be more capable to provide diagnostic
data beyond what is strictly available via command outputs.
As such, add a new step to the collection phase that allows plugins to
perform these manual data collections. Plugins may now define their own
`collect()` method to do so, thus moving the existing exceptions out of
`setup()` phase execution.
To aide in writing these collections to the plugin directory, a new
`collection_file()` generator has been added which will handle creating,
managing, and closing the new file so that plugins only need to be
concerned with the content being written to such files.
Plugin contributors should note that these manual collections are
executed at the end of the collection phase - meaning they are more
likely to be skipped or interrupted due to plugin timeouts.
Closes: #2992
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
- test for #3022 .
- fixup of opt_parser being a tuple with a string
Relates: #3022
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As highlighted in #1817, `pkg_by_name()` could provide unpredictable
results, when using wildcards. As such, limited this method to only
returning package info for exact package name matches. In turn, change
`Plugin.is_installed()` to leverage `PackageManager.all_pkgs_by_name()`
which does explicitly support wildcards and returns information on _all_
matching packages, not just the last one found.
In so doing, clean up the `PackageManager` design to use a new
`packages` property for these lookups, and update the former usage of
`all_pkgs()` accordingly. Similarly, signal `get_pkg_list()` should be
private (in any sense that a python method can be) by renaming to
`_get_pkg_list()` and update the single Plugin (`etcd`) referencing this
method.
Closes: #1817
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
When a file collection will reach its sizelimit, don't tail the file if
it has binary content, as this will be useless collection.
Closes: #2851
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Adds the ability for plugins to define a default set of environment vars
to pass to all commands executed by the plugin. This may be done either
via the new `set_default_cmd_environment()` or
`add_default_cmd_environment()` methods. The former will override any
previously set values, whereas the latter will add/update/modify any
existing values.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
First, update the regexes to account for possible quotes wrapping the
mac address to match.
Second, fix an edge case with these quoted mac addresses in our check
for avoiding duplicating obfuscations of already obfuscated addresses by
checking the stripped mac address instead of the raw one.
Closes: #2873
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For parsers such as the username and keyword parsers, we don't discover
new items through parsing archives - these parsers use static lists
determined before we begin the actual obfuscation process.
As such, we can build a list of regexes for these static items once, and
then reference those regexes during execution, rather than rebuilding
the regex for each of these items for every obfuscation.
For use cases where hundreds of items, e.g. hundreds of usernames, are
being obfuscated this results in a significant performance increase.
Individual per-file gains are minor - fractions of a second - however
these gains build up over the course of the hundreds to thousands of
files a typical archive can be expected to contain.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`nose` is no longer maintained, and as of python-3.10 is functionally
broken. As such, instead transition to running those tests via avocado,
like we do with our integration test suite.
The tests themselves do not need much modification, however due to the
isolation provided for executing the tests we do need to explicitly set
a new PYTHONPATH env var for those executions. This means we still need
to run the unit tests as a separate step from the stageone tests.
The changes needed are mostly around file paths relative to the pwd
where the tests are executed from originally.
Additionally, remove the sosreport_pexpect unit test as it is no longer
useful in its own right, would need more significant changes to run
properly with avocado, and the integration test suite provides better
coverage for what it was testing.
Closes: #2716
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit allows plugins that call `add_string_as_file` to specify if
the string should be written to `sos_strings/$name/` (current behavior)
or if it should be written to `sos_commands/$name/` which may be
desireable for organizational purposes for plugin collections.
`add_string_as_file()` has also been updated to write to a plugin's
manifest section for any files written this way. Accordingly, the method
will now accept a `tags` parameter to add specified tags to the manifest
entry.
Certain plugins directly calling this method have been updated, but the
existing logic to write truncated data to `sos_strings/` remains
untouched, and will not generate manifest entries (as those should
already be handled by the method that trigged the truncated collection).
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Up until now plugin options were defined via tuples, with positional
significance of the tuple elements. This made plugin option creation
fairly easy, but option handling could easily become confusing.
Instead, create a new `PluginOpt` class that plugin options from here on
out will need to build from in order to function. These will still be
applied to plugins by inserting them into the `option_list` class attr
in order to retain an easy way to expand plugin options for authors.
Internally, options are now assigned to a dict which is then directly
accessed for plugin option manipulations. PluginOpt default values are
retained separate from their current value, and elements are assigned
directly to meaningful identifiers within the class. This should
alleviate some of the overhead when handling plugin options within sos.
Not all current tuple elements have been carried over into the new
`PluginOpt` class - for example, the 'speed' attribute has been dropped
as it does not have a current function. In the planned `sos info`
component, the time effects of a plugin option should be documented in
the `long_desc` attribute instead.
Additionally, the `Plugin.get_option_as_list()` method has been removed
as it was not being used anywhere.
Note that this particular commit only introduces the new class, and the
loading options used by `SoSReport()`. As such, plugins using options
currently will report errors during test runs. A commit following this
one will shuffle existing plugin options into the new class structure
and allow the plugins to execute normally.
Resolves: #274
Resolves: #452
Resolves: #1597
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a global `--namespaces` option that can be used to limit the
number of namespaces all plugins will iterate over. If a plugin provides
a specific plugin option, such as the `networking.namespaces` option,
then if set that plugin option will override the global option value.
The global option defaults to not limiting namespaces, matching current
behavior.
Closes: #2092
Resolves: #2547
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
Instead of re-reading the mapping file for each parser, read it once and
store the contents, then hand those contents over to each parser.
This allows us to side-step handling the same exception for malformed
config files over and over for each parser loaded.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds wrapper methods to `sos.utilities` that allows the common `os`
methods, such as `os.path.exists()` to account for the setting of
`sysroot`. This will allow sos, and particularly plugins, to more
easily and reliably function as expected when sos is run within a
container.
Helpers have been added directly to `Plugin` that automatically pass the
set `sysroot` option, so plugin authors do not need to handle that
option or directly import these path functions from `sos.utilities`.
Closes: #494
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Adds an enablement trigger tuple, `containers`, that allows for enabling
a plugin based on the presence of a _running_ container matching a given
name or regex. Note that this is only functional for the default runtime
discovered by a policy during initialization.
Resolves: #2561
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
Adds a unittest that ensures conformance for plugin design, e.g.
ensuring that enablement triggers are proper tuples and not
inadvertent strings due to a missing trailing comma for single-tuples.
Resolves: #2549
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
As sos is now python3-only, we can avoid shelling-out to compression
utilities like `xz` or `gzip`, and instead use the method provided by
the built-in `tarfile` module.
Resolves: #2523
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
| |
|
|
|
|
|
|
|
| |
Updates the filepaths in our unittests for the temp data files to
reflect their new location in the updated directory layout.
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
This commit represents the start of an overhaul of the test suite used
by sos. Note that several more commits to follow will be required in
order for the test suite to be considered stable.
The new test suite will use the avocado-framework to build out new
tests.
This first part adopts a new 'stageX' naming scheme for our tests as
follows:
stage0 -> Unittests
stage1 -> Basic function tests, no mocking allowed
stage2 -> Mocked tests for specific scenarios/regressions
stage3 -> Complex setups for layered products/environments
At the moment, these unittests are not updated for avocado, though most
should still work with `nosetest` directly.
A new set of base classes is defined in tests/sos_tests.py which provide
the foundation for actual tests cases. This approach entails new test
cases subclassing a base class, such as the new `StageOneReportTest`,
and setting the `sos_cmd` class attr to the _options_ for an sos report
run. By default `sos report --batch` will be run, and targeted to the
test job's directory as a tmpdir.
Each sos command will be executed once, and all test_* methods within a
test case that subclasses `StageOneReportTest` will be checked against
the output of that execution. Note that this diverges from avocado's
typical approach where each test_* method is performed against a brand
new instance of the class (thus meaning any setup including our sos
report run would normally be run fresh). However, after speaking with
the avocado devel team, this is still seen as a valid pattern for the
framework.
The current organizational approach is to separate the tests by
component rather than stage. For example. `tests/report_tests/` should
hold any report-centric tests, and the `plugin_tests` directory therein
should be used for plugin-specific tests. As of this commit, there are
basic functionality tests under `tests/report_tests/` and a single
plugin test under `tests/report_tests/plugin_tests/` to act as a POC.
Further, there is a `tests/vendor_tests/` directory for organizing
vendor-specific bug/feature tests that are not covered by the generic
project-wide tests. A POC test from RHBZ1928628 is available with this
commit.
Note that in order for these tests to be run properly _without_
installing the current branch to the local system, you will need to run
the following command:
`PYTHONPATH=tests/ avocado run -t stageone tests/`
Related: #2431
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
