| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Extra changes required due to module and function changes in new
avocado
* CentOS Stream is now seen as centos-stream in avocado instead of
centos
* Fine tune tox for stage tests
Closes: #3588
Resolves: #3594
Co-authored-by: Jake Hunsaker <jacob.r.hunsaker@gmail.com>
Signed-off-by: Arif Ali <arif.ali@canonical.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Consolidate testing, so that things can be run from CLI, CI and
various other means in a consistent way by using tox.
* Updated all the flake8 tests in the tests folder.
* Added pylint as a neutral test so that we can work on this in stages
and have some collaboration on what we test and don't
* The tox tests for unit, stageone, stagetwo testing makes is easier
for users to know how to run tests, and not have to do things
manually
* Using tox for CI doesn't make sense, as that will create virtual
envs and will disregard system/snap based python modules so may not
work
Signed-off-by: Arif Ali <arif.ali@canonical.com>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix the issues reported by the tests on the new version of ubuntu
Resolves: #3587
Closes: #3588
Related: SET-594,SET-595
Co-authored-by: David Negreira <david.negreira@canonical.com>
Signed-off-by: Arif Ali <arif.ali@canonical.com>
Signed-off-by: David Negreira <david.negreira@canonical.com>
|
|
|
|
|
|
| |
Closes: #3469
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
| |
A puppet regression in version 7.29.0 prevents foreman-installer to run.
Let prevent installing the buggy puppet-agent-7.29.0* .
Resolves: #3542
Closes: #3555
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
| |
Related: Discussion #3513
Signed-off-by: Arif Ali <arif.ali@canonical.com>
|
|
|
|
|
|
|
|
|
| |
Some parsers should skip strings <=3 length and some mappings should
obfuscate just full words.
Relevant: #3403
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
`unittest.TestCase.assertEquals()` was deprecated in Python 3 [1], and
finally dropped in Python 3.12. This now causes the unit tests to fail [2].
[1] https://docs.python.org/3.5/library/unittest.html#deprecated-aliases
[2] https://bugs.debian.org/1058214
Signed-off-by: Martin Pitt <mpitt@debian.org>
|
|
|
|
|
|
|
|
|
|
|
|
| |
New version of juju uses colorisation, and therefore juju status
and json.loads doesn't load the juju status correctly. By using
--no-color based on the version of juju this should fix this
particular use-case
Resolves: #3399
Resolves: SET-339
Signed-off-by: Arif Ali <arif.ali@canonical.com>
|
|
|
|
|
|
|
|
|
| |
Bump tested foreman versions - use current latest (3.7) on Ubuntu and
CentOS and previous 3.5 and 3.3 to match supported Satellite versions.
This requires enabling proper module streams for foreman-3.3 as well.
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Change grep_for_content to grep for a fixed string by default.
The reason is tests would match 1a2b3c4 as IP address 1.2.3.4 (which it
is not).
Add regexp option to grep_for_content, to allow the default "grep" search.
Resolves: #3320
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
|
|
| |
Add a test that cleaner keeps permissions of a sanitised file
Relevant to: #3292
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
options_tests.py tests -n option which means tens of plugins are
redundantly collected. Testing -o option is effectively the same while
the sosreport run needs a half time only.
The functionality of -n option is tested elsewhere, thus the change does
not shrink test coverage.
Resolves: #3288
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
|
|
|
| |
Adds a new Prepper for handling hostname determination for preparing the
mapping and parser.
As part of this new prepper, pass the CLI options to each prepper for use.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Up until now, the archive abstractions have defined what files `sos
clean` will use to prepare the mappings for obfuscation before entering
the normal obfuscation loop over every file in every archive.
While this is straight forward enough, it is not particularly flexible,
and prevents us from easily using other approaches for preparing the
mappings beyond what is directly obtained via the parsers (which in some
cases need special handling to be prepared at all).
Change this by introducing `SoSPrepper`s which will be used to determine
now only what files to pass to which parsers on an archive-by-archive
basis, but will also allow for manually retrieving items from disaparate
sources within the archive(s) and handing those directly to the
mappings, without the need for those items to first pass the parser
check.
Related: RH: SUPDEV-135
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When collecting an sos report from witin a container, and intending to
collect host-level information, symlink collection would previously
collect the container's version of the symlink target due to how they
would resolve within the `Archive.add_file()` flow.
Fix this by simply forcing the follow up collection of the sysroot
location of the symlink target, which would previously be aborted due to
the fact that sos had already collected the container's matching file.
This introduces a certain level of inefficiency by way of doubling the
copy operations, however ensuring that we only make a single copy
operation originally involves a quite fragile set of path mangling that
depends heavily upon how the symlinks are created on the host's
filesystem. For example a symlink using a relative path could be
resolved somewhat easily, but a symlink using an absolute path
introduces a certain amount of indeterminate resolution, which we cannot
allow for the vast majority of use cases that are performed outside of a
container.
Resolves: rhbz#2075720
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
| |
Adds basic unit tests for the fitness of `PackageManager`
implementations for rpm and dpkg, as well as the new
`MultiPackageManager` using both of those.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
Fix below two scenarios:
1. If subordinate's parent is missing.
2. If subordinate's parent's units is missing.
Closes: #3213
Signed-off-by: jneo8 <james.lin@canonical.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a new `arch` class attr that allows test classes to specify which
architecture(s) they are allowed/designed to execute on. If the test is
attempted to be executed on an architecture that is not in the specified
list, we will skip the test before any setup or sos executions are done.
By default, the `arch` class attr is an empty list, which implies that
the test can run on _any_ architecture. Tests should specify `arch` as a
list even for single-arch tests.
Included in this change is restricting the Foreman tests to x86_64 only,
as that is the only architecture on which that product is supported.
Closes: #3186
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add a new cluster profile and transport called "juju" for `sos collect`.
Both the profile and transport are intended to be used on juju managed
environments which assumes that `juju` is installed on the machine where
`sos collect` is called, and that the juju user has superuser privilege
to the current controller.
When using the "juju" cluster profile, the sos reports will be collected
from all the applications within the current model by default. If
necessary, one can filter the nodes by models / applications / units /
machines with cluster options. For example `-c "juju.models=sos" -c
"juju.apps=a,b,c"`. Moreover, transport will also be dynamically changed
to "juju" when cluster type is juju.
If not using "juju" cluster profile, one can still choose to use the
"juju" transport by specifying --transport option. However, not that the
--nodes option will be expected to be a comma separated machine IDs ,
**not** IP addr, since `juju ssh` identifies the ssh target by machine
ID. For example, `sos collect --nodes 0,1,2`
Closes: #2668
Signed-off-by: Chi Wai Chan <chiwai.chan@canonical.com>
Signed-off-by: jneo8 <james.lin@canonical.com>
Co-authored-by: Chi Wai Chan <chiwai.chan@canonical.com>
Co-authored-by: jneo8 <james.lin@canonical.com>
|
|
|
|
|
|
|
|
|
|
|
| |
Avocado tests generate netdevs by listing /sys/class/net which can
contain also bonding_masters that does not refer to any netdev. In such
a case, the test case fails.
Resolves: #3176
Closes: #3178
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
When running avocado tests in a sequence on the same host, further tests
are affected by cleaner default_mapping built from obfuscating specific
keywords also. Prevent adding these keywords to the mapping.
Resolves: #3165
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
|
|
|
| |
Run the cleaner tests with obfuscating (also) "tmp" to cover files
handling under sys_tmp.
Related to: #3160
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a new `--low-priority` option to report, which will attempt to
constrain the process priority for the report generation. We do this by
attempting to set ourselves to an 'idle' IO class, as well as setting
our niceness to 19 to avoid contending for CPU time.
This is also exposed via `sos collect`, however users should note that
this will not be effective until the sos-4.5.1 release.
Closes: #3127
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Historically, journal sizes have been limited to the *higher* of 100MB
or `--log-size`. While this had the benefit of potentially controlling
both logs and journals with the same option, it was not immediately
intuitive to end users and downright prevented collecting less than
100MB of journals.
Address this by separating journal size limiting from `--log-size` by
adding a new `--journal-size` option (default 100). This will allow
users to individually control journal sizes without any "gotcha"
scenarios with relation to general log size limiting.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
| |
Commits a downstream test from RHEL to ensure that a given release has a
consistent version string in the UI report, manifest, and most
importantly for the package nvr for an sos package built from the
current branch. Note that this test is only implemented for RPM
installations at this time.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
| |
Add test for #3071 : Prevent obfuscating tmpDir
Resolves: #3136
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
|
|
|
| |
Add unit test for #3030.
Relevant: #3030
Resolves: #3135
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
| |
Relates: #3060
Signed-off-by: Miroslav Hradilek <mhradile@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, mocked files were kept under the `tests/test_data/`
directory and generally mimic'd the file location they would be
temporarily copied to during the execution of their relevant tests.
This has a few maintainability drawbacks, and the handling of the
`files` attribute for test cases as either strings or tuples is at best
confusing.
Improve on this by instead making the `files` references relative to
where the test case file is written. This enables easier maintenance by
keeping all test requirements closer together, rather than spread across
the repo. As such, the `files` attribute now requires a list of tuples,
taking the form `(relative_src, absolute_dest)`. Additionally, fake
plugins for tests that need them to artificially test a specific
criteria should also be included in the test's subdir now.
Along with this change, move several StageTwo tests to their own subdirs
that now contain both the test cases and the needed files for mocking.
This should be the new design pattern going forward - if a test needs to
mock files of any kind, put it in a new subdirectory (and if it doesn't
need to mock files, continue to keep it in the relevant directory within
the test suite).
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
| |
Resolves: #3109
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit adds a new parser and accompanying map for obfuscating IPv6
addresses.
This new parser will attempt to capture valid IPv6 networks and
addresses, and produce a mostly-randomized obfuscated pair. Due to the
multiple formats an IPv6 address can take, some identifiers are
necessary to preserve relevant information while still obfuscating
actual addresses and networks.
For example, global unicast addresses that have more than one defined
hextet (greater than /16 prefix) will always generate an obfuscated
address starting with `534f` (or 'so', continuing the style of our mac
address handling that uses 'sos' as an identifier). Addresses with a /16
prefix or less, will start with simply '53'. Private addresses, which
start with `fd` will generate an obfuscated address starting with
`fd53`, so that the contextual understanding that it is a private
network/address can remain. Link-local addresses which start with
`fe80::` will remain that way, only having the device hextets obfuscated
- again, keeping the contextual information that it is a link-local
interface intact, as otherwise these obfuscations may confuse end
users reviewing an sos report for problems.
Note that the address `::1` and `::/0` are explicitly skipped and never
obfuscated, for the same reasons given above.
Additionally, this parser/map will write data to the default map (and
any per-run private maps) differently than previous parsers. Rather than
simply dumping the obfuscation pairs into the map, it is broken up via
network, with hosts belonging to that network nested inside those
network entries (still being json-formatted). Users will also note that
the ipv6 entries in the map also have a `version` key, which is intended
to be used for handling future updates to the parser/map when upgrading
from an older sos version to a newer one. This may or may not be carried
over to future updates to other parsers.
Closes: #3008
Related: RHBZ#2134906
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Hostname parser treats strings like 'host.domain.com' with precompiled
domain 'domain.com' in a wrong way. It first obfuscates the domain while
subsequent _parse_line skips host obfuscation.
Calling _parse_line before _parse_line_with_compiled_regexes does clean
both the host name and the domain name well.
Adding a unittest with a reproducer.
Resolves: #3054
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
After all plugins have run, sos will now generate a "tag summary" and
add it to the report section of the manifest. This summary will be a
json-formatted entry that details all collections within the report that
have any tag associated with them at all. This should allow for easier
parsing of the manifest by analyzers such as Red Hat Insights.
As part of this change, commands will no longer be automatically tagged
with the name of the binary used in the command collection.
Additionally, manual collections performed by a plugin's `collect()`
method will now be recorded in the manifest in the same manner as file
and command output collections.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It was found that the `krb5` plugin was not executing on Ubuntu/Debian
systems due to a legacy change that inadvertently removed support for
those distributions.
Re-enable support by defining a new plugin class for those distros,
setting the kdc directory appropriately. Additionally, expand the
package tuple to include newer package names.
Closes: #3041
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a new manual collection to the `process` plugin, that tries to
compile a mapping of running processes' binaries to an owning package
via the package manager. As such, package managers now have a new
`pkg_by_path()` method that serves this purpose.
Closes: #1350
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Historically, sos has limited collections to existing files and command
output. While there have been many reasons for this, there have also
been several exceptions made to current plugins as well as ongoing
requests for data that is not currently provided via existing commands.
As sos evolves, it should in turn be more capable to provide diagnostic
data beyond what is strictly available via command outputs.
As such, add a new step to the collection phase that allows plugins to
perform these manual data collections. Plugins may now define their own
`collect()` method to do so, thus moving the existing exceptions out of
`setup()` phase execution.
To aide in writing these collections to the plugin directory, a new
`collection_file()` generator has been added which will handle creating,
managing, and closing the new file so that plugins only need to be
concerned with the content being written to such files.
Plugin contributors should note that these manual collections are
executed at the end of the collection phase - meaning they are more
likely to be skipped or interrupted due to plugin timeouts.
Closes: #2992
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Foreman has since made several releases since we initially added
integration testing for it. Rebuilding the images is maintenance
overhead we can reduce by automating the installation of foreman on
images.
Provide this scripted installation, and expand our testing matrix to
additional versions of Foreman. As of this commit, we will be testing
versions 2.5 and 3.1 on CentOS Stream 8 to cover Red Hat Satellite
features, as well as Foreman version 3.4 (the latest current upstream)
for CentOS 8 and Debian 11.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A scenario was found that if a file had encoding issues when `sos clean`
went to open the file for obfuscation, we would stop processing the file
but leave it in the archive, which had the potential to leave
unobfuscated information in that file in the archive.
Fix this, by using the `errors='replace'` parameter when opening archive
files. This allows us to continue parsing the file normally, while
replacing the problematic characters with `?`s.
Closes: #3015
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
| |
- test for #3022 .
- fixup of opt_parser being a tuple with a string
Relates: #3022
Signed-off-by: Pavel Moravec <pmoravec@redhat.com>
|
|
|
|
|
|
|
|
| |
The Fedora 36 image is now available on GCP, so update our testing
matrix to use it. Accordingly, this means we no longer need to build or
maintain our own Fedora images going forward.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Up until now, the test suite was forced to be run using the git
checkout. While this is useful for on-the-fly testing, it does miss an
important use case of building a test package from the checkout, and
running it using that as the system installation of sos.
This commit allows the use of an installed version of sos to test
against. This may be leveraged by adding `-p TESTLOCAL=true` in the
`avocado run` command used to launch the test suite. Setting this
parameter to any other value, or omitting it entirely, will continue the
current behavior or using the git checkout for running tests.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a new test case for the `teamd` plugin that also inherently acts to
test team device enumeration.
Included with this test case is an addition to the base test classes
that allows for tests to define a `post_test_tear_down()` method that
will be run at the end of each test execution to allow for manual
cleanup - in this case deleting a 'fake' team device created for the
test.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
| |
Adds a new test to the networking plugin test to ensure we iterate
correctly over network devices.
|
|
|
|
|
|
|
|
|
|
|
| |
For files that can be considered obvious version files - those that end
specifically with either `version` or `version.txt` - skip processing
via the IP parser, as this may lead to improper obfuscation of certain
version strings. This is also applied to files ending in `release`.
Closes: #2962
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
| |
Removes catalog entries from the journal collection for `this` and
`last` boot collections.
Closes: #2132
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For our CI tests, CPU resources are not guaranteed which in turn can
cause longer-than-expected run times for test executions. In practice,
this is only seen occasionally but it requires manual intervention when
the timeouts are hit (and thus far it has been seen that the changes
these timeouts are hit on are not actually causing the timeouts).
Previous conversations have revolved around improving test efficiency,
however this seems unlikely given the nature of some of the test setup
and further, even with the most efficienct approach possible we would
still be at the whim of resource availability.
As such, increase the default timeout to account for this resource
consideration.
Closes: #2700
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As highlighted in #1817, `pkg_by_name()` could provide unpredictable
results, when using wildcards. As such, limited this method to only
returning package info for exact package name matches. In turn, change
`Plugin.is_installed()` to leverage `PackageManager.all_pkgs_by_name()`
which does explicitly support wildcards and returns information on _all_
matching packages, not just the last one found.
In so doing, clean up the `PackageManager` design to use a new
`packages` property for these lookups, and update the former usage of
`all_pkgs()` accordingly. Similarly, signal `get_pkg_list()` should be
private (in any sense that a python method can be) by renaming to
`_get_pkg_list()` and update the single Plugin (`etcd`) referencing this
method.
Closes: #1817
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
| |
Adds a new `--disable-parsers` option that allows users to selectively
disable parsers for a given execution of `sos clean`.
This may be useful in specific scenarios where obfuscation is not
strictly needed for all the types of data we obfuscate, and where the
user trusts whomever may be receiving the archive for review.
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|
|
|
|
|
|
|
|
|
| |
When a file collection will reach its sizelimit, don't tail the file if
it has binary content, as this will be useless collection.
Closes: #2851
Signed-off-by: Jake Hunsaker <jhunsake@redhat.com>
|