aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* Do not allow duplicate destinations in TarFileArchive.add_file()Bryn M. Reeves2012-12-121-0/+2
| | | | | | | | | | | | | The logic to ensure we have a proper tar record for each directory in the path of files collected with addCopySpec*() ends up generating lots of duplicate calls to add_file(). Filter them out by checking for the existence of the new destination in tarfile.getnames(). This reduces peak resident memory consumption by 50-75% and cuts run times by 50-66%. Related to Issue #81
* Add profiling to command and file substitution methodsBryn M. Reeves2012-12-121-7/+21
| | | | | | | | | | | | | | | | | | | | | Add profiling calls around the regex substitution methods. Shows that substitution costs are considerable compared to copy: copied: /root/anaconda-ks.cfg time: 0.002293 copied: /var/log/anaconda/anaconda.log time: 0.002204 copied: /var/log/anaconda/syslog time: 0.002709 copied: /var/log/anaconda/anaconda.packaging.log time: 0.002658 copied: /var/log/anaconda/ks-script-YGuewK.log time: 0.002483 copied: /var/log/anaconda/anaconda.program.log time: 0.002416 copied: /var/log/anaconda/anaconda.storage.log time: 0.002263 copied: /var/log/anaconda/anaconda.xlog time: 0.002299 copied: /var/log/anaconda/anaconda.ifcfg.log time: 0.002461 copied: /var/log/anaconda/ks-script-kdAaa4.log time: 0.002558 subst : /root/anaconda-ks.cfg time: 0.021056 I.e. subsituting one file is on average eight times more costly than simple collection.
* Work around non-writable directories in host file systemsBryn M. Reeves2012-12-121-7/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Distributions that have moved to reduced capabilities and replaced suid and sgid binaries with fscaps have tightened the permissions on several standard system paths. For e.g.: dr-xr-xr-x. 18 root root 4096 Nov 23 19:04 / -rw-r--r--. 1 root root 1174 Dec 11 18:05 /etc/passwd ----------. 1 root root 742 Dec 11 18:05 /etc/shadow dr-xr-x---. 8 root root 4096 Dec 12 19:06 /root Processes that need to write to these paths on the host system must possess cap_dac_override in order to work but in the archive this creates problems when unpacking the archive without this capability. For files this is not a problem since the user only requires write permissions to the containing directory to remove the file. For directories it causes real problems for unprivileged users working with sosreport archives. This includes problems unpacking the archive (since directories are created without write permissions causing subsequent file creation beneath that path to fail) as well as problems cleaning up archives as a typical 'rm -rf' fails to remove these paths. These problems make it impossible to create archives that are both failthful to the host system and easy to work with for typical analysis users. Ultimately this may necessitate dropping permission preservation in the archive and instead storing these (and other information like ACLs and SELinux context) in files within the archive instead). Works around Issue #85
* Add logging to regex subsitution methodsBryn M. Reeves2012-12-121-0/+9
| | | | | | | | | | | | | | Currently we don't log anything when applying file or command output regex substitutions and any errors are silently discarded. This could cause a user to think that passwords etc. have been obscured when in fact they have not. Log our intent to substitute at debug level and log any exception at error level along with the path or glob, module and exception text. Fixes Issue #84
* Re-work sunrpc module to make porting easierBryn M. Reeves2012-12-121-4/+28
| | | | | | | | | | | | | | Re-organise the sunrpc module so that the service check is in the base (Plugin-derived) class but uses a service string defined only in the subclasses. This allows the same checkenabled() to be used for any policy that implements the needed runlevelByService() API. This can serve as a prototype for a generic enabled-by-service facility (mimicking existing files and packages checks) although it currently needs considerable work to review and revise the service and runlevel interfaces in the policy modules. Related to Issue #77
* Use paths list in anaconda pluginBryn M. Reeves2012-12-111-3/+1
| | | | | The previous commit introduced a paths list for the module to collect but did not use it: fix the addCopySpecs call.
* Update anaconda supportBryn M. Reeves2012-12-111-7/+18
| | | | | Update the anaconda module to support new log locations in recent versions.
* Fix regression TarFileArchive regressionBryn M. Reeves2012-12-111-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Commit 179d9bb introduced a regression in the TarFileArchive class: (Pdb) bt /usr/sbin/sosreport(23)<module>() -> main(sys.argv[1:]) /usr/lib/python2.7/site-packages/sos/sosreport.py(908)main() -> sos.execute() /usr/lib/python2.7/site-packages/sos/sosreport.py(901)execute() -> return self.final_work() /usr/lib/python2.7/site-packages/sos/sosreport.py(773)final_work() -> self._finish_logging() /usr/lib/python2.7/site-packages/sos/sosreport.py(376)_finish_logging() -> self.archive.add_file(self.sos_log_file.name, dest=os.path.join('sos_logs', 'sos.log')) /usr/lib/python2.7/site-packages/sos/utilities.py(280)add_file() -> self.tarfile.addfile(tar_info, fileobj) /usr/lib64/python2.7/tarfile.py(2015)addfile() -> buf = tarinfo.tobuf(self.format, self.encoding, self.errors) /usr/lib64/python2.7/tarfile.py(996)tobuf() -> return self.create_gnu_header(info) /usr/lib64/python2.7/tarfile.py(1027)create_gnu_header() -> return buf + self._create_header(info, GNU_FORMAT) /usr/lib64/python2.7/tarfile.py(1112)_create_header() -> itn(info.get("mtime", 0), 12, format), > /usr/lib64/python2.7/tarfile.py(212)itn() -> raise ValueError("overflow in number field") The tarinfo mtime field is a float but the pax headers take a string-encoded value. The tarinfo field was inadvertently converted to a formatted string. The problem is hard to track down because it's not always triggered; the bug depends on the value of the string-encoded mtime date.
* Collect file lists for /var/libBryn M. Reeves2012-12-111-0/+2
| | | | | Collect a list of files and directories under /var/lib using ls -lR and tree (if available).
* Collect additional files and command output for infinibandBryn M. Reeves2012-12-111-2/+9
| | | | | Collect output of ibstat, ibstatus, ibhosts and the opensm and openib configuration files.
* Add mkqdisk -L output to cluster moduleBryn M. Reeves2012-12-111-0/+1
|
* Add /sys/firmware/acpi/tables to hardware moduleBryn M. Reeves2012-12-111-0/+1
| | | | Collect the ACPI tables present on the host system
* Add SysV IPC moduleBryn M. Reeves2012-12-111-0/+29
| | | | | Add a module to collect SysV IPC related information. Collect the files in /proc/sysvipc/* as well as the traditional ipcs -a output.
* Add more commands to iscsi initiator moduleBryn M. Reeves2012-12-111-1/+7
| | | | | Add additional iscsiadm command collection to the iscsi module and check for the iscsi-initiator-utils package on Red Hat systems.
* Remove analyze() wiringBryn M. Reeves2012-12-114-16/+3
| | | | | | | The analyze() methods haven't been used since commit 6ea48cb and their functonality has now been moved or removed in all modules. Delete the last bits of wiring and docs references.
* Remove diagnose() wiringBryn M. Reeves2012-12-113-68/+1
| | | | | | Complete the removal of the diagnose functionality by removing the methods in the Plugin class and sosreport module, command line options and documentation.
* Remove analyze() method from yum moduleBryn M. Reeves2012-12-111-9/+0
| | | | | The yum module's analyze method is pretty useless. It just checks for extra /etc/yum.repos.d files on RHEL systems. Delete it.
* Delete kernel analyze() method and add more sysfs module filesBryn M. Reeves2012-12-111-41/+3
| | | | | | | | | The analyze methods are no longer used and the kernel module's regex-table based taint checking is kinda klunky and a maintenance problem. Delete this code but collect the initstate, refcnt and taint files from /sys/modules/*.
* Delete selinux module's analyze() methodBryn M. Reeves2012-12-111-8/+9
| | | | | | | | | | | | The analyze machinery hasn't been hooked up since commit 6ea48cb. The use of these methods has also not been especially clear: some modules trigger additional file/command collection here while others only add alerts (the original purpose of analyze()). Other module call the addAlert() interface from their setup() method. Move the SELinux module's analyze functionality into the main setup() method but make it conditional on setroubleshoot being installed as it's necessary for the current implementation.
* Remove diagnose() methods from pluginsBryn M. Reeves2012-12-114-57/+0
| | | | | | Delete the diagnose() methods from all modules still using it. The diagnostics feature never worked well and is better implemented to run after-the-fact on the data collected in the tarball.
* Do not attempt to store SELinux context for /proc and /sysBryn M. Reeves2012-12-101-3/+7
| | | | | | | | | | | | | | | | | | | | | | | | | Storing a (correct) SELinux context for files in procfs and sysfs leads to tonnes of ugly spew when the tarball is unpacked as these contexts are not permitted on "normal" file systems: tar: sosreport-rhel7-vm1-20121210145629/proc/sys/vm: Cannot setfilecon: Permission denied tar: sosreport-rhel7-vm1-20121210145629/proc/sys/vm/percpu_pagelist_fraction: Cannot setfilecon: Permission denied tar: sosreport-rhel7-vm1-20121210145629/proc/sys/vm: Cannot setfilecon: Permission denied tar: sosreport-rhel7-vm1-20121210145629/proc/sys/vm/scan_unevictable_pages: Cannot setfilecon: Permission denied Etc. Check for these path prefixes in TarFileArchive.add_file() and skip generating a pax header for them. This isn't a perfect fix; it may be better to move this up to a higer layer (i.e. within the Plugin class's file handling routines) and also to filter by context rather than path (since SELinux contexts are a property of an inode not a path and a user could mount these file systems at other locations). This partially fixes Issue #79 but I'll keep it open for the time being to track these improvements.
* Fix typo in systemd moduleBryn M. Reeves2012-12-101-4/+2
| | | | | Forgot to refresh the copy of the module in HEAD; pickup two additional fixes added in testing.
* Add systemd moduleBryn M. Reeves2012-12-101-0/+46
| | | | | | | | | Add a new module to collect systemd information. This is the start of a general re-organisation of the init related information captured by sos. Initially collect systemctl and journalctl command output as well as (some) configuration files owned by the package.
* Move sysctl collection to kernel module and add new locationsBryn M. Reeves2012-12-102-1/+3
| | | | | | | The sysctl configuration and state belongs to the kernel. Reflect this by moving the collection of sysctl.conf to the kernel module (where sysctl -a is already captured) and add collection of the /etc/sysctl.d and /lib/sysctl.d directories.
* Remove obsolete sysreport references from sos.specBryn M. Reeves2012-12-071-2/+0
|
* Obscure password in /etc/keystone/keystone.confBryn M. Reeves2012-12-071-0/+5
|
* Make sar module more robustBryn M. Reeves2012-12-071-7/+27
| | | | | Make the sar module more tolerant of broken environments and users force-enabling the module when its own enabled check fails.
* Remove stray debug code in copy_permissions_filter()Bryn M. Reeves2012-12-071-1/+0
|
* Make TarArchive preserve permissions and SELinux contextBryn M. Reeves2012-12-071-18/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Make the TarArchive class preserve permissions and SELinux context on all files and directories it collects. This requires us to recursively add the parent directories of any files copied in by full path (or the directory would only be implicitly created and so receive default ownership and permissions and no SELinux context). To make this work, don't call tarfile.add() to recursively add directories - the doCopyFileOrDir machinery in the Plugin class already handles directory recursion properly. Instead when a directory is passed to add_file(), just create a TarInfo record of type tarfile.DIRTYPE and propagate the permissions from the host file system. The SELinux contexts have the unfortunate side-effect of spewing errors if unpacked without --no-selinux as some contexts cannot be placed on some file systems. It may be more useful to eventually store the contexts in the in a file rather than directly in the tarball. Before: drwxr-xr-x. 9 root root 640 Dec 7 00:57 proc lrwxrwxrwx. 1 root root 30 Dec 7 00:57 ps -> sos_commands/process/ps_auxwww drwxr-xr-x. 2 root root 60 Dec 7 00:57 root drwxr-xr-x. 2 root root 80 Dec 7 00:57 sbin After: dr-xr-xr-x. 9 root root 640 Dec 5 19:07 proc lrwxrwxrwx. 1 root root 30 Dec 7 00:58 ps -> sos_commands/process/ps_auxwww dr-xr-x---. 2 root root 60 Dec 7 00:48 root dr-xr-xr-x. 2 root root 80 Dec 7 00:57 sbin More fixes issue #76
* Fix typo in i18n moduleBryn M. Reeves2012-12-061-1/+1
|
* Add /etc/localtime to general moduleBryn M. Reeves2012-12-061-0/+1
|
* Add /etc/locale.conf to i18n moduleBryn M. Reeves2012-12-061-1/+1
|
* Fix exception when sar module enabled on system without sysstatBryn M. Reeves2012-12-061-3/+5
| | | | | The sar module assumes that /var/log/sa exists when it is enabled. Check for its presence and exit from setup early.
* Preserve permissions and ownership for files stored as stringsBryn M. Reeves2012-12-061-1/+11
| | | | | | | | | | Files that are stored as strings bypass the permissions code added in commit 74c1476 since they are added to the archive via add_string(). Add a check to see if a destination exists in the host file system. If it does propagate the permissions to the stored string's tar record.
* Preserve permissions, ownership and times for files in archiveBryn M. Reeves2012-12-061-2/+6
| | | | | | | Stat files during archiving and propagate the mode, atime, uid and gid to the archive. Fixes issue #76
* Handle non-readable files better in doCopyFileOrDirBryn M. Reeves2012-12-061-1/+7
| | | | | | | | | | | Currently a non-readable file causes an exception and resulting error logging as the path is passed blindly to archive.add_file(). There are legitimate reasons why directories colleceted by sos may contain files not readable by root, e.g. in sysfs where there are numerous write-only trigger files. Check the permissions via os.stat() before reading the file and substitute unreadable files with the empty string.
* Move Debian iSCSI target plugin to iscsitarget moduleBryn M. Reeves2012-12-062-16/+25
| | | | | | | | | The Debian iSCSI class implements iscsi target data collection. Move it to the iscsitarget module where it belongs. This needs further work to add an iSCSI initiator implementation for Debian and Ubuntu. That should be very straightforward but needs a system for testing.
* Remove redundant exception handling in postgresql moduleBryn M. Reeves2012-12-061-5/+1
| | | | | | | | | | | | | | | | | If we pass a 'None' into shutil.rmtree() it will os.path.join() that with a string leading to a TypeError: Traceback (most recent call last): File "/usr/sbin/sosreport", line 23, in <module> main(sys.argv[1:]) File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 961, in main sos.execute() TypeError: coercing to Unicode: need string or buffer, NoneType found > /usr/lib64/python2.7/posixpath.py(133)islink() -> st = os.lstat(path) Rather than catch the exception test tmp_dir before calling rmtree.
* Fix PacakgeManager.pkgByName()Bryn M. Reeves2012-12-061-4/+8
| | | | | | | Fix a typo that prevents any package manager queries from succeeding and change the wrapper function to explicitly test the results list rather than relying on an exception to detect an empty result set.
* Restore exception handling and logging in copy_stuff()Bryn M. Reeves2012-12-061-6/+6
| | | | | Now that collectOutputNow is fixed restore the exception handling block and logging in copy_stuff().
* Fix exception in postgresql postproc methodBryn M. Reeves2012-12-061-1/+6
| | | | | Do not assume that tmp_dir has been set and wrap tree deletion in a try block.
* Collect rhsm log files in general moduleBryn M. Reeves2012-12-061-1/+3
|
* Collect subscription-manager output in yum moduleBryn M. Reeves2012-12-061-0/+2
|
* Add /proc/iomem to the hardware moduleBryn M. Reeves2012-12-061-0/+1
|
* Add Red Hat Update Infrastructure moduleBryn M. Reeves2012-12-061-0/+43
|
* Fix line lengths and whitespace in rhn.pyBryn M. Reeves2012-12-061-3/+6
|
* Update RHN module and add spacewalk-debug collectionBryn M. Reeves2012-12-061-12/+14
| | | | | Update the RHN module to match current RHEL6 where relevant and add collection of spacewalk-debug on appropriate systems.
* Do not collect archived accounting files in psacct moduleBryn M. Reeves2012-12-061-1/+9
|
* Set default console loglevel to ERROR not FATALBryn M. Reeves2012-12-061-1/+1
| | | | | | | | | | | | | | | | | | | | | Setting the default console loglevel to FATAL causes important log messages to be discarded leading to mysterious failures for e.g. when an unknown option is specified: [root@hex tmp]# sosreport --batch --debug -k psacct.all=True sosreport (version 2.3) [root@hex tmp]# Lolwhut? [root@hex tmp]# sosreport -vvvv --batch --debug -k psacct.all=True sosreport (version 2.3) no such option "all" for plugin (psacct) [root@hex tmp]#
* Add sanlock moduleBryn M. Reeves2012-12-061-0/+36
|
generated by cgit (git 2.34.1) at 2024-11-16 12:19:16 +0000